Multi-level caching to deploy local volatile memory, local persistent memory, and remote persistent memory

A technique is introduced for applying multi-level caching to deploy various types of physical memory to service captured memory calls from an application. The various types of physical memory can include local volatile memory (e.g., dynamic random-access memory), local persistent memory, and/or remote persistent memory. In an example embodiment, a user-space page fault notification mechanism is used to defer assignment of actual physical memory resources until a memory buffer is accessed by the application. After populating a selected physical memory in response to an initial user-space page fault notification, page access information can be monitored to determine which pages continues to be accessed and which pages are inactive to identify candidates for eviction.

TECHNICAL FIELD

The present disclosure relates generally to memory management in a computer system, and more particularly, to the processing of memory calls from an application.

BACKGROUND

The computer industry continues to develop and refine solid-state storage devices and media, moving closer and closer to achieving memory-class storage. In the past several years the access latency of affordable storage devices has dramatically decreased and is expected to continue to decrease in coming years. At the same time, advances in networking technology have led to increases in bandwidth with commensurate decreases in latency. Further, the emergence of standardized remote direct memory access (RDMA) functionality has led to improvements in communication efficiency and further reduced latency.

These faster computing infrastructures demand new data infrastructures where both memory-speed data access and disk-like high storage density are strongly desired at the same time. Such new data infrastructures promise to bring significant performance improvements to computing tasks whose working data sets exceed dynamic random-access memory (DRAM) capacity, and where highly frequent data movements between DRAM and lower storage tiers, such as solid-state drive (SSD) and hard disk drive (HDD), are therefore required.

To provide the lowest possible access latency, operating system support of emerging persistent memory (PMEM) technology has created mechanisms for a user-space application to have direct access (DAX) to persistent memory media (i.e., without the access being performed by operating system software). PMEM generally refers to solid-state byte addressable memory devices that reside on a memory bus of a given system. Being on the memory bus allows PMEM to have similar speed and latency as DRAM as well as the non-volatility of SSDs and HDDs. Examples of existing solutions from providing DAX to PMEM include “NOVA,” “Strata,” “Octopus,” “Hotpot,” and “FluidMem.”

DETAILED DESCRIPTION

Overview

Computer systems typically provide mechanisms to allocate volatile memory to user-space applications. For example, to obtain an allocation in volatile memory to facilitate execution, an application may submit, transmit, generate, or otherwise communicate a memory allocation request. Depending on the architecture of the computing system, this memory allocation request may be communicated in the form of a system call to a core of the computer system's operating system (i.e., the kernel) and/or through the use of a higher-level library function. Volatile memory can include, for example, DRAM, synchronous DRAM (SDRAM), and/or static random-access memory (SRAM). For illustrative simplicity, certain embodiments of the introduced technique are described herein with respect to DRAM; however, a person having ordinary skill will recognize that the introduced technique can be applied to systems that have other types of volatile memory including SDRAM, SRAM, etc.

In some embodiments, a computer system can be configured to provide user-space applications with direct access to DRAM and/or PMEM. In some embodiments, PMEM can be implemented in a distributed manner through the use of a distributed memory object (DMO) system that provides a cluster of DMOs implemented across multiple physical devices. Although PMEM is persistent by nature, there are situations in which persistence is not necessary and PMEM can instead be used in a volatile mode to, for example, provide byte-addressable memory to an application when memory requirements exceed available volatile memory such as DRAM. Using PMEM in volatile mode may present a better performance alternative to memory that would otherwise be swapped using virtual memory. Examples where use of PMEM in volatile mode may be advantageous include computations on large social media graphs and various machine learning application.

Using direct access to PMEM in volatile mode presents several challenges. For example, in many cases it is not feasible to modify an application to make use of volatile mode PMEM. Similarly, reconfiguring a computer system at the kernel level to offload certain memory requests to PMEM can introduce various security issues. Further, while byte-addressable like DRAM, PMEM is typically not as fast as DRAM and may not be suitable for replacement of all the memory buffers requested by a memory application programming interface (API) such as malloc( ) since that may cause the application to execute at a slower rate. Also, when an application forks a child operation while one of its private mappings has been mapped to a DAX PMEM device, the copy-on-write functionality that would normally accompany that buffer is not provided. As a result, changes made to a mapped buffer in the child operation would incorrectly be visible in the parent operation, and vice versa.

Introduced herein is a technique for implementing PMEM in volatile mode that addresses the above-mentioned challenges. In an example embodiment, a memory allocation capture library is implemented to intercept memory calls from an application and whether such calls are to be handled using volatile memory such as DRAM or whether such calls are to be handled using volatile mode local and/or remote PMEM, for example, that is part of a DMO. The memory allocation capture library can apply an allocation policy to intercepted calls to determine whether to capture and process such calls. In some embodiments, the memory allocation capture library can be configured to use a multi-level caching mechanism to deploy volatile memory (e.g., DRAM), local PMEM, and remote PMEM, for example, in accordance with resource availability and real-time (or near-real-time) monitoring of page accesses by an application. In some embodiments, the memory allocation capture library can be configured to handle application forks by, for example, cloning (pre-fork or post-fork) a separate copy of PMEM for a child operation based on monitored PMEM utilization by a parent operation.

Allocating Memory to an Application—Existing Technique

FIG.1shows diagram illustrating a typical process100for allocating memory to an application in a computer system. As shown inFIG.1, an application110executing a user-space task may submit, transmit, generate, communicate, invoke, or otherwise make a memory call102to a memory function112configured to submit, transmit, generate, communicate, invoke, or otherwise make a system call104(e.g., sbrk) to an operating system kernel116to allocate and/or manage an allocated portion119of memory118.

The memory function112may include one or more functions for performing memory management (e.g., memory allocation, reallocation, release, etc.). In some embodiments, the memory function112includes one or more functions in a software library of a standardized programming language (e.g., C, C++, etc.). For example, the C standard library includes C dynamic memory allocation functions such as malloc( ) which allocates a specified number of bytes, realloc( ) which increases or decreases the size of a specified block of memory, calloc( ) which allocates a specified number of bytes and initializes them to zero and free( ) which releases a specified block of memory back to the system. These are just examples of memory functions and are not to be construed as limiting. Other memory functions include mmap( ), mmap64( ), munmap( ), mprotect( ), madvise( ), etc.

In an illustrative embodiment, a user process executing in the application110calls a memory function112(e.g., malloc( ). This memory allocation function112then invokes an appropriate kernel service using a system call104to allocate the appropriate portion119of memory118. In other words, there is a separation of duties between the user-space application110and the lower-level operating system kernel116. The system call104causes the operating system kernel116to allocate memory on behalf of the user-space application110.

Applying an Allocation Policy to Capture Memory Calls

As previously mentioned, PMEM is typically not as fast as DRAM and may not be suitable for replacement of all the memory buffers requested by a memory application programming interface (API) such as malloc( ) since that may cause the application to execute at a slower rate. In other words, situations can arise in which certain memory requests can be handled using PMEM while other memory requests should instead be handled using volatile memory such as DRAM.

For example, many machine learning applications based on TensorFlow use 256 KB buffers for preprocessing of data and 2 MB buffers for the computation. Replacing the 2 MB buffers with PMEM can have a large negative impact on performance, while replacing the 256K buffers with PMEM has little negative impact on performance. In such a case, it may be preferable to handle the 2 MB buffer using DRAM while offloading the 256 k buffer to be handled using PMEM to free up the limited DRAM capacity for other tasks.

One possible approach to address this issue includes modifying an application to allocate memory through a custom API such that certain memory requests are handled using PMEM and others are handled using DRAM. While such an approach may be effective in certain cases, many modern applications are too complicated to allocate memory though a custom API. For example, it is likely that many such applications are calling APIs such as malloc( ) and/or mmap( ) through a dynamically loaded memory function library to access available DRAM in a computer system.

Another possible approach to address this issue includes providing the application with access to the operating system kernel to manage memory allocations. As with modifying the application to allocate memory through a custom API, this approach also introduces the complication of having to modify the application to manage memory allocations in DRAM and PMEM and further introduces security concerns. The kernel space has access to everything in a computer system. It is difficult to sell applications that go into the kernel because any adopter (especially an enterprise, government, or other type of organization) of the application will have to vet the application to make sure the application will not take over their systems in a hostile manner.

To address these challenges, a technique can be implemented to intercept memory calls from a user-space application and apply an allocation policy to determine whether such calls are handled in DRAM or in PMEM. In an example embodiment, memory calls from an application are intercepted by a memory allocation capture library. Such calls may be to a memory function such as malloc( ) or mmap( ) and may be configured to cause a portion of volatile memory (e.g., DRAM) to be allocated to the application to process a task. The memory allocation capture library will determine whether the intercepted call satisfies one or more capture criteria associated with an allocation policy. If the intercepted call does satisfy the one or more capture criteria, the memory allocation capture library processes the call to cause a portion of PMEM to be allocated to the application instead of DRAM. In some embodiments, the allocated portion of PMEM is or is part of a DMO.

FIG.2shows a diagram illustrating an example process200for applying an allocation policy to capture memory calls, according to an embodiment of the introduced technique. As shown inFIG.2, an application110executing a user-space task may submit, transmit, generate, communicate, or otherwise invoke calls to a memory function112, for example, as described with respect toFIG.1. In other words, the application110shown inFIG.2can be the same as the application110shown inFIG.1and does not need to be specifically configured for implementation according to the introduced technique.

The introduced technique includes implementation of a memory allocation capture library220configured to capture calls by application110to one or more APIs such as memory function112. For example, the memory allocation capture library220can be configured to intercept calls by the application110to a memory function such as malloc( ), calloc( ), realloc( ), mmap( ), mmap64( ), munmap( ), mprotect( ), madvise( ), etc. A “call” refers to any type of communication between entities such as the application110and memory function112. For example, a memory call may include a digital message that includes data (e.g., a set of parameter values) configured to cause another entity (i.e., the memory function112) to perform certain operations such as making a subsequent system call to cause a portion of memory118to be allocated to the application110. “Intercepting” a call refers to the act of receiving, retrieving, or otherwise obtaining a call from the application110that was intended for another destination (e.g., memory function112).

In some embodiments, the memory allocation capture library220described herein may represent one or more software components. For example, the memory allocation capture library220may group together multiple compiled object code files in a single file that can be linked by multiple applications. In some embodiments, the memory allocation capture library220may be implemented as a static library (e.g., as a “.a” file) that is linked with and effectively part of the application110. Alternatively, in other embodiments, the memory allocation capture library220may be implemented as a dynamically linked (or “shared object”) library (e.g., as a “.so” file) that can be dynamically linked to the application110during execution. For illustrative simplicity, certain processes, operations, steps, and/or functions are described herein as being performed by the memory allocation capture library220; however, a person having ordinary skill in the art will recognize that, in some embodiments, such processes, operations, steps, and/or functions may actually be performed by an executable program that uses the memory allocation capture library220.

When the application110makes a memory call, the memory allocation capture library220can intercept the call and determine whether to ignore the call or to capture the call. For example,FIG.2depicts a scenario in which the application110has made two memory calls202aand202b. In this example, both memory calls202aand202bare calls to a memory function112such as malloc( ). Here, a first memory call202ais intercepted by the memory allocation capture library220and ignored. In other words, the first memory call202acontinues to the appropriate memory function112which may in turn make a system call204ato the kernel116to allocate a portion of memory118. Depending on the configuration of the computer system, an ignored memory call202amade through this standard path will typically be handled using DRAM. For example, an ignored memory allocation call202a(e.g., a call to malloc( ) may cause an allocation of a memory chunk in DRAM218a(e.g., memory chunk219a).

As alluded to previously, certain memory calls can be handled using PMEM instead of DRAM. If the memory allocation capture library220determines that an intercepted call should be handled using PMEM instead of DRAM, the memory allocation capture library220can capture the call and handle the call in a customized manner to take advantage of what is likely the much larger byte-addressable space of PMEM. For example, as shown inFIG.2, a second memory call202bis captured by the memory allocation capture library220. Again, the captured memory call202bmay be a call to a memory function112, for example, similar to memory call220a. In other words, from the point of view of the application110, memory call202bmay be no different than memory call202a. The application110has no knowledge of the memory allocation capture library220and is not specifically making a call to the memory allocation capture library220.

Memory calls that are captured by the memory allocation capture library220are handled in a customized manner to allocate or otherwise manage a portion of memory118. For example, in some embodiments, response to capturing a memory call, the memory allocation capture library220may submit, transmit, generate, communicate, or otherwise invoke a system call204bto the operating system kernel116to handle the request using an alternative to DRAM such as PMEM218b. In other words, the system call204bmay cause the allocation of a portion of PMEM218b(e.g., memory chunk219b). In other embodiments, in response to capturing a memory call, the memory allocation capture library220may submit, transmit, generate, communicate, or otherwise invoke a call203to another API212to handle using an alternative to DRAM such as local or remote PMEM218b. The other API212may rely on RDMA or other mechanisms to allocate the portion of PMEM218bwithout involving the operating system kernel116. For example, as will be described in more detail, in some embodiments, a computer system may be configured to enable applications with direct access to DMOs. Such DMOs may include local and/or remote PMEM218b. In such an example, the memory allocation capture library220can process a captured memory call202bby submitting, transmitting, generating, communicating, or otherwise invoking a call to an API212associated with a DMO system to allocate a DMO that is accessible to the application110.

The memory allocation capture library220can apply one or more allocation policies222to determine whether to capture (i.e., process) an intercepted memory call from application110or to ignore such a call. The allocation policy222may specify various capture criteria that, when satisfied, cause the memory allocation capture library220to capture an intercepted memory call. Accordingly, the process of applying the one or more allocation policies222may include intercepting a memory call, processing the memory call to identifying parameters of the memory call, and determining if the parameters of the memory call satisfy one or more capture criteria specified by the one or more allocation policies222.

Parameters of an intercepted memory call may include, for example, the type of memory call (e.g., malloc( ), mmap( ), etc.), a size of a mapping associated with the memory call (e.g., 256 KB vs. 2 MB, etc.), certain flags in mappings associated with the memory call (e.g., MAP_STACK, MAP_NORESERVE, etc.), the application where the memory call originated, the type of the application where the memory call originated (e.g., a machine learning application vs. other types of applications), etc. In some embodiments, the parameters of an intercepted memory call may further include timing information associated with the call such as a time of day when the call was intercepted, a period of time elapsed since the call was intercepted, etc. These are just examples of certain parameters of an intercepted memory call that can be considered and are not intended to be limiting. Other types of parameters can similarly be considered when determining whether to capture an intercepted memory call from an application.

In some embodiments, the allocation policy222may be hard coded into the memory allocation capture library220. Alternatively, in other embodiments, the allocation policy222may be generated, updated, stored, managed, etc. independent of the memory allocation capture library220. For example, the allocation policy222may be stored in a database that is accessible to the memory allocation capture library220. Further, the allocation policy222may be independently updated (manually and/or automatically) without updating the memory allocation capture library220.

In some embodiments, the one or more allocation policies222applied by the memory allocation capture library220can be modified by a user by adjusting one or more configurable parameters associated with the capture criteria. For example, a user may specify whether to capture malloc( ) calls, mmap( ) calls, or both. The user can also specify a minimum, maximum, and/or range of mapping sizes to capture. The user can also specify whether to capture or ignore mappings with certain flags. The user can also specify a particular period of time (e.g., a number of seconds) to wait after intercepting a memory call before capturing the call (assuming other capture criteria are satisfied). Again, these are just examples of certain parameters that a user can configure to control which memory calls are captured by the memory allocation capture library220and which calls are ignored.

As shown inFIG.2, a user252may adjust capture criteria of the allocation policy222, for example, by providing inputs via user interface253that set certain parameter values. The user252in this context may be an end user of the application110, an administrator user (e.g., of a DMO system), etc. The user interface253may include a GUI configured to receive user inputs and present visual outputs. The user interface253may be accessible via a web browser, desktop application, mobile application, or over-the-top (OTT) application, or any other type of application at the user computing device. The user computing device displaying user interface253may include, for example, a desktop computer, a laptop computer, a server computer, a smartphone, a tablet computer, a wearable device, or computing device capable of presenting user interface253, and/or communicating over a computer network.

In an example embodiment, a computer system receives an input from a user252, for example, via interface253. The input includes a user selection of various parameters such as a type of call (e.g., malloc( ), mmap( ), etc.), a size of a mapping associated with the memory call (e.g., 256 KB vs. 2 MB, etc.), certain flags in mappings associated with the memory call (e.g., MAP_STACK, MAP_NORESERVE, etc.), the application where the memory call originated (e.g., a specific application identifier), the type of the application where the memory call originated (e.g., a machine learning application vs. other types of applications), timing information (e.g., a time of day, time delay, etc.) or any other type of parameter. For example, using interface253, a user252may select the type of call to include malloc( ) and a maximum mapping size of 256 KB. The computer system can process the parameters included in the user's input to generate an updated capture criterion. The updated capture criterion in this example would specify that all calls to malloc( ) for mappings that are 256 KB or less are to be captured. The allocation policy222can then be configured based on the updated capture criterion, for example, by replacing previous capture criteria and/or supplementing previous capture criteria.

In some embodiments, the allocation policy222can be automatically updated using machine learning techniques. For example, various performance metrics (e.g., processing time, DRAM utilization, etc.) associated with previous application of the introduced technique can be tracked and used as feedback data that is used to train a machine learning model to determine updated capture criteria for an allocation policy222. The capture criteria associated with an allocation policy222may therefore continually update as the system learns which calls can be captured and handled using PMEM without significantly impacting the performance of applications.

In an example embodiment, a computer system may track the performance of processing one or more application tasks. Application tasks may include previous tasks by application110and/or tasks by other applications executed by the computer system. For example, a computer system may track how long each task takes to perform over some period of time, how much DRAM is utilized to perform the tasks, etc. Based on this tracking, the computer system can generate performance metric data that is indicative of this processing performance. Performance metric data may include, for example, aggregations of various performance metrics such as an average processing time and/or average DRAM utilization. The computer system can then use such historical performance metric data to train a machine learning model to determine capture criteria based on one or more inputs such as current capture criteria, current performance metrics, etc. For example, a machine learning model may be trained to produce values for the one or more configurable parameters through the use of tools such as Naïve Bayes classifiers, support vector machines, random forests, artificial neural networks, etc. The parameter values output by the machine learning model can be used to generate an updated capture criterion. The allocation policy222can then be configured based on the updated capture criterion, for example, by replacing previous capture criteria and/or supplementing previous capture criteria.

The introduced technique for applying an allocation policy to captured memory calls allows DRAM, which would normally be allocated, to be freed in favor of local and/or remote PMEM. By offloading certain tasks to be handled by PMEM (which is typically available in greater abundance than DRAM), the introduced technique can significantly lower overall DRAM utilization without negatively impacting application performance.FIG.3shows a chart300that illustrates how DRAM utilization can be reduced by using the introduced technique for capturing memory calls from an application. Specifically,FIG.3charts DRAM utilization over time by a GrapSAGE machine learning application using a baseline technique and the introduced capture technique. As illustrated inFIG.3, the introduced capture technique results in significantly lower DRAM peak utilization and does not lead to a significant increase in overall processing time.

The chart300depicted inFIG.3is provided to illustrate how the introduced technique can improve the operation of a computer system by reducing DRAM utilization, but is not intended to represent results in all configurations. Actual processing results in any given embodiment will depend on a number of different factors including the allocation policy222applied to capture memory calls from an application. For example, a timing study was conducted to determine how capture of various types of memory calls from GraphSAGE affected overall performance. A machine learning application such as GraphSAGE can make various types of memory allocations. As part of the timing study, an allocation policy was configured to capture various combinations of one or more of the following types of allocations: 256 KB allocations for loading, MAP_STACK allocations, MAP_NORESERVE allocations, and the malloc( ) family of allocations. Enabling capture of 256 KB allocations slowed down processing time by about 12% over baseline (i.e., no capture) but resulted in DRAM savings of about 65% over baseline. Enabling capture of malloc( ) calls slowed down processing time by about 17% over baseline but resulted in DRAM savings of about 23% over baseline. Conversely, enabling capture of MAP_STACK calls resulted in little impact on DRAM usage or processing time and enabling capture of MAP_NORESERVE calls increased processing time by about 10% without significantly reducing DRAM usage. The result of the timing study conducted using GraphSAGE demonstrated that for the tested system, the most significant benefit was realized by applying an allocation policy to only capture 256 KB allocations and to capture malloc( ) calls, when needed. The results from this example timing study are provided to demonstrate the benefit of a configurable allocation policy for selectively capturing memory calls over capturing all memory calls from an application. However, the results of this timing study are not intended to represent the performance for all applications. A different application and/or a different type of processing job using GraphSAGE may lead to different results.

FIG.4shows a flow diagram of an example process400for applying an allocation policy to capture memory calls from an application. Certain operations of the example process400are described with reference to components described with respect toFIG.2and/orFIG.11. Example process400can be executed by one or more of the components of a computer system such as the example processing system1400described with respect toFIG.14. For example, in some embodiments, the example process400depicted inFIG.4may be represented in instructions stored in memory that are then executed by a processor. The process400described with respect toFIG.4is an example provided for illustrative purposes and is not to be construed as limiting. Other processes may include more or fewer operations than depicted, while remaining within the scope of the present disclosure. Further, the operations depicted in example process400may be performed in a different order than is shown.

Example process400begins at operation402with intercepting a call from an application. For example, as described with respect toFIG.2, a memory allocation capture library220may be configured to intercept one or more calls from an application110that are intended for a memory function112such as malloc( ) or mmap( ). Such calls may be configured to cause a portion (i.e., a chunk) of a first type of memory to be allocated to the application110to process a task. In some embodiments, the first type of memory is DRAM that is local to an execution computer system that is executing the application110.

Example process400continues at operation404with identifying one or more parameters associated with the intercepted call. The one or more parameters may include, for example, the type of call (e.g., malloc( ), mmap( ), etc.), a size of a mapping associated with the memory call (e.g., 256 KB vs. 2 MB, etc.), certain flags in mappings associated with the memory call (e.g., MAP_STACK, MAP_NORESERVE, etc.), the application where the memory call originated (e.g., a specific application identifier), the type of the application where the memory call originated (e.g., a machine learning application vs. other types of applications), etc. In some embodiments, the one or more parameters of an intercepted call may further include timing information associated with the call such as a time of day when the call was intercepted, a period of time elapsed since the call was intercepted, etc. Other types of parameters can similarly be determined based on the intercepted call.

In some embodiments, identifying the one or more parameters associated with the intercepted call may include processing data included in the call to determine, extract, infer, or otherwise obtain information indicative of the one or more parameters. For example, an intercepted call may include data indicative of a source of the call (e.g., the application110), a destination for the call (e.g., a specific memory function112such as malloc( ) or mmap( ), a size of a mapping (e.g., 256 KB)), etc. In some embodiments, certain parameters may not be immediately evident based on data included in the call. For example, the memory allocation capture library220may infer that the call is from a machine learning application such as TensorFlow based on available information included in the call and/or external to the call even if such information does not specifically identify the application as a machine learning type application.

Example process400continues at operation406with accessing an allocation policy that specifies one or more capture criteria. For example, as described with respect toFIG.2, a memory allocation capture library220may access an allocation policy that specifies one or more capture criteria that can be applied to determine whether to capture an intercepted call.

Example process400continues at operation408with determining, based on the one or more parameters of the intercepted call whether the intercepted call satisfies the one or more capture criteria specified by the allocation policy. In some embodiments, operation408may include comparing the one or more parameters associated with the intercepted call to the capture criteria specified by the allocation policy and determining, based on the comparison, whether the one or more parameters substantially match the capture criteria. For example, a capture criterion may specify that all calls to the malloc( ) family of memory functions are to be captured. The malloc( ) family refers to a set of multiple functions for dynamic memory allocation in the C programming language that include, for example, malloc( ), realloc( ), callocO, and free( ). Accordingly, if the intercepted call is to any of these memory functions, the captured criterion is satisfied. Conversely, if the intercepted call is to mmap( ) the capture criterion is not satisfied. As another illustrative example, a capture criterion may specify that all calls for mappings under 1 MB are to be captured. Accordingly, if a memory call is intercepted for a 256 KB mapping (e.g., for loading), the capture criterion is satisfied. Conversely, if the intercepted memory call is for a 2 MB mapping, the capture criterion is not satisfied.

If, based on the determination at operation408, the one or more capture criteria are not satisfied, example process400continues to operation410with ignoring the intercepted call and at operation412with allowing the ignored call to be processed by the intended destination function (e.g., malloc( ), mmap( ), etc.) to allocate a portion of the first type of memory (e.g., DRAM) to the application. In other words, if the one or more capture criteria are not satisfied, the memory allocation capture library220will take no further affirmative actions with regard to the intercepted call and will instead allow the call to proceed through the normal memory allocation and management channels associated with the execution computer system where the application is executing. In most cases, this will mean that the memory call from the application will be handled using local DRAM at the execution computer system.

If, based on the determination at operation408, the one or more capture criteria are satisfied, example process400instead continues to operation414with capturing the intercepted call. In this context, the operation of “capturing” a call means that the call is not allowed to directly proceed through the normal memory allocation and management channels associated with the execution computer system and is instead handled using an alternative approach. For example, as shown inFIG.4, operation400continues at operation416with processing the captured call to cause a portion of a second type of memory (different than the first type) to be allocated to the application. In embodiments where the first type of memory is DRAM, the second type of memory may include local and/or remote PMEM. In some embodiments, the second type of memory may include a DMO which itself includes local and/or remote PMEM distributed across multiple nodes.

In some embodiments, processing the captured call at operation416may include transmitting, generating, communicating, or otherwise invoking a system call to an operating system kernel associated with the execution computer system to allocate the portion of the second type of memory (assuming it is locally available). For example, as described with respect toFIG.2, the memory allocation capture library220may make a system call204bto the operating system kernel116to cause a chunk219bof PMEM218to be allocated to application110. Similar to a memory function112, in such embodiments, the memory allocation capture library220will process one or more parameters associated with the captured memory call and then generate a separate second call to the operating system kernel (e.g., system call204b). However, while the system call204afrom the memory function112may, by default, cause an allocation in the first type of memory (i.e., DRAM), the system call204bfrom the memory allocation capture library220will, by default, cause an allocation in the second type of memory (e.g., local PMEM218b).

In some embodiments, processing the captured call at operation416may instead include transmitting, generating, communicating, or otherwise invoking a call to some other API (e.g., a different memory function) to handle the memory allocation request. For example, as described with respect toFIG.2, in some embodiments, the memory allocation capture library220may make a second memory function call203to another API212. This second memory function call203may be based on the one or more parameters associated with the captured call202bfrom the application110.

In some embodiments, this other API212may be an API associated with a DMO system (e.g., DMO system1100described with respect toFIG.11). In other words, in response to determining that a captured call satisfies the one or more capture criteria specified by an allocation policy222, the memory allocation capture library220may make a second call203to a DMO API that causes a DMO system1100to create a new memory mode (volatile or persistent) DMO and/or allocate a portion of an existing memory mode DMO to handle the request from the application.

In many computing systems, the address space available to applications can be increased beyond the limits of local physical DRAM through a memory management technique generally referred to as ‘virtual memory.” In a typical virtual memory context, a kernel-level virtual memory manager can create and manage address maps for application operations, leverage available local HDD to store inactive pages, and manage the copying of pages into physical memory when needed. As alluded to above, such virtual memory management is implemented at the kernel level of an operating system which presents a challenge when allocating alternative memory resources such as PMEM using, for example, the previously described memory allocation capture library220.

One solution to address this issue includes reconfiguring the kernel-level virtual memory manager and/or providing an application with access to the operating system kernel to manage memory allocations. As previously discussed, making changes to and/or providing access to an operating system kernel introduces security concerns since the operating system kernel has access to everything in a computer system.

Instead, in some embodiments, a technique for multi-level caching can be applied as a use-space process to deploy various types of physical memory such as volatile memory (e.g., DRAM), local PMEM, and/or remote PMEM, for example, based on resource availability and application page accesses. In some embodiment, the introduced technique for multi-level caching can be performed using a memory allocation capture library, for example, similar to the memory allocation capture library220described with respect toFIG.2. Notably, the introduced technique for multi-level caching, which can perform much of the functionality of a kernel-level virtual memory manager, can be implemented completely in user mode with no need for custom kernel components.

FIG.5Ashows a diagram illustrating an example process500afor multi-level caching, according to an example embodiment. As described with respect toFIG.2, a memory allocation capture library220can be configured to intercept, and in some cases, capture memory calls from an application110. For example,FIG.5Adepicts the memory allocation capture library220capturing a memory call502from application110. In some embodiments, the memory call502is captured in response to applying an allocation policy, for example, as described with respect toFIG.2

In response to capturing the memory call502, the memory allocation capture library220may, at operation504, arm a memory buffer530for user-space page fault notification. For example, the Linux operating system provides a mechanism generally referred to as userfault that enables certain virtual memory functionalities (e.g., on-demand paging) to be performed in user-space as opposed to at the kernel level. While certain embodiments are described in the context of the userfault feature of Linux, a person having ordinary skill in the art will recognize that other mechanisms for user-space page fault notification that are specific to other operating systems or environments can similarly be implemented.

A “page fault” occurs when an application attempts to access a block of memory (i.e., a page) that is not stored in physical memory (e.g., DRAM). In a virtual memory context, a page fault notifies an operating system that it must locate the data in virtual memory (e.g., at a physical storage device such as an HDD) and transfer that data into physical memory (e.g., DRAM). The userfault mechanism of Linux can be applied to enable a user-space process to receive page fault notifications when an application attempts to access a block of memory (i.e., a page) that is not stored in physical memory.

In some embodiments, arming the memory buffer530for user-space page fault notification at operation504may include registering an address range in the memory buffer530for page fault notification. The address range may include one or more pages in the memory buffer such as page532depicted inFIG.5A. Note, the arrow associated with operation504is depicted as directed to the memory buffer530to illustrate the arming of the memory buffer for page fault notification; however, this is not to be construed as limiting. For example, in the case of userfault, operation504may include invoking a userfaultfd system call to the operating system kernel116.

Once armed, the memory buffer530will generate an initial page fault notification in response to the application110attempting to access, at operation506, data within the registered address range of the buffer (e.g., page532). Notably, by responding to an initial page fault notification, the memory allocation capture library can defer assignment of actual memory resources (e.g., local volatile memory518a, local PMEM518, and or remote PMEM518cuntil the application110actually needs to access data from memory. In other words, in response to receiving a user-space page fault notification at operation508, the memory allocation capture library220will at operation510copy, swap, or otherwise move data for the memory buffer530into one of several available physical memory devices such as local DRAM518a, local PMEM518b, and/or remote PMEM518c. For example, in response to receiving an initial user-space page fault notification indicating that the application has attempted to access page532, the memory allocation capture library220may cause the data associated with page532to be copied, swapped, or otherwise moved into one of several available physical memory devices such as local DRAM518a, local PMEM518b, and/or remote PMEM518c.

In some embodiments, in response to receiving an initial user-space page fault notification indicating that the application has attempted to access page532, the memory allocation capture library220may cause the data associated with page532to be copied, swapped, or otherwise moved into one of the local memory devices such as local DRAM518aor local PMEM518b, but not into remote PMEM518c. That is, in some embodiments the system may be configured to fault into only local memory resources and not into remote memory resources such as remote PMEM518c. In such embodiments, data may be evicted from a local memory device (e.g., local DRAM518aor local PMEM518b) into remote PMEM518cand can then be fault-restored back into local memory, for example, by monitoring page accesses.

The specific physical memory resource selected may depend on several factors such as the relative capacities of each of the available physical memory resources, real-time usage of each of the available physical memory resources, parameters associated with the portion of the memory buffer530(e.g., page532) to be placed into memory (e.g., size, fragmentation, etc.), the type of application110requesting access (e.g., machine-learning vs. other applications), etc. For example, in some embodiments, the memory allocation capture library220will default to move data into local DRAM518aas long as available capacity and current usage permits. If local DRAM518adoes not have available capacity and/or no other data can be evicted from local DRAM518a, the memory allocation capture library220may instead elect to move data into local PMEM518b. Similarly, if local PMEM518bdoes not have available capacity and/or no other data can be evicted from local PMEM518b, the memory allocation capture library220may instead elect to move data into remote PMEM518c. This is just an example allocation scheme provided for illustrative purposes and is not to be construed as limiting. For example, as mentioned, in some embodiments, the memory allocation capture library220may select a particular physical memory resource based on certain parameters associated with the portion of the memory buffer530(e.g., page532) to be placed into memory. Recall, that in the context of one TensorFlow application, experimentation revealed that replacing a 2 MB buffer used for computation with PMEM had a large negative impact on performance while replacing a 256 KB buffer for preprocessing with PMEM had little negative impact on performance.

The userfault mechanism was originally developed to enable post-copy migration of a virtual machine from one node to another. For example, the one or more memory mappings associated with a virtual machine running at a first node could be armed for userfault to allow a user-space process to migrate each mapping to a second node in response to an application accessing the memory mappings. Accordingly, the userfault mechanism is configured to only issue a single notification in response to detected access of a page that is not in memory. This works for migration because, once the userfault mechanism is triggered, the data is migrated, and the process concludes.

The single trigger aspect of the userfault mechanism presents a challenge in a memory allocation context where you may need to evict underutilized data from memory. For example, to optimize use of limited amounts of physical memory (e.g., local DRAM518a, local PMEM518b, and/or remote PMEM518c), a memory allocation process may need to continually identify data that applications are accessing to place into memory and identify data that applications are not accessing to evict from memory. Accordingly, a solution is needed to identify candidates for eviction.

One solution involves an active approach that relies on the application110actively identifying portions of in-memory data that are no longer needed and communicating that information to a memory management process, for example, performed by the memory allocation capture library220. However, this solution requires modification by an application developer of the application110to cause it to take an active role in identifying eviction candidates. For reasons stated earlier, relying on customization of an application110to make use of the introduced techniques may not be practical or feasible in many situations.

Alternatively, in some embodiments, a process for rearming the memory buffer530for user-space page fault notification can be performed to enable the use, for example, of the userfault mechanism to automatically identify candidates for eviction without the need for any active steps by the application110. For example, after populating the selected physical memory with data from the memory buffer530in response to an initial userfault, the memory allocation capture library220may rearm the memory buffer530for user-space page fault notification. In an example embodiment, rearming the memory buffer530for user-space page fault notification includes creating a new demand-zero page outside the memory buffer530mapped range, arming this new demand-zero page for user-space page fault notification (e.g., using userfault), and moving the newly armed page onto the memory buffer530, page to be evicted, for example, using a mremap( ) call.

Rearming the user-space page fault notification enables the memory allocation capture library220to monitor page fault information, for example, received from the operating system kernel116at operation512, to identify candidates for eviction. In some embodiments, the memory allocation capture library220may monitor page fault information exported by a kernel memory manager through interfaces such as /proc/kpageflags and/or/sys/kemel/mm/page_idle. This enables the memory allocation capture library220to determine which pages continued to be accessed and which pages are candidates for eviction (e.g., due to lack of access by the application110) if memory requirements exceed capacity.

In some embodiments, the introduced technique for multi-level caching can include multiple mappings of the allocated space: one for an application's110view and one management activity, for example, by the allocation capture library220. In such embodiments, the application view only includes mapping operations applied to it whereas any movement of data occurs in the management view.FIG.5Bshows a diagram500bthat illustrates this concept involving multiple mappings.

As shown inFIG.5B, certain embodiments of the introduced technique may apply multiple mappings to an application address space540, namely, application access mappings550and capture library access mappings552. The application access mappings550may be part of a first view of the application address space540that is viewable to an application such as application110(i.e., an application view). Conversely, the capture library access mappings552may be part of a second view of the application address space540that is viewable to a user-space memory management function (i.e., management view). In some embodiments, this user-space memory management function is, or is part of, the memory allocation capture library220.

Within the applications access mappings550shown inFIG.5Bare multiple different pages560,562,564,566, and568. In some cases, pages in the application access mappings550may be mapped into a local memory pool. For example, page560is mapped into the local PMEM pool570as indicated by line580. In other words, the data mapped to page560is located in an allocation in the local PMEM pool570. As another example, page562is mapped into the local DRAM pool572as indicated by line582.

In some cases, pages in the application access mappings550may not yet be mapped into a local allocation memory pool, but may be armed for user-space page fault notification. Such pages may be mapped in several different cases. For example, page564is initially mapped but without a backing store allocation. As another example, page566is soft-evicted (i.e., temporarily unmapped from the application view) from one of the local memory pool allocations to, for example, evaluate if the application110is actively using it and/or to move the page from one pool to another. In the example, depicted inFIG.5B, page566was previously allocated to the DRAM pool572(as indicated by line584), but has been soft-evicted to move the page into the local PMEM pool570(as indicated by the arrow at the end of line584). Soft-eviction, in this context means that the mapping has changed without necessarily moving the physically allocated data. During the temporary soft-eviction, page566may be a demand-zero page576from the capture library view as indicated by the as indicated by eviction map line586. As another example, page568has been hard-evicted from the local memory into a remote PMEM pool574as indicated by line588. Hard-eviction, in this context, means that the mapping has changed and that the physically allocated data has been moved to an unmappable resource (e.g., remote PMEM). Again, from the capture library access view, page568may be a demand zero page576as indicated by line598.

FIG.6Ashows a flow diagram of an example process600afor applying multi-level caching according to an embodiment of the introduced technique. Certain operations of the example process600aare described with reference to components described with respect toFIG.5Aand/orFIG.11. Example process600acan be executed by one or more of the components of a computer system such as the example processing system1400described with respect toFIG.14. For example, in some embodiments, the example process600adepicted inFIG.6Amay be represented in instructions stored in memory that are then executed by a processor. The process600adescribed with respect toFIG.6Ais an example provided for illustrative purposes and is not to be construed as limiting. Other processes may include more or fewer operations than depicted, while remaining within the scope of the present disclosure. Further, the operations depicted in example process600amay be performed in a different order than is shown.

Example process600abegins at operation602with receiving a memory call from an application. For example, as described with respect toFIG.5A, a memory allocation capture library220may be configured to intercept one or more calls from an application110that are intended for a memory function112such as malloc( ) or mmap( ). Such calls may be configured to cause a portion of memory to be allocated to the application110to process a task.

Example process600acontinues at operation604with arming a memory buffer for user-space page fault notification. In some embodiments, arming the memory buffer for user-space page fault notification may include registering an address range in the memory buffer for page fault notification, for example, by invoking a userfault system call. The address range may include one or more pages in the memory buffer such as page532depicted inFIG.5A.

Example process600acontinues at operation606with receiving an initial user-space page fault notification (e.g., an initial userfault). The initial user-space page fault notification may indicate a detected initial access by an application of one or more pages in the memory buffer that are not yet populated in a physical memory device such as a local volatile memory device, a local PMEM device, or a remote PMEM device.

In response to the initial user-space page fault notification, example process600acontinues at operation608with copying, swapping, or otherwise moving data for the memory buffer into any one of a local volatile memory device (e.g., DRAM), a local PMEM device, or a remote PMEM device. For example, in response to receiving an initial userfault indicating an initial access by an application of a particular page in the memory buffer, data associated with that particular page may be copied, swapped, or otherwise moved into physical memory.

In some embodiments, before moving data into physical memory, example process600amay include an operation (not depicted inFIG.6A) for selecting between two or more levels of physical memory (e.g., any two or more of: local volatile memory device (e.g., DRAM), local PMEM, or remote PMEM). The specific physical memory resource selected may depend on several factors such as the relative capacities of each of the available physical memory resources, real-time usage of each of the available physical memory resources, parameters associated with the portion of the memory buffer to be placed into memory (e.g., size, fragmentation, etc.), the type of application requesting access (e.g., machine-learning vs. other applications), etc.

In some embodiments, for captured allocation calls, local PMEM may be selected by default and allocated to the application. While an application task is running using the allocated local PMEM, the application task may be promoted into an allocation in local volatile memory (e.g., DRAM) in response to determining, for example, that the application task's level of access of the allocated local PMEM satisfies a specified access criterion (e.g., exceeds a threshold number of page accesses). Further, while the application task is running in the allocated local volatile memory, the application task may be demoted back into local PMEM in response to determining that the application task's level of access of the allocated local volatile memory does not satisfy a specified access criterion. Note that in some embodiments, the specified access criteria used for promoting into local volatile memory and demoting out of local volatile memory may be different. Similarly, if local PMEM is under pressure (e.g., demand is nearing or exceeds capacity), one or more local PMEM allocations (e.g., least used, most recently allocated, etc.) can be evicted to remote PMEM. Pages in remote PMEM can be brought back into either local volatile memory or local PMEM, for example, in response to detecting (e.g., based on a heuristic using a page's history) that the page is re-accessed by the application and/or satisfies some specified access criterion.

In other embodiments, local volatile memory (e.g., DRAM) may be selected by default provided the local volatile memory has sufficient capacity to handle the data associated with an accessed page. The local PMEM and/or remote PMEM can therefore serve as an alternative when the local volatile memory does not have sufficient capacity. For example, in some embodiments, selecting a particular physical memory device may include determining that the local volatile memory (e.g., DRAM) does not have capacity to handle the data associated with an accessed page and then selecting one of the local PMEM or remote PMEM instead. In such embodiments, selecting the particular physical memory device may further include determining that the local PMEM does not have sufficient capacity to handle the data associated with the particular page and then selecting remote PMEM instead.

Returning toFIG.6A, example process600acontinues at operation610with rearming the memory buffer (or a particular page in the memory buffer) for user-space page fault notification. As previously discussed, in some embodiments, rearming the memory buffer for user-space page fault notification may include creating a new demand-zero page in the memory buffer that is outside a mapped range, arming this new demand-zero page for user-space page fault notification (e.g., using userfault), and moving the newly armed page into the evited buffer530, for example, using a mremap( ) call.

Example process600acontinues at operation612with monitoring page accesses by an application to, for example, identify candidates for eviction. As previously mentioned, in some embodiments, monitoring page access may include monitoring page fault information exported by a kernel memory manager via an interface such as /proc/kpageflags or/sys/kemel/mm/page_idle. In some embodiments, the monitoring is performed in real-time, or near-real-time (i.e., within seconds or fractions of a second), as an application is accessing pages in memory. In some embodiments, monitoring the page fault information may include continually retrieving, receiving, or otherwise accessing page fault information over a specified or open-ended period of time, processing the page fault information accessed over that period of time, and generating values for one or more metrics related to levels of access based on the processing. Such metrics may include, for example, a calculated total number of times an application accessed a particular page over the period of time, an average number of times the application accessed the particular page per time period (e.g., per minute, etc.), a maximum/minimum number of times the application accessed the particular page per time period (e.g., per minute), a total amount of time a page has remained idle (i.e., not accessed by the application), etc. These are just example metrics related to levels of access and are not to be construed as limiting. Other metrics may similarly be determined by processing page fault information.

Example process600acontinues at operation614with evicting data from local physical memory (i.e., any of the local volatile memory (e.g., DRAM) or local PMEM). In some embodiments, operation614may include determining, for example, based on the monitoring performed at operation612, that a level of access by the application does not satisfy a specified access criterion. For example, the determined level of access may be represented by any one or more of the aforementioned metrics (e.g., number of pages accesses, amount of time a page has remained idle, etc.). A corresponding access criterion may include a threshold value associated with any one or more of the metrics. For example, operation614may include comparing a determined value for a given metric (e.g., amount of time a page has remained idle) against a threshold value for that metric (e.g., 1 minute). In this example, if the value for the metric exceeds the threshold value, the access criterion is not satisfied. This is just an example access criterion and is not to be constructed as limiting. Other types of access criterion may similarly be specified including target values for a metric (as opposed to thresholds), specific ranges of values for a metric, etc.

In some embodiments, one or more access criteria used to identify eviction candidates may be specified based on inputs from a user252, for example, via interface253. For example, using interface253, a user252may provide an input that specifies a threshold value for a particular metric related to memory access by an application. An access criterion may then be generated based on the input threshold value for the particular metric.

In some embodiments, one or more access criteria used to identify eviction candidates may be automatically generated or updated using machine learning techniques. For example, various other performance metrics (e.g., processing time, DRAM utilization, etc.) may be used as feedback data to train a machine learning model to determine and/or update one or more access criteria. The access criteria for identifying eviction candidates may therefore continually update as the system learns to identify eviction candidates that result in optimal system performance.

In some embodiments, the decision to evict data from physical memory may further be based on a current available capacity at the physical memory device. In other words, even if the access criteria are not satisfied, data may be left in physical memory as long as the physical memory has sufficient available capacity. As such, in some embodiments, operation614may also include determining an available capacity of a physical memory device (e.g., in terms of bytes, frames, etc.), determining that the available capacity in the physical memory does not satisfy one or more specified capacity criteria (e.g., a threshold level of available capacity), and electing to evict data from physical memory in response to determining that the available capacity does not satisfy the one or more specified capacity criteria. As with the access criteria, in some embodiments, the capacity criteria may be generated and/or updated based on inputs from a user and/or using machine learning.

FIGS.6B-6Dshow series of flow diagrams of a set of example processes600b-dfor applying multi-level caching according to another embodiment of the introduced technique. Certain operations of the example processes600b-dare described with reference to components described with respect toFIGS.5A-5Band/orFIG.11. The example processes600b-dcan be executed by one or more of the components of a computer system such as the example processing system1400described with respect toFIG.14. For example, in some embodiments, the example processes600b-ddepicted inFIGS.6B-6Dmay be represented in instructions stored in memory that are then executed by a processor. The processes600b-ddescribed with respect toFIGS.6B-Dare examples example provided for illustrative purposes and are not to be construed as limiting. Other processes may include more or fewer operations than depicted, while remaining within the scope of the present disclosure. Further, the operations depicted in example processes600b-dmay be performed in a different order than is shown.

Example process600bdepicted inFIG.6Bbegins at operation642with receiving a memory call from an application. For example, as described with respect toFIG.5A, a memory allocation capture library220may be configured to intercept one or more calls from an application110that are intended for a memory function112such as malloc( ) or mmap( ). Such calls may be configured to cause a portion of memory to be allocated to the application110to process a task.

Example process600bcontinues at operation644with arming a page for user-space page fault notification. In some embodiments, arming the memory buffer for user-space page fault notification may include registering an address range in an application address space540associated with the page for user-space page fault notification, for example, by invoking a userfault system call. As described with respect toFIG.5B, the application address space540may include two mappings: one for an application view and another for a management view (e.g., for the allocation capture library220). The page armed at operation644may be initially mapped in the application view, but without any backing store allocated, for example, as described with respect to page564inFIG.5B.

Example process600bcontinues at operation646with receiving an initial user-space page fault notification (e.g., an initial userfault). The initial user-space page fault notification may indicate a detected initial access by an application of the page armed at operation644.

In response to the initial user-space page fault notification, example process600bcontinues at operation648with mapping the page into a local memory pool allocation. Specifically, in the example embodiment depicted inFIG.6B, the page is mapped to an allocation in local PMEM. For example, the page may be mapped to a local PMEM pool570as shown inFIG.5B. In some embodiments, mapping the page to the local PMEM may include copying, swapping, or otherwise moving data associated with the page into a local PMEM device. Following operation648, the page is mapped to the local PMEM pool (i.e., part of capture library mappings552), for example, as described with respect to page560inFIG.5B.

Although not depicted inFIG.6B, in some embodiments, example process600bmay continue with rearming the page for user-space page fault notification, for example, as described with respect to operation610of process600a.

Example process600bcontinues at operation650with monitoring page access by the application, for example, as described with respect to operation612of process600a.

Example process600bcontinues with determining whether an access criterion is satisfied. For example, a determined level of access may be represented by any one or more metrics (e.g., number of pages accesses, amount of time a page has remained idle, etc.). A corresponding access criterion may include a threshold value associated with any one or more of the metrics. For example, if the value for a metric (e.g., number of page accesses) exceeds the threshold value, the access criterion is satisfied. This is just an example access criterion and is not to be constructed as limiting. Other types of access criterion may similarly be specified including target values for a metric (as opposed to thresholds), specific ranges of values for a metric, etc.

If the access criterion is satisfied, the page may be a candidate for promotion into local volatile memory (e.g., DRAM) to improve processing performance. Before moving the page into local volatile memory, the example process600bmay include determining if the local volatile memory has adequate capacity. In other words, example process may include determining whether a capacity criterion (e.g., a threshold level of available capacity) associated with the local volatile memory is satisfied. If the capacity criterion associated with the local volatile memory is not satisfied (e.g., because local DRAM is overutilized), the page remains mapped to the local PMEM and example process600breturns to operation650to continue monitoring the page accesses by the application. If the capacity criterion is satisfied, example process600bcontinues with promoting the page into local volatile memory, for example, by soft-evicting the page from local PMEM (at operation652) and moving the evicted data associated with the page from local PMEM into local volatile memory (e.g., local DRAM) (at operation654). In some embodiments, soft-evicting the page includes temporarily unmapping the page from the application view, for example, as described with respect to page566inFIG.5B. Note, although not depicted inFIG.6B, in some embodiments, example process600bmay include allocating a portion of local volatile memory (e.g., based on a size of the page) before soft-evicting the page from local PMEM and moving the page to local volatile memory. Following operation654, the page is mapped (in the application view) to local volatile memory, for example, as described with respect to page562inFIG.5B.

If the access criterion is not satisfied, for example, because the application is infrequently accessing the page, the page may be a candidate for hard-eviction. In some embodiments, a determination on whether to hard-evict a page may also depend on the available capacity in local PMEM. For example, as shown inFIG.6B, example process600bcontinues with determining whether a capacity criterion associated with the local PMEM is satisfied. If the local PMEM capacity criterion is satisfied (e.g., because local PMEM has above a threshold level of capacity) the system may elect to keep the page mapped to local PMEM whereby example process600breturns to operation650to monitoring the page accesses by the application. If the local PMEM capacity criterion is not satisfied, example process600bcontinues at operation656with hard-evicting the page from local PMEM and at operation658with moving data associated with the page to remote PMEM. Note, although not depicted inFIG.6B, in some embodiments, example process600bmay include allocating a portion of remote PMEM (e.g., based on a size of the page) before hard-evicting the page from local PMEM and moving the page to remote PMEM. Following operation658, the page is no longer mapped to the application view and is instead moved to remote PMEM (i.e., an unmappable resource), for example, as described with respect to page568inFIG.5B.

FIG.6Cshows a flow chart of an example process600cwhich represents a continuation of example process600bdepicted inFIG.6B. Specifically, the example process600cdepicted inFIG.6Cmay occur after a page has been promoted from local PMEM into local volatile memory (e.g., DRAM), for example, at operation654in example process600b.

As shown inFIG.6C, example process600ccontinues from operation654with monitoring, at operation660, page access by the application. Such monitoring may be performed, for example, as described with respect to operation612of process600a.

Example process600ccontinues with determining whether an access criterion is satisfied based on the monitoring. For example, a determined level of access may be represented by any one or more metrics (e.g., number of pages accesses, amount of time a page has remained idle, etc.). A corresponding access criterion may include a threshold value associated with any one or more of the metrics. For example, if the value for a metric (e.g., number of page accesses) exceeds the threshold value, the access criterion is satisfied. This is just an example access criterion and is not to be constructed as limiting. Other types of access criterion may similarly be specified including target values for a metric (as opposed to thresholds), specific ranges of values for a metric, etc.

If the access criterion is satisfied (e.g., because the application has accessed the page a threshold number of times), example process600ccontinues with determining if the local volatile memory has adequate capacity to continue handling the page. In other words, example process600cmay include determining whether a capacity criterion (e.g., a threshold level of available capacity) associated with the local volatile memory (e.g., DRAM) is satisfied. If the capacity criterion associated with the local volatile memory is satisfied, the page remains in local volatile memory (e.g., DRAM) and the example process returns to operation660to continue monitoring the page accesses by the application. If the capacity criterion associated with the local volatile memory is not satisfied (e.g., because local DRAM is overutilized), the page may be a candidate for soft-eviction to local PMEM and/or hard-eviction to remote PMEM.

If the access criterion is not satisfied (e.g., because the application has not accessed the page above a threshold number of times), the page may be a candidate for soft-eviction into local PMEM and/or hard-eviction to remote PMEM. As shown inFIG.6C, the decision on whether to soft-evict the page to local PMEM or hard-evict the page to remote PMEM may be based on available capacity in local PMEM. In other words, example process600cmay include determining whether a capacity criterion (e.g., a threshold level of available capacity) associated with the local PMEM is satisfied. If the capacity criterion associated with the local PMEM is satisfied (e.g., because the local PMEM has sufficient available capacity), example process600cmay continue with soft-evicting the page from local volatile memory (e.g., DRAM) (at operation662) and moving the soft-evicted page into local PMEM (at operation664). Note, although not depicted inFIG.6C, in some embodiments, example process600cmay include allocating a portion of local PMEM (e.g., based on a size of the page) before soft-evicting the page from local volatile memory and moving the page to local PMEM. Following operation664, the page is mapped (in the application view) to local PMEM, for example, as described with respect to page560inFIG.5B. If the capacity criterion associated with the local PMEM is not satisfied (e.g., because local PMEM is overutilized), example process600cmay continue with hard-evicting the page from local volatile memory (e.g., DRAM) (at operation666) and moving data associated with the page to remote PMEM (at operation668). Note, although not depicted inFIG.6C, in some embodiments, example process600cmay include allocating a portion of remote PMEM (e.g., based on a size of the page) before hard-evicting the page from local volatile memory and moving the page to remote PMEM. Following operation668, the page is no longer mapped to the application view and is instead moved to remote PMEM (i.e., an unmappable resource), for example, as described with respect to page568inFIG.5B.

FIG.6Dshows a flow chart of an example process600cdwhich represents a continuation of example process600bdepicted inFIG.6B. Specifically, the example process600ddepicted inFIG.6Dmay occur after a page has been hard-evicted from local PMEM to remote PMEM, for example, at operation658in example process600b. Note, example process600dmay also be performed after a page has been hard-evicted from local volatile memory to remote PMEM, for example, at operation668in example process600c.

As shown inFIG.6D, example process600dcontinues from operation658(or668) with monitoring, at operation670, page access by the application. Such monitoring may be performed, for example, as described with respect to operation612of process600a.

Example process600ccontinues with determining whether an access criterion is satisfied based on the monitoring. For example, a determined level of access may be represented by any one or more metrics (e.g., number of pages accesses, amount of time a page has remained idle, etc.). A corresponding access criterion may include a threshold value associated with any one or more of the metrics. For example, if the value for a metric (e.g., number of page accesses) exceeds the threshold value, the access criterion is satisfied. This is just an example access criterion and is not to be constructed as limiting. Other types of access criterion may similarly be specified including target values for a metric (as opposed to thresholds), specific ranges of values for a metric, etc. In some embodiments, the access criterion is satisfied in response to determining, for example, based on a heuristic using a page's history, that the page has been re-accessed by the application. In other, a single page access by the application after a dormant period may be enough to satisfy the access criterion.

If the access criterion is satisfied, the page may be a candidate for re-mapping back into local volatile memory (e.g., DRAM) or local PMEM. As shown inFIG.6D, the decision on whether to remap the page back into local memory may be based on available capacity in local memory. In other words, example process600dmay include determining whether a capacity criterion (e.g., a threshold level of available capacity) associated with the local volatile memory and/or local PMEM is satisfied. If the capacity criterion is satisfied, example process600dcontinues at operation672with remapping the page back into local volatile memory (e.g., DRAM) or local PMEM. This is also referred to as a fault-restore.

In some embodiments, example process600dmay default to local PMEM (as opposed to local volatile memory) when remapping the page back into local memory. Alternatively, example process600dmay include selecting one of local volatile memory (e.g., DRAM) or local PMEM based on the relative capacity of each resource when remapping the page back into local memory. In any case, in some embodiments, example process600dmay include allocating a portion of the local memory resource (local volatile memory or local PMEM) before remapping the page back into local memory.

Fork Handling

When an application forks a child operation (e.g., when a parent operation makes a copy of itself) while one of its private mappings have been mapped to a DAX device, the copy-on-write functionality that would normally accompany the memory buffer is not provided. As a result, changes made to the mapped buffer for the child operation would be incorrectly visible to the parent, and vice versa.

To prevent such behavior, a memory allocation library can be configured to handle application forks by, for example, cloning a separate copy of the PMEM for the child operation before the fork occurs (i.e., before the child operation is created or otherwise initiated) or after the fork occurs, for example, upon initial access by the child operation of the memory buffer.

FIGS.7A-7Bshow a sequence of diagrams that illustrate an example process700a-bfor application fork handing, according to an example embodiment. As described with respect toFIG.2, a memory allocation capture library220can be configured to intercept, and in some cases, capture memory calls from an application110. For example,FIG.7Adepicts the memory allocation capture library220capturing a memory call702from application110. In some embodiments, the memory call702is captured in response to applying an allocation policy, for example, as described with respect toFIG.2.

In response to capturing the memory call702, the memory allocation capture library220may, at operation704, allocate PMEM718for use by the application. For example, the memory call702may be associated with a request by the application110for memory to facilitate processing by a parent operation192. In response, the memory allocation capture library may allocate a portion719of the PMEM718for use by the parent operation192. At operation706, the parent operation is enabled to access the allocated portion719of PMEM718, for example, to facilitate processing associated with an application task.

The amount of the allocated portion719of PMEM718may be based on a request included in the captured memory call702. For example, the parent operation192may include in the memory call702an upfront request for a particular amount of PMEM. Under existing techniques, if that parent operation eventually forked into a child operation it would require creating a clone of the full amount of PMEM allocated to the parent operation. Depending on the amount initially allocated, this may represent a significant amount of data to copy. Consider, for example, a large-scale data processing operation with 40 GB of PMEM allocated to it. If that operation forks to create a child operation, it would require duplicating all 40 GB which may take a minute or so, thereby causing the application to freeze for a minute while the clone is created.

To improve processing efficiency, the introduced technique can instead create a clone of the PMEM that is based on a portion of the allocated PMEM that is actually being used by the parent operation192. Accordingly, as depicted inFIG.7A, example process700amay include, at operation708, monitoring the actual usage of the allocated PMEM719by the parent operation192. For example, based on the monitoring at operation708, the memory allocation capture library220may determine that the parent operation192is only utilizing a portion720of the allocated PMEM719. Note, the allocated PMEM719and portion720in use are depicted inFIGS.7A-7Bas rectangles of differing area to illustrate relative amounts of data. For example, based on the depiction inFIG.7A, the portion720of PMEM that is in use by the parent operation192is smaller than the portion719of PMEM allocated to the parent operation192; however, this not intended to convey anything about how the memory is allocated or used. For example, although depicted as a single block, allocated portion719may actually represent multiple chunks (e.g., frames) at non-consecutive address spaces.

Turning toFIG.7B, at some point during execution of the parent operation192, the memory allocation capture library220may intercept or otherwise receive a fork call710by the application indicating that the parent operation192will copy itself to create a child operation193(i.e., that the parent operation192will fork).

In response to receiving the fork call710, the memory allocation capture library220may cause, at operation712, a clone of PMEM to be created to facilitate the processing by the child operation193. In some embodiments, the clone created at operation712may be based on a portion720of the allocated PMEM719that is actually in use by the parent operation192. For example, as shown inFIG.7B, a clone721of the portion720of PMEM is created and placed into memory.

As previously alluded to, the point at which the clone721is created at operation712may occur pre-fork (i.e., just before the child operation193is created) or may occur post-fork (i.e., after the child operation193is created). In a pre-fork configuration, the clone721is created in response to receiving the fork call710, but just before the actual child operation193is created. Accordingly, once created, the clone721is available for access by the newly-created child operation193. In a post-fork configuration, the clone721may instead be created after the child operation193is created and in response to detecting an initial access by the child operation193to a memory buffer. In some embodiments, user-space page fault notification (e.g., implemented using the userfault mechanism) may be used to determine when the child operation193first attempts to access memory. For example, in response to receiving an initial user-space page fault notification (i.e., indicating initial access by the child operation193), the memory allocation capture library220may cause the clone721to be created in PMEM.

In some embodiments, the memory allocation capture library220may be configured for either pre-fork or post-fork cloning. In other words, the memory allocation capture library220will handle application forks using one of the two techniques regardless of outside factors. In some embodiments, the memory allocation capture library220may be configured by a user252, for example via interface253, to apply either pre-fork or post-fork cloning. Alternatively, the memory allocation capture library220may be configured to dynamically select either pre-fork or post-fork cloning based on one or more parameters associated with the fork call such as: the type of application initiating the fork, the type of application operation forking, the level utilization by the parent operation193of allocated PMEM719, the available capacity in PMEM718, etc. For example, in cases where it is known that both the parent operation192and child operation193will access most of the memory, it may be advantageous to create the copy pre-fork all at once to avoid future page faults. Alternatively, if it is known that the parent operation192and/or child operation193will only access a small portion of the memory, applying a post-fork cloning strategy will result in fewer memory copies.

In any case, once the application fork occurs, the parent operation192and child operation193may access data from PMEM718at operations714aand714b(respectively). For example, the parent operation192may at operation714acontinue to access data from the allocated portion719of PMEM, while the child operation193may, at operation714b, access data from the clone721created at operation712.

FIG.8shows a flow diagram of an example process800for handing application forks according to an embodiment of the introduced technique. Certain operations of the example process800are described with reference to components described with respect toFIGS.7A-7Band/orFIG.11. Example process800can be executed by one or more of the components of a computer system such as the example processing system1400described with respect toFIG.14. For example, in some embodiments, the example process800depicted inFIG.8may be represented in instructions stored in memory that are then executed by a processor. The process800described with respect toFIG.8is an example provided for illustrative purposes and is not to be construed as limiting. Other processes may include more or fewer operations than depicted, while remaining within the scope of the present disclosure. Further, the operations depicted in example process800may be performed in a different order than is shown.

Example process800begins at operation802with receiving a memory call from an application. For example, as described with respect toFIG.7A, a memory allocation capture library220may be configured to intercept and capture a memory call702from application110. The memory call702may be intended for a memory function112such as malloc( ) or mmap( ). Such calls may be configured to cause a portion of memory to be allocated to the application110to process a task.

Example process800continues at operation804with allocating PMEM to the application to facilitate processing. For example, as described with respect toFIG.7A, in response to capturing memory call702, the memory allocation capture library220may cause a portion719of PMEM718to be allocated to the application to facilitate processing of an operation (i.e., the pre-fork parent operation192).

Example process800continues at operation806with monitoring the use by the application110of the PMEM allocated at operation804to determine a portion of the allocated PMEM that is in use by application110. In particular, example operation808may include monitoring the use by the parent operation192of the allocated portion719of PMEM718. As previously discussed, at any given time, the parent operation192of application110may utilize less than all the allocated portion719of PMEM718. Accordingly, operation806may include determining a portion of the allocated PMEM that is in use by the parent operation192based on the monitoring. For example, the determined portion720in use may represent a subset of the allocated portion719of PMEM718.

In some embodiments, operation806may be performed continually after the portion of PMEM719is allocated to the application110. For example, the memory allocation capture library220may be configured to periodically poll (at regular or irregular intervals) information regarding page accesses by application110. Such polling may be performed, for example, by monitoring page fault information exported by a kernel memory manager through interfaces such as/proc/kpageflags and/or/sys/kemel/mm/page_idle.

In some embodiments, this page fault information may be processed to generate one or more metrics related to levels of access by the application110. Such metrics may include, for example, a calculated total number of times the application110accessed a particular page over the period of time, an average number of times the application110accessed the particular page per time period (e.g., per minute, etc.), a maximum/minimum number of times the application accessed the particular page per time period (e.g., per minute), a total amount of time a page has remained idle (i.e., not accessed by the application), etc.

The one or more metrics related to levels of access by the application110may then be utilized to determine which portion720of the allocated PMEM719is in use and/or is predicted to be in use by the application over some time horizon. For example, the memory allocation capture library220may determine that a particular portion720is in use and/or predicted to be in use based on a tracked average number of page accesses by the parent operation192over a period of time.

As alluded to above, in some embodiments, the portion720of PMEM determined at operation806to be in use by an application110(or more specifically, parent operation192) may actually represent a prediction of the portion of allocated PMEM719that will be in use. In some embodiments, this prediction can be made, for example, by processing one or more metrics related to levels of access using a machine learning model.

In some embodiments, the portion720of PMEM determined at operation806may correspond to an actual observed portion in use multiplied by some safety factor to account for unexpected spikes in usage by the parent operation. Consider, for example, a scenario in which the memory allocation capture library220determines, based on monitoring actual access, that the parent operation is using (or is likely to use) 2 MB out of a 10 MB allocation of PMEM. In such a scenario, the portion of allocated PMEM determined at operation806may represent that amount of PMEM determined based on direct monitoring (i.e., 2 MB) multiplied by a safety factor (e.g., 1.1).

Example process800continues at operation808with receiving a fork call indicative that a currently executing operation will fork to create a copy of itself. For example, with reference toFIG.7B, operation808may include the memory allocation capture library220, intercepting and capturing a fork call710from application110. This fork call710may indicate that a currently executing parent operation192will fork to create a copy of itself (i.e., child operation193).

Example process800continues at operation810with creating a clone of the portion of PMEM determined at operation806in response to receiving the fork call at operation808. For example, with reference toFIG.7B, operation810may include creating a clone721of the portion720of the PMEM719allocated to the parent operation192.

The clone721created at operation810can then be allocated to the child operation193to facilitate processing of the child operation193. Accordingly, example process800concludes at operation812with enabling the child operation193to access the clone721to facilitate such processing. Similarly, the parent operation192can continue to access the portion719of PMEM718allocated at operation804.

The creation of the clone721at operation810may be performed before the application fork occurs or after the application fork occurs. For example, in some embodiments, the clone721may be created in response to receiving the fork call (i.e., at operation808) but just before the parent operation192actually forks to create the child operation193. Alternatively, in some embodiments, the clone721may be created after the fork occurs (i.e., after the child operation193is created).

In some embodiments, the memory allocation capture library220may be configured for either pre-fork or post-fork cloning. In other words, the memory allocation capture library220will handle application forks using one of the two techniques regardless of outside factors. In some embodiments, the memory allocation capture library220may instead dynamically select either pre-fork or post-fork cloning based on one or more factors.FIG.9shows a flow diagram of an example process900for dynamically selecting pre-fork or post-fork cloning according to an embodiment of the introduced technique. As with the example process800ofFIG.8, example process900can be executed by one or more of the components of a computer system such as the example processing system1400described with respect toFIG.14. For example, in some embodiments, the example process900depicted inFIG.9may be represented in instructions stored in memory that are then executed by a processor. The process900described with respect toFIG.9is an example provided for illustrative purposes and is not to be construed as limiting. Other processes may include more or fewer operations than depicted, while remaining within the scope of the present disclosure. Further, the operations depicted in example process900may be performed in a different order than is shown.

Example process900begins at operation902with receiving a fork call from an application, for example, as described with respect to operation808of example process800.

Example process900continues at operation904with determining a parameter associated with the fork call in response to receiving the fork call. The parameter determined at operation904may specifically relate to any one or more of the application110originating the fork call, the parent operation192to be forked, and/or the child operation193that will result from the fork. Such parameters may include, for example, the type of application initiating the fork (e.g., machine learning vs. non machine learning), the type of application operation forking, the level PMEM utilization by the operation that will be forked (e.g., in absolute terms or as a percentage of allocated PMEM), a predicted level of utilization by a child operation that will result from the fork, the available capacity in PMEM718, etc.

In some embodiments, operation904may include processing data associated with the fork call received at operation902to determine the one or more parameters associated with the fork call. For example, the fork call may include such parameters or other data from which such parameters may be derived. In some embodiments, operation904may include receiving, retrieving, or otherwise accessing additional data indicative of the parameters in response to receiving the fork call at operation902. For example, in response to receiving a fork call, a system performing operation900may query the application110for additional information (e.g., a type of the application and/or operation to be forked), calculate a measure (e.g., a percentage) of allocated PMEM used by a parent operation based on monitored page access information, predict a measure (e.g., a percentage) of the allocated PMEM that the parent operation will use based on the current use, etc. These are just examples of how one or more parameters associated with a fork call may be determined. Other types of operations may similarly be performed to determine such parameters.

Example process900continues at operation906with determining whether the one or more parameters satisfy a specified criterion that governs whether cloning is to be performed pre-fork or post fork. In an example embodiment, if the criterion is satisfied, example process900continues to operation908with creating the clone before the child operation is created (i.e., pre-fork). Alternatively, if the criterion is not satisfied, example process900continues to operation910with creating the clone after the child operation is created (i.e., post-fork).

The one or more criteria applied at operation906may differ in various embodiments. In an example embodiment based on the percentage of allocated PMEM accessed by the parent operation, a specified criteria may set a threshold percentage value. Such an embodiment may include, for example, determining, at operation904, a percentage of the allocated PMEM719that a parent operation will access during processing and determining, at operation906, whether the percentage of allocated PMEM satisfies a specified threshold criterion (e.g., 50%). In response to determining that the percentage of allocated PMEM satisfies the specified threshold criterion (e.g., is above 50%), process900may continue to operation908with creating the clone before the child operation is created (i.e., pre-fork). Conversely, in response to determining that the percentage of allocated PMEM does not satisfy the specified threshold criterion (e.g., is at or below 50%), process900may continue at operation910with creating the clone after the child operation is created. This example scenario is based on an assumption that it is advantageous to create the clone pre-fork all at once to avoid future page faults in cases where it is known that the parent operation192and child operation193will access most of their allocated memory and that it is similarly advantageous to create clone post-fork to make fewer memory copies in cases where it is known that the parent operation192and/or child operation193will only access a small portion of their allocated memory. Again, this scenario is an example provided for illustrative purposes. The one or more criteria for determining whether to create the clone pre-fork or post-fork may different in various embodiments.

In any case, once the clone is created (pre-fork or post-fork), example process900continues at operation912with enabling the child operation to access the cloned PMEM, for example, as described with respect to operation812of example process800.

In some embodiments, performing post-fork cloning may include setting up data structures that enable the memory allocation capture library220to perform data copy on-demand once the child operation is created. In some embodiments, user-space page fault notification (e.g., using the userfault mechanism) can be applied to enable on-demand post-fork PMEM cloning.FIG.10shows a flow diagram of an example process1000for performing post-fork cloning according to an embodiment of the introduced technique. As with the example process800ofFIG.8, example process1000can be executed by one or more of the components of a computer system such as the example processing system1400described with respect toFIG.14. For example, in some embodiments, the example process1000depicted inFIG.10may be represented in instructions stored in memory that are then executed by a processor. The process1000described with respect toFIG.10is an example provided for illustrative purposes and is not to be construed as limiting. Other processes may include more or fewer operations than depicted, while remaining within the scope of the present disclosure. Further, the operations depicted in example process1000may be performed in a different order than is shown.

Example process1000begins at operation1002with receiving a fork call from an application, for example, as described with respect to operation808of example process800.

Example process1000continues at operation1004with arming a memory buffer for user-space page fault notification, for example, as described with respect to operation604of example process600a. As described with respect to operation604, arming the memory buffer for user-space page fault notification may include registering an address range in the memory buffer for page fault notification, for example, by invoking a userfault system call.

Example process1000continues at operation1006with receiving an initial user-space page fault notification (e.g., an initial userfault), for example, as described with respect to operation606of example process600a. In this case, the initial user-space page fault notification may indicate a detected initial access by the newly-created child operation193of the one or more memory pages in the memory buffer that are not yet populated in memory (e.g., local or remote PMEM).

In response to the initial user-space page fault notification indicating initial access by the child operation193, example process1000continues at operation1008with creating the clone (e.g., clone721) in PMEM.

Distributed Memory Object Architecture

In some embodiments, one or more of the introduced techniques can be applied in a distributed system. For example, a DMO system can provide persistent DMOs that can be accessed in either in-memory or file-storage mode, and may be implemented in low-latency RDMA. Thus, the DMO system enables use of DMOs both as memory and storage. The DMO system may also enable data in the system to be converted between in-memory and file-storage modes. In general, a DMO system can provide close-to-memory-speed data access which in turn can significantly relieve data bottlenecks observed at upper layer applications. Furthermore, embodiments may be built in user-space, thus obviating the need to install a customized operating system kernel.

FIG.11is an illustration of an example embodiment of a DMO system1100that provides persistent DMOs that can be accessed in either memory mode or file-storage mode. In the example DMO system1100, a system cluster1105is formed by a number of nodes. Each node in the system cluster1105may include a memory, a processor, and a network interface through which the node may send and receive messages and data. The illustrated DMO system1100provides for the creation of sharable memory spaces, each space being a DMO with a single owner node such as DMO owner node1142. In this example a node that uses a DMO is referred to herein as a client proxy node1140. In the example embodiment depicted inFIG.11, a system cluster1105within which the DMO system1100may be implemented includes an object node group1130, a name node group1120, a node manager1110, and a cluster manager1114.

Address space for a DMO may be partitioned into equal size chunks, with each chunk being stored on one or more chunk replica nodes1144included in the cluster of nodes1105. The chunks are distributed among a subset of the cluster nodes in such a manner as to: 1) focus locality of the chunks for performance efficiency, 2) provide sufficient availability of address space, and to 3) balance resources among the cluster of nodes. Furthermore, any node in a cluster using a DMO can locally keep a copy of a page.

The object owner node1142is responsible for coordinating updates to the client proxy nodes1140as well as the chunk replica nodes1144. The object owner node1142is also responsible for maintaining a configurable replication factor per DMO. The object owner node1142and chunk replica nodes1144can migrate to deal with failures, performance, or resource constraints. Client proxy nodes1140and chunk replica nodes1144cooperate with the object owner node1142in implementing protocols to make coherent updates and thereby provide a crash consistent view in the face of failures.

A node manager1110operates on each node in the DMO system1100. Once a node manager1110starts on a node, it can start or stop all other services associated with a node. Some services associated with a node may be started or stopped automatically or by request. The node manager1110is responsible for finding or electing the cluster manager (CM)1114and notifying its existence and node health to the cluster manager1114. Hence the node manager1110has access to performance and exception information from other components in the DMO system1100.

The cluster manager1114runs on a single node in the DMO system1100. The single node on which the cluster manager1114runs may be elected by a consensus algorithm of the node managers1110. The cluster manager1114mediates cluster membership, node ID assignment, and the name service (NS) group1120. The cluster manager1114also chooses nodes to satisfy allocation request constraints against cluster resource loading.

The DMO name service1124is a hash-distributed service which provides mapping of a DMO name string to its object ID and the object owner. The service is hash distributed across a set of nodes in the system cluster1105. In the present example, the set of nodes is a name service group that is determined by the cluster manager1114.

The DMO owner1142is a single-node service that manages a DMO. The node corresponding to the client proxy1140that creates the DMO becomes the DMO owner1142. The DMO owner1142is responsible for selecting (via a cluster manager1114) an initial object node group1130to contain the DMO and for assigning the chunk replicas1144within that node group1130. Some embodiments may contain additional object node groups1132,1134, etc. The DMO owner1142also manages growing, shrinking, migrating, and recovering both the object node group1130as a whole, and the chunk replica1144assignments within that group, as required to meet the DMO's size and replication requirement, or to optimize its usage efficiency. The DMO owner1142can choose to move to another node (e.g., to be on the same node as a write client proxy). If the DMO owner1142node fails, the DMO's node group will re-elect a new DMO owner. The DMO owner1142keeps track of client proxies and orchestrates all updates affecting the DMO (e.g., configuration changes as well as data writes (msync commits and/or write10)).

The chunk replica1144is a slave entity to the object owner1142and client proxy1140. The DMO owner1142and client proxy1140read from and write to the chunk replica1144. The chunk replica1144owns some amount of storage devices (PMEM, SSD, etc.) on its node and manages the details of how/where a chunk of address space is stored therein.

The client proxy1140performs all input/output operations for the client and locally materializes and synchronizes/persists any object that the client requests to be memory mapped. To do that materialization, the client proxy1140creates a local cache for pieces of remote chunks that are in use and manages selection and eviction of pieces that are unused (or less actively used) as capacity constraints require. In some embodiments, the client proxy1140has code to specifically handle page fault notifications received, for example, from the userfaultfd mechanism of Linux. The client proxy1140may similarly be configured to handle other types of page fault notifications in other operating environments.

FIG.12is an illustration of a DMO in a client address space. When a client proxy1140opens a DMO, the client proxy1140allocates a logical address region1210or space for that DMO and registers the region to monitor for page faults. The client proxy1140then direct maps for any local chunks1230at their appropriate offset within the logical address region. Next, the client proxy1140acquires an RDMA access descriptor to an instance of each remote chunk. The client proxy1140then creates and maps one or more persistent memory files to use as a cache1250. Now when the application accesses a region of that DMO space that is not direct mapped, a page fault is signaled and the client proxy's page fault handler will allocate an area of the cache file, fill the cache file via an RDMA read of the appropriate remote chunk area1220, and then map that area of the cache file into its appropriate offset of the DMO region, thus completing the handling of the page fault.

In some embodiments, management of the cache capacity may require that a previously allocated area of cache be removed from its current role in the DMO address space (i.e., evicted) in order to reassign it for a new role. This eviction process can typically happen as a background task where an eviction candidate is selected, unmapped from the DMO space, and written back via an RDMA write to its remote location if required. The cache area of that candidate is then freed for reallocation.

With continued reference toFIG.12and additional reference toFIG.11, a client application installed in a client node or local node, which may be any node in the system cluster ofFIG.11, opens a DMO name. For example, an application may transmit, generate, communicate, or otherwise invoke a memory call to a memory function library or other API associated with DMO system1100(herein referred to as a “DMO API” for illustrative simplicity) which may in turn call a client proxy1140. As previously discussed, in some embodiments, the call by the application may be a captured call to a memory function such as malloc( ). For example, with reference toFIG.2, the other API212may represent a DMO API through which chunks of PMEM from a DMO may be allocated. In this example, the local and/or remote PMEM218bofFIG.2would be part of a DMO. Again, with reference toFIGS.12and11, the DMO API is configured to map an anonymous memory region equal to the size of the DMO, to register that memory region for user page faults, to over map1240the local chunk files on that memory region, and to remember the cache file for later use. The client proxy1140is configured to call the DMO name service1124to get the DMO owner1142, call the DMO owner1142to get table of chunk nodes, to open “local chunk” files that are on the local node, to open an empty “cache file” or “cache object” on the local node, and to reply to the DMO API in the local node with file information including: a file descriptor for the local chunk files on the local node and a file descriptor for the cache file/object. The file descriptor for the local chunks may include an offset within the logical address space for the DMO and a size for the local chunk.

The client application starts using the DMO. In other words, the client application can perform load/store references to the DMO, and/or read/write input/output calls to/from the DMO. If a load/store reference from the client application accesses a DMO region that is not over mapped, the client application may take or otherwise receive a page fault. The DMO API may receive a page fault notification and calls to the client proxy1140. The client proxy1140caches the needed region into the cache file and replies to the DMO API. The DMO API then can over map the new region onto an appropriate local DMO space.

Thus, from a client application perspective, the DMO system1100enables a user, via the client application in conjunction with a client proxy, to initiate the use of a DMO, have data placed in one or more memory regions mapped to the DMO by either of a store call or a write call, and access data stored in one or more memory regions mapped to the DMO by a load call or a read call.

A DMO can be accessed in one of the two modes, namely, storage mode and memory mode. In general, storage mode allows a user to perform conventional file and object operations on DMOs. Such operations include open, read, write, close, memory map operations, and directory operations, etc. A DMO in storage mode can be replicated for high availability. Memory mode allows a user to access data using memory semantics such as memory allocation, deallocation, and accessing data using pointer semantics. Therefore, reading from and writing to memory mode objects are achieved via memory load and store semantics. Captured memory calls from an application may be handled using a DMO in memory mode.

FIG.13is a block diagram that illustrates various APIs that may be associated with the various data access modes of an example DMO system1100, namely a DMO storage mode1310and DMO memory mode1320.

Storage mode1310is designed for “write-once, read many times” objects. The core storage mode APIs are shown in the left side ofFIG.13. A storage mode DMO can be created using the function mvfs create( ) An application can then write data to the storage mode DMO using mvfs write( ) and close the storage mode DMO using mvfs close( ) After a storage mode DMO is closed, the storage mode DMO cannot be rewritten. To read data from a storage mode DMO, an application may first open the storage mode DMO using mvfs open( ) then read data from the storage mode DMO using mvfs read( ) A storage mode DMO can be removed with mvfs unlink( ) An application can also map a storage mode DMO into its virtual memory address using the function mvfs mmap( ) and read data through a memory copy. Note, since storage mode DMOs are write-once, the resulting mapped addresses are read-only. A storage mode DMO can be replicated to different cluster nodes to increase availability. For example, this can be performed by passing a replication policy for the storage mode DMO when the function mvfs create( ) is called by an application. Additionally, storage mode APIs further allow users to create objects under directories.

Various example APIs associated with a DMO memory mode620are shown on the right side ofFIG.13. These APIs allow for the creation and destruction of memory mode DMOs via memory allocation and deallocation operations, respectively. In an example embodiment, read and write operations are performed through memory copy realized by CPU load and store instructions.

A new memory mode DMO may be created by allocating it with the function dmo_malloc( ). As alluded to previously, in some embodiments, a DMO may be created and allocated in response to capturing a call by an application to a memory function112such as malloc( ). With reference toFIG.2, an intercepted call to malloc( ) may be captured by memory allocation capture library220and translated into a call to dmo_malloc( ) to create a memory mode DMO and/or allocate a chunk of PMEM associated with an existing memory mode DMO. In other words, the dmo_malloc( ) function associated with the DMO API may be similar to the malloc( ) function, but may instead cause memory allocation in a DMO system instead of DRAM. Instead of returning a pointer to a first byte in an allocated chunk of DRAM, the dmo_malloc( ) function may return a pointer to the first byte of a newly allocated memory mode DMO, upon a successful operation. The application can then start writing and reading data by performing memory copy operations to and from the memory address range of the allocated memory mode DMO.

An allocated memory mode DMO can be destroyed with the function dmo_free( ) The allocated memory space will then be reclaimed by DMO system1100. The function dmo_msync( ) may be called after writing data via memory copy to make sure data are fully written into PMEM (as data may also partially stay in CPU cache).

A DMO system1100may support both volatile memory mode DMOs as well as persistent memory mode DMOs. Although DMOs are persistent by nature, there may be situations when persistence is not necessary, and the memory is instead used in a volatile mode. This may occur, for example, when a memory mode DMO is needed to provide additional byte-addressable memory to an application that has exceeded available DRAM. Note, the term “volatile memory mode” is not to be confused with “volatile memory” (e.g., DRAM, SDRAM, and SRAM) which is volatile by nature of the hardware. A volatile memory mode DMO may be implemented using non-volatile memory hardware such as PMEM. One way to implement these two kinds of memory mode DMOs is to include an additional input parameter for the dmo_malloc( ) function. The parameter can be a string, representing the name of the memory mode DMO to be allocated. When the input name is empty, the DMO system1100generates an internal unique name and allocates a volatile memory mode DMO. In turn, a volatile memory mode DMO is destroyed when a user deallocates the object or disconnects from a DMO system1100. A volatile memory mode DMO will also be destroyed after DMO itself reboots. When input name passed to dmo_malloc( ) is non-empty, a persistent memory mode DMO will be allocated. Since a volatile memory mode DMO does not have a user-given name and therefore cannot be described, other processes cannot access it.

As opposed to a volatile memory mode DMO, a persistent memory mode DMO survives across user disconnection and/or reboot of the DMO system1100. As long as the object is not deallocated, a persistent memory mode DMO can be retrieved using a dmo_recall function by passing in the name of the object. The function dmo_recall then looks up the object in DMO system1100. Upon success, it returns the pointer to the first byte of the allocated persistent memory mode DMO. A user can then use the returned pointer to continue data access. As a persistent memory mode DMO has a user-given name, the object can be accessed from all the DMO nodes.

In some embodiments, all captured memory calls from an application may be handled using volatile memory mode DMOs. Alternatively, in some embodiments, certain captured memory calls may be handled using volatile memory mode DMOs while others are handled using persistent memory model DMOs. This selective allocation of volatile and persistent memory mode DMOs can be configured using the allocation policy associated with the memory allocation capture library. For example, the allocation policy may specify that a first type of memory call is handled using a volatile memory mode DMO while a second type of memory call is handled using a persistent memory mode DMO.

Memory mode APIs provide a user data access experience that is close to conventional local memory access (i.e., data locality, low latency, pointer semantics). Therefore, a memory mode DMO object can always be rewritten. However, storage features such as replication and directory support may not be available for such objects for performance and usability considerations.

In some embodiments, the DMO system1100may enable DMOs to be converted between different modes. For example, a persistent memory mode DMO can be converted to a storage mode DMO using the to_storage_mode API. A user can further pass in a parameter that specifies the replication policy. In this case, the DMO system400will switch the DMO from memory mode to storage mode, and start replicating the DMO across nodes following the specified replication policy. After conversion, the DMO may only be accessed using one or more of the storage mode APIs. Similarly, a storage mode DMO can be converted to a memory mode DMO using the to_memory_mode API. In doing so, all the replicas of the DMO will be invalidated, becoming point-in-time snapshots. Upon success, the function returns a pointer pointing to the first byte of the converted memory mode DMO. After the conversion, the memory mode DMO becomes writeable and only memory mode APIs can be used for accessing the DMO's data. Replication support may no longer be available to the DMO.

Computer Processing System

FIG.14is a block diagram illustrating an example of a computer processing system1400in which at least some operations described herein can be implemented. For example, some components of the computer processing system1400may be part of a computer system executing an application (e.g., application110) and/or any one or more of the nodes associated with a distributed computing cluster such as DMO system1100described with respect toFIG.11.

The processing system1400may include one or more central processing units (“processors”)1402, main memory1406, non-volatile memory1410, network adapter1412(e.g., network interface), video display1418, input/output devices1420, control device1422(e.g., keyboard and pointing devices), drive unit1424including a storage medium1426, and signal generation device1430that are communicatively connected to a bus1416. The bus1416is illustrated as an abstraction that represents one or more physical buses and/or point-to-point connections that are connected by appropriate bridges, adapters, or controllers. The bus1416, therefore, can include a system bus, a Peripheral Component Interconnect (PCI) bus or PCI-Express bus, a HyperTransport or industry standard architecture (ISA) bus, a small computer system interface (SCSI) bus, a universal serial bus (USB), IIC (I2C) bus, or an Institute of Electrical and Electronics Engineers (IEEE) standard 1394 bus (also referred to as “Firewire”).

The processing system1400may share a similar computer processor architecture as that of a server computer, a desktop computer, a tablet computer, personal digital assistant (PDA), mobile phone, a wearable electronic device (e.g., a watch or fitness tracker), network-connected (“smart”) device (e.g., a television or home assistant device), virtual/augmented reality systems (e.g., a head-mounted display), or any other electronic device capable of executing a set of instructions (sequential or otherwise) that specify action(s) to be taken by the processing system1400.

While the main memory1406, non-volatile memory1410, and storage medium1426(also called a “machine-readable medium”) are shown to be a single medium, the term “machine-readable medium” and “storage medium” should be taken to include a single medium or multiple media (e.g., a centralized/distributed database and/or associated caches and servers) that store one or more sets of instructions1428. The term “machine-readable medium” and “storage medium” shall also be taken to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by the processing system1400.

Moreover, while embodiments have been described in the context of fully functioning computing devices, those skilled in the art will appreciate that the various embodiments are capable of being distributed as a program product in a variety of forms. The disclosure applies regardless of the particular type of machine or computer-readable media used to actually effect the distribution.

Further examples of machine-readable storage media, machine-readable media, or computer-readable media include recordable-type media such as volatile and non-volatile memory devices1410, floppy and other removable disks, hard disk drives, optical discs (e.g., Compact Disc Read-Only Memory (CD-ROMS), Digital Versatile Discs (DVDs)), and transmission-type media such as digital and analog communication links.

The network adapter1412enables the processing system1400to mediate data in a network1414with an entity that is external to the processing system1400through any communication protocol supported by the processing system1400and the external entity. The network adapter1412can include a network adaptor card, a wireless network interface card, a router, an access point, a wireless router, a switch, a multilayer switch, a protocol converter, a gateway, a bridge, a bridge router, a hub, a digital media receiver, and/or a repeater.

The network adapter1412may include a firewall that governs and/or manages permission to access/proxy data in a computer network, as well as tracks varying levels of trust between different machines and/or applications. The firewall can be any number of modules having any combination of hardware and/or software components able to enforce a predetermined set of access rights between a particular set of machines and applications, machines and machines, and/or applications and applications (e.g., to regulate the flow of traffic and resource sharing between these entities). The firewall may additionally manage and/or have access to an access control list that details permissions including the access and operation rights of an object by an individual, a machine, and/or an application, and the circumstances under which the permission rights stand.

The techniques introduced here can be implemented by programmable circuitry (e.g., one or more microprocessors), software and/or firmware, special-purpose hardwired (i.e., non-programmable) circuitry, or a combination of such forms. Special-purpose circuitry can be in the form of one or more application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), etc.

Remarks

Although the Detailed Description describes certain embodiments and the best mode contemplated, the technology can be practiced in many ways no matter how detailed the Detailed Description appears. Embodiments may vary considerably in their implementation details, while still being encompassed by the specification. Particular terminology used when describing certain features or aspects of various embodiments should not be taken to imply that the terminology is being redefined herein to be restricted to any specific characteristics, features, or aspects of the technology with which that terminology is associated. In general, the terms used in the following claims should not be construed to limit the technology to the specific embodiments disclosed in the specification, unless those terms are explicitly defined herein. Accordingly, the actual scope of the technology encompasses not only the disclosed embodiments, but also all equivalent ways of practicing or implementing the embodiments.