Systems and methods for preventing kernel stalling attacks

Systems and methods for preventing kernel stalling attacks. An example method may comprise receiving, by a kernel, an address range associated with a data store of an application program; mapping, by the kernel, a portion of random access memory (RAM) to the address range; disabling page fault handling with respect to addresses falling within the address range; and responsive to receiving, from the application program, a memory access request specifying an address outside of the address range, returning a memory access error to the application program.

TECHNICAL FIELD

The present disclosure is generally related to virtualized computer systems, and more particularly, to preventing the kernel from stalling due to malicious attacks.

BACKGROUND

Modern computer systems include memory management features that provide memory protection. The memory management features may restrict particular processes from accessing particular portions of memory. The restriction may be enforced using a combination of hardware features and kernel features that work together to enable or restrict an executing process from accessing memory resources. The memory resources are often separated into user space and kernel space and when a user space process attempts to access memory resources in kernel space the memory management features may generate a segmentation fault. The segmentation fault may indicate that an access violation occurred so that it can be handled accordingly.

DETAILED DESCRIPTION

Described herein are systems and methods for preventing the kernel from stalling due to malicious third party actions.

Modern computer systems may include memory virtualization features that contain security enhancements to reduce the computer systems' vulnerability to attack. An example of a security enhancement involves segregating virtual memory into kernel space and user space. The kernel space may be reserved for running a privileged operating system kernel, kernel extensions, and most device drivers. The user space may be a memory area where application software (e.g., application programs) and some drivers execute. Memory virtualization details are often implemented by features of a kernel and may be hidden from a processing thread executing in user space. Application programs running on the computer system may communicate with the kernel using system calls. A system call is a programmatic way in which an application program may request a service from the kernel. For example, in response to a request by the application program through a system call, the kernel may provide services such as memory management, process management, file management, and input/output (I/O) management.

In current computer systems, in response to the kernel receiving a request to access the user space (memory) via a system call, the kernel may first verify whether the application program attempting to access the memory location has permission to access the memory location. For example, the kernel may determine that the pointer associated with the memory access request received includes a restricted address, such as an address associated with the kernel space or another application program's user space. A pointer is an object (e.g., variable, data structure, function, etc.) that can store a memory address from which data is to be read or to which data of one or more data streams is to being programmed (written). If the pointer does include a restricted address, the kernel may generate a segmentation fault (e.g., a memory access violation error), which is a failure condition notifying the operating system that the application program attempted to access a restricted area of memory. Otherwise, the kernel may process the memory access request. When the user application requests access to a portion of virtual memory that is not present in (loaded into) physical memory, the underlying hardware may generate a hardware event (e.g., page fault). The kernel may detect the hardware event and handle the hardware event by retrieving the requested portion of virtual memory from a secondary storage device (e.g., swap space of a hard disk). The process of detecting the hardware events and retrieving the data corresponding to the portion of virtual memory is often handled by the kernel and is typically hidden from user space processing threads.

However, in some instances, malicious software may intentionally trigger a hardware event, such as a page fault, by using a system call to target data stored on the hard disk. Since retrieving data from the hard disk is a relatively lengthy process, triggering a series of page faults may allow the malicious software to effectively stall the kernel while the kernel processes the page faults. Stalling the kernel renders it unresponsive and, thus, inaccessible to other computer applications, which can be considered as a denial-of-service (DoS) situation.

Aspects of the present disclosure address the above and other deficiencies by providing technology that can protect the kernel from malicious stalling attacks. In particular, aspects of the present disclosure enable an application program to first generate a memory buffer in user space. The buffer may be used for system calls generated by the application program. The application program can send a virtual address range of the memory buffer to the kernel and indicate to the kernel that the parameters of all system calls generated by the application program should reside within the buffer. The kernel can first perform a verification check on the memory buffer. The verification check can include determining that the size of the memory buffer is below a threshold criterion (e.g., below a memory size value), is within a permissible address range (e.g., within the application program's allowable user space). Once the buffer is validated, the kernel may map a portion of random access memory (RAM) to the address range of the memory buffer. With the address range mapped to the RAM, the kernel can disable page fault handling with respect to the virtual addresses within the address range. By mapping the address range to the RAM, the kernel may ensure that any valid system call from the application program would not require a page fault to retrieve data from a hard disk. In one example, the kernel may disable page fault handling by pinning memory pages associated with the virtual memory addresses using a pinning function. Pinning the memory pages prevents the memory pages from being evicted from the RAM. Thus, pinning the memory pages disables page fault handling with respect to the memory pages associated with the virtual addresses, which the kernel prevents itself from triggering a hardware event, such as a page fault to retrieve a requested portion of virtual memory from a cache in a hard disk.

The application program may copy parameter values to the memory buffer. The application program may then invoke a system call referencing the addresses associated with the parameter values. The kernel may validate the addresses referenced by the system call against the address range of the reserved memory buffer. In response to the addresses being within the address range, the kernel may execute the system call. In response to the system call referencing an address outside the address range of the memory buffer, the kernel may return a memory access error to the application program. Additionally, in response to the system call referencing addresses inside the address range of the memory buffer, but a memory page is not present for a referenced address, the kernel may also return a memory access error to the application program without handling the page fault, thus ensuring that the application program cannot stall the kernel. Accordingly, aspects of the present disclosure protect the kernel from intentionally stalling due to malicious attacks.

Various aspects of the above referenced methods and systems are described in details herein below by way of examples, rather than by way of limitation. The examples provided below discuss a computer system where the computing processes may be managed by aspects of a kernel, a hypervisor, a host operating system, a virtual machine, or a combination thereof. In other examples, the computing processes may be performed in a computer system that is absent a hypervisor or other hardware virtualization features (e.g., virtual machines) discussed below.

FIG.1depicts an illustrative architecture of elements of a computing system100, in accordance with an embodiment of the present disclosure. Computing system100may be a single host machine or multiple host machines arranged in a heterogeneous or homogenous group (e.g., cluster) and may include one or more rack mounted servers, workstations, desktop computers, notebook computers, tablet computers, mobile phones, palm-sized computing devices, personal digital assistants (PDAs), etc. It should be noted that other architectures for computing system100are possible, and that the implementation of a computing system utilizing embodiments of the disclosure are not necessarily limited to the specific architecture depicted. In one example, computing system100may be a computing device implemented with x86 hardware. In another example, computing system100may be a computing device implemented with PowerPC®, SPARC®, or other hardware. In the example shown inFIG.1, computing system100may include a supervisor110, computing processes120A-C, computing resources130, and a network140.

Supervisor110may manage the execution of one or more computing processes and provide the computing processes with access to one or more underlying computing devices (e.g., hardware or virtualized resources). Supervisor110may be the same or similar to a kernel and may be a part of an operating system, hypervisor, or a combination thereof. Supervisor110may interact with hardware devices130and provide hardware virtualization, operating-system virtualization, other virtualization, or a combination thereof. Hardware virtualization may involve the creation of one or more virtual machines that emulate an instance of a physical computing machine. Operating-system-level virtualization may involve the creation of one or more containers that emulate an instance of an operating system. In one example, supervisor110may be part of a non-virtualized operating system that is absent hardware virtualization and operating-system-level virtualization and each of the computing processes120A-C may be an application process managed by the non-virtualized operating system. In another example, supervisor110may be a hypervisor or include hypervisor functionality and each of computing processes120A-C may execute within a separate virtual machine or container. In either example, the supervisor may be implemented as part of a kernel and execute as one or more processes in kernel space (e.g., privileged mode, kernel mode, root mode).

In the example, shown inFIG.1, supervisor110may include memory isolation component112. Memory isolation component112may enable supervisor110to manage access to user space memory116by disabling hardware event handling (e.g., page fault handling) during a system call and restricting supervisor110from accessing user space memory outside of a selected address range. Memory isolation component112is discussed in more detail in regards toFIG.2and may be used individually or in combination to provide enhanced memory management features for computing processes120A-C.

Computing processes120A-C may include a sequence of instructions that can be executed by one or more processing devices (e.g., physical processing devices134). A computing process may be managed by supervisor110or may be a part of supervisor110. For example, supervisor110may execute as one or more computing processes that cooperate to manage resource accessed by computing processes120A-C. Each computing process may include one or more threads, processes, other stream of executable instructions, or a combination thereof. A thread may any computer based “thread of execution” and may be the smallest sequence of programmed instructions managed by supervisor110. A process may include one or more threads and may be an instance of an executable computer program.

Computing processes120A-C may be associated with a particular level of privilege that may be the same or similar to protection levels (e.g., processor protection rings). The privilege level may indicate an access level of a computing process to computing devices (e.g., memory, processor, or other virtual or physical resources). There may be multiple different privilege levels assigned to the computing processes120A-C. In one example, the privilege levels may correspond generally to a user mode (e.g., reduced privilege mode, non-root mode, non-privileged mode) and a supervisor mode (e.g., enhanced privilege mode, kernel mode, root mode). The computing process executing in user mode may access resources assigned to the computing processes and may be restricted from accessing resources associated with kernel space or with another user space process (e.g., other portion of user space). For example, each computing process may have its own address space protected from other computing processes. The supervisor mode may enable the computing process to access resources associated with the kernel space and the user space. In other examples, there may be a plurality of privilege levels, and the privilege levels may include a first level (e.g., ring 0) associated with a supervisor/kernel, a second and third level (e.g., ring 1-2), and a fourth level (e.g., ring 3) that may be associated with user space applications.

A computing process may be referred to as a user space process when the computing process is executing with a user mode privilege level. In one example, the privilege level associated with a computing process may change during execution and a computing process executing in user space (e.g., userland) may request and be subsequently granted enhanced privileges by supervisor110. Modifying the privilege level is often associated with a context switch (e.g., system call or hypercall).

User space memory116may be a portion of virtual memory that is assigned to a particular computing process (e.g.,120A). The virtual memory may be managed by supervisor110and may be segregated into kernel space (not shown) and user space. The user space may be further segregated into individual portions that are assigned to respective computing processes120A-C. To simplify the illustration, the portions of the user space assigned to computing process120A is illustrated (e.g., user space memory116) and the portions of user space assigned to computing processes120B and120C are not shown. During execution of computing process120A, the user space memory116may be updated to add or remove executable data and non-executable data.

Hardware devices130may provide hardware resources and functionality for performing computing tasks. Hardware devices130may include one or more physical storage devices132, one or more physical processing devices134, other computing devices, or a combination thereof. One or more of hardware devices130may be split up into multiple separate devices or consolidated into one or more hardware devices. Some of the hardware device shown may be absent from hardware devices130and may instead be partially or completely emulated by executable code.

Physical storage devices132may include any data storage device that is capable of storing digital data and may include volatile or non-volatile data storage. Volatile data storage (e.g., non-persistent storage) may store data for any duration of time but may lose the data after a power cycle or loss of power. Non-volatile data storage (e.g., persistent storage) may store data for any duration of time and may retain the data beyond a power cycle or loss of power. In one example, physical storage devices132may be physical memory and may include volatile memory devices132A (e.g., random access memory (RAM)), non-volatile memory devices132B (e.g., flash memory, NVRAM), and/or other types of memory devices. In another example, physical storage devices132may include one or more mass storage devices, such as hard drives, solid state drives (SSD)), other data storage devices, or a combination thereof. In a further example, physical storage devices132may include a combination of one or more memory devices, one or more mass storage devices, other data storage devices, or a combination thereof, which may or may not be arranged in a cache hierarchy with multiple levels.

Physical processing devices134may include one or more processors that are capable of executing the computing tasks. Physical processing devices134may be a single core processor that is capable of executing one instruction at a time (e.g., single pipeline of instructions) or may be a multi-core processor that simultaneously executes multiple instructions. The instructions may encode arithmetic, logical, or I/O operations. In one example, physical processing devices134may be implemented as a single integrated circuit, two or more integrated circuits, or may be a component of a multi-chip module (e.g., in which individual microprocessor dies are included in a single integrated circuit package and hence share a single socket). A physical processing device may also be referred to as a central processing unit (“CPU”).

Network140may be a public network (e.g., the internet), a private network (e.g., a local area network (LAN), a wide area network (WAN)), or a combination thereof. In one example, network140may include a wired or a wireless infrastructure, which may be provided by one or more wireless communications systems, such as a wireless fidelity (WiFi) hotspot connected with the network140and/or a wireless carrier system that can be implemented using various data processing equipment, communication towers, etc.

FIG.2depicts a block diagram illustrating an exemplary supervisor110that includes technology for providing memory isolation for a computing process, in accordance with one or more aspects of the present disclosure. Supervisor110may be the same or similar to supervisor110ofFIG.1and may include memory isolation component112, and a data store230. The components and modules discussed herein may be performed by any portion of supervisor110(e.g., kernel/hypervisor) or by an application, virtual machine, other portion of a computing system, or a combination thereof. More or less components or modules may be included without loss of generality. For example, two or more of the modules may be combined into a single modules, or features of a module may be divided into two or more modules. In one implementation, one or more of the modules may reside on different computing devices (e.g., a client device and a server device).

Memory isolation component112may be a hardware or software component that enables supervisor110to manage access to a user space memory by disabling hardware event handling (e.g., page fault handling) during system calls and restricting supervisor110from accessing user space memory outside of a selected address range. In one example, memory isolation component112may include mapping module210, fault management module212, and memory access control module214. Each module may include executable code to perform the one or more functions or processes discussed below. Data store118, which may include mapping data220and transfer data222, may be configured by supervisor110and/or computing process120A-C. Data store118may be used for system calls generated by computing process120A-C.

Mapping module210may enable supervisor110to configure a portion of the user space memory of a computing process for memory management. In some embodiments, configuring the portion of user space memory involves mapping virtual memory pages of the user space memory to a portion of volatile memory device132A (e.g., RAM). For example, a computing process (e.g., computing process120A, computing process120B, computing process120C, etc.) can generate data store118(e.g., a memory buffer) in user space (e.g., user space memory116) and send one or more virtual address ranges relating to data store118to supervisor110. The computing process may also indicate to the supervisor110that all system calls from the computing process are to be directed only towards data store118. The virtual memory addresses may be related to an application (e.g., a web browser), a processing thread, a virtual machine, or any other computer software. In other embodiments, the virtual address range relating to data store118may be retrieved, by mapping module210, from a configuration file, received from another computing process, etc.

Mapping module210may perform a verification check on data store118. The verification check can include determining that the size of data store118is below a threshold criterion (e.g., below a memory size value), is within a permissible address range (e.g., within the computing process's allowable user space). Mapping module210may then map a portion of the physical memory (e.g., volatile memory device132A) to the virtual memory addresses. To map the virtual memory addresses, mapping module210may configure one or more page table structures. The page table structures may include mapping data composed of multiple records, where each record correlates an address in user space memory (e.g., a virtual address to an address in physical memory (e.g., a physical address). Mapping module210may generate the page table structures or update existing page table structures. Updating the page tables may involve adding, removing, or replacing mapping data220. Mapping data220may include permission data, location data, other data or a combination thereof. The permission data may indicate the permissions associated with particular locations in virtual memory and whether the data at the locations is executable, non-executable, privileged, non-privileged, or a combination thereof. The location data may identify one or more locations in virtual memory (e.g., virtual memory addresses) and one or more locations in physical memory (e.g., guest physical memory addresses or host physical memory addresses). In one example, the location data may include one-to-one mapping between a location in virtual memory and a location to physical memory. In another example, the location data may include many-to-one mapping and multiple locations in virtual memory may map to the same location in physical memory. In either example, the one or more page table structures may each be a nested page table structure comprising a mapping between a guest virtual memory addresses and host physical memory addresses.

Fault management module212may enable and disable one or more fault handling capabilities (e.g., page fault handling) of supervisor110. In some embodiments, fault management module212may disable one or more fault handling capabilities of supervisor110in response to a system call from the computing process, and enable the one or more fault handling capabilities of supervisor110upon completion of the system call. In one example, fault management module212may disable fault handling capabilities by executing a pinning function (e.g., executing an “mlock( )” function) to pin memory pages associated with the virtual memory addresses to physical memory. To enable the fault handling capabilities, fault management module212may unpin the memory pages. In another example, a hardware element or software function may be used by fault management module212to disable and enable the fault handling capabilities of supervisor110.

Memory access control module214may enable supervisor110to restrict or enable access to data associated with the memory access request from a system call. For example, the computing process may require memory access and copy parameter values to data store118. The computing process may then invoke a system call referencing the addresses associated with the parameter values. Supervisor110may receive the system call and memory access control module214may determine whether a pointer associated with the system call include one or more addresses that are outside the address range of data store118(e.g., validate the addresses referenced by the system call against the address range of the reserved memory buffer). In response to the addresses being within the address range, memory access control module214may execute the system call. In response to determining that at least one address associated with the system call is outside the address range of data store118, memory access control module214may return a memory access error to the computing process. The memory access error may indicate that supervisor110is unable to retrieve the requested data, that the requested data is restricted, that the supervisor110is unable to program (write) data to a storage device associated with the one or more addresses. In some embodiments, in response to the system call referencing one or more addresses inside the address range of data store118, but a memory page is not present for the referenced address inside data store118, memory access control module214can also return a memory access error to the computing process,

In some embodiments, where the system call includes a request to retrieve data, in response to determining that each address associated with the system call is within the address range of data store118, memory access control module214may retrieve the data from the addresses associated with the system call. In some embodiments, where the system call includes a request to program (write) data, in response to determining that each address associated with the system call is within the address range of data store118, memory access control module214may program data to data store118as transfer data222. Once the system call is processed, memory access control module214may unpin the memory pages of data store118and copy (page out) the data from data store118to a hard disk (e.g., non-volatile memory device132B). memory access control module214may re-pin the memory pages in response to the completion of the page out, in response to another system call, etc.

In some embodiments, the processes performed by memory isolation component112may be applied to one or more specific computing processes. For example, the processes performed by memory isolation component112may be applied to computing process120A, but not to computing process120B and120C. In other embodiments, the processes performed by memory isolation component112may be applied to all of the computing processes hosted by supervisor110. For example, the processes performed by memory isolation component112may be applied to computing process120A-C.

In some embodiments, memory isolation component112can use memory protection keys to enable or restrict supervisor110from accessing user space memory outside of a selected address range (e.g., the virtual addresses associated with data store118). Memory protection keys provide a mechanism for enforcing page-based protection without modifying page tables. In particular, one or more bits in each page-table entry can be used to assign a “key” values to any given page. The key values may indicate whether a memory page is readable, writable, executable, protected (e.g., blocks reads), etc. Thus, the memory protection keys allow the kernel to selectively disable or enable access to memory pages.

In one example, memory access control module214can maintain two different memory protection keys for supervisor110. The first protection key can be used to access all of the user space memory116. The second protection key can be used for accessing selected address range only (e.g., data store118). During normal operation, supervisor110can use the first protection key. In response to a system call from, for example, computing process120A, memory access control module214can switch to using the second protection key. As such, during the system call, supervisor110will be unable to access data outside of the selected address range. Upon completion of the system call, memory access control module214can switch back to using the first protection key.

In some embodiments, memory isolation component112can switch page tables to enable or restrict supervisor110from accessing user space memory outside of a selected address range (e.g., the virtual addresses associated with data store118). In some embodiments, an instruction from supervisor110may cause one or more physical processing devices134to switch from one page table structure to another page table structure. The instruction may be exposed to code executing at a user mode privilege level (e.g., non-root), a kernel privilege level (e.g., root), other privilege level, or a combination. As a result, the instruction may be invoked (e.g., called) by computing processes120A-C, supervisor110, or a combination thereof. In one example, the instruction may switch between multiple page table structures by updating processor configuration data in one or more control registers.

In some embodiments, memory isolation component112may maintain two page table structures. The first page table structure may include a first set of mapping records, where each mapping record correlates an address in user space memory and kernel space memory to an address in physical memory. The second page table structure may include a second set of mapping records, where each mapping record correlates an address in the selected address range of user space memory to an address in physical memory. During normal operation, supervisor110can use the first page table structure (a user space and kernel space page table). In response to a system call from, for example, computing process120A, memory access control module214can perform a page table switch by switching to using the second page table structure (selected address range of user space memory page table) instead of the first page table structure. As such, during the system call, supervisor110will be unable to access data outside of the selected address range. Upon completion of the system call, memory access control module214can switch back to using the first page table structure.

FIG.3depicts a flow diagram of an illustrative example of a method300for providing memory isolation for a computing process, in accordance with one or more aspects of the present disclosure. Method300and each of its individual functions, routines, subroutines, or operations may be performed by one or more processors of the computer device executing the method. In certain implementations, method300may be performed by a single processing thread. Alternatively, method300may be performed by two or more processing threads, each thread executing one or more individual functions, routines, subroutines, or operations of the method. In an illustrative example, the processing threads implementing method300may be synchronized (e.g., using semaphores, critical sections, and/or other thread synchronization mechanisms). Alternatively, the processes implementing method300may be executed asynchronously with respect to each other.

Method300may be performed by processing devices of a server device or a client device and may begin at block302. At block302, a kernel may receive an address range associate with an application program. At block304, the kernel may map a portion of random access memory (RAM) to the address range. In some embodiments, the kernel may pin memory pages associated with the addresses falling within the address range by executing a pinning function. At block306, the kernel may disable page fault handling with respect to addresses falling within the address range.

At block308, responsive to receiving from the application program a memory access request specifying an address outside of the address range, the kernel may return a memory access error to the application program. Alternatively, responsive to receiving, from the application program, a memory access request specifying an address within the address range, the kernel may process the memory request (e.g., program data to memory associated with the address range, retrieve data from memory associated with the address range, etc.). Responsive to completion of the processing of the memory request, the kernel may enable page fault handling with respect to the address falling within the address range. In some embodiments, responsive to processing the memory access request, the kernel may page out data (to a hard disk) associated with the memory access request (e.g., a write request).

In some embodiments, the kernel may maintain a first memory protection key and a second memory protection key. The first memory protection key may be used to access user space memory while the second memory protection key may be used to access only the address range in the user space memory. Responsive to receiving, from the application program, the memory access request, the kernel may switch from using the first protection key to using the second protection key.

In some embodiments, the kernel may maintain a first page table structure and a second page table structure. The first page table structure may include a first set of mapping records, where each mapping record of the first set correlates an address in user space memory to an address in physical memory. The second page table structure may include a second set of mapping records, where each mapping record of the second set correlates an address in the address range of the user space memory to an address in the physical memory. Responsive to receiving, from the application program, the memory access request, the kernel may switch from using the first page table structure to using the second page table structure. Responsive to completing the operations described herein above with references to block408, the method may terminate.

FIG.4depicts a block diagram of a computer system400operating in accordance with one or more aspects of the present disclosure. Computer system400may be the same or similar to supervisor110and computer system100and may include one or more processing devices and one or more memory devices. In the example shown, computer system400may include mapping module410, fault management module420, and memory access control module430.

Mapping module410may receive an address range associate with an application program. The mapping module may then map a portion of random access memory (RAM) to the address range. In some embodiments, mapping module410may pin memory pages associated with the addresses falling within the address range by executing a pinning function.

Fault management module420may disable page fault handling with respect to addresses falling within the address range. Responsive to receiving from the application program a memory access request specifying an address outside of the address range, memory access control module430may return a memory access error to the application program. Alternatively, responsive to receiving, from the application program, a memory access request specifying an address within the address range, memory access control module430may process the memory request (e.g., program data to memory associated with the address range, retrieve data from memory associated with the address range, etc.). Responsive to completion of the processing of the memory request, memory access control module430may enable page fault handling with respect to the address falling within the address range. In some embodiments, responsive to processing the memory access request, memory access control module430may page out data (to a hard disk) associated with the memory access request (e.g., a write request).

In some embodiments, memory access control module430may maintain a first memory protection key and a second memory protection key. The first memory protection key may be used to access user space memory while the second memory protection key may be used to access only the address range in the user space memory. Responsive to receiving, from the application program, the memory access request, memory access control module430may switch from using the first protection key to using the second protection key.

In some embodiments, memory access control module430may maintain a first page table structure and a second page table structure. The first page table structure may include a first set of mapping records, where each mapping record of the first set correlates an address in user space memory to an address in physical memory. The second page table structure may include a second set of mapping records, where each mapping record of the second set correlates an address in the address range of the user space memory to an address in the physical memory. Responsive to receiving, from the application program, the memory access request, memory access control module430may switch from using the first page table structure to using the second page table structure.

FIG.5depicts a flow diagram of one illustrative example of a method500for providing memory isolation for a computing process, in accordance with one or more aspects of the present disclosure. Method500may be similar to method500and may be performed in the same or a similar manner as described above in regards to method500. Method500may be performed by processing devices of a server device or a client device and may begin at block502.

At block502, a processing device may receive an address range associate with an application program. At block504, the processing device may map a portion of random access memory (RAM) to the address range. In some embodiments, the kernel may pin memory pages associated with the addresses falling within the address range by executing a pinning function. At block506, the processing device may disable page fault handling with respect to addresses falling within the address range.

At block508, responsive to receiving from the application program a memory access request specifying an address outside of the address range, the processing device may a memory access error to the application program. Alternatively, responsive to receiving, from the application program, a memory access request specifying an address within the address range, the processing device may process the memory request (e.g., program data to memory associated with the address range, retrieve data from memory associated with the address range, etc.). Responsive to completion of the processing of the memory request, the processing device may enable page fault handling with respect to the address falling within the address range. In some embodiments, responsive to processing the memory access request, the processing device may page out data (to a hard disk) associated with the memory access request (e.g., a write request).

In some embodiments, the processing device may maintain a first memory protection key and a second memory protection key. The first memory protection key may be used to access user space memory while the second memory protection key may be used to access only the address range in the user space memory. Responsive to receiving, from the application program, the memory access request, the kernel may switch from using the first protection key to using the second protection key.

In some embodiments, the processing device may maintain a first page table structure and a second page table structure. The first page table structure may include a first set of mapping records, where each mapping record of the first set correlates an address in user space memory to an address in physical memory. The second page table structure may include a second set of mapping records, where each mapping record of the second set correlates an address in the address range of the user space memory to an address in the physical memory. Responsive to receiving, from the application program, the memory access request, the processing device may switch from using the first page table structure to using the second page table structure. Responsive to completing the operations described herein above with references to block508, the method may terminate.

FIG.6depicts a block diagram of a computer system operating in accordance with one or more aspects of the present disclosure. In various illustrative examples, computer system600may correspond to computing device100ofFIG.1or computer system200ofFIG.2. The computer system may be included within a data center that supports virtualization. Virtualization within a data center results in a physical system being virtualized using virtual machines to consolidate the data center infrastructure and increase operational efficiencies. A virtual machine (VM) may be a program-based emulation of computer hardware. For example, the VM may operate based on computer architecture and functions of computer hardware resources associated with hard disks or other such memory. The VM may emulate a physical computing environment, but requests for a hard disk or memory may be managed by a virtualization layer of a computing device to translate these requests to the underlying physical computing hardware resources. This type of virtualization results in multiple VMs sharing physical resources.

In a further aspect, the computer system600may include a processing device602, a volatile memory604(e.g., random access memory (RAM)), a non-volatile memory606(e.g., read-only memory (ROM) or electrically-erasable programmable ROM (EEPROM)), and a data storage device616, which may communicate with each other via a bus608.

Computer system600may further include a network interface device622. Computer system600also may include a video display unit610(e.g., an LCD), an alphanumeric input device612(e.g., a keyboard), a cursor control device614(e.g., a mouse), and a signal generation device620.

Data storage device616may include a non-transitory computer-readable storage medium624on which may store instructions626encoding any one or more of the methods or functions described herein, including instructions for implementing methods300or500and for memory isolation component112, and modules illustrated inFIGS.1and2.

Instructions626may also reside, completely or partially, within volatile memory604and/or within processing device602during execution thereof by computer system600, hence, volatile memory604and processing device602may also constitute machine-readable storage media.