CACHING MEMORY MAPPED I/O EMULATION FOR VIRTUAL MACHINES

A method includes selecting a guest address of an address space of a virtual machine for optimizing memory mapped input/output (MMIO) emulation and initializing a page modification log associated with the virtual machine as full. The method further includes receiving an access of the guest address by the virtual machine and, in response to receiving the access to the guest address, triggering a page modification log full event, and exiting to a hypervisor of the virtual machine to perform a task associated with the guest address.

TECHNICAL FIELD

Aspects of the present disclosure relate to memory mapped input/output (MMIO) emulation in virtual environments, and more particularly, caching MMIO address translations using a page modification log for a virtual machine.

BACKGROUND

A hypervisor abstracts the physical resources of a computing device such as physical processing devices (e.g., processors, CPUs, etc.) and physical memory (e.g., RAM) into virtual resources. The hypervisor manages and assigns the virtual resources to virtual machines (VMs) running on the computing device. A virtual machine (VM) is the virtualization or emulation of a computer system, and may be run on a server. Virtual machines are based on computer architectures and provide functionality of a physical computer. VM implementations may involve specialized hardware, software, or a combination of the two.

DETAILED DESCRIPTION

Memory mapped input/output (MMIO) is a method for using the same address space for main memory and memory of an i/o device of a computing system. In the context of virtualized environments, a virtual machine (VM) may emulate an MMIO scheme to signal events or requests to a hypervisor managing the virtual machine. For example, when the virtual machine accesses a specific subset of memory (e.g., MMIO memory) of the guest address space, an event is signaled to the hypervisor. In conventional MMIO emulation, the access of the MMIO memory by the guest virtual machine triggers a page table violation or misconfiguration fault to cause the central processing unit (CPU) to exit to the context of the hypervisor. The hypervisor may then handle the corresponding event. However, when a fault triggers the exit to the hypervisor, the address translation from the guest address space to the host address space is not cached. Therefore, each time the MMIO guest address is accessed an address translation has to reoccur causing significant latency and additional computation costs for the VM exits.

While conventional systems may include VM exit capabilities beyond MMIO emulation, such as a hypercall to the hypervisor, MMIO emulation has several distinct advantages over such methods. In particular, an MMIO exit allows for transparent virtualization because the address can belong to the hypervisor, a physical device, or a virtual device in a nested VM configuration. Additionally, a guest VM can provide specific components within the VM (e.g., applications, VMs, virtual devices, etc.) access to the MMIO address using guest page table entries (PTEs).

Aspects of the disclosure address the above-noted and other deficiencies by providing techniques for caching MMIO emulation address translations. Embodiments may utilize the page modification log capabilities of a processing device to log and cache the address translations upon accessing an MMIO emulated address. For example, a hypervisor may identify at least one guest page address that is used for MMIO emulation to be optimized for address translation caching. In some examples, the hypervisor may select the MMIO guest page address that is accessed most frequently. The hypervisor may mark the selected MMIO guest page address as valid (e.g., by clearing the accessed and dirty bits in the host page table corresponding to that address). The hypervisor may also initialize each of the other host page table entries corresponding to the guest address space of the virtual machine by setting either one or both of the accessed and dirty bits of each of the host page table entries. The hypervisor may further initialize a page modification log as full.

In some examples, when the guest virtual machine accesses the MMIO address, processing logic may determine that the accessed or dirty bits are not set in the host page table entry corresponding to the guest MMIO address and attempt to increment the page modification log. Because the PML is initialized to full, a PML full event will be triggered. The PML full event may log the address translation (e.g., guest address to host address translation) and the processing logic may then cache the address translation in the MMU of the CPU. Accordingly, future accesses to the MMIO addresses for VM exits to the hypervisor may be performed more quickly using the cached address translation. In some examples, the processor executing the virtual machine may perform the address translation and log it in the PML. The hypervisor or host may then retrieve the address translation from the PML. In another example, the guest virtual machine may provide the guest address directly to the hypervisor via para-virtualization. In other examples, where only a single address is being optimized for MMIO emulation, the hypervisor may already be aware of the selected address because only one address will be capable of triggering the corresponding PML full event and exit to hypervisor.

In another example, the hypervisor may initialize the page modification log to the last index or last entry such that one more log may occur before a PML full event is triggered. Therefore, upon access to a first MMIO address, information may be logged in the last PML entry for use by the hypervisor in one or more VM management tasks. Then upon the second guest access to an MMIO address, the PML full event may be triggered, causing the log of the address translations for the MMIO addresses and the exit to the hypervisor. Accordingly, additional information may be passed to the hypervisor at the VM exit.

Embodiments of the present disclosure provide for reduced latency for VM exits while maintaining transparent virtualization of fault based VM exits. The caching of an MMIO address translation reduces latency as well the use of computing resources required to perform a page walk for address translations. Additionally, the dedicated exit reasons for MMIO access exits to the hypervisor provide for various potential optimizations of VM exits by using the PML full event to cache address translations.

FIG.1is a system diagram of an example computer system architecture100for cacheable MMIO emulation for virtual machines, in accordance with some embodiments of the present disclosure. One skilled in the art will appreciate that other computer system architectures are possible, and that the implementation of a computer system utilizing examples of the invention are not necessarily limited to the specific architecture depicted byFIG.1.

As shown inFIG.1, computer system architecture100includes host system110. The host system110includes one or more processing devices160, memory170, which may include volatile memory devices (e.g., random access memory (RAM)), non-volatile memory devices (e.g., flash memory) and/or other types of memory devices, a storage device180(e.g., one or more magnetic hard disk drives, a Peripheral Component Interconnect [PCI] solid state drive, a Redundant Array of Independent Disks [RAID] system, a network attached storage [NAS] array, etc.), and one or more devices190(e.g., a Peripheral Component Interconnect [PCI] device, network interface controller (NIC), a video card, an I/O device, etc.). In certain implementations, memory170may be non-uniform access (NUMA), such that memory access time depends on the memory location relative to processing devices160A. It should be noted that although, for simplicity, host system110is depicted as including a single processing device160, storage device180, and device190inFIG.1, other embodiments of host systems110A may include a plurality of processing devices, storage devices, and devices. The host system110may be a server, a mainframe, a workstation, a personal computer (PC), a mobile phone, a palm-sized computing device, etc. In embodiments, host system110may be a standalone computing device. In some embodiments, host system110may be included in a cluster of computing devices. Accordingly, embodiments of the disclosure may utilize any number of host systems. For example, computer system architecture100may include a cluster of computing nodes in which host system110is included.

Host system110may additionally include one or more virtual machines (VMs)130A-B and host operating system (OS)120. VMs130A-B may be a software implementation of a machine that executes programs as though it were an actual physical machine. Host OS120manages the hardware resources of the computer system and provides functions such as inter-process communication, scheduling, memory management, and so forth.

Host OS120may include a hypervisor125(which may also be known as a virtual machine monitor (VMM)), which provides a virtual operating platform for VMs130A-B and manages their execution. Hypervisor125may manage system resources, including access to physical processing devices (e.g., processors, CPUs, etc.), physical memory (e.g., RAM), storage device (e.g., HDDs, SSDs), and/or other devices (e.g., sound cards, video cards, etc.). The hypervisor125, though typically implemented in software, may emulate and export a bare machine interface to higher level software in the form of virtual processors and guest memory. Higher level software may comprise a standard or real-time OS, may be a highly stripped down operating environment with limited operating system functionality, and/or may not include traditional OS facilities, etc. Hypervisor125may present other software (i.e., “guest” software) the abstraction of one or more VMs that provide the same or different abstractions to various guest software (e.g., guest operating system, guest applications). It should be noted that in some alternative implementations, hypervisor125may be external to host OS120, rather than embedded within host OS120, or may replace host OS120.

The host system110may be coupled (e.g., may be operatively coupled, communicatively coupled, may communicate data/messages with each other) with one or more client devices via a network. A network may be a public network (e.g., the internet), a private network (e.g., a local area network (LAN) or wide area network (WAN)), or a combination thereof. In one embodiment, a network may include a wired or a wireless infrastructure, which may be provided by one or more wireless communications systems, such as a WiFi™ hotspot connected with the network and/or a wireless carrier system that can be implemented using various data processing equipment, communication towers (e.g., cell towers), etc. The network may carry communications (e.g., data, message, packets, frames, etc.) between the various components of host system110and the one or more client devices.

In some examples, the hypervisor125may include processing logic (e.g., VM exit component115) to select an MMIO emulated address of a guest address space132A-B of a virtual machine130A-B for VM exit optimization. The VM exit component115of the hypervisor125may further initialize a host page table entry corresponding to the selected MMIO emulated address host page table entry of a host page table126as valid. The VM exit component115may further set a valid or accessed bit of the remaining host page table entries of the host page table126that correspond to the guest address space132A-B of the virtual machine130A-B. The VM exit component115may also initialize a page modification log128as full. Upon access of the selected MMIO emulated address by the VM130, the VM exit component115may attempt to decrement an index in a page modification log128. Since the page modification log128is initialized as full, the access to the MMIO emulated address will trigger a PML full event. The PML full event will cause an exit to the hypervisor125and cache the address translation of the MMIO emulated address. Additionally, the index of the page modification log128will not be decremented upon access of the remaining host page table entries (e.g., any address other than the optimized address) of the guest address space132. Therefore, the VM130will only exit to the hypervisor125upon access to the MMIO emulated address. Further details regarding the VM exit component115will be discussed atFIGS.2-8below.

FIG.2depicts an example page table initialization and process flow for cacheable MMIO emulation for virtual machines, in accordance with embodiments of the disclosure. The host page table entries corresponding to guest addresses of a virtual machine may be initialized as provided in host page table230ofFIG.2when a guest virtual machine210is in an MMIO optimized mode or setting. The host page table230includes multiple host page table entries232A-E each mapping a guest memory address to a physical memory address as well as host page table entries234A-B mapping MMIO emulated guest addresses to a physical memory address. Each page table entry232A-E of the host page table230that is mapped to by a guest address space of a virtual machine may be initialized with the dirty and accessed bits set to a Boolean “true” (e.g., “1”) while the host page table entries234A-B mapped by one or more emulated MMIO addresses of the guest virtual machine are set as valid (e.g., valid and dirty bits set to “0”). In addition, a page modification log240may be initialized as full or nearly full.

During execution, the guest virtual machine210may access guest memory in a virtual guest address space for the guest virtual machine210. The guest page table220may map the guest address space to a host address space via guest page table entries222A-E. Each guest page table entry222A-E may map to a host page table entry232A-E which may in turn map the corresponding host address to a physical memory address. Upon mapping the guest memory access from the guest address space to the host address space using the guest page table entry (e.g.,222C), processing logic may determine whether the accessed host page table entry indicates that the page is dirty, modified, or accessed (e.g., whether the corresponding bit is set to “1”). Upon determining that page has the respective bit or bits set, the processing logic then provides the memory access to the guest virtual machine210without a VM exit. When the guest virtual machine210accesses the MMIO guest PTE224A or MMIO guest PTE224B, the processing logic determines that the respective valid or accessed bits are not set and therefore attempts to log the memory access to a page modification log240. Because the page modification log240has been initialized as full, upon attempting to log the memory access to the page modification log240, a page modification log full event242is triggered. The page modification log full event242causes the address translation from the guest address space to the physical address space to be cached (e.g., at a translation lookaside buffer of a memory management unit of a processing device). The page modification log full event242may also cause a VM exit to the hypervisor245. The hypervisor245may then perform the event or request corresponding to the MMIO emulated memory access. In some examples, the hypervisor245may obtain the guest address and its translation from a PML entry. In another example, the hypervisor245may obtain the guest address directly from the guest VM (e.g., via para-virtualization). In other embodiments, the hypervisor245may identify the guest address as the only address being used for MMIO emulation optimization.

FIG.3system diagram that illustrates an example computing system300providing cacheable MMIO emulation for virtual machines, in accordance with embodiments of the disclosure. System300includes a processing device310and a memory320. Processing device310be the same or similar to processing device160ofFIG.1and memory320may be the same or similar to memory170ofFIG.1. The processing device310may execute an MMIO address selection component312, a PML initialization component314, a PML full trigger component316, a VM exit component318, and a hypervisor319. In some examples, the memory320may include a page modification log322, a guest address space324, and one or more emulated MMIO addresses326.

In some examples, the MMIO address selection component312may identify and select one or more MMIO emulated addresses326of a guest address space324of a virtual machine. For example, the MMIO address selection component312may identify the most frequently accessed MMIO emulated address or a number of the most frequently accessed MMIO emulated addresses to be optimized for address translation caching. In some examples, the MMIO address selection component312may provide an interface for a user or administrator to select the MMIO emulated address or MMIO functionality to be optimized. The PML initialization component314may initialize a PML322for a virtual machine as full or nearly full (e.g., only one or two log entries or indexes remaining). For example, the PML initialization component314may initialize the index of the PML322to a negative value to indicate that it is full. Therefore, the next attempt to log to the PML322or decrement the PML index will cause a PML full event. The PML full trigger component316may determine that a guest virtual machine has accessed the optimized MMIO emulated address326or one of the optimized MMIO emulated addresses and attempt to log the access to the PML322. Because the PML322is set to a negative value, the PML full trigger component316may determine that the PML is full and cause an exit from the VM to the hypervisor319by the VM exit component318. The VM exit component318may also cache the address translation from the guest address space to the physical memory address in the case of a PML full event. Thus, upon future accesses to the optimized MMIO emulated address, the cached translation may be retrieved rather than performing a page walk of both the guest page table and the host page table for the MMIO emulated address translation. In some examples, firmware of the processing device310may be modified to cause the processing device to report the guest page address of the MMIO address to the host (e.g., the hypervisor).

FIG.4is a flow diagram of a method400of caching address translations in MMIO emulation for virtual machines, in accordance with some embodiments. Method400may be performed by processing logic that may comprise hardware (e.g., circuitry, dedicated logic, programmable logic, a processor, a processing device, a central processing unit (CPU), a system-on-chip (SoC), etc.), software (e.g., instructions running/executing on a processing device), firmware (e.g., microcode), or a combination thereof. In some embodiments, at least a portion of method400may be performed by VM exit component115ofFIG.1.

With reference toFIG.4, method400illustrates example functions used by various embodiments. Although specific function blocks (“blocks”) are disclosed in method400, such blocks are examples. That is, embodiments are well suited to performing various other blocks or variations of the blocks recited in method400. It is appreciated that the blocks in method400may be performed in an order different than presented, and that not all of the blocks in method400may be performed.

Method400begins at block410, where the processing logic selects one or more guest addresses of an address space of a virtual machine for memory mapped i/o emulation. In some examples, the processing logic may select the guest address corresponding to the most used MMIO address to be used as the optimized MMIO address. Selecting the most used MMIO address to be optimized may result in the best overall optimization of VM exits by reducing the address translation time (e.g., required page walks) for the VM exits using that MMIO address.

At block420, the processing logic initialized a page modification log associated with the virtual machine as full. In some examples, the processing logic may also initialize an accessed or valid bit of the host page table entries associated with the guest address for the MMIO emulation to a first value (e.g., “0”) indicating that the pages are valid and have not been accessed or modified. The processing logic may also initialize the accessed or valid bit for all other host page table entries corresponding to the guest VM address space to a second value (e.g., “1”) indicating that the host page table entries have been accessed or are dirty (e.g., the bit is set to a Boolean “yes”).

At block430, the processing logic triggers a page modification log full event in response to receiving a guest access to the selected one or more MMIO guest addresses. The page modification log full event may cause the processor (e.g., CPU) to cache an address translation of the MMIO address. For example, the page modification log full event may cause an exit to the hypervisor of the virtual machine and cache an address translation of the MMIO guest address to physical memory. In some examples, the processing logic may cache the address translation at a memory management unit (e.g., in a translation lookaside buffer (TLB)) of a respective processing device executing the VM.

In another example, the processing logic may initialize the page modification log for the virtual machine to the last available index (e.g., the zeroth index) allowing a first guest access to an optimized MMIO address to pass information to the PML. An additional MMIO access will then trigger the PML full event and cause an exit to the hypervisor. Thus, additional information may be passed to the hypervisor via the PML in addition to caching of the MMIO address translation upon the PML full event. In some examples, the processing logic can use paravirtualization to report the guest physical address, and an instruction length of the memory access, to the host (e.g., the hypervisor). For example, the guest virtual machine may send the address and the instruction length directly to the host. In some examples, only a single MMIO address is optimized per vCPU allowing the identification of the accessed MMIO address upon the PML full event. In other examples, the processing logic may modify CPU firmware to report the guest page address and the instruction length in response to a PML log full event.

At block440, the processing logic exits to the hypervisor to perform the event requested by the virtual machine. For example, each MMIO address may correspond to an event or request to the hypervisor to be performed by the hypervisor on behalf of the virtual machine. Accordingly, the processing logic may identify the corresponding event or request based on the MMIO address accessed by the virtual machine. The hypervisor may then perform the corresponding event that has been identified from the accessed MMIO address.

Because all other PTEs besides the optimized MMIO address have the accessed/dirty bits set at all times during the optimization, memory overcommit and migration solutions that use the accessed/dirty bits may become non-functional. Therefore, in some examples, the processing logic may disable memory overcommit and migration capabilities while the MMIO optimization is active. In another example, the processing logic may use page faults for memory overcommit, where the non-MMIO PTEs are marked as non-present or read-only. Additionally, in some examples, the processing logic may disable the MMIO optimization such as when a migration of a virtual machine is requested.

FIG.5is a flow diagram of a method500of caching address translations in MMIO emulation for virtual machines, in accordance with some embodiments. Method500may be performed by processing logic that may comprise hardware (e.g., circuitry, dedicated logic, programmable logic, a processor, a processing device, a central processing unit (CPU), a system-on-chip (SoC), etc.), software (e.g., instructions running/executing on a processing device), firmware (e.g., microcode), or a combination thereof. In some embodiments, at least a portion of method400may be performed by a VM exit component115ofFIG.1.

With reference toFIG.5, method500illustrates example functions used by various embodiments. Although specific function blocks (“blocks”) are disclosed in method500, such blocks are examples. That is, embodiments are well suited to performing various other blocks or variations of the blocks recited in method400. It is appreciated that the blocks in method500may be performed in an order different than presented, and that not all of the blocks in method500may be performed.

Method500begins at block502, where the processing logic selects at least one MMIO emulated address of a guest virtual machine for optimization. In some examples, an address space of the guest virtual machine may include many MMIO emulated addresses that are accessed by the virtual machine to cause a VM exit to the hypervisor for performance of a particular task. The processing logic may select one of the MMIO emulated addresses to optimize exits to the hypervisor upon access of that address. In some examples, the processing logic may select a particular MMIO emulated address based on the hypervisor task corresponding to the address. In some examples, the processing logic may select an MMIO emulated address that is accessed most frequently by a virtual machine. In another example, the processing logic may receive input (e.g., from a user or administrator) indicating which MMIO emulated address for which to optimized VM exits.

At block504, the processing logic initialized the page modification log and host page table entries corresponding to the guest VM address space. The processing logic may initialize the host page table entries for the guest address space of the VM to indicate that the pages are dirty or access (e.g., by setting the dirty and valid bits to “1”). The processing logic may initialize only the optimized MMIO emulated address or addresses as valid (e.g., setting the dirty and valid bits to “0”). Upon access to an address in the guest address space, a PML log will not occur because the dirty and accessed bits are set while accesses to the optimized MMIO emulated addresses will cause an attempted log to the PML because the dirty and valid bits are not set. It should be noted that although the dirty and accessed bits are described herein, embodiments of the present disclosure may operate the same or similar by using other PTE bits or an additional dedicated bit.

The processing logic may also initialize the page modification log for the guest VM to either full or nearly full. Initializing the PML to full may include setting an index of the PML to a negative value. Initializing the PML to nearly full may include setting the index of the PML to “0” or otherwise the last index of the PML to allow one entry to the PML without a VM exit and then causing the next attempted entry to trigger a VM exit via a PML log full event.

At block506, the processing logic receives a guest VM access of the optimized MMIO address. The guest VM may request access to the memory address of the optimized MMIO emulated address in the guest address space. Processing logic may then perform a page walk of the guest page table to identify a translation of the guest address to a host address. The processing logic may then perform an additional page walk on a host page table to identify a page table entry for a translation of the host address to a physical memory address. The processing logic may further determine whether the page table entry of the host page table indicates that dirty, valid, or other specified bit, is set or is not set. If the bit is not set (e.g., the page is indicated as valid), the processing logic then attempts to log the memory access to a page modification log. Because the optimized MMIO emulated addresses are the only addresses set as valid, only accesses to the MMIO emulated addresses will cause a log to the PML.

At block508, the processing logic determines, in response to receiving the guest access of the optimized MMIO address, whether the PML is indicated as full. For example, the processing logic may determine whether an index of the PML is set to a negative value, indicating that the PML is full. If the PML is full, the processing logic continues to block512. If the PML is not indicated as full, such as if the PML index is set to 0 (e.g., the last available index), then the processing logic continues to block510.

At block510, the processing logic logs information associated with the guest access of the MMIO address and then decrements the PML index. The processing logic may log the request, the accessed memory, and any other information to be passed to a hypervisor for performing one or more tasks.

At block512, the processing logic triggers a PML full event in response to determining that the PML is indicated as full. The PML full event may indicate that the PML is full and provide the PML log to the host. At block514, the processing logic caches an address translation for the MMIO address at a memory management unit of a processing device executing the guest VM. For example, the processing logic may cache the guest to physical address translation at a translation lookaside buffer of the memory management unit of the processing device executing the guest VM. The address translation may then be quickly retrieved upon future access to the optimized MMIO emulated address. In some examples, the processing logic also logs the MMIO address and the translation in an entry of the PML.

At block516, in response to the PML full event, the processing logic exits to the hypervisor of the guest VM to allow the hypervisor to perform the event or request associated with the MMIO address that was accessed by the guest VM. In some examples, the hypervisor may identify the MMIO address from the PML entry. In another example, the guest VM may provide the MMIO address directly to the hypervisor (e.g., via para-virtualization). In other examples, the hypervisor may identify the MMIO address upon a PML full event where the MMIO address is the only address used for MMIO emulation optimization. The MMIO address that is accessed may correspond to a particular task, event, or request to be performed by the hypervisor on behalf of the guest VM. For example, the MMIO address may cause the hypervisor to perform memory management tasks, virtual device management tasks, or any other virtualization management tasks for the guest VM that the guest VM does not have permissions to access or perform.

The example computing device600may include a processing device (e.g., a general-purpose processor, a PLD, etc.)602, a main memory604(e.g., synchronous dynamic random-access memory (DRAM), read-only memory (ROM)), a static memory606(e.g., flash memory and a data storage device618), which may communicate with each other via a bus630.

Computing device600may further include a network interface device608which may communicate with a network620. The computing device600also may include a video display unit610(e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device612(e.g., a keyboard), a cursor control device614(e.g., a mouse) and an acoustic signal generation device616(e.g., a speaker). In one embodiment, video display unit610, alphanumeric input device612, and cursor control device614may be combined into a single component or device (e.g., an LCD touch screen).

Data storage device618may include a computer-readable storage medium628on which may be stored one or more sets of instructions625that may include instructions for a virtual machine exit component, e.g., VM exit component115, for carrying out the operations described herein, in accordance with one or more aspects of the present disclosure. Instructions625may also reside, completely or at least partially, within main memory604and/or within processing device602during execution thereof by computing device600, main memory604and processing device602also constituting computer-readable media. The instructions625may further be transmitted or received over a network620via network interface device608.