Patent Description:
Assigning unpinned memory to the virtual machines allows the host to "overcommit" memory; that is, to allocate an amount of memory to each virtual machine that, in aggregate, exceeds the total available memory of the host. The host overcommits memory based on an assumption that a virtual machine generally will not use its entire allocation of memory all of the time. Thus, overcommitting memory allows for higher utilization and/or a need for less total memory in the host.

<CIT> discloses systems and methods for directly updating the virtual machine memory by interrupt handlers. An example method may comprise: receiving, by a computer system, an interrupt triggered by a physical device; receiving, by an interrupt handling routine, a data frame from the physical device; identifying a virtual machine to receive the interrupt; and responsive to determining that an active memory context on the computer system matches a memory context of the virtual machine, writing, by the interrupt handling routine, the data frame into a memory of the virtual machine.

<CIT> discloses systems and methods for lazy memory transformation in virtual machine live migration. An example method may comprise: receiving, by a computer system, a plurality of transformed memory blocks, each transformed memory block comprising one or more memory pages mapped into an address space of a virtual machine being migrated to the computer system; storing, in a memory data structure, one or more mappings, each mapping comprising a guest virtual address of a memory page and an identifier of a transformed memory block containing the memory page; responsive to detecting an access to a memory page by the virtual machine, identifying, using the memory data structure, a transformed memory block containing the memory page being accessed; and storing in a memory mapped into the address space of the virtual machine the memory page produced by performing a reverse transformation of the transformed memory block.

<CIT> discloses a recoverable I/O request processor includes computer-executable instructions for processing I/O requests, such as requests to send or receive data through a network. The recoverable I/O request processor translates virtual memory addresses to physical memory addresses utilizing translation tables local to an I/O device. If a local translation fails, the recoverable I/O request processor requests virtual address mapping information from the operating system.

The claimed subject-matter is defined in the independent claims. The dependent claims define embodiments of the claimed subject-matter.

At least one aspect is directed to a method of error handling in a network interface card of a host device. The method includes receiving, at a processor of the network interface card, a data packet addressed to a virtual machine executing on the host device. The method includes selecting, by the processor, a receive queue corresponding to the virtual machine and residing in a memory of the network interface card. The method includes retrieving, by the processor, a buffer descriptor from the receive queue. The buffer descriptor includes a virtual memory address. The method includes, in response to determining that the virtual memory address is not associated with a valid translated memory address associated with the virtual machine, retrieving, by the processor, a backup buffer descriptor from a hypervisor queue residing in the network interface card memory and corresponding to a hypervisor executing on the host device; and storing, by the processor, contents of the data packet in a host device memory location indicated by a backup memory address in the backup buffer descriptor.

At least one aspect is directed to a system comprising a host device executing one or more virtual machines and a hypervisor, a host device memory, and a network interface card. The network interface card is configured to receive a data packet addressed to a virtual machine executing on the host device. The network interface card is configured to select a receive queue corresponding to the virtual machine and residing in a memory of the network interface card. The network interface card is configured to retrieve a buffer descriptor from the receive queue. The buffer descriptor includes a virtual memory address. The network interface card is configured to, in response to determining that the virtual memory address is not associated with a valid translated memory address associated with the virtual machine retrieve a backup buffer descriptor from a hypervisor queue residing in the network interface card memory and corresponding to a hypervisor executing on the host device; and store contents of the data packet in a host device memory location indicated by a backup memory address in the backup buffer descriptor.

At least one aspect is directed to a method of error handling in a network interface card of a host device. The method includes receiving, at a processor of the network interface card, a transmission request from a virtual machine executing on the host device. The method includes retrieving, by the processor, a packet descriptor from a transmit queue corresponding to the virtual machine and residing in a memory of the network interface card. The packet descriptor includes a virtual memory address. The method includes, in response to determining that the virtual memory address is not associated with a valid translation memory address associated with the virtual machine, sending, by the processor, a message to a backup queue residing in the network interface card memory and corresponding to a hypervisor executing on the host device. The message includes an identifier of the transmit queue and the virtual memory address. The method includes receiving, by the processor, a notification from the hypervisor that a host device memory region has been reallocated to the virtual machine. The method includes retrieving, by the processor from the transmit queue, a second packet descriptor including a second virtual memory address. The method includes translating, by the processor, the second virtual memory address. The method includes generating, by the processor, a data packet using data from a host device memory location indicated by the translated second virtual memory address. The method includes transmitting, by the processor, the data packet.

At least one aspect is directed to a system comprising a host device executing one or more virtual machines and a hypervisor, a host device memory, and a network interface card. The network interface card is configured to receive a transmission request from a virtual machine executing on the host device. The network interface card is configured to retrieving, by the processor, a packet descriptor from a transmit queue corresponding to the virtual machine and residing in a memory of the network interface card. The packet descriptor includes a virtual memory address. The network interface card is configured to, in response to determining that the virtual memory address is not associated with a valid translation memory address associated with the virtual machine, send a message to a backup queue residing in the network interface card memory and corresponding to a hypervisor executing on the host device. The message includes an identifier of the transmit queue and the virtual memory address. The network interface card is configured to receive a notification from the hypervisor that a host device memory region has been reallocated to the virtual machine. The network interface card is configured to retrieve, from the transmit queue, a second packet descriptor including a second virtual memory address. The network interface card is configured to translate the second virtual memory address. The network interface card is configured to generate a data packet using data from a host device memory location indicated by the translated second virtual memory address. The network interface card is configured to transmit the data packet.

This disclosure relates generally to systems and methods for providing input/output to and from unpinned memory used by virtual machines. The techniques described herein allow a host or server executing virtual machines to overcommit memory to the virtual machines; that is, allowing the host to allocate an amount of memory to each virtual machine that, in aggregate, would exceed a total amount of host memory available to the virtual machines. The techniques also facilitate live migration of the virtual machines within the host and between the host and a second host.

According to embodiments described herein, systems and methods of error handling in a network interface card (NIC) are provided. For a data packet destined for a local virtual machine (VM), if the NIC cannot determine a valid translation memory address for a virtual memory address in a buffer descriptor from a receive queue of the VM, the NIC can retrieve a backup buffer descriptor from a hypervisor queue, and store the packet in a host memory location indicated by an address in the backup buffer descriptor. For a transmission request from a local VM, if the NIC cannot determine a valid translated address for a virtual memory address in the packet descriptor from a transmit queue of the VM, the NIC can send a message to a hypervisor backup queue, and generate and transmit a data packet based on data in a memory page reallocated by the hypervisor.

For unpinned memory of a host-that is, not permanently assigned to a virtual machine-communication to and from the virtual machines may require the host to translate virtual machine memory addresses to physical memory addresses and vice versa. If the host attempts to translate a virtual machine address to a physical memory address that is no longer allocated to that virtual machine, the host may get a translation error. For example, the host device may experience errors when its network interface card tries to assemble and transmit packets from physical memory addresses no longer allocated to the virtual machine that requested transmission of that packet. This can be due to, for example, a period of time elapsing between when the requested transmission is scheduled to when the requested transmission is executed. During this period of time, physical memory addresses within the host memory may be reallocated among the virtual machines. In such a case, the network interface card will not be able to translate the descriptor from the virtual machine's transmit queue to a physical memory address. Similarly, the host device can experience errors when the network interface card receives a packet for a destination virtual machine, but cannot properly translate a virtual machine address to a host device memory location assigned to the destination virtual machine. This disclosure therefore presents systems and methods for graceful handling of translation errors caused by reallocation of unpinned physical memory among the virtual machines executing on the host.

Similar failures of address translation can occur during live migration of a virtual machine, as the virtual machine migrates to a different memory space or different host device altogether. In such cases, the same systems and methods can be used to provide for graceful handling of translation errors during virtual machine live migration. The systems and methods can include addition features that provide for dirty bit or dirty page tracking; that is, keeping track of additions to, or modifications of, memory allocated to the virtual machine at its original location as the memory contents are copied to the virtual machine's new location. Once the virtual machine is up and running in its new location or host, the dirty bits/pages can be copied to the virtual machine's new memory space.

<FIG> is a block diagram of a network environment <NUM> with a network device <NUM>, according to some implementations. In broad overview, the illustrated network environment <NUM> includes a network <NUM> of interconnected network nodes <NUM>. The network nodes <NUM> participate in the network <NUM> as data sources, data destinations (or data sinks), and/or intermediary nodes such as switches, routers, and/or gateways propagating data from sources towards destinations through the network <NUM>. The network <NUM> includes the network device <NUM> with links <NUM> to various other participating network nodes <NUM>.

Referring to <FIG> in more detail, the network <NUM> is a network facilitating interactions between participant devices. An illustrative example network <NUM> is the Internet; however, in other implementations, the network <NUM> may be another network, such as a local network within a data center, a network fabric, or any other local area or wide area network. The network <NUM> may be composed of multiple connected sub-networks or autonomous networks. The network <NUM> can be a local-area network (LAN), such as a company intranet, a metropolitan area network (MAN), a wide area network (WAN), an inter-network such as the Internet, or a peer-to-peer network; for example, an ad hoc WiFi peer-to-peer network. Any type and/or form of data network and/or communication network can be used for the network <NUM>. It can be public, private, or a combination of public and private networks. In general, the network <NUM> is used to convey information between computing devices; for example, network nodes <NUM>, and the network device <NUM>.

As shown in <FIG>, the network device <NUM> is a host device or server executing one or more virtual machines 150a-150c (generally virtual machines <NUM>) on one or more CPUs of the network device <NUM>. In some implementations, the network device <NUM> has configuration similar to that of a computing system <NUM> as shown in <FIG>. For example, the memory <NUM> can have a configuration similar to that of a memory <NUM> as shown in <FIG>, and the network interface card <NUM> can have configuration similar to that of a network interface card <NUM> and/or a network interface controller <NUM> as shown in <FIG>. The computing system <NUM> is described in more detail below, in reference to <FIG>. The elements shown in the computing system <NUM> illustrated in <FIG> do not all need to be present in some implementations of the network device <NUM> illustrated in <FIG>.

The network device <NUM> includes a network interface driver <NUM>, a memory <NUM>, a network interface card <NUM>, a real OS <NUM>, a hypervisor <NUM>, and the virtual machines <NUM>. The hypervisor <NUM> manages operation of the virtual machines <NUM>, including spinning up and terminating the virtual machines <NUM>, managing allocation of memory <NUM> to the virtual machines <NUM>, and migrating the virtual machines <NUM> to other network devices. For example, the hypervisor <NUM> can allocated pinned and unpinned memory to the virtual machines <NUM>. In some implementations, the hypervisor <NUM> can allocate memory to the virtual machines that in aggregate exceeds a total host device memory available to the virtual machines.

The network device <NUM> includes the memory <NUM>. In some implementations, the memory <NUM> may store computer executable instructions for a real operating system (OS) <NUM> and a transport protocol module <NUM> (such as a TCP protocol module or the TCP layer of the network stack) to be executed on a processor of the network device <NUM>. In some implementations, the memory <NUM> may store computer executable instructions related to the operation and use of a network interface driver <NUM>. In some other implementations, the memory <NUM> may store computer executable instructions for the hypervisor <NUM> to be executed on a processor of the network device <NUM>. In some other implementations, the memory <NUM> may store computer executable instructions for the virtual machines <NUM> executing on the network device <NUM>. In some other implementations, the memory <NUM> may store data for forwarding to the virtual machines <NUM>, processing by the virtual machines <NUM>, and/or transmission from the virtual machines <NUM>. The memory <NUM> may include, for example, a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a synchronous dynamic random access memory (SDRAM), a ferroelectric random access memory (FRAM), a read only memory (ROM), a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), and/or a flash memory.

Functionality implemented within the TCP layer of a network device can additionally or alternatively execute in another network protocol module within the transport layer, the network layer or a combined transport/network layer of a network protocol stack. For example, the functionality can be implemented in a user datagram protocol (UDP) module, reliable datagram protocol (RDP) module, reliable user datagram protocol (RUDP) module, or a datagram congestion control protocol (DCCP) module. As used herein, a network layer, a transport layer, or a combined transport/network layer will generally be referred to as a packet layer of the network protocol stack.

The network interface driver <NUM> can include a network interface driver software module running on a real OS. A network interface driver, such as the network interface driver <NUM>, can be a collection of computer executable instructions stored in the memory <NUM> that when executed by a processor help facilitate network communications. In some other implementations, the network interface driver <NUM> may be implemented as logic implemented in a hardware processor or other integrated circuit, or as a combination of hardware and software logic. The network interface driver <NUM> can communicate with one of the software virtual machines <NUM> (or applications executing on the virtual machines <NUM>), via a guest OS of a virtual machine (or in some implementations, through a hypervisor and the guest OS if operating in a virtual machine environment). In some implementations, the network interface driver <NUM> is included within a first layer of a transmission control protocol (TCP) stack of the real OS <NUM> of the network device <NUM> and communicates with a software module or application that is included in an upper layer of the TCP stack. In one example, the network interface driver <NUM> is included within a transport layer of a TCP stack and communicates with a software module or application that is included in an application layer of the TCP stack. In another example, the network interface driver <NUM> is included within a link layer of a TCP stack and communicates with a TCP/IP module that is included in an internet/transport layer of the TCP stack. In some implementations, the functionality is additionally or alternatively configured to receive packets from another network or transport layer protocol module, such as a user datagram protocol (UDP) module, reliable datagram protocol (RDP) module, reliable user datagram protocol (RUDP) module, or a datagram congestion control protocol (DCCP) module. In some other implementations, the network interface driver <NUM> can be included as a portion of the network interface card <NUM>.

The network interface card <NUM> includes a scheduler <NUM>, a packet builder <NUM>, a packet forwarder <NUM>, and a memory <NUM>. The scheduler <NUM>, packet builder <NUM>, and packet forwarder <NUM> can be functional blocks of one or more processors of the network interface card <NUM>, such as an application-specific integrated circuit (ASIC) having discrete functional blocks. In some implementations, the scheduler <NUM>, packet builder <NUM>, and packet forwarder <NUM> can be functional blocks of software executing on one or more general-purpose processors of the network interface card. In some implementations, the functions of the scheduler <NUM>, packet builder <NUM>, and packet forwarder <NUM> can be combined of further subdivided into different functional blocks.

The memory <NUM> can store computer executable instructions for execution by network interface card <NUM>, such as instructions relating to the operation of the scheduler <NUM>, packet builder <NUM>, and packet forwarder <NUM>. The memory <NUM> can provide for caches, buffers, translation tables, and/or time-indexed data structures used by the network interface card <NUM>, virtual machines <NUM>, hypervisor <NUM> for the transmission and receipt of data packets over the network <NUM>. For example, the memory <NUM> can store, for each virtual machine <NUM>, one or more receive queues and transmit queues containing packet descriptors. Each packet descriptor can contain a virtual memory address corresponding to a virtual machine memory region. In another example, the memory <NUM> can store tables containing translations of virtual memory addresses associated with virtual machines to physical memory addresses of regions of the network device <NUM> memory <NUM> respectively allocated to the virtual machines. The memory <NUM> may include, for example, a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a synchronous dynamic random access memory (SDRAM), a ferroelectric random access memory (FRAM), a read only memory (ROM), a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), and/or a flash memory.

The network interface card <NUM> includes a scheduler <NUM>. The scheduler <NUM> can schedule transmission of a corresponding data packet based on a transmission request received from a virtual machine <NUM>. In some implementations, the scheduler <NUM> is utilized to manage a sequence of packet identifiers inserted into and extracted from a time-indexed data structure. The time-indexed data structure can be a data structure in the network interface card <NUM> processor or memory <NUM> that facilitates the scheduling of data packet transmissions. In some implementations, before making a scheduling, or any other decision with respect to a data packet, the scheduler <NUM> can i) copy at least the header of the packet to the network interface card memory <NUM>, ii) prepare descriptors that point to the packet header and packet payload (which may or may not be copied to the network interface card memory <NUM>), and iii) execute scheduling decisions and queue an identifier of the descriptor in the time-indexed data structure. The scheduler <NUM> need not perform these operations in this exact order and may, in some implementations, perform two or more of these operations during overlapping time periods.

The network interface card <NUM> includes the packet builder <NUM>. The packet builder <NUM> can receive a transmission request from a virtual machine <NUM> (or the scheduler <NUM>), and execute packet generation and transmission. The packet builder <NUM> includes packet generation logic that can generate the data packet using a set of pointers stored in the memory <NUM> and associated with the communication. Each pointer can be a memory descriptor. The memory descriptor can describe a packet; for example, by specifying a memory location and length of packet data. In some implementations, the packet data can be in the host memory <NUM>. In some implementations, the packet data can be in the network interface card memory <NUM>. The packet builder <NUM> can build packets for transmission based on the identifiers in the time-indexed data structure in the memory <NUM>.

An example operation of the packet builder <NUM> is as follows. In some implementations, a data packet transmission is scheduled by the scheduler <NUM>. The scheduler <NUM> would have previously inserted a packet identifier corresponding to a transmission request received from a virtual machine <NUM> in the time-indexed data structure in the memory <NUM>. When a scheduled time for transmission of the packet arrives, the packet builder <NUM> can receive or retrieve the packet identifier from the time-indexed data structure in the memory <NUM>, and use the packet identifier to generate the packet for transmission. In some implementations, however, the packet builder <NUM> can receive the transmission request directly from the virtual machine <NUM>, where the transmission request is or contains the packet identifier. Based on the transmission request or packet identifier, the packet builder <NUM> can retrieve a packet descriptor from a transmit queue corresponding to the virtual machine <NUM> identified in the transmission request. The transmit queue can reside in the memory <NUM>. The packet descriptor can include a virtual memory address associated with the virtual machine. The virtual memory address can correspond to a region of the memory <NUM> that is supposed to contain data for the data packet. The packet descriptor may also include a length that can define how many contiguous data blocks the packet builder <NUM> to should retrieve from the memory <NUM>. The packet builder <NUM> can attempt to translate the virtual memory address. If the packet builder <NUM> is able to determine a valid translated address for the virtual memory address, the packet builder <NUM> can retrieve the data from the specified region of the memory <NUM> to generate the data packet. The network interface card <NUM> can then transmit the generated data packet.

The packet builder <NUM> can translate virtual memory addresses to translated addresses (physical memory addresses or specific memory locations) using one or more tables stored in the network interface card memory <NUM>. For example, the memory <NUM> may maintain a table including columns for virtual memory addresses and physical memory addresses, respectively. The table may also include a column for the virtual machine associated with the virtual memory address and/or the virtual machine to which the physical memory address is currently allocated. In some implementations, address correlations and virtual machine correlations may be stored in separate tables. A translation errors can occur in several scenarios. For example, when the translation tables do not list the virtual memory address, or do not list a corresponding physical memory address for that virtual memory address. In another example, either the virtual memory address or corresponding physical address may not be associated with the virtual machine who sent the request (or, in the case of receiving a data packet, the physical memory address corresponding to the destination virtual memory address may not be associated with the destination virtual machine). These scenarios can occur when memory is reallocated among virtual machines between a request for transmission and packet building, when memory is reallocated due to migration of a virtual machine, or when memory is reallocated prior to receipt of a data packet addressed to a virtual memory address that was previously valid.

If the packet builder <NUM> is unable to determine a valid translated address for the virtual memory address, however, the packet builder <NUM> can enter an error handling routine. Such an error handling routine is described in further detail below with respect to <FIG> and <FIG>. Once the packet builder <NUM> has obtained a second virtual memory address for which it can determine a valid translated address, the packet builder <NUM> can generate a data packet using data from a host device memory location indicated by the translated address. The network interface card <NUM> can then transmit the data packet.

The network interface card <NUM> includes the packet forwarder <NUM>. The packet forwarder <NUM> can receive a data packet over one of the lines <NUM>, translate the destination address, and forward the data packet to a location in the memory <NUM> allocated to a destination virtual machine <NUM>. An example operation of the packet forwarder <NUM> is as follows. The packet forwarder <NUM> receives the data packet addressed to a destination virtual machines <NUM> executing on the network device <NUM>. The packet forwarder <NUM> can select a receive queue assigned to the virtual machine <NUM>, and retrieve a buffer descriptor from the selected receive queue. The buffer descriptor includes a virtual memory address associated with the virtual machine. The packet forwarder <NUM> can attempt to translate the virtual memory address to a physical address in the memory <NUM>. If the packet forwarder <NUM> is able to determine a valid translated address for the virtual memory address, the packet forwarder <NUM> can store the data packet in the location in memory <NUM> specified by the translated address. The packet forwarder <NUM> can translate virtual memory address in a manner similar to that described for the packet builder <NUM>.

If the packet forwarder <NUM> is unable to determine a valid translated address for the virtual memory address, however, the packet forwarder <NUM> can enter an error handling routine. Such an error handling routine is described in further detail below with respect to <FIG> and <FIG>. Once the packet forwarder <NUM> is able to determine a valid memory region in the network device memory <NUM> associated with the destination virtual machine <NUM>, the packet forwarder <NUM> can store contents of the data packet in the memory region.

In some implementations, following, or concurrent with, forwarding of the contents of the data packet to the memory <NUM> by the packet forwarder <NUM>, the hypervisor <NUM> can allocate a new page of the memory <NUM> to the virtual machine <NUM> that was the intended destination of the data packet. The hypervisor can then update the memory <NUM> of the network interface card with the memory address of the newly allocated page of memory, and associate the newly allocated page with the virtual machine.

<FIG> is a swim lane diagram <NUM> illustrating operations for handling translation errors when receiving packets for a virtual machine on a host device, according to some implementations. The swim lane diagram <NUM> shows the interaction of various components of the system during an example operation of error handling for a received packet.

The packet forwarder of the network interface card receives a data packet (<NUM>) over the network. The packet forwarder determines a destination virtual machine for the data packet, and selects (<NUM>) a virtual machine receive queue associated with the destination virtual machine. The packet forwarder retrieves (<NUM>) a buffer descriptor from a head of the receive queue and extracts a virtual memory address from the buffer descriptor. The packet forwarder attempts to translate (<NUM>) the virtual memory address to determine a valid translated address to a host device memory location associated with the destination virtual machine. If the packet forwarder experiences a translation error; for example, because the virtual memory address no longer translates to a physical memory address to a host device memory location that is still assigned to the destination virtual machine, the packet forwarder can fall back to a hypervisor queue. In some implementations, the network interface card will maintain a separate hypervisor queue for each virtual machine receive queue. In some implementations, the network interface card will maintain a separate hypervisor queue for each virtual machine. In some implementations the network interface card may maintain one or more shared hypervisor queues shared for multiple virtual machines. The packet forwarder can retrieve (<NUM>) a backup buffer descriptor from the hypervisor queue, and extract a backup memory address from the backup buffer descriptor. The packet forwarder can then store (<NUM>) data from the receive data packet at a host device memory location indicated by the backup memory address. The destination virtual machine can then retrieve (<NUM>) the data from the host device memory and process it (<NUM>).

<FIG> is a flowchart showing operations of an example method <NUM> of handling translation errors when receiving packets for a virtual machine on a host device, according to some implementations. The method <NUM> can be performed by a network interface card, such as the network interface card <NUM> previously described. The method <NUM> includes receiving a data packet addressed to a virtual machine (stage <NUM>). The method <NUM> includes selecting a receive queue assigned to the virtual machine (stage <NUM>). The method <NUM> includes retrieving a buffer descriptor including a virtual memory address from the receive queue (stage <NUM>). The method <NUM> includes determining whether the virtual memory address is associated with a valid translated memory address (decision block <NUM>). The method <NUM> includes retrieving a backup buffer descriptor from a hypervisor queue (stage <NUM>). The method <NUM> includes storing the data packet in a host device memory location indicated by a backup memory address (stage <NUM>).

The method <NUM> includes receiving a data packet addressed to a virtual machine (stage <NUM>). The data packet can be received by a packet forwarder of the network interface card, such as the packet forwarder <NUM> previously described. The data packet can be received over a network connection from a remote client device, peer network device, controller device, or the like.

The method <NUM> includes selecting a receive queue assigned to the virtual machine (stage <NUM>). The packet forwarder can select a receive queue from the network interface card memory, such as the memory <NUM> previously described. The receive queue can include one or more buffer descriptors.

The method <NUM> includes retrieving a buffer descriptor including a virtual memory address from the receive queue (stage <NUM>). The packet forwarder can retrieve the buffer descriptor from the head of the receive queue. The buffer descriptor includes a virtual memory address that indicates a memory location associated with the virtual machine. The virtual memory address is not a physical memory address, and therefore must be translated before the network interface card can access the corresponding memory region.

The method <NUM> includes determining whether the virtual memory address is associated with a valid translated memory address (decision block <NUM>). If the network interface cannot determine a valid translated address for the virtual address, the method <NUM> continues with stage <NUM> as described below. If the network interface can determine a valid translated address, the packet forwarder can store the data packet in a memory location indicated by the translated memory address according to the normal operation.

If the network interface cannot determine a valid translated address for the virtual memory address, the method <NUM> continues by retrieving a backup buffer descriptor from a hypervisor queue (stage <NUM>). The packet forwarder <NUM> can retrieve the backup buffer descriptor from the head of the hypervisor queue associated with the destination virtual machine. The backup buffer descriptor can include a backup memory address that indicates a memory location in the memory <NUM> that can accept the received data packet and store it for processing by the destination virtual machine.

The method <NUM> includes storing the data packet in a host device memory location indicated by a backup memory address (stage <NUM>). The hypervisor can notify the virtual machine of the location of the data packet so the virtual machine may retrieve and process it.

<FIG> is a swim lane diagram <NUM> illustrating operations for handling translation errors when transmitting packets from a virtual machine on a host device, according to some implementations. The swim lane diagram <NUM> shows the interaction of various components of the system during an example operation of error handling for a transmitted packet.

Prior to operation of the translation error handling, a virtual machine executing on a host device will begin the process of transmitting a packet via a network interface card of the host device. The virtual machine will store (<NUM>) data for the data packet to be generated in the host device memory. The virtual machine will then send a transmission request (<NUM>) to a packet builder of the network interface card.

In some implementations, the transmission request (<NUM>) can be delayed, prioritized, or otherwise scheduled awaiting a predetermined or indefinite period of time (<NUM>). Such scheduling can be accomplished using, for example, the scheduler <NUM> and/or the time-indexed data structure previously described. Processing the transmission request for transmission can recommence at a predetermined time, or when all higher priority requests and/or previously transmitted requests have been processed.

The packet builder receives the transmission request (<NUM>) from the virtual machine (or, in some implementations, the scheduler <NUM> or time-indexed data structure). The packet builder retrieves a packet descriptor (<NUM>) from a transmit queue corresponding to the virtual machine and residing in the memory of the network interface card, and extracts a virtual memory address from the buffer descriptor. The packet builder will attempt to translate (<NUM>) the virtual memory address to determine a valid translated address to a host device memory location associated with the virtual machine that send the transmission request. If the packet builder experiences a translation error; for example, because the virtual memory address no longer translates to a physical memory address to a host device memory location that is still assigned to the virtual machine, the packet builder sends a message (<NUM>) to a backup queue in the network interface card memory. The message will include an identifier of the transmit queue and the virtual memory address that could not be translated successfully. The packet builder can disable the transmit queue until the packet builder receives a notification that memory has been reallocated to the transmitting virtual machine.

A hypervisor executing on the host device can retrieve the message (<NUM>) from the backup queue. The hypervisor can reallocate (<NUM>) a host device memory region to the virtual machine. The hypervisor, which can be configured to back up the contents of an old page of memory allocated to the virtual machine when reallocating the memory to a different virtual machine, can retrieve the backup and store its contents in the new page of memory. The hypervisor can send a notification (<NUM>) to the packet builder that the host device memory region has been reallocated, and the transmit queue can be reactivated.

The packet builder can receive the notification (<NUM>) from the hypervisor that the hypervisor has reallocated host device memory to the transmitting virtual machine. The packet builder can then retrieve a second packet descriptor (<NUM>) from the transmit queue, and extract a second virtual memory address from the second packet descriptor. The packet builder can attempt to translate (<NUM>) the second virtual memory address. If the translation is successful, the packet builder can retrieve the data (<NUM>) from a location of the host device memory indicated by the translated second virtual memory address. The packet builder can use the retrieved data to generate (<NUM>) the data packet. The packet builder can then transmit the generated data packet (<NUM>) over a network.

<FIG> is a flowchart showing operations of an example method <NUM> of handling translation errors when transmitting packets from a virtual machine on a host device, according to some implementations. The method <NUM> can be performed by a network interface card, such as the network interface card <NUM> previously described. The method <NUM> can include receiving a transmission request from a virtual machine (stage <NUM>). The method <NUM> can include retrieving a packet descriptor including a virtual memory address from a transmit queue (stage <NUM>). The method <NUM> can include determining whether virtual memory address is associated with a valid translated address (decision block <NUM>). If the network interface card cannot determine a valid translated address for the virtual memory address, the method can continue with stage <NUM> described below. If the network interface can determine a valid translated address, the network interface card can generate and transmit the data packet normally. The method <NUM> can include sending a message to a backup queue of a hypervisor (stage <NUM>). The method <NUM> can include receiving a notification that memory has been reallocated (stage <NUM>). The method <NUM> can include retrieving a second packet descriptor including a second virtual memory address from the transmit queue (stage <NUM>). The method <NUM> can include translating the second virtual memory address (stage <NUM>). The method <NUM> can include generating a data packet using data from the translated address (stage <NUM>). The method <NUM> can include transmitting the data packet (stage <NUM>).

The method <NUM> can include receiving a transmission request from a virtual machine (stage <NUM>). The transmission request can be received by a scheduler of the network interface card, such as the scheduler <NUM> previously described. In some implementations, the scheduler <NUM> can maintain a time-indexed data structure in a network interface card memory, such as the memory <NUM> previously described. The scheduler <NUM> can insert a packet identifier corresponding to the transmission request into the time-indexed data structure. When the scheduled time for data packet transmission arrives, a packet builder, such as the packet builder <NUM> can retrieve the packet identifier and commence generating a data packet corresponding to the transmission request. In some implementations, the packet builder <NUM> can receive the transmission request directly from the virtual machine, and proceed with generating the corresponding data packet immediately.

The method <NUM> can include retrieving a packet descriptor including a virtual memory address from a transmit queue (stage <NUM>). When the scheduled time for transmission of the data packet arrives, the packet builder can use information in the packet identifier and/or the transmission request to retrieve the packet descriptor from the transmit queue associated with the virtual machine that submitted the transmission request. The packet descriptor can include a virtual memory address that indicates a memory location associated with the virtual machine. The virtual memory address is not a physical memory address, and therefore must be translated before the network interface card can access the corresponding memory region.

The method <NUM> can include determining whether virtual memory address is associated with a valid translated address (decision block <NUM>). If the network interface card cannot determine a valid translated address for the virtual memory address, the method can continue with stage <NUM> described below. If the network interface can determine a valid translated address, the network interface card can generate the data packet based on data in a memory location indicated by the translated memory address, and transmit the data packet according to the normal operation.

If the network interface card cannot determine a valid translated address for the virtual memory address, the method <NUM> can include sending a message to a backup queue of a hypervisor (stage <NUM>). The message can include an identifier of the transmit queue and the virtual memory address. Based on this information, the hypervisor can create a new, valid queue, with new packet descriptors with updated virtual memory addresses that will have valid translated addresses. The hypervisor can then notify the packet builder that the new queue has been created. While the network interface card awaits notification of the creation of a new queue from the hypervisor, it can cease using the old transmit queue.

The method <NUM> can include receiving a notification that host device memory has been reallocated to the transmitting virtual machine (stage <NUM>). The packet builder can receive the notification regarding the memory reallocation from the hypervisor. Upon receiving the notification, the packet builder can reactivate the transmit queue.

The method <NUM> can include retrieving a second packet descriptor including a second virtual memory address from the transmit queue (stage <NUM>). Once the packet builder has received the notification, the packet builder can reactivate the transmit queue retrieve the second packet descriptor from the head of the transmit queue, and extract the second virtual memory address.

The method <NUM> can include translating the second virtual memory address (stage <NUM>). The second virtual memory address should in almost every case be valid, having been very recently provided by the hypervisor; that is, without a delay awaiting the scheduled time for transmission.

The method <NUM> can include generating a data packet using data from the translated address (stage <NUM>). The packet builder can retrieve the data from the memory location indicated by the translated address and use the data to generate the data packet. In some implementations, the translated address can point to a location in memory (either network interface card memory or host device memory) containing a scatter-gather list. The scatter-gather list can include memory pointers to additional memory locations containing data for the packet payload and/or header.

The method <NUM> can include transmitting the data packet (stage <NUM>). The network interface card can then transmit the data packet through one the links to the network. In some implementations, the network interface card can send a transmission completion message to the virtual machine indicating that the packet has been successfully transmitted.

<FIG> is a block diagram illustrating a general architecture for a computer system <NUM> that may be employed to implement elements of the systems and methods described and illustrated herein, according to an illustrative implementation.

In broad overview, the computing system <NUM> includes at least one processor <NUM> for performing actions in accordance with instructions and one or more memory devices <NUM> or <NUM> for storing instructions and data. The illustrated example computing system <NUM> includes one or more processors <NUM> in communication, via a bus <NUM>, with at least one network interface driver controller <NUM> with one or more network interface cards <NUM> connecting to one or more network devices <NUM>, memory <NUM>, and any other devices <NUM>, e.g., an I/O interface. The network interface card <NUM> may have one or more network interface driver ports to communicate with the connected devices or components. Generally, a processor <NUM> will execute instructions received from memory. The processor <NUM> illustrated incorporates, or is directly connected to, cache memory <NUM>.

In more detail, the processor <NUM> may be any logic circuitry that processes instructions, e.g., instructions fetched from the memory <NUM> or cache <NUM>. In many embodiments, the processor <NUM> is a microprocessor unit or special purpose processor. The computing device <NUM> may be based on any processor, or set of processors, capable of operating as described herein. The processor <NUM> may be multiple processors. In some implementations, the processor <NUM> can be configured to run multi-threaded operations. In some implementations, the processor <NUM> may host one or more virtual machines or containers, along with a hypervisor or container manager for managing the operation of the virtual machines or containers. In such implementations, the methods shown in <FIG> and <FIG> can be implemented within the network interface card in conjunction with the virtualized or containerized environments provided on the processor <NUM>.

The memory <NUM> may be any device suitable for storing computer readable data. The memory <NUM> may be a device with fixed storage or a device for reading removable storage media. Examples include all forms of non-volatile memory, media and memory devices, semiconductor memory devices (e.g., EPROM, EEPROM, SDRAM, and flash memory devices), magnetic disks, magneto optical disks, and optical discs (e.g., CD ROM, DVD-ROM, and Blu-ray® discs). A computing system <NUM> may have any number of memory devices <NUM>. In some implementations, the memory <NUM> supports virtualized or containerized memory accessible by virtual machine or container execution environments provided by the computing system <NUM>.

The cache memory <NUM> is generally a form of computer memory placed in close proximity to the processor <NUM> for fast read times. In some implementations, the cache memory <NUM> is part of, or on the same chip as, the processor <NUM>. In some implementations, there are multiple levels of cache <NUM>, e.g., L2 and L3 cache layers.

The network interface driver controller <NUM> manages data exchanges via the network interface driver <NUM> (also referred to as network interface driver ports). The network interface driver controller <NUM> handles the physical and data link layers of the OSI model for network communication. In some implementations, some of the network interface driver controller's tasks are handled by the processor <NUM>. In some implementations, the network interface driver controller <NUM> is part of the processor <NUM>. In some implementations, a computing system <NUM> has multiple network interface driver controllers <NUM>. The network interface driver ports configured in the network interface card <NUM> are connection points for physical network links. In some implementations, the network interface controller <NUM> supports wireless network connections and an interface port associated with the network interface card <NUM> is a wireless receiver/transmitter. Generally, a computing device <NUM> exchanges data with other network devices <NUM> via physical or wireless links that interface with network interface driver ports configured in the network interface card <NUM>. In some implementations, the network interface controller <NUM> implements a network protocol such as Ethernet.

The other network devices <NUM> are connected to the computing device <NUM> via a network interface driver port included in the network interface card <NUM>. The other network devices <NUM> may be peer computing devices, network devices, or any other computing device with network functionality. For example, a first network device <NUM> may be a network device such as a hub, a bridge, a switch, or a router, connecting the computing device <NUM> to a data network such as the Internet.

The other devices <NUM> may include an I/O interface, external serial device ports, and any additional co-processors. For example, a computing system <NUM> may include an interface (e.g., a universal serial bus (USB) interface) for connecting input devices (e.g., a keyboard, microphone, mouse, or other pointing device), output devices (e.g., video display, speaker, or printer), or additional memory devices (e.g., portable flash drive or external media drive). In some implementations, a computing device <NUM> includes an additional device <NUM> such as a coprocessor, e.g., a math co-processor can assist the processor <NUM> with high precision or complex calculations.

Implementations of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software embodied on a tangible medium, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Implementations of the subject matter described in this specification can be implemented as one or more computer programs embodied on a tangible medium, i.e., one or more modules of computer program instructions, encoded on one or more computer storage media for execution by, or to control the operation of, a data processing apparatus. The computer storage medium can also be, or be included in, one or more separate components or media (e.g., multiple CDs, disks, or other storage devices). The computer storage medium may be tangible and non-transitory.

The operations may be executed within the native environment of the data processing apparatus or within one or more virtual machines or containers hosted by the data processing apparatus.

A computer program can be deployed to be executed on one computer or on multiple computers or one or more virtual machines or containers that are located at one site or distributed across multiple sites and interconnected by a communication network.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions or of what may be claimed, but rather as descriptions of features specific to particular implementations of particular inventions. Conversely, various features that are described in the context of a single implementation can also be implemented in multiple implementations separately or in any suitable sub-combination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a sub-combination or variation of a sub-combination.

References to "or" may be construed as inclusive so that any terms described using "or" may indicate any of a single, more than one, and all of the described terms. The labels "first," "second," "third," and so forth are not necessarily meant to indicate an ordering and are generally used merely to distinguish between like or similar items or elements.

Claim 1:
A method of error handling in a network interface card (<NUM>) of a host device, the method comprising:
receiving (<NUM>), at a processor of the network interface card (<NUM>), a data packet addressed to a virtual machine (<NUM>) executing on the host device;
selecting (<NUM>), by the processor, a receive queue corresponding to the virtual machine (<NUM>) and residing in a memory (<NUM>) of the network interface card (<NUM>);
retrieving (<NUM>), by the processor, a buffer descriptor from the receive queue, wherein the buffer descriptor includes a virtual memory address; and
in response to determining (<NUM>) that the virtual memory address does not translate to a valid physical memory address to a memory location of a host device memory (<NUM>) assigned to the virtual machine (<NUM>):
retrieving (<NUM>), by the processor, a backup buffer descriptor from a hypervisor queue residing in the network interface card (<NUM>) memory (<NUM>) and corresponding to a hypervisor executing on the host device (<NUM>);
storing (<NUM>), by the processor, contents of the data packet in a memory location of the host device memory (<NUM>)indicated by a backup memory address in the backup buffer descriptor; and
retrieving by the virtual machine (<NUM>) the contents of the data packet from the host device memory location for processing.