Virtual machine networking

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for receiving one or more respective socket commands from a virtual machine socket core, the one or more respective socket commands being associated with a respective socket destination address; determining, based on the one or more respective socket commands and the respective socket destination address, whether to communicate data associated with the one or more respective socket commands to a host virtual socket device; and in response to determining to communicate the data associated with the one or more respective socket commands to the host virtual socket device, converting the socket destination address to a Uniform Resource Identifier (URI) address and communicating the data associated with the one or more respective socket commands and the URI address between the host virtual socket device.

BACKGROUND

This specification relates to networking of virtual machines.

Cloud computing is network-based computing in which one or more servers provide computational resources and data storage as needed to remote end users. Some cloud computing services provide access to software applications such as word processors and other commonly used applications to end users who interface with the applications through web browsers or other client-side software. Users' electronic data files are usually stored in the server farm rather than on the users' computing devices. Maintaining software applications and user data on a server farm simplifies management of end user computing devices. Some cloud computing services allow end users to execute software applications in virtual machines. In a public cloud computing environment, multiple users are able to launch virtual machines (VMs), and each VM launched by a user is included in a cluster of other VMs launched by the user.

SUMMARY

In general, one innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of receiving one or more socket commands from a virtual machine socket core, each of the one or more socket commands being associated with a respective socket destination address; determining, based on the one or more socket commands and the respective socket destination addresses, whether to communicate data associated with the one or more respective socket commands to a host virtual socket device; and in response to determining to communicate the data associated with the one or more respective socket commands to the host virtual socket device, converting the socket destination addresses to a respective Uniform Resource Identifier (URI) address and communicating the data associated with the one or more socket commands and their respective URI addresses between the host virtual socket device.

These and other embodiments can each optionally include one or more of the following features. The host virtual socket device is implemented using a virtio socket device, wherein the data associated with the one or more respective socket commands is communicated to the virtio socket device using one or more virtio queues. The URI address specifies a protocol, a destination end point address, and a port number. Based on the URI address, the host virtual socket device provides the data associated with the one or more respective socket commands to a socket mapper backend, the socket mapper backend being configured to map, based on the URI address, each of the one or more respective socket commands to a particular socket handler for processing. Each socket handler is configured to communicate the data associated with the respective one or more respective socket commands to the specified destination end point address using the specified protocol. The socket mapper backend is configured to evaluate the URI address based on an access control list. The socket mapper backend is configured to rewrite the URI address based on one or more rules specified in the access control list. The data associated with the one or more respective socket commands are not encapsulated using a networking protocol stack in the virtual machine.

Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. Network data can be communicated from a virtual machine, i.e., a guest, to a virtual machine manager, i.e., host, to a host virtual socket device. Network data being communicated from a guest to a host can bypass the networking protocol stack in the virtual machine and is communicated from the guest, using a virtual socket coprocessor device, to the host through a host virtual socket device. Thus, the guest is no longer tasked with the segmentation, fragmentation, dispatching, or re-transmission of data when communicating data between the guest and the host. Data communicated using the host virtual socket device does not rely on timers, e.g., re-transmission timers, as is typically the case in TCP/IP communication.

DETAILED DESCRIPTION

FIG. 1is a schematic illustration of an example virtual machine system100. The system100includes one or more host machines such as, for example, host machine102and host machine104. Generally speaking, a host machine is one or more data processing apparatuses such as rack mounted servers or other computing devices. The data processing apparatuses can be in different physical locations and can have different capabilities and computer architectures. Host machines can communicate with each other through an internal data communications network116. The internal network can include one or more wired (e.g., Ethernet) or wireless (e.g., WI-FI) networks, for example. In some implementations the internal network116is or includes an intranet. Host machines can also communicate with devices on one or more external networks122, such as the Internet. In an embodiment, the host machines may communicate with the one or more external networks122through one or more gateways120which are data processing apparatuses responsible for routing data communication traffic between the internal network116and the external network122. Other types of external networks are possible.

Each host machine102,104executes a host operating system106,108. A host operating system106,108manages host machine resources. In this example, host operating systems106,108run software that virtualizes the underlying host machine hardware and manages concurrent execution of one or more virtual machines. For example, the host operating system106manages virtual machine (VM)110and VM112, while a different host operating system108manages a single VM114. VMs can be migrated from one host machine to another host machine. In addition, a single VM can be managed by multiple host machines. Each VM includes a simulated version of the underlying host machine hardware, or a different computer architecture. The simulated version of hardware is referred to as virtual hardware (e.g., virtual hardware110a,112aand114a). Software that is executed by the virtual hardware is referred to as guest software. In some implementations, guest software cannot determine if it is being executed by virtual hardware or by a physical host machine. If guest software executing in a VM, or the VM itself, malfunctions or aborts, other VMs executing on the host machine will not be affected. A host machine's microprocessor(s) can include processor-level mechanisms to enable virtual hardware to execute software applications efficiently by allowing guest software instructions to be executed directly on the host machine's microprocessor without requiring code-rewriting, recompilation, or instruction emulation.

Each VM (e.g., VMs110,112and114) is allocated a set of virtual memory pages from the virtual memory of the underlying host operating system. Each VM is also allocated virtual disk blocks from one or more virtual disk drives for use by the guest software executing on the VM. The term “virtual disk drive” includes, but is not limited to, a simulated version of a physical disk drive. For example, host operating system106allocates memory pages and disk blocks to VM110and VM112, and host operating system108does the same for VM114. In some implementations, a given VM cannot access the virtual memory pages assigned to other VMs. For example, VM110cannot access memory pages that have been assigned to VM112. A virtual disk drive can be persisted across VM restarts. Virtual disk blocks are allocated on physical disk drives coupled to host machines or available over the internal network116, for example.

In addition to virtual memory and virtual disk resources, VMs can be allocated network addresses through which their respective guest software can communicate with other processes reachable through the internal network116or the external network122. The network addresses can be, for example, Internet Protocol (IP) addresses. In some implementations, each VM is allocated one or more unique Internet Protocol (IP) version 4 or version 6 addresses, one or more User Datagram Protocol (UDP) port numbers, or both. Other address schemes are possible.

A VM's guest software can include a guest operating system (e.g., guest operating systems110b,112band114b) which is software that controls the execution of respective guest software applications (e.g., guest applications110c,112cand114c), within the VM and provides services to those applications. For example, a guest operating system could be a variation of the UNIX operating system. Other operating systems are possible. Each VM can execute the same guest operating system or different guest operating systems. In further implementations, a VM does not require a guest operating system in order to execute guest software applications. A guest operating system's access to resources such as networks and virtual disk storage is controlled by the underlying host operating system, other cluster management software executing on a separate computing device, or both.

By way of illustration, and with reference to virtual machine110, when the guest application110cor guest operating system110battempts to perform an input/output operation on a virtual disk, initiate network communication, or perform a privileged operation, for example, the virtual hardware110ais interrupted so that the host operating system106can perform the action on behalf of the virtual machine110. The host operating system106can perform these actions with a process that executes in kernel process space106b, user process space106a, or both.

The kernel process space106bis virtual memory reserved for the host operating system106's kernel106dwhich can include kernel extensions and device drivers, for instance. Similarly, the kernel process space108bis virtual memory reserved for the host operating system108's kernel108d.

The kernel process space has elevated privileges (sometimes referred to as “supervisor mode”); that is, the kernel106dcan perform certain privileged operations that are off limits to processes running in the user process space106a. Examples of privileged operations include access to different address spaces, access to special functional processor units in the host machine such as memory management units, and so on. The user process space106ais a separate portion of virtual memory reserved for user mode processes. User mode processes cannot perform privileged operations directly.

In various implementations, a portion of VM network communication functionality is implemented in a communication process (e.g., communication process106cor communication process108c). In some implementations, the communication process executes in the user process space (e.g., user process space106aor user process space108a) of a host operating system (e.g., host operating system106or host operating system108). In other implementations, the communication process can execute in the kernel process space (e.g., kernel process space106dor kernel process space108d) of the host operating system. There can be a single communication process for all VMs executing on a host machine or multiple communication processes, for the VMs executing on a host machine. In yet further implementations, some portion of the communication process executes in the user process space and another portion executes in the kernel process space. For example, the communications process106creceives packets sent by VM110or VM112and forwards the packets to the destination identified in the packet.

In various implementations, a VM can be configured so that some or all socket commands from the VM (e.g., socket commands originating from guest software applications) are offloaded to a virtual socket coprocessor device150that resides within the VM. The term “socket command” may refer to socket operations that are performed by the user process space component in a guest operating system, e.g., operations performed using the VM's socket core. Some example socket operations include create, connect, bind, release, sendmsg, recvmsg_nosec, splice_read, do_ioctl, poll, mmap, listen, accept, getsockname, getpeername, setsockopt, getsockopt, and shutdown. Other socket operations may also be included.

The virtual socket coprocessor device150can be, e.g., a virtual PCI card. In such implementations, data associated with the offloaded socket commands is not encapsulated by a networking protocol stack (e.g., the Open System Interconnection, or OSI, model) in the VM, i.e., the data associated with the socket commands bypasses the VM networking protocol stack. Instead, the data associated with the socket commands is communicated directly to the host using the virtual socket coprocessor device150. The virtual socket coprocessor device150is configured to communicate data associated with the offloaded socket commands to the host operating system that is managing the VM (i.e., the Virtual Machine Manager, or VMM) for further processing. The VMM receives the offloaded socket commands at a host virtual socket device that resides in the host operating system.

In some implementations, both the guest operating system and the host operation system maintain respective states for each socket command being communicated through the virtual socket coprocessor device150. In such implementations, the guest operating system and the host operating system coordinate their respective states to communicate socket commands asynchronously.

By offloading the socket commands to the virtual socket coprocessor device150, the VM can communicate data to the host operating system through, e.g., the transport layer, or Layer 4, in the OSI model. Thus, socket commands can be communicated without having to process and communicate the data through a host virtual network (e.g., Ethernet) device, where the data is sent, e.g., through the data link layer, or Layer 2, in the OSI model.

Thus, for example, when an application running in the guest attempts to communicate network data to a device over a network, e.g., the Internet, the create, connect, and bind socket operations that are typically handled by the VM socket core are offloaded to the virtual socket coprocessor device. The virtual socket coprocessor device then communicates data being sent by the guest application to the host operating system through a host virtual socket device. Therefore, the data being sent by the guest application is not processed by the VM networking protocol stack before being sent to the host operating system. Once the data being sent by the guest application is received by the host, the host can process the data using the host networking protocol stack, and can communicate the data to the device over the network.

As a result, the VM is no longer tasked with the responsibility of constructing and communicating data packets to the host operating system, which are then evaluated by the host operating system or some other underlying network fabric. Instead, the host operating system constructs data packets using the data received at the host virtual socket device, and communicates the data packets to a destination end point (e.g., another VM, another machine accessible on the Internet, or an application that is running on the host operating system) that is specified by the VM.

The term “socket” may refer to a bidirectional communication channel between a VM and another end point. In some implementations, a socket is configured as a unidirectional communication channel between a VM and another end point. In other words, a socket can have one direction of communication disabled, e.g., an incoming direction, while the other direction, e.g., an outgoing direction, is enabled. Network communication between the host and guest operating systems can be accomplished by accessing the VM's socket core using, e.g., generally known socket Application Programming Interfaces (APIs) (e.g., the Berkeley sockets API or the Windows Sockets API).

In some implementations, the VM's socket core is configured so that some or all socket-based communication (i.e., socket commands) is offloaded to the virtual socket coprocessor device150. In some implementations, operations within a socket core are configured to offload socket commands to the virtual socket coprocessor device150by including a SOCK_HWASSIST flag in the type of available socket types. For example, when a guest application114cattempts to create a stream socket, e.g., using the SOCK_STREAM flag, or a datagram socket, e.g., using the SOCK_DGRAM flag, the socket core can be configured to offload the socket to the virtual socket coprocessor device150by using a SOCK_HWASSIST flag instead. As a result, socket commands configured using the SOCK_HWASSIST flag are offloaded by the VM to the virtual socket coprocessor device150for processing, rather than creating the stream or datagram socket to a destination end point.

In some implementations, the virtual socket coprocessor device150has first right of refusal on handling certain global socket functions. For example, the socket core can be configured so that the virtual socket coprocessor device150has first right of refusal for handling certain operations, e.g., socket, connect, bind, listen, accept, send, sendto, shutdown, that are made by a guest application114c. In evaluating its first right of refusal, the virtual socket coprocessor device150determines whether socket commands originating from a guest application114cshould be offloaded to the virtual socket coprocessor device150for handling.

The VM's virtual hardware also includes a virtual socket device driver that can communicate with a host virtual socket device that resides in the VMM. The virtual socket device driver can be included in the virtual machine's guest software, e.g., as part of the guest operating system. In some implementations, the virtual socket coprocessor device150communicates offloaded socket commands to the host virtual socket device in the VMM using the virtual socket device driver. Socket commands being offloaded are tagged with a SOCK_HWASSIST flag, as described above, to indicate that the socket commands should be processed by the virtual socket coprocessor device150, and not the VM's networking protocol stack. If the virtual socket coprocessor device150returns an error code during the offloading of socket commands, the socket commands can be processed using the VM's networking protocol stack. Similarly, in a situation where the virtual socket coprocessor device150determines that the socket commands should not be offloaded, the socket commands are processed by the VM's networking protocol stack and passed to the host operating system through the host virtual network (e.g., Ethernet) device.

In some implementations, the host virtual socket device is implemented using a virtio socket device. The virtio socket device can be operated by the VM using a virtio socket device driver. Socket commands can be passed between the VM's virtual socket coprocessor device and the VMM's host virtual socket device using the virtio socket device driver and shared memory, for example, as implemented using one or more virtio queues. In some implementations, socket commands that are received by the VMM's host virtual socket device are evaluated by a socket mapper backend, as described below.

In some implementations, socket destination addresses specified by a guest application114care converted by the virtual socket coprocessor device150to Uniform Resource Identifier (URI) addresses. This conversion allows the virtual socket coprocessor device150to communicate to the host operating system the type of communication protocol (e.g., Transmission Control Protocol, or TCP, or User Datagram Protocol, or UDP), a destination end point address (e.g., Internet Protocol, or IP, address), and a port number. For example, when creating a socket, a guest application114ccan specify a socket destination address that uses the IPv4 protocol (e.g., AF_INET) and is a stream socket (e.g., SOCK_STREAM). In such cases, the virtual socket coprocessor device150can convert the socket destination address to a URI address, e.g., having a “tcp://<IPv4 address>:port” format, where “tcp” is the communication protocol, where “IP address” is the end point address, and where “port” is the port number.

An example URI conversion table is:

In some implementations, the socket mapper is configured to map incoming socket commands to one or more socket handlers based on their respective URI addresses. For example, the socket mapper can map socket commands associated with a TCP URI address (e.g., “tcp://”) to a TCP socket handler. Socket handlers can provide operations for handling a particular protocol. Each socket handler is configured to facilitate communication of socket commands, e.g., by creating sockets, using the protocol specified in a URI address. For example, a UDP socket handler can be configured to process socket commands associated with UDP URI addresses (e.g., “udp://”) by converting the socket commands into UDP datagram packets. A TCP socket handler can be configured to process socket commands associated with TCP URI addresses (e.g., “tcp://”) by converting the socket commands into TCP packets.

Additional types of socket handlers can be created. For example, a HTTP socket hander can be configured to handle Hypertext Transfer Protocol (HTTP) requests sent by the VM. A DNS socket handler can be configured to respond to Domain Name System (DNS) queries sent by the VM.

In some implementations, the socket mapper backend evaluates the URI addresses provided by the VM based on an access control list (ACL). The host operating system can be configured to perform ACL validation based on regular expression pattern matching of the URI addresses. In some implementations, the host operating system evaluates the URI addresses using a blacklist of regular expressions. In some implementations the host operating system evaluates the URI addresses using a whitelist of regular expressions.

In some implementations, the socket mapper backend is configured implement ACL validation by rewriting URI addresses that were provided by the VM. For example, if the VM provided a URI address for “tcp://foo.bar:80” (i.e., a TCP connection to a site “foo.bar” through port 80), the host operating system can rewrite the URI address to “tcp://foo.bar.secure:1234”, (i.e., a secure TCP connection to a different site “foo.bar.secure” through port 1234), so that the URI address is redirected to a secure version of the site “foo.bar” site.

FIG. 2is a schematic illustration of an example virtual machine system200describing network communication between a virtual machine (VM), i.e., a guest,202and a virtual machine manager (VMM), i.e., a host,252. Depending on the implementations, a virtual machine system can be configured to perform socket offload using a virtual socket coprocessor device, as described in reference toFIG. 1, or socket offload using a virtual socket coprocessor device in combination with a virtual networking (e.g., Ethernet) device.

The VM202includes a socket core204, a virtual networking device driver206, a guest networking stack208, a virtual socket coprocessor device210, and a virtual socket device driver212. The VMM252includes a host networking stack254, a virtual networking device256, a host virtual socket device258, a socket mapper260, and handlers262. The VM202can communicate with the VMM by constructing packet data using the guest networking stack208and sending the packets to the VMM using the virtual networking device256.

In some implementations, socket commands from the VM (e.g., socket commands originating from guest software applications) are offloaded to the virtual socket coprocessor device210, as described above. The virtual socket coprocessor device210can convert the socket destination address corresponding to the socket commands to a URI string, as described above. The data associated with offloaded socket commands is communicated to the VMM using the host virtual socket device258. In some implementations, a socket mapper260is configured to map incoming socket commands to one or more socket handlers262based on their respective URI addresses, as described above. The socket handlers262process and communicate the data to the VMM's hardware264.

FIG. 3is a flow diagram illustrating an example technique300of offloading socket commands to a virtual socket coprocessor device. The technique300can be performed by one or more data processing apparatuses. For example, a virtual machine (e.g., virtual machine114ofFIG. 1) having a virtual socket coprocessor device that is executing on a host machine (e.g., host machine104ofFIG. 1) can be used to perform the technique300.

The virtual socket coprocessor device receives302socket commands from a virtual machine socket core, each of the socket commands being associated with a respective socket destination address. In some implementations, the virtual machine socket core is configured to offload, e.g., provide, some or all of the socket commands to the virtual socket coprocessor device for processing, as described above.

The virtual socket coprocessor device determines304, based on the socket commands and the respective socket destination addresses, whether to communicate data associated with the respective socket commands to a host virtual socket device. In some implementations, the virtual socket coprocessor device communicates data associated with the respective socket commands to the host virtual socket device based on a protocol specified in the respective socket destination address. For example, the virtual socket coprocessor device can offload all socket commands associated with a socket address that specifies the TCP protocol. In some implementations, the virtual socket coprocessor device offloads socket commands that include one or more operations specified in a socket core, e.g., socket, connect, bind, listen, accept, send, sendto, or shutdown. In some implementations, socket commands that relate to recv or read, as specified in a socket core, are not offloaded. In such implementations, the virtual machine, i.e., guest, is tasked with processing these socket commands.

In response to determining that data associated with the respective socket commands should be communicated to the host virtual socket device, the virtual socket coprocessor device communicates306the data associated with the socket commands to the host virtual socket device using the virtual socket device driver. In some implementations, the virtual socket coprocessor device converts the socket destination address to a URI address, as described above. In some implementations, the host virtual socket device communicates the data associated with the socket commands to a socket mapper backend that resides on the VMM. The socket mapper backend can evaluate URI addresses based on an access control list, as described above.

FIG. 4is a block diagram of computing devices400,450that may be used to implement the systems and methods described in this document, as either a client or as a server or plurality of servers. The phrase ‘computing device’ as used herein encompasses its plain and ordinary meaning, including, but not limited to, various forms of digital computers and mobile devices. Computing device400is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device450is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, and other similar computing devices. Additionally computing device400or450can include Universal Serial Bus (USB) flash drives. The USB flash drives may store operating systems and other applications. The USB flash drives can include input/output components, such as a wireless transmitter or USB connector that may be inserted into a USB port of another computing device. The components shown here, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations described and/or claimed in this document.

The memory404stores information within the computing device400. In one implementation, the memory404is a volatile memory unit or units. In another implementation, the memory404is a non-volatile memory unit or units. The memory404may also be another form of computer-readable medium, such as a magnetic or optical disk.

The storage device406is capable of providing mass storage for the computing device400. In one implementation, the storage device406may be or contain a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. A computer program product can be tangibly embodied in an information carrier. The computer program product may also contain instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory404, the storage device406, or memory on processor402.

The high speed controller408manages bandwidth-intensive operations for the computing device400, while the low speed controller412manages lower bandwidth-intensive operations. Such allocation of functions is exemplary only. In one implementation, the high-speed controller408is coupled to memory404, display416(e.g., through a graphics processor or accelerator), and to high-speed expansion ports410, which may accept various expansion cards (not shown). In the implementation, low-speed controller412is coupled to storage device406and low-speed expansion port415. The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

The computing device400may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server420, or multiple times in a group of such servers. It may also be implemented as part of a rack server system424. In addition, it may be implemented in a personal computer such as a laptop computer422. Alternatively, components from computing device400may be combined with other components in a mobile device (not shown), such as device450. Each of such devices may contain one or more of computing device400,450, and an entire system may be made up of multiple computing devices400,450communicating with each other.

Computing device450includes a processor452, memory464, an input/output device such as a display454, a communication interface466, and a transceiver468, among other components. The device450may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the components450,452,464,454,466, and468, are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other manners as appropriate.

The processor452can execute instructions within the computing device450, including instructions stored in the memory464. The processor may be implemented as a chipset of chips that include separate and multiple analog and digital processors. Additionally, the processor may be implemented using any of a number of architectures. For example, the processor410may be a CISC (Complex Instruction Set Computers) processor, a RISC (Reduced Instruction Set Computer) processor, or a MISC (Minimal Instruction Set Computer) processor. The processor may provide, for example, for coordination of the other components of the device450, such as control of user interfaces, applications run by device450, and wireless communication by device450.

Processor452may communicate with a user through control interface458and display interface456coupled to a display454. The display454may be, for example, a TFT (Thin-Film-Transistor Liquid Crystal Display) display or an OLED (Organic Light Emitting Diode) display, or other appropriate display technology. The display interface456may comprise appropriate circuitry for driving the display454to present graphical and other information to a user. The control interface458may receive commands from a user and convert them for submission to the processor452. In addition, an external interface462may be provide in communication with processor452, so as to enable near area communication of device450with other devices. External interface462may provide, for example, for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces may also be used.

The memory465stores information within the computing device450. The memory464can be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. Expansion memory474may also be provided and connected to device450through expansion interface472, which may include, for example, a SIMM (Single In Line Memory Module) card interface. Such expansion memory474may provide extra storage space for device450, or may also store applications or other information for device450. Specifically, expansion memory474may include instructions to carry out or supplement the processes described above, and may include secure information also. Thus, for example, expansion memory474may be provide as a security module for device450, and may be programmed with instructions that permit secure use of device450. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.

Device450may also communicate audibly using audio codec460, which may receive spoken information from a user and convert it to usable digital information. Audio codec460may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of device450. Such sound may include sound from voice telephone calls, may include recorded sound (e.g., voice messages, music files, etc.) and may also include sound generated by applications operating on device450.