Patent Publication Number: US-11394786-B2

Title: Zero-copy forwarding for network function virtualization

Description:
TECHNICAL FIELD 
     The present disclosure is generally related to virtualized computer systems, and is more specifically related to zero-copy forwarding for network function virtualization (NFV). 
     BACKGROUND 
     Network interface controllers (NIC) implement the OSI layer 1 (physical layer) and OSI layer 2 (data link layer standards), thus providing physical access to a networking medium and a low-level addressing system using media access control (MAC) addresses, in order to allow computer systems to communicate over a wired or wireless network. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The present disclosure is illustrated by way of examples, and not by way of limitation, and may be more fully understood with references to the following detailed description when considered in connection with the figures, in which: 
         FIG. 1  depicts a high-level component diagram of an example host computer system operating in accordance with one or more aspects of the present disclosure; 
         FIG. 2  schematically illustrates an example zero-copy forwarding for NFV implemented by a host computer system operating in accordance with one or more aspects of the present disclosure; 
         FIG. 3  depicts a flowchart of an example method of zero-copy forwarding for NFV, in accordance with one or more aspects of the present disclosure; 
         FIG. 4  depicts a flowchart of another example method of zero-copy forwarding for NFV, in accordance with one or more aspects of the present disclosure 
         FIG. 5  depicts a high-level component diagram of an example computer system, which may be employed to implement the systems and methods described herein; and 
         FIG. 6  depicts a high-level component diagram of another example computer system, which may be employed to implement the systems and methods described herein. 
     
    
    
     DETAILED DESCRIPTION 
     Described herein are methods and systems for zero-copy forwarding for network function virtualization (NFV). A host computer system may be equipped with one or more network interface controllers (NICs) providing physical access to a networking medium and a low level addressing system (e.g., using media access control (MAC) addresses), in order to allow the host computer system to communicate over a wired or wireless network. Each of one or more virtual machines running on the host computer system may have one or more virtual network interface controllers (vNICs). 
     In some implementations, the data transmitted by and addressed to one or more vNICs may be handled by a hypervisor-managed proxy application, which may run within a privileged or non-privileged context on the host computer system. In an illustrative example, in order to support a Transmission Control Protocol (TCP) connection initiated by a vNIC, the proxy application would create a pair of sockets, including one socket for communicating with the vNIC driver and another socket for communicating with the destination. Responsive to receiving data on one of the sockets, the proxy applications would forward it to another socket of the pair of sockets. When one of the sockets is eventually closed, the proxy applications would close the remaining socket of the pair of sockets. However, the above-described data forwarding scheme involves creating multiple data copies on all stages of the communication process, and is prone to losing packet boundaries, retransmission/bandwidth information, etc. 
     Implementations of the present disclosure alleviates the above-noted and other deficiencies by providing methods and systems for implementing zero-copy forwarding for network function virtualization (NFV). In an illustrative example, each virtual machine running on a host computer system may create, for one or more network connections associated with each vNIC of the virtual machine, a packet filter (such as a Berkeley Packet Filter (BPF)) for matching network packets, based on their link layer protocol fields, to the respective network connections. The virtual machine may forward the packet filter definition to the hypervisor. Responsive to validating the packet filter definition, the hypervisor may associate the packet filter with the vNIC, and may run the packet filtering program within the hypervisor context. 
     For a network packet initiated by the vNIC, the packet filter may compare one or more data link layer fields of the network packet (e.g., the protocol, the port, and the destination IP address) with the corresponding data link layer parameters of existing network connections maintained by the proxy application on behalf of the vNIC. Should a matching connection be identified, the packet filter would forward the network packet directly to that connection, bypassing the proxy application. Forwarding the network packet to the identified connection may involve modifying certain fields of the network packet (e.g., the source Internet Protocol (IP) address). An incoming packet received on the connection may be similarly modified (e.g., by modifying the destination IP address), and may be forwarded to the vNIC, again bypassing the proxy application. 
     Conversely, should no existing connection be found that would match the destination address of an outgoing network packet, the hypervisor would notify the proxy application thus causing it to create a new connection to the specified destination and modify the filter accordingly. Notably, since every vNIC of the host computer system may have a corresponding packet filter associated with it, the above-described packet filtering scheme may be implemented in a multi-tenant environment. 
     Various aspects of the methods and systems are described herein by way of examples, rather than by way of limitation. The methods described herein may be implemented by hardware (e.g., general purpose and/or specialized processing devices, and/or other devices and associated circuitry), software (e.g., instructions executable by a processing device), or a combination thereof. 
       FIG. 1  depicts a high-level component diagram of an example host computer system operating in accordance with one or more aspects of the present disclosure. The example computer system  100  may comprise one or more processors  120 A- 120 B communicatively coupled to one or more memory devices  130  and two or more NICs  140 A- 140 B via a system bus  150 . 
     “Processor” or “processing device” herein refers to a device capable of executing instructions encoding arithmetic, logical, or I/O operations. In one illustrative example, a processor may follow the von Neumann architectural model and may comprise an arithmetic logic unit (ALU), a control unit, and a plurality of registers. In a further aspect, a processor may be a single core processor which is typically capable of executing one instruction at a time (or process a single pipeline of instructions), or a multi-core processor which may simultaneously execute multiple instructions. In another aspect, a processor may be implemented as a single integrated circuit, two or more integrated circuits, or may be a component of a multi-chip module (e.g., in which individual microprocessor dies are included in a single integrated circuit package and hence share a single socket). A processor may also be referred to as a central processing unit (CPU). “Memory device” herein refers to a volatile or non-volatile memory device, such as RAM, ROM, EEPROM, or any other device capable of storing data. “I/O device” herein refers to a device capable of providing an interface between a processor and an external device capable of inputting and/or outputting binary data. In various implementations, computer system  100  may further comprise various other devices, such as peripheral device controllers, which are omitted from  FIG. 1  for clarity and conciseness. 
     The example computer system  100  may be employed as a host system configured to run multiple virtual machines  170 , by executing a software layer  180 , referred to as “hypervisor,” above the hardware and below the virtual machines. In one illustrative example, the hypervisor  180  may be a component of an operating system  185  executed by host computer system  100 . Alternatively, the hypervisor  180  may be provided by an application running under the host operating system  185 , or may run directly on host computer system  100  without an operating system beneath it. The hypervisor  180  may abstract the physical layer, including processors, memory, and I/O devices, and present this abstraction to virtual machines  170  as virtual devices. 
     A virtual machine  170  may comprise one or more virtual processors (vCPUs)  190 . Processor virtualization may be implemented by the hypervisor  180  scheduling time slots on one or more physical processors (CPUs)  120  such that, from the guest operating system&#39;s perspective, those time slots are scheduled on a virtual processor  190 . The virtual machine  170  may execute a guest operating system  196 , which may utilize the underlying virtual devices, including the virtual memory  192 , virtual I/O devices  195 , and vNICs  194 . One or more applications  198  may be running on virtual machine  170  under the guest operating system  196 . 
     As noted herein above, the data transmitted by and addressed to vNICs  194  may be handled by a hypervisor-managed proxy application  182 . In the example implementation of  FIG. 1 , the proxy application  182  runs within the context of the hypervisor  180 . Alternatively, the proxy application may run within an unprivileged context of the host computer system  100 , e.g., on a virtual machine running on the host computer system  100  or a standalone application running under the host OS  185 . For each vNIC  194 , the respective virtual machine  170  may create an associated packet filter  184  for matching the network packets transmitted by and/or addressed to the vNIC to active connections maintained by the proxy application  182 , as described in more detail herein below with reference to  FIG. 2 . 
       FIG. 2  schematically illustrates an example zero-copy forwarding for NFV implemented by a host computer system operating in accordance with one or more aspects of the present disclosure. As shown in  FIG. 2 , a proxy application  182  may run in the context of the hypervisor  180  or in an unprivileged context of the host computer system  100 , e.g., on a virtual machine running on the host computer system  100  or a standalone application running under the host OS. For one or more network connections associated with each vNIC  194 , the respective virtual machine  170  may create a packet filter for matching network packets, based on their link layer protocol fields, to existing network connections. Upon creating the packet filter  184 , the virtual machine  170  may forward the packet filter definition to the hypervisor  180 . 
     In an illustrative example, the packet filter  184  may be implemented as a Berkeley Packet Filter (BPF), which is a pseudo-device that may be bound to a network interface, such that reading from the pseudo-device would return packets received on the network interface, while writing to the device would inject packets on the network interface. Accordingly, responsive to validating the packet filter  184 , the hypervisor  180  may associate the packet filter  184  with the vNIC  194 , and may run the packet filtering program within the hypervisor context. Validating the packet filter  184  may involve ensuring that the packet filtering rules encoded by the packet filter definition are not mutually-exclusive and do not specify an infinite loop or infinite recursion. 
     In operation, responsive to receiving a network packet  210  initiated by the vNIC  194 , the hypervisor  180  may identify the packet filter  184  associated with the vNIC  194  and apply the identified packet filter  184  to the network packet  210 . The packet filter  184  may compare one or more data link layer fields of the network packet  210  with the corresponding data link layer parameters of existing network connections  220 A- 220 N maintained by the proxy application  182  on behalf of the vNIC  194 . In an illustrative example, the data link layer parameters may include the protocol (e.g., TCP or UDP), the port, and the destination IP address. Should a matching connection  220 N be identified, the packet filter  184  may forward the network packet  210  directly to the identified connection  220 N, thus bypassing the proxy application  182 . 
     Forwarding the outgoing network packet  210  to the identified connection  220 N may involve modifying certain fields of the network packet (e.g., setting the source IP address of the network packet to the IP address of the host NIC  140  which is employed for sending and receiving packets on the identified connection  220 N). An incoming network packet (not shown in  FIG. 2 ) received on the connection  220 N may be similarly modified by the packet filter  184  (e.g., by setting the destination IP address of the incoming network packet to the IP address assigned to the vNIC  194 ), and may be forwarded to the vNIC  194 , again bypassing the proxy application  182 . 
     Conversely, should applying the packet filter  184  to the outgoing network packet  210  yield no existing connection that would match the specified link layer parameters of the outgoing network packet  210 , the hypervisor  180  may notify the proxy application  182 , thus causing it to create a new connection to the specified destination and modify the packet filter  184  accordingly. 
       FIG. 3  depicts a flowchart of an example method  300  of zero-copy forwarding for NFV, in accordance with one or more aspects of the present disclosure. In some implementations, method  300  may be performed by a single processing thread executed by a processing device. Alternatively, method  300  may be performed by two or more processing threads executed by one or more processing devices, such that each thread would execute one or more individual functions, routines, subroutines, or operations of the method. In an illustrative example, the processing threads implementing method  300  may be synchronized (e.g., using semaphores, critical sections, and/or other thread synchronization mechanisms). Alternatively, the processing threads implementing method  300  may be executed asynchronously with respect to each other. Therefore, while  FIG. 3  and the associated description lists the operations of method  300  in certain order, various implementations of the method may perform at least some of the described operations in parallel and/or in arbitrary selected orders. 
     At block  310 , the hypervisor running on a host computer system implementing the method may receive, from a virtual machine running on the host computer system, a definition of a packet filter for matching network packets, based on their link layer protocol fields, to existing network connections. The link layer protocol fields may include the protocol (e.g., TCP or UDP), the port, and the destination IP address, as described in more detail herein above. 
     At block  320 , the hypervisor may validate the packet filter. Validating the packet filter may involve ensuring that the packet filtering rules encoded by the packet filter definition are not mutually-exclusive and do not specify an infinite loop or infinite recursion, as described in more detail herein above. 
     At block  330 , the hypervisor may associate the packet filter with a vNIC of the virtual machine that has produced the packet filter definition. Accordingly, reading from the pseudo-device implemented by the packet filter would return packets received on the network connection selected by the filter, while writing to the device would inject packets into the selected network connection. 
     The operations of blocks  310 - 330  may be repeated for associating packet filters with one or more vNICs of one or more virtual machines running on the host computer system implementing the method. 
     At block  340 , the hypervisor may receive a network packet originated by a vNIC of a virtual machine running on the host computer system. 
     Responsive to successfully matching, at block  350 , the network packet to a network connection specified by the packet filter associated with the vNIC, the hypervisor may, at block  360 , cause the packet filter to forward the network packet via the identified network connection, and the method may loop back to block  340 . 
     Alternatively, should the packet filter associated with the vNIC fail to match, at block  350 , an existing network connection to the outgoing network packet, the hypervisor may, at block  370 , cause the network proxy application to create a new network connection to the destination specified by the network packet, and the method may loop back to block  310 . 
       FIG. 4  depicts a flowchart of an example method  400  of zero-copy forwarding for NFV, in accordance with one or more aspects of the present disclosure. In some implementations, method  400  may be performed by a single processing thread executed by a processing device. Alternatively, method  400  may be performed by two or more processing threads executed by one or more processing devices, such that each thread would execute one or more individual functions, routines, subroutines, or operations of the method. In an illustrative example, the processing threads implementing method  400  may be synchronized (e.g., using semaphores, critical sections, and/or other thread synchronization mechanisms). Alternatively, the processing threads implementing method  400  may be executed asynchronously with respect to each other. Therefore, while  FIG. 4  and the associated description lists the operations of method  400  in certain order, various implementations of the method may perform at least some of the described operations in parallel and/or in arbitrary selected orders. 
     At block  410 , the hypervisor running on a host computer system implementing the method may receive, from a virtual machine running on the host computer system, a definition of a packet filter for matching network packets, based on their link layer protocol fields, to existing network connections. The link layer protocol fields may include the protocol (e.g., TCP or UDP), the port, and the destination IP address, as described in more detail herein above. 
     At block  420 , the hypervisor may validate the packet filter. Validating the packet filter may involve ensuring that the packet filtering rules encoded by the packet filter definition are not mutually-exclusive and do not specify an infinite loop or infinite recursion, as described in more detail herein above. 
     At block  430 , the hypervisor may associate the packet filter with a vNIC of the virtual machine that has produced the packet filter definition. Accordingly, reading from the pseudo-device implemented by the packet filter would return packets received on the network connection selected by the filter, while writing to the device would inject packets into the selected network connection. 
     At block  440 , the hypervisor may receive a first network packet originated by a vNIC of a virtual machine running on the host computer system. 
     At block  450 , the hypervisor may identify, by applying the packet filter associated with the vNIC, a network connection matching the data link layer fields (e.g., the protocol, the port, and the destination address) of the first network packet, as described in more detail herein above. 
     At block  460 , the hypervisor may cause the packet filter to forward the first network packet via the identified network connection, as described in more detail herein above. 
     At block  470 , the hypervisor may receive a second network packet originated by the vNIC. 
     Responsive to failing to match, at block  480 , an existing network connection to the second network packet, the hypervisor may, at block  490 , cause the network proxy application to create a new network connection to the destination specified by the second network packet. 
       FIG. 5  depicts a block diagram of an illustrative computer system  500  operating in accordance with one or more aspects of the disclosure. In various implementations, computer system  1000  may perform the functions of to the host computer system  100  of  FIG. 1 . Computer system  500  comprises a memory  510  and one or more physical processors  520 A- 520 N, that are operatively coupled to the memory  510  and execute the code implementing the methods  300  and/or  400  for zero-copy forwarding for NFV. The memory  510  may further store definitions of packet filters  530 A- 530 N associated with respective vNICs of one or more virtual machines running on the host computer system  500 . 
       FIG. 6  depicts a high-level component diagram of an example computer system which may be employed to implement the systems and methods described herein. In various implementations, computer system  1000  may perform the functions of host computer system  100  of  FIG. 1 . In some implementations, computer system  1000  may be connected (e.g., via a network  1030 , such as a Local Area Network (LAN), an intranet, an extranet, or the Internet) to other computer systems. Computer system  1000  may operate in the capacity of a server or a client computer in a client-server environment, or as a peer computer in a peer-to-peer or distributed network environment. Computer system  1000  may be provided by a personal computer (PC), a tablet PC, a set-top box (STB), a Personal Digital Assistant (PDA), a cellular telephone, a web appliance, a server, a network router, switch or bridge, or any device capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that device. Further, the term “computer” shall include any collection of computers that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methods described herein. 
     In a further aspect, the computer system  1000  may include a processing device  1002 , a volatile memory  1004  (e.g., random access memory (RAM)), a non-volatile memory  1009  (e.g., read-only memory (ROM) or electrically-erasable programmable ROM (EEPROM)), and a data storage device  1016 , which may communicate with each other via a bus  1008 . 
     Processing device  1002  may be provided by one or more processors such as a general purpose processor (such as, for example, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a microprocessor implementing other types of instruction sets, or a microprocessor implementing a combination of types of instruction sets) or a specialized processor (such as, for example, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), or a network processor). 
     Computer system  1000  may further include a network interface device  1022 . Computer system  1000  also may include a video display unit  1010  (e.g., an LCD), an alphanumeric input device  1012  (e.g., a keyboard), a cursor control device  1014  (e.g., a mouse), and a signal generation device  1020 . 
     Data storage device  1016  may include a non-transitory computer-readable storage medium  1024  on which may store instructions  1026  encoding any one or more of the methods or functions described herein, including instructions for implementing methods  300  and/or  400  of zero-copy forwarding for NFV. 
     Instructions  1026  may also reside, completely or partially, within volatile memory  1004  and/or within processing device  1002  during execution thereof by computer system  1000 , hence, volatile memory  1004  and processing device  1002  may also constitute machine-readable storage media. 
     While computer-readable storage medium  1024  is shown in the illustrative examples as a single medium, the term “computer-readable storage medium” shall include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of executable instructions. The term “computer-readable storage medium” shall also include any tangible medium that is capable of storing or encoding a set of instructions for execution by a computer that cause the computer to perform any one or more of the methods described herein. The term “computer-readable storage medium” shall include, but not be limited to, solid-state memories, optical media, and magnetic media. 
     The methods, components, and features described herein may be implemented by discrete hardware components or may be integrated in the functionality of other hardware components such as ASICS, FPGAs, DSPs or similar devices. In addition, the methods, components, and features may be implemented by firmware modules or functional circuitry within hardware devices. Further, the methods, components, and features may be implemented in any combination of hardware devices and software components, or only in software. 
     Unless specifically stated otherwise, terms such as “updating”, “identifying”, “determining”, “sending”, “assigning”, or the like, refer to actions and processes performed or implemented by computer systems that manipulates and transforms data represented as physical (electronic) quantities within the computer system registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices. 
     Examples described herein also relate to an apparatus for performing the methods described herein. This apparatus may be specially constructed for performing the methods described herein, or it may comprise a general purpose computer system selectively programmed by a computer program stored in the computer system. Such a computer program may be stored in a computer-readable tangible storage medium. 
     The methods and illustrative examples described herein are not inherently related to any particular computer or other apparatus. Various general purpose systems may be used in accordance with the teachings described herein, or it may prove convenient to construct more specialized apparatus to perform methods  400 ,  500  and/or each of their individual functions, routines, subroutines, or operations. Examples of the structure for a variety of these systems are set forth in the description above. 
     The above description is intended to be illustrative, and not restrictive. Although the present disclosure has been described with references to specific illustrative examples and implementations, it will be recognized that the present disclosure is not limited to the examples and implementations described. The scope of the disclosure should be determined with reference to the following claims, along with the full scope of equivalents to which the claims are entitled.