Patent Description:
<CIT> describes hosting virtual memory backed kernel isolated containers. A server includes at least one physical processor and at least one physical computer memory addressable via physical memory addresses. The at least one physical computer memory stores executable code configured to provide at least one host including a kernel and at least one kernel isolated container within the at least one host. The host allocates virtual memory having virtual memory addresses to a respective container of the at least one kernel isolated container. The host pins a subset of the virtual memory addresses to a subset of the physical memory addresses. The host performs a direct memory access operation or device memory-mapped input- output operation of the respective container on the subset of the physical memory addresses. At least part of the physical computer memory that is not pinned is oversubscribed.

A computer system is provided. The computer system may include at least one host device comprising at least one processor. The at least one processor may be configured to implement, in a host operating system (OS) space, a teamed network interface card (NIC) software program that provides a unified interface to host OS space upper layer protocols including at least a remote direct memory access (RDMA) protocol and an Ethernet protocol. The teamed NIC software program may provide multiplexing for at least two data pathways. The at least two data pathways may include an RDMA data pathway that transmits communications to and from an RDMA interface of a physical NIC, the RDMA data pathway being within the host OS space. The at least two data pathways may include an Ethernet data pathway that transmits communications to and from an Ethernet interface of the physical NIC through a virtual switch that is implemented in a host user space and a virtual NIC that is implemented in the host OS space.

Cloud computing usage continues to grow as more enterprises have moved toward hosting their services and applications on cloud-based datacenters. Kernel space network virtualization encounters difficult challenges when used for cloud computing solutions. For example, kernel space network virtualization places greater burden on the host central processing units (CPUs), and has become cost prohibitive. Further, kernel space network virtualization faces network and data security challenges. For example, a guest user may potentially escape and land in global kernel address space, which may present a vulnerability that, in an extreme case, may be exploited to take over the host. Lastly, the kernel of the operating system has a large amount of overhead from decades of legacy software that has built up to achieve broad applicability goals that may degrade performance and increase servicing costs.

Due to these challenges, cloud computing has increasingly moved toward user space network virtualization, which enables high-performance, secure, and reliable packet processing for modern datacenter workloads. However, conventional implementations of user space network virtualization suffer from several drawbacks. For example, such implementations typically do not allow for user space network virtualization to participate in the same virtual fabric and coexist with kernel space remote direct memory access (RDMA). These user space network virtualization implementations typically do not address the RDMA over Converged Networking requirements, and do not have a notion of a Converged device. Thus, these user space network virtualization implementations typically preclude RDMA enablement and user space networking for the same host address space.

To address these issues, the following disclosure describes a server architecture for enabling both user space network virtualization and RDMA that does not need to co-locate the Ethernet device (e.g., the network interface card) of a compute node with the Transmission Control Protocol (TCP) and the Internet Protocol (IP) stack used to service incoming data packets, thus allowing for a user space Ethernet packet processing data pathway while simultaneously allowing for kernel space consumption of RDMA protocols.

<FIG> illustrates a computer system <NUM> for a cloud platform <NUM> that may implement the server architecture described herein for enabling both user space network virtualization and RDMA. The computer system <NUM> includes a hardware plane <NUM>, a virtual machine plane <NUM>, a hypervisor plane <NUM>, and network infrastructure <NUM> that are collectively configured to operate the cloud platform <NUM>. The hardware plane <NUM> includes a collection of host devices <NUM> (each denoted by the symbol "N" for compute node in <FIG>) that may include processors, graphics processing units (GPU), volatile memory, and other computer components configured to run a host operating system (OS). The host OS executed by the host devices <NUM> of the hardware plane <NUM> are configured to communicate with one or more hypervisors of the hypervisor plane <NUM>. The one or more hypervisors of the hypervisor plane <NUM> may create, handle, and monitor a plurality of virtual machines <NUM> (each denoted by the symbol "VM" in <FIG>) of the virtual machine plane <NUM>. Through the hypervisor plane <NUM>, each virtual machine <NUM> of the virtual machine plane <NUM> may be hosted and run by the hardware components of one or more host devices <NUM> of the hardware plane <NUM>. In this manner, the plurality of virtual machines <NUM> of the virtual machine plane <NUM> may share virtualized hardware resources managed by the hypervisor plane <NUM>. Each virtual machine <NUM> provides a simulated computer environment within which guest software, such as, for example, cloud applications may be executed.

In one example, the computer system <NUM> corresponds to a data center environment configured to operate the cloud platform <NUM> that communicatively couples the plurality of host devices <NUM> via standard network infrastructure. Turning to <FIG>, the plurality of host devices <NUM> may be organized into a plurality of host device clusters <NUM>. Each host device cluster <NUM> may include a top of rack (TOR) network switch <NUM>, two or more host devices of the plurality of host devices <NUM>, and a backplane <NUM> communicatively coupling the top of rack network switch <NUM> and host devices <NUM>. For example, each host device cluster <NUM> may correspond to a server rack that provides physical structure, ventilation, etc., for a TOR switch <NUM> and a plurality of host devices <NUM> that are located physically proximate to each other in the same server rack. The backplane <NUM> communicatively coupling each host device in the server rack may facilitate a low latency and high bandwidth exchange of network packets between host devices in the same server rack. It should be appreciated that the host device cluster <NUM> shown in <FIG> is merely exemplary, and that the host devices <NUM> may be organized in any other suitable configuration in the computer system <NUM>.

As illustrated in <FIG>, each host device <NUM> in the host device cluster <NUM> includes at least one processor <NUM> communicatively coupled to other hardware components by an internal data bus <NUM>. The at least one processor <NUM> may execute a host OS <NUM>. As shown, each host device <NUM> may include more than one processor <NUM> that may each execute a separate host OS <NUM>, or may collectively execute a single host OS. In one example, the internal data bus <NUM> may take the form of a Peripheral Component Interconnect Express (PCIe) link, for example. Data buses of other formats may alternatively be used. It should be appreciated that "internal" as used in the term "internal data bus" refers to the fact that at least a portion of the data bus is typically housed in the same housing (which serves as a Faraday cage) as the processor <NUM> of the host device <NUM>, and should be understood to encompass a data bus that connects a processor of a host device in a housing with internally mounted hardware components and/or to externally coupled hardware components plugged into, e.g., a port on an external surface of the housing of the host device. As illustrated, each host device <NUM> may include other suitable hardware components, such as, for example, a hardware acceleration device <NUM> that may be used to provide hardware acceleration for applications or modules of the host OS <NUM>, a physical network interface card (NIC) <NUM>, volatile and non-volatile memory <NUM>, etc. It should be appreciated that the host devices <NUM> are not limited to the illustrated hardware components, but may include any suitable configuration of hardware components configured for operating the cloud platform <NUM>. Additionally, it should be appreciated that while the host devices <NUM> are illustrated as being clustered in a server rack configuration, other types of network infrastructure and housing configurations may be utilized to couple the plurality of host devices <NUM> and operate the cloud platform <NUM>.

Turning back to <FIG>, the network infrastructure <NUM> may include typical network infrastructure to couple the host devices <NUM> within a host device cluster together, such as server racks including TOR network switches. The computer system <NUM> may include a plurality of host device clusters that each have an associated TOR network switch, and may have the architecture described in <FIG>. Network infrastructure <NUM> may further include higher-level switching infrastructure <NUM> (L1) and (L2) that connects the TOR network switches together. The higher-level switching infrastructure <NUM> may take the form of any suitable networking architecture, and may be driven by any suitable routing protocol(s). In the illustrated example, the higher-level infrastructure <NUM> includes a collection of aggregation switches L1 and core switches L2. However, it will be appreciated that the higher-level switching infrastructure may include any suitable number of levels of switches.

Each host OS <NUM> executed via processors <NUM> of the host devices <NUM> may communicate with other host server instances <NUM> through the network infrastructure <NUM>. Additionally, the plurality of NICs <NUM> of the plurality of host devices <NUM> may include remote direct access memory (RDMA) capabilities that allow applications and modules running on the cloud platform <NUM> to directly access memory devices across the cloud platform <NUM> without passing through a host OS <NUM>.

The collective host OSs <NUM> manages the collective hardware resources of the hardware plane <NUM>, which may be utilized to run the virtual machines <NUM> of the virtual machine plane <NUM> through the hypervisor plane <NUM>. In one example, the virtual machines <NUM> utilization of the hardware resources of host devices the hardware plane <NUM> is controlled by the hypervisor plane <NUM>, and the virtual machines <NUM> may not directly access the host devices <NUM> themselves. The virtual machines <NUM> of the virtual machine plane <NUM> provide a virtual computing environment within which users of the cloud platform <NUM> may execute cloud applications. During execution of a cloud application, the hypervisor plane <NUM> may allocate hardware resources of one or more host devices <NUM> of the hardware plane <NUM> to run the cloud application. The hypervisor plane <NUM> may allocate the hardware resources of the host devices <NUM> in a changeable and scalable manner, such that additional host devices <NUM> may be allocated to a particular virtual machine <NUM>, and already allocated host devices <NUM> may be reduced, transferred, or otherwise changed for that particular virtual machine <NUM> while the cloud application is running.

It should be appreciated that the cloud platform <NUM> infrastructure described above and illustrated in <FIG> and <FIG> are merely exemplary, and that other networking infrastructures and organization methods not specifically described herein may also be utilized.

<FIG> shows an example host architecture <NUM> for enabling both user space network virtualization and RDMA. The example host architecture <NUM> may be implemented for the host devices <NUM> of the computer system <NUM> of <FIG>. However, it should be appreciated that the example host architecture <NUM> may be implemented for any suitable computer system. For example, the host architecture <NUM> shown in <FIG> and described herein may be implemented in another example computer system that includes at least one host device <NUM> including at least one processor <NUM>. In some examples, the host device <NUM> is one of a plurality of host devices <NUM>. The computer system may further include a network, such as the network infrastructure <NUM>, that connects the plurality of host devices <NUM> via a respective plurality of physical NICs <NUM> of each of the host devices <NUM>. Each of the plurality of host devices <NUM> may include at least one processor <NUM>, at least one memory device <NUM>, and at least one physical NIC <NUM>, as shown in the example of <FIG>.

Each of the plurality of host devices <NUM> are configured to execute a respective host OS <NUM>. Each host OS <NUM> allocates a portion of system memory from a respective memory device <NUM> to a host user space <NUM> for guest applications <NUM> and a portion of system memory to a host OS space <NUM> for a kernel of the host OS <NUM>. The host user space <NUM> is allocated for execution of programs, such as the guest applications <NUM>, by authorized and authenticated users of the computer system <NUM>. The host user space <NUM> may include the code that runs outside the operating system's kernel, and may include various programs and libraries that the host OS uses to interact with the kernel, such as, for example, software for input/output (IO), software for manipulating file system objects, etc..

The host OS space <NUM>, which may also be referred to as the host kernel space, is allocated for a kernel of the host OS <NUM> for execution of threads by OS processes. The host OS space <NUM> is separate from the host user space and excludes application space where application software is typically executed. The kernel code of the host OS <NUM> may, for example, be executed under central processing unit (CPU) Protection Ring <NUM> in the host OS space <NUM>, and may have access to all of the machine's instructions and system memory. In contrast, the programs and applications run in the host user space <NUM> may be executed under, for example, CPU Protection Ring <NUM>, which limits access to system resources. Programs in the host user space <NUM> may access system resources using a set of API calls, also referred to as the system calls, that are sent to the kernel to request memory and physical hardware access. It should be appreciated that other types of memory and hardware protection architectures may be implemented to separate the applications running in host user space <NUM> and the OS processes run in host OS space <NUM>.

The physical NIC <NUM> of the host device <NUM> may include a NIC switch <NUM>, which is a physical embedded layer for performing switching on the physical NIC. The NIC switch <NUM> may provide functionality to create virtual ports that connect to virtual cables that are mapped to a virtual NIC, such as, for example, the host kernel space virtual NIC (vNiC) <NUM> that will be described in more detail below. These vNICs, such as the host kernel space vNIC <NUM>, may be assigned a destination MAC address, and may operate in a similar manner to a physical network counterpart. The software programs and applications of the physical NIC <NUM>, and other hardware components may operate in a physical address space <NUM>, as shown in <FIG>. Each physical NIC <NUM> may transmit communications over a network fabric <NUM> of the computer system <NUM> to interact with other physical NICs <NUM> and hardware components across the computer system <NUM>.

As discussed above, moving toward user space network virtualization may provide the potential advantage of enabling high-performance, secure, and reliable packet processing. As shown in <FIG>, to achieve user space network virtualization, a virtual switch <NUM> is implemented in the host user space <NUM>. The virtual switch <NUM> may provide functionality for enabling virtual machines, such as the virtual machines <NUM> of the virtual machine plane <NUM>, to access the capabilities of the physical NICs <NUM> to communicate with other virtual machines <NUM>, applications, and other types of software using Ethernet protocols. The virtual switch <NUM> may communicate with the physical NIC <NUM>, use the physical NIC <NUM> as an uplink, and may create virtual NICs that logically lie on top of the physical NIC <NUM> to provide an abstraction layer for downstream software. The virtual switch <NUM> may handle handshakes, exchanges, configurations, and function pointer tables for communication for the Ethernet traffic flowing through the host user space.

In conventional implementations, RDMA protocols would typically use a kernel space switch to configure associated vNICs, handshake, get function pointer tables for communication, etc. In the example shown in <FIG>, rather than a kernel space switch, the server architecture uses user space virtualization that includes the virtual switch <NUM> implemented in the host user space. However, the RDMA protocols are kernel space modules that typically must stay within the host OS space to achieve suitable data security and protection requirements. Thus, the RDMA protocols are unable to use the virtual switch <NUM> that is implemented in the host user space for configure associated vNICs, handshaking, and other capabilities needed to perform RDMA.

As discussed above, conventional user space network virtualization implementations focus on servicing virtual machines and guests, and do not encompass storage protocols, such as RDMA. Thus, these conventional implementations are unable to provide functionality for performing both Ethernet traffic that flows through the host user space as well as RDMA accesses that flows through the host OS space.

To address this issue, the example host architecture <NUM> shown in <FIG> includes a teamed NIC software program <NUM> that provides functionality for enabling a user space Ethernet packet processing data pathway while simultaneously allowing for kernel space consumption of RDMA protocols. The teamed NIC software program <NUM> is configured to provide a unified interface to host OS space <NUM> upper layer protocols. These upper layer protocols include at least an RDMA protocol <NUM> and an Ethernet protocol <NUM>. To provide the unified interface, the teamed NIC software program <NUM> provides multiplexing for at least two data pathways including an RDMA data pathway <NUM> and an Ethernet data pathway <NUM>. As shown in <FIG>, RDMA data pathway <NUM> transmits communications to and from an RDMA interface of a physical NIC <NUM> through the host OS space <NUM>. On the other hand, the Ethernet data pathway transmits communications to and from an Ethernet interface of the physical NIC <NUM> through the virtual switch <NUM> that is implemented in the host user space <NUM> and a host kernel space virtual NIC <NUM> that is implemented in the host OS space <NUM>. The RDMA interface and Ethernet interface described herein refers to the drivers, adapters, and other software and hardware constructs of the physical NIC <NUM>. These two data pathways will be described in more detail below.

In one example, the unified interface provided to the host OS space upper layer protocols by the teamed NIC software program <NUM> includes a single unified media access control (MAC) address and internet protocol (IP) address that is used to transmit data with the physical NIC <NUM>. That is, the kernel space applications running the host OS space <NUM> managing the RDMA protocol <NUM> and the Ethernet protocol <NUM> may use the same MAC address and IP address when transmitting data for both RDMA and Ethernet traffic. Further, the teamed NIC software program <NUM> aggregates data traffic from both the RDMA data pathway <NUM> and the Ethernet data pathway <NUM>, and provides the aggregated traffic to the host OS space upper layer protocols through the unified interface. In this manner, the aggregated traffic appears to the upper layer protocols to be originating from a same device. The upper layer protocols, such as the RDMA protocol <NUM> and the Ethernet protocol <NUM> are unaware that the data from the RDMA data pathway <NUM> and the data from the Ethernet data pathway <NUM> are being aggregated, and only sees that the data is being transmitted using the same MAC address and IP address of the unified interface presented by the teamed NIC software program <NUM>. Thus, these upper layer protocols are unaware that the virtual switch <NUM> is not co-located with the TCP/IP stack, and are unaware that the Ethernet traffic is being routed through the host user space <NUM>. This data transport architecture provides the benefit of enabling a user space Ethernet packet processing data pathway while simultaneously allowing for kernel space consumption of RDMA protocols.

Turning to <FIG>, the Ethernet data pathway <NUM> flows through the host user space <NUM> via several components. To provide access to the physical NIC <NUM>, a software interface <NUM> for the NIC switch <NUM> of the physical NIC <NUM> is run in the host user space <NUM>. The software interface <NUM> implements queues and resources <NUM> for transmitting data to and from the physical NIC <NUM>. The software interface <NUM> may includes multiple transmit and receive queues <NUM>, allotting packets received by the physical NIC <NUM> to be assigned to one of the queues. The queues and resources <NUM> may also include a hardware poll mode driver associated with the physical NIC <NUM>, and multiple uplink virtual ports <NUM> that are used to route traffic to corresponding downlink virtual ports <NUM>. It should be appreciated that the software interface <NUM> for the physical NIC <NUM> may implement other software constructs not specifically described herein.

As illustrated in <FIG>, the virtual switch <NUM> transmits data packets to and from the NIC switch <NUM> using the software interface <NUM> implemented in the host user space <NUM>. For example, the virtual switch <NUM> may be communicatively coupled to the uplink virtual ports <NUM> of the software interface <NUM>. The virtual switch <NUM> may provide capabilities handling handshakes, exchanges, and configurations for the data traffic. Further, the virtual switch <NUM> may be configured to create and manage virtual NICs that will logically lie on top of the physical NIC <NUM>, and manages the function pointer tables for communication using those virtual NICs. Communication using these virtual NICs may be routed using the uplink virtual ports <NUM> and downlink virtual ports <NUM>. These virtual NICs may be used to route traffic between virtual machines <NUM> running in the virtual machine plane <NUM>. Additionally, these virtual NICs managed by the virtual switch <NUM> include the host kernel space virtual NIC <NUM> that lies under the teamed NIC software program <NUM>.

In this manner, Ethernet traffic may be routed to the host kernel space virtual NIC <NUM> through the host user space <NUM> via the uplink virtual port <NUM> and downlink virtual port <NUM> associated with the host kernel space virtual NIC <NUM>. As illustrated in <FIG>, the virtual switch <NUM> may be configured to transmit data packets to and from the virtual NIC <NUM> implemented in the host OS space <NUM> using a shared memory communication between the host user space <NUM> and the user OS space <NUM>. The shared memory communication between the host user space <NUM> and the host OS space <NUM> may be controlled by a poll mode driver (PMD) <NUM> implemented in the host user space <NUM>. The PMD <NUM> may use shared memory queues to transmit and receive data with the host kernel space virtual NIC <NUM> associated with the downlink virtual port <NUM>.

Using the software and hardware components described above, the Ethernet data pathway <NUM> includes steps (<NUM>)-(<NUM>). At (<NUM>), transmitting communications between the Ethernet interface of the physical NIC <NUM> and a software interface <NUM> implemented in the host user space <NUM> for the physical NIC <NUM>. At (<NUM>), transmitting communications between the software interface <NUM> of the physical NIC <NUM> and the virtual switch <NUM> implemented in the host user space <NUM>. At (<NUM>) transmitting communications between the virtual switch <NUM> and the PMD <NUM> configured for shared memory communication between the host user space <NUM> and the host OS space <NUM>. At (<NUM>) transmitting communications between the PMD <NUM> and the virtual NIC <NUM> implemented in the host OS space <NUM>. At (<NUM>) transmitting communications between the virtual NIC <NUM> and the teamed NIC software program <NUM>. A network driver interface specification (NDIS) driver <NUM> may be used to logically link the teamed NIC software program <NUM> with the upper layer Ethernet protocol. The NDIS driver <NUM> specifies a standard interface between layered network drivers, thereby abstracting lower-level drivers that manage hardware from upper level drivers, such as network transports.

Turning to <FIG>, the RDMA data pathway <NUM> flows through the host OS space <NUM>, and does not flow through the host user space <NUM> like the Ethernet data pathway <NUM>. As discussed previously, the RDMA protocol <NUM> typically goes through a host OS space virtual switch for handshaking and getting function pointer tables for communication. However, in this example, the virtual switch <NUM> resides in the host user space <NUM>. Thus, instead of using the virtual switch <NUM>, the RDMA protocol <NUM> will access RDMA capabilities of the physical NIC <NUM> through the teamed NIC software program <NUM>. The teamed NIC software program <NUM> may be configured to access an RDMA interface of the physical NIC <NUM> through direct access to a virtual port <NUM> of the physical NIC <NUM> using a hardware driver <NUM>. The hardware driver <NUM> of the physical NIC <NUM> may provide functionality for managing the physical NIC <NUM>, including sending and receiving data through the physical NIC. The hardware driver <NUM> may also provide functionality for interfacing with higher-level drivers, such as the RDMA protocol <NUM>. Through the virtual port <NUM> and the hardware driver <NUM> of the physical NIC <NUM>, the teamed NIC software program <NUM> may use a MAC and IP address assignment for the RDMA interface of the physical NIC <NUM> for the RDMA data pathway <NUM>. Using the RDMA data pathway <NUM>, the RDMA protocol <NUM> may be used to directly read or write to a memory device without being processed by a host OS of the computer system <NUM>.

The hardware driver <NUM> of the physical NIC <NUM> may implement an RDMA network adapter <NUM> to provide the RDMA functionality of the physical NIC <NUM>. As a specific example, the RDMA network adapter <NUM> may take the form of a Network Direct Kernel Provider Interface (NDKPI) that provides an interface for kernel-mode RDMA support. As illustrated in <FIG>, the RDMA network adapter <NUM> may be used by the teamed NIC software program <NUM> to expose the RDMA functionality of the physical NIC <NUM>.

As shown in <FIG>, the RDMA data pathway <NUM> may include, at (<NUM>), accessing the RDMA interface of the physical NIC <NUM> through using the hardware driver <NUM>, and at (<NUM>), transmitting communications between the RDMA interface of the physical NIC <NUM> and the teamed NIC software program <NUM>. The teamed NIC software program <NUM> may aggregate the traffic from the Ethernet data pathway <NUM> with any RDMA processes for the RDMA pathway <NUM>, and present a unified interface to the upper layer protocols such as the RDMA protocol <NUM> and the Ethernet protocol <NUM>. The RDMA protocol <NUM> and the Ethernet protocol <NUM> are peer protocols expecting to communicate with the same device. These upper layer protocols may in turn communicate with an agent in the host user space <NUM>, such as-, for example, a guest application being run in the host user space <NUM>. The upper layer protocols may service the guest applications through specific OS requests and API calls. <FIG> shows a flowchart for an example method <NUM> for enabling both user space network virtualization and RDMA processes. The method <NUM> may be implemented by each host device <NUM> of the computer system <NUM> shown in <FIG>, or by another example computer system for hosting a cloud platform.

At <NUM>, the method <NUM> may include executing a teamed network interface card (NIC) software program in a host operating system (OS) space. Executing the teamed NIC software program may include providing a unified interface to host OS space upper layer protocols including at least a remote direct memory access (RDMA) protocol and an Ethernet protocol. The RDMA protocol and the Ethernet protocol are peer protocols that expect to lie above the same device. The unified interface provided to the host OS space upper layer protocols by the teamed NIC software program may include a single unified media access control (MAC) address and internet protocol (IP) address that is used to transmit data with the physical NIC. Thus, from the perspective of the upper layer protocols, a single device is sending and receiving data traffic for both RDMA processes and Ethernet traffic.

At <NUM>, the method <NUM> may include multiplexing for at least two data pathways using the teamed NIC software program. The two data pathways include at least an RDMA data pathway and an ethernet data pathway. The RDMA data pathway is implement in the host OS space. The Ethernet data pathway flows through the host user space.

The RDMA data pathway includes steps <NUM> and <NUM> of the method <NUM>. At <NUM>, the method <NUM> may include accessing an RDMA interface of a physical NIC through using a hardware driver. Step <NUM> may correspond to step (<NUM>) shown in <FIG>. At <NUM>, the method <NUM> may include transmitting communications between the RDMA interface of the physical NIC and the teamed NIC software program. Step <NUM> may correspond to step (<NUM>) shown in <FIG>.

The Ethernet data pathway includes steps <NUM>-<NUM>. At <NUM>, the method <NUM> may include transmitting communications between an Ethernet interface of the physical NIC and a virtual switch implemented in a host user space. In some examples, step <NUM> may include sub-steps corresponding to steps (<NUM>) and (<NUM>) shown in <FIG>, and may include transmitting communications between the Ethernet interface of the physical NIC and a software interface implemented in the host user space for the physical NIC, and transmitting communications between the software interface of the physical NIC and the virtual switch implemented in the host user space.

At <NUM>, the method <NUM> may include transmitting communications between the virtual switch and a virtual NIC implemented in the host OS space. In some examples, step <NUM> may include sub-steps corresponding to steps (<NUM>) and (<NUM>) shown in <FIG>, and may include transmitting communications between the virtual switch and a poll mode driver configured for shared memory communication between the host user space and the host OS space, and transmitting communications between the poll mode driver and the virtual NIC implemented in the host OS space.

At <NUM>, the method <NUM> may include transmitting communications between the virtual NIC and the teamed NIC software program, which may correspond to step (<NUM>) shown in <FIG>.

At <NUM>, the method <NUM> may include aggregating data traffic from both the RDMA data pathway and the Ethernet data pathway.

At <NUM>, the method <NUM> may include providing the aggregated traffic to the host OS space upper layer protocols through the unified interface such that the aggregated traffic appears to be originating from a same device.

In this manner, the method and systems described above may provide the potential benefit of enabling both user space network virtualization and RDMA processes that does not need to co-locate the Ethernet device with the TCP/IP stack, thus allowing for a user space Ethernet packet processing data path while simultaneously allowing for OS space consumption of RDMA protocols. By providing both of these data paths, the method and systems described above may achieve the potential benefits of user space network virtualization of high-performance, secure, and reliable packet processing for modern datacenter workloads, while simultaneously enabling RDMA which provides the potential benefits of low latency transfer of information between compute nodes at the memory-to-memory level, without burdening the CPU.

Computing system <NUM> may embody one or more of the host devices <NUM> of the computer system <NUM> described above and illustrated in <FIG>. Computing system <NUM> may take the form of one or more personal computers, server computers, tablet computers, home-entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phone), and/or other computing devices, and wearable computing devices such as smart wristwatches and head mounted augmented reality devices.

Processors of the logic processor <NUM> may be single-core or multicore, and the instructions executed thereon may be configured for sequential, parallel, and/or distributed processing.

The following paragraphs provide additional support for the claims of the subject application. One aspect provides a computer system comprising at least one host device comprising at least one processor. The at least one processor is configured to implement, in a host operating system (OS) space, a teamed network interface card (NIC) software program that provides a unified interface to host OS space upper layer protocols including at least a remote direct memory access (RDMA) protocol and an Ethernet protocol. The teamed NIC software program provides multiplexing for at least two data pathways including an RDMA data pathway that transmits communications to and from an RDMA interface of a physical NIC. The RDMA data pathway being within the host OS space. The at least two data pathways include an Ethernet data pathway that transmits communications to and from an Ethernet interface of the physical NIC through a virtual switch that is implemented in a host user space and a virtual NIC that is implemented in the host OS space.

In this aspect, additionally or alternatively, the at least one processor may be configured to implement, in a host user space, a software interface for a NIC switch of the physical NIC, and the virtual switch that transmits data packets to and from the NIC switch using the software interface implemented in the host user space. In this aspect, additionally or alternatively, the software interface implemented in host user space for the NIC switch of the physical NIC may include queues and resources for transmitting data to and from the physical NIC. In this aspect, additionally or alternatively, the virtual switch may transmit data packets to and from the virtual NIC implemented in the host OS space using a shared memory communication between the host user space and the user OS space. In this aspect, additionally or alternatively, the shared memory communication between the host user space and the host OS space may be controlled by a poll mode driver implemented in the host user space. In this aspect, additionally or alternatively, the Ethernet data pathway may transmit communications between the Ethernet interface of the physical NIC and the teamed NIC software program through the software interface for the NIC switch of the physical NIC, the virtual switch implemented in the host user space, the shared memory communication between the host user space and the host OS space, and the virtual NIC that is implemented in the host OS space.

In this aspect, additionally or alternatively, the RDMA data pathway may use a media access control (MAC) address and an internet protocol (IP) address assignment for the RDMA interface of the physical NIC. In this aspect, additionally or alternatively, the teamed NIC software program may access the RDMA interface of the physical NIC through direct access to a virtual port of the physical NIC using a hardware driver. In this aspect, additionally or alternatively, the RDMA protocol may be used to directly read or write to a memory device without being processed by a host OS of the computer system. In this aspect, additionally or alternatively, the unified interface provided to the host OS space upper layer protocols by the teamed NIC software program may include a single unified media access control (MAC) address and internet protocol (IP) address that is used to transmit data with the physical NIC. In this aspect, additionally or alternatively, the teamed NIC software program may aggregate data traffic from both the RDMA data pathway and the Ethernet data pathway, and may provide the aggregated traffic to the host OS space upper layer protocols through the unified interface such that the aggregated traffic appears to be originating from a same device.

In this aspect, additionally or alternatively, the host device may be one of a plurality of host devices, and0 the computer system may further comprise a network that connects the plurality of host devices via a respective plurality of physical NICs of each of the host devices. Each of the plurality of host devices may include at least one processor, at least one memory device, and at least one physical NIC. Each of the plurality of host devices may be configured to execute a respective host operating system that allocates a portion of system memory from a respective memory device to a host user space for guest applications and a portion of system memory to a host OS space for a kernel of the host operating system. In this aspect, additionally or alternatively, the host user space may be allocated for execution of programs by authorized and authenticated users of the computer system, and the host OS space may be allocated for a kernel of the host OS for execution of threads by OS processes.

Another aspect provides a method comprising, at a processor of a host device, executing a teamed network interface card (NIC) software program in a host operating system (OS) space, wherein the teamed NIC software program includes providing a unified interface to host OS space upper layer protocols including at least a remote direct memory access (RDMA) protocol and an Ethernet protocol. The method may further comprise multiplexing for at least two data pathways using the teamed NIC software program. The at least two data pathways include an RDMA data pathway implemented in the host OS space that includes accessing an RDMA interface of a physical NIC through using a hardware driver, and transmitting communications between the RDMA interface of the physical NIC and the teamed NIC software program. The at least two data pathways include an Ethernet data pathway that includes transmitting communications between an Ethernet interface of the physical NIC and a virtual switch implemented in a host user space, transmitting communications between the virtual switch and a virtual NIC implemented in the host OS space, and transmitting communications between the virtual NIC and the teamed NIC software program.

In this aspect, additionally or alternatively, transmitting communications between the Ethernet interface of the physical NIC and the virtual switch implemented in a host user space may further include transmitting communications between the Ethernet interface of the physical NIC and a software interface implemented in the host user space for the physical NIC, and transmitting communications between the software interface of the physical NIC and the virtual switch implemented in the host user space. In this aspect, additionally or alternatively, transmitting communications between the virtual switch and the virtual NIC implemented in the host OS space may further include transmitting communications between the virtual switch and a poll mode driver configured for shared memory communication between the host user space and the host OS space, and transmitting communications between the poll mode driver and the virtual NIC implemented in the host OS space.

In this aspect, additionally or alternatively, the unified interface provided to the host OS space upper layer protocols by the teamed NIC software program may include a single unified media access control (MAC) address and internet protocol (IP) address that is used to transmit data with the physical NIC. In this aspect, additionally or alternatively, executing the teamed NIC software program may include aggregating data traffic from both the RDMA data pathway and the Ethernet data pathway, and providing the aggregated traffic to the host OS space upper layer protocols through the unified interface such that the aggregated traffic appears to be originating from a same device. In this aspect, additionally or alternatively, transmitting communications between the RDMA interface of the physical NIC and the teamed NIC software program may include directly reading or writing to a memory device without being processed by a host OS.

Another aspect provides a computer system comprising at least one host device comprising at least one processor and at least one physical network interface card (NIC), the at least one processor being configured to execute a virtual switch in a host user space, the virtual switch being configured to transmit data packets to and from the physical NIC through the user space, execute a teamed NIC software program in a host operating system (OS) space, the teaming NIC software program being configured to provide a unified interface to host OS space upper layer protocols including at least a remote direct memory access (RDMA) protocol and an Ethernet protocol, transmit communications to and from an RDMA interface of the physical NIC and the teamed NIC software program using an RDMA data pathway that flows through the host OS space, transmit communications to and from an Ethernet interface of the physical NIC through the virtual switch that is implemented in a host user space and a virtual NIC that is implemented in the host OS space using an Ethernet data pathway that flows through the host user space, aggregate data traffic for at least the RDMA data pathway and the Ethernet data pathway, and provide the aggregated data traffic to the RDMA protocol and the Ethernet protocol using the unified interface.

Claim 1:
A host device (<NUM>) comprising at least one processor (<NUM>) configured to implement:
in a host operating system, OS, space (<NUM>);
a teamed network interface card, NIC, software program (<NUM>) that provides a unified interface to host OS space upper layer protocols including at least a remote direct memory access, RDMA, protocol (<NUM>) and an Ethernet protocol (<NUM>), the teamed NIC software program (<NUM>) provides multiplexing for at least two data pathways including:
an RDMA data pathway (<NUM>) that transmits communications to and from an RDMA interface of a physical NIC (<NUM>), the RDMA data pathway (<NUM>) being within the host OS space (<NUM>); and
an Ethernet data pathway (<NUM>) that transmits communications to and from an Ethernet interface of the physical NIC (<NUM>) through a virtual switch (<NUM>) that is implemented in a host user space (<NUM>) and a virtual NIC (<NUM>) that is implemented in the host OS space (<NUM>),
wherein the unified interface provided to the host OS space upper layer protocols by the teamed NIC software program includes a single unified media access control, MAC, address and internet protocol, IP, address that is used to transmit data with the physical NIC, and
wherein the teamed NIC software program aggregates data traffic from both the RDMA data pathway and the Ethernet data pathway, and provides the aggregated traffic to the host OS space upper layer protocols through the unified interface such that the aggregated traffic appears to be originating from a same device.