Scalable multi-tenant underlay network supporting multi-tenant overlay network

Techniques are disclosed for scalable virtualization of tenants and subtenants on a virtualized computing infrastructure. In one example, a first controller for the virtualized computing infrastructure configures underlay network segments in the virtualized computing infrastructure by configuring respective Virtual Extensible Local Area Network (VXLAN) segments of a plurality of VXLAN segments of a VXLAN in a switch fabric comprising network switches. Each VXLAN segment provides underlay network connectivity among a different subset of host computing devices of the virtualized computing infrastructure to enable orchestration of multiple tenants in the VXLAN. A second controller for a first subset of the host computing devices has underlay network connectivity through operation of a first VXLAN segment. The second controller configures overlay networks in the first subset of the host computing devices to enable orchestration of multiple subtenants in the first subset of the host computing devices.

TECHNICAL FIELD

The disclosure relates to virtualized computing infrastructures.

BACKGROUND

Virtualized data centers are becoming a core foundation of the modern information technology (IT) infrastructure. In particular, modern data centers have extensively utilized virtualized environments in which virtual hosts, such virtual machines or containers, are deployed and executed on an underlying compute platform of physical computing devices.

Virtualization within a data center can provide several advantages. One advantage is that virtualization can provide significant improvements to efficiency. As the underlying physical computing devices (i.e., servers) have become increasingly powerful with the advent of multicore microprocessor architectures with a large number of cores per physical CPU, virtualization becomes easier and more efficient. A second advantage is that virtualization provides significant control over the computing infrastructure. As physical computing resources become fungible resources, such as in a cloud-based computing environment, provisioning and management of the compute infrastructure becomes easier. Thus, enterprise IT staff often prefer virtualized compute clusters in data centers for their management advantages in addition to the efficiency and increased return on investment (ROI) that virtualization provides.

A computing infrastructure that manages deployment and infrastructure for application execution may involve two main roles: (1) orchestration—for automating deployment, scaling, and operations of applications across clusters of hosts and providing computing infrastructure, which may include container-centric computing infrastructure; and (2) network management—for creating virtual networks in the network infrastructure to enable communication among applications running on virtualized computing environments, such as containers or VMs, as well as among applications running on legacy (e.g., physical) environments. Software-defined networking contributes to network management.

SUMMARY

In general, techniques are described for scalable virtualization of a plurality of tenant underlay networks and a plurality of subtenant overlay networks executed by a virtualized computing infrastructure so as to enable multi-tenancy overlay networks supported by multi-tenancy underlay networks. For example, an underlay controller for the virtualized computing infrastructure configures a Virtual Extensible Local Area Network (VXLAN) on a switch fabric of network switches. The underlay controller further configures a plurality of underlay network segments, each underlay network segment configured as a VXLAN segment of a plurality of VXLAN segments of the VXLAN. Each VXLAN segment of the plurality of VXLAN segments provides underlay network connectivity among a different subset of host computing devices of the virtualized computing infrastructure. The underlay controller assigns a different tenant of a plurality of tenants to each VXLAN segment of the plurality of VXLAN segments such that a different subset of host computing devices supports the VXLAN segment to which the tenant is assigned. Thus, the underlay controller may enable orchestration of multiple tenants in the VXLAN, each tenant having hardware within the VXLAN that is isolated from each other tenant.

Furthermore, because the underlay segments provide effective network isolation, a different overlay controller may control each of the subsets of the host computing devices of the virtualized computing infrastructure. With respect to a first subset of the host computing devices having underlay network connectivity by operation of a first VXLAN segment of the VXLAN segments, an overlay controller configures, e.g., a plurality of overlay networks on the first subset of the host computing devices. Thus, the overlay controller for the first subset of the host computing devices may enable orchestration of multiple subtenants in the first subset of the host computing devices supporting the underlay segment, each subtenant having a private network that is isolated from each other subtenant of the first subset of host computing devices.

The techniques of the disclosure may provide specific improvements to the computer-related field of virtualized computing infrastructure. Furthermore, the techniques of the disclosure may be integrated into numerous practical applications. For example, the techniques of the disclosure may allow for the robust and efficient scaling of tenants in an underlay network and subtenants in a plurality of overlay networks, thereby allowing for a much greater number of tenants and subtenants than supported by the use of VXLAN or VLAN alone. For example, the techniques of the disclosure may allow for the segmentation of each tenant of the underlay network to provide a private underlay environment or isolated hardware environment to each tenant. Additionally, the techniques of the disclosure may reduce the complexity of configuring overlay and underlay networks across the virtualized computing infrastructure and avoid the use of the inefficient spanning tree protocol to flood broadcast, unknown, and multicast (BUM) traffic. The techniques of the disclosure also allow for simplified management of access to isolated overlay networks from external networks as well as providing private access to hardware administration interfaces of servers (e.g., Intelligent Platform Management Interface (IPMI)). Furthermore, the techniques of the disclosure may enable the use of isolated environments for tenants and subtenants that allows for increased agility in the development and maintenance of application services within the virtualized computing infrastructure, as well as enabling service providers to offer improved white label services to other service providers and/or offer Bare Metal Server as a Service (BMSaaS) with agility and flexibility that is not currently possible with conventional techniques.

In one example, this disclosure describes a method comprising: configuring, by a first controller for a virtualized computing infrastructure, a plurality of underlay network segments in the virtualized computing infrastructure by configuring respective Virtual Extensible Local Area Network (VXLAN) segments of a plurality of VXLAN segments of a VXLAN in a switch fabric comprising network switches for the virtualized computing infrastructure to enable orchestration of multiple tenants in the VXLAN, wherein each VXLAN segment of the plurality of VXLAN segments provides underlay network connectivity among a different subset of host computing devices of the virtualized computing infrastructure; and configuring, by a second controller for a first subset of the host computing devices having underlay network connectivity by operation of a first VXLAN segment of the VXLAN segments, a plurality of overlay networks in the first subset of the host computing devices to enable orchestration of multiple subtenants in the first subset of the host computing devices.

In another example, this disclosure describes a system comprising: a first controller for a virtualized computing infrastructure, the first controller executing on processing circuitry and configured to configure a plurality of underlay network segments in the virtualized computing infrastructure by configuring respective Virtual Extensible Local Area Network (VXLAN) segments of a plurality of VXLAN segments of a VXLAN in a switch fabric comprising network switches for the virtualized computing infrastructure to enable orchestration of multiple tenants in the VXLAN, wherein each VXLAN segment of the plurality of VXLAN segments provides underlay network connectivity among a different subset of host computing devices of the virtualized computing infrastructure; and a second controller for a first subset of the host computing devices having underlay network connectivity by operation of a first VXLAN segment of the VXLAN segments, the second controller configured to configure a plurality of overlay networks in the first subset of the host computing devices to enable orchestration of multiple tenants in the first subset of the host computing devices.

In another example, this disclosure describes a non-transitory, computer-readable medium comprising instructions that, when executed, cause processing circuitry to execute: a first controller for a virtualized computing infrastructure, the first controller configured to configure a plurality of underlay network segments in the virtualized computing infrastructure by configuring respective Virtual Extensible Local Area Network (VXLAN) segments of a plurality of VXLAN segments of a VXLAN in a switch fabric comprising network switches for the virtualized computing infrastructure to enable orchestration of multiple tenants in the VXLAN, wherein each VXLAN segment of the plurality of VXLAN segments provides underlay network connectivity among a different subset of host computing devices of the virtualized computing infrastructure; and a second controller for a first subset of the host computing devices having underlay network connectivity by operation of a first VXLAN segment of the VXLAN segments, the second controller configured to configure a plurality of overlay networks in the first subset of the host computing devices to enable orchestration of multiple tenants in the first subset of the host computing devices.

DETAILED DESCRIPTION

FIG. 1is a block diagram illustrating an example computing infrastructure8in which examples of the techniques described herein may be implemented. In general, data center10provides an operating environment for applications and services for a customer sites11(illustrated as “customers11”) having one or more customer networks coupled to the data center by service provider network7. Data center10may, for example, host infrastructure equipment, such as networking and storage systems, redundant power supplies, and environmental controls. Service provider network7is coupled to public network15, which may represent one or more networks administered by other providers, and may thus form part of a large-scale public network infrastructure, e.g., the Internet. Public network15may represent, for instance, a local area network (LAN), a wide area network (WAN), the Internet, a virtual LAN (VLAN), an enterprise LAN, a layer 3 virtual private network (VPN), an Internet Protocol (IP) intranet operated by the service provider that operates service provider network7, an enterprise IP network, or some combination thereof.

Although customer sites11and public network15are illustrated and described primarily as edge networks of service provider network7, in some examples, one or more of customer sites11and public network15may be tenant networks within data center10or another data center. For example, data center10may host multiple tenants (customers) each associated with one or more virtual private networks (VPNs), each of which may implement one of customer sites11. Service provider network7offers packet-based connectivity to attached customer sites11, data center10, and public network15. Service provider network7may represent a network that is owned and operated by a service provider to interconnect a plurality of networks. Service provider network7may implement Multi-Protocol Label Switching (MPLS) forwarding and in such instances may be referred to as an MPLS network or MPLS backbone. In some instances, service provider network7represents a plurality of interconnected autonomous systems, such as the Internet, that offers services from one or more service providers.

In some examples, data center10may represent one of many geographically distributed network data centers. As illustrated in the example ofFIG. 1, data center10may be a facility that provides network services for customers. A customer of the service provider may be a collective entity such as enterprises and governments or individuals. For example, a network data center may host web services for several enterprises and end users. Other exemplary services may include data storage, virtual private networks, traffic engineering, file service, data mining, scientific- or super-computing, and so on. Although illustrated as a separate edge network of service provider network7, elements of data center10such as one or more physical network functions (PNFs) or virtualized network functions (VNFs) may be included within the service provider network7core.

In this example, data center10includes storage and/or compute servers interconnected via switch fabric14provided by one or more tiers of physical network switches and routers, with servers12A-12N (herein, “servers12”) depicted as coupled to top-of-rack switches16A-16N. Servers12are computing devices and may also be referred to herein as “hosts” or “host devices.”

Switch fabric14in the illustrated example includes interconnected top-of-rack (TOR) (or other “leaf”) switches16A-16N (collectively, “TOR switches16”) coupled to a distribution layer of chassis (or “spine” or “core”) switches18A-18N (collectively, “chassis switches18”). Although not shown, data center10may also include, for example, one or more non-edge switches, routers, hubs, gateways, security devices such as firewalls, intrusion detection, and/or intrusion prevention devices, servers, computer terminals, laptops, printers, databases, wireless mobile devices such as cellular phones or personal digital assistants, wireless access points, bridges, cable modems, application accelerators, or other network devices. Data center10may also include one or more physical network functions (PNFs) such as physical firewalls, load balancers, routers, route reflectors, broadband network gateways (BNGs), Evolved Packet Cores or other cellular network elements, and other PNFs.

In this example, TOR switches16and chassis switches18provide servers12with redundant (multi-homed) connectivity to IP fabric20and service provider network7. Chassis switches18aggregate traffic flows and provides connectivity between TOR switches16. TOR switches16may be network devices that provide layer 2 (MAC) and/or layer 3 (e.g., IP) routing and/or switching functionality. TOR switches16and chassis switches18may each include one or more processors and a memory and can execute one or more software processes. Chassis switches18are coupled to IP fabric20, which may perform layer 3 routing to route network traffic between data center10and customer sites11by service provider network7. The switching architecture of data center10is merely an example. Other switching architectures may have more or fewer switching layers, for instance.

In the example ofFIG. 1, IP fabric20may utilize Ethernet VPN (E-VPN) technology to provide an EVPN data center interconnect (DCI) that allows data center10to perform multi-tenancy of a plurality of tenants within data center10. An EVPN connects dispersed customer sites using a Layer 2 virtual bridge. As compared with other types of Layer 2 VPNs, an EVPN consists of customer edge (CE) devices, such as hosts, routers, or switches, such as switches16,18, connected to network access devices (not depicted) of IP fabric20. The network access devices of IP fabric20may include an MPLS edge switch (MES) that acts at the edge of the MPLS infrastructure. In another example, a standalone switch can be configured to act as the MES. Multiple EVPNs may be deployed within a service provider network, such as network system2ofFIG. 1, each providing network connectivity to a customer while ensuring that the traffic sharing on that network remains private. An EVPN may define multiple types of routes, such as, e.g., Ethernet AD routes, MAC/IP advertisement routes, and Ethernet Segment routes. In this way, IP fabric20provides EVPN23to transport L2 communications for customer networks while maintaining virtual isolation of the customer networks. In particular, IP fabric20enables EVPN23to transport L2 communications, such as Ethernet packets or “frames,” through service provider network7for different customers of data center10.

The term “packet flow,” “traffic flow,” or simply “flow” refers to a set of packets originating from a particular source device or endpoint and sent to a particular destination device or endpoint. A single flow of packets may be identified by the 5-tuple: <source network address, destination network address, source port, destination port, protocol>, for example. This 5-tuple generally identifies a packet flow to which a received packet corresponds. An n-tuple refers to any n items drawn from the 5-tuple. For example, a 2-tuple for a packet may refer to the combination of <source network address, destination network address> or <source network address, source port> for the packet.

Servers12may each represent a compute server, switch, or storage server. For example, each of servers12may represent a computing device, such as an x86 processor-based server, configured to operate according to techniques described herein. Servers12may provide Network Function Virtualization Infrastructure (NFVI) for an NFV architecture.

Any server of servers12may be configured with virtual execution elements by virtualizing resources of the server to provide an isolation among one or more processes (applications) executing on the server. “Hypervisor-based” or “hardware-level” or “platform” virtualization refers to the creation of virtual machines that each includes a guest operating system for executing one or more processes. In general, a virtual machine (“VM” ofFIG. 1) provides a virtualized/guest operating system for executing applications in an isolated virtual environment. Because a virtual machine is virtualized from physical hardware of the host server, executing applications are isolated from both the hardware of the host and other virtual machines. Each virtual machine may be configured with one or more virtual network interfaces for communicating on corresponding virtual networks.

Virtual networks, such as overlay networks22A-22N (hereinafter, “overlay networks22”) are logical constructs implemented on top of the physical networks. Additionally, or alternatively, virtual networks may be used to replace VLAN-based isolation and provide multi-tenancy in a virtualized data center, e.g., data center10. Each tenant or an application can have one or more virtual networks. Each virtual network may be isolated from all the other virtual networks unless explicitly allowed by security policy.

Virtual networks can be connected to, and extended across, physical Multi-Protocol Label Switching (MPLS) Layer 3 Virtual Private Networks (L3VPNs) and Ethernet Virtual Private Networks (EVPNs) networks using a datacenter10edge router (not shown inFIG. 1). Virtual networks may also be used to implement Network Function Virtualization (NFV) and service chaining.

Virtual networks can be implemented using a variety of mechanisms. For example, each virtual network could be implemented as a Virtual Local Area Network (VLAN), Virtual Private Networks (VPN), etc. A virtual network can also be implemented using two networks—the physical underlay network made up of IP fabric20and switching fabric14and a virtual overlay network. The role of the physical underlay network is to provide an “IP fabric,” which provides unicast IP connectivity from any physical device (server, storage device, router, or switch) to any other physical device. The underlay network may provide uniform low-latency, non-blocking, high-bandwidth connectivity from any point in the network to any other point in the network.

Servers12may execute one or more virtual routers21A-21N (hereinafter, virtual routers21) and one or more virtual machines (“VMs” inFIG. 1). Virtual routers21running in the kernels or hypervisors of servers12create virtual overlay networks22on top of servers12using a mesh of dynamic “tunnels” amongst themselves. These overlay tunnels can be MPLS over GRE/UDP tunnels, or VXLAN tunnels, or NVGRE tunnels, for instance. The underlay physical routers and switches may not contain any per-tenant state, such as any Media Access Control (MAC) addresses, IP address, or policies for virtual machines or other virtual execution elements. The forwarding tables of the underlay physical routers and switches may only contain the IP prefixes or MAC addresses of the physical servers12. (Gateway routers or switches that connect a virtual network to a physical network are an exception and may contain tenant MAC or IP addresses).

Virtual routers21of servers12do contain per tenant state. They contain a separate forwarding table (a routing-instance) per virtual overlay network22. That forwarding table contains the IP prefixes (in the case of a layer 3 overlays) or the MAC addresses (in the case of layer 2 overlays) of the virtual machines or other virtual execution elements (e.g., pods of containers). No single virtual router21needs to contain all IP prefixes or all MAC addresses for all virtual machines in the entire data center. A given virtual router21only needs to contain those routing instances that are locally present on the server12(i.e. which have at least one virtual execution element present on the server12.)

In some examples, instead of virtual-machine-based virtualization, servers12implement container-based virtualization. “Container-based” or “operating system” virtualization refers to the virtualization of an operating system to run multiple isolated systems on a single machine (virtual or physical). Such isolated systems represent containers, such as those provided by the open-source DOCKER Container application or by CoreOS Rkt (“Rocket”). Like a virtual machine, each container is virtualized and may remain isolated from the host machine and other containers. However, unlike a virtual machine, each container may omit an individual operating system and provide only an application suite and application-specific libraries. In general, a container is executed by the host machine as an isolated user-space instance and may share an operating system and common libraries with other containers executing on the host machine. Thus, containers may require less processing power, storage, and network resources than virtual machines. A group of one or more containers may be configured to share one or more virtual network interfaces for communicating on corresponding virtual networks.

In some examples, containers are managed by their host kernel to allow limitation and prioritization of resources (CPU, memory, block I/O, network, etc.) without the need for starting any virtual machines, in some cases using namespace isolation functionality that allows complete isolation of an application's (e.g., a given container) view of the operating environment, including process trees, networking, user identifiers and mounted file systems. In some examples, containers may be deployed according to Linux Containers (LXC), an operating-system-level virtualization method for running multiple isolated Linux systems (containers) on a control host using a single Linux kernel. LXC is an operating-system-level virtualization method for running multiple isolated Linux systems (containers) on a single control host (LXC host). An LXC does not use a virtual machine (although an LXC may be hosted by a virtual machine). Instead, an LXC uses a virtual environment with its own CPU, memory, block I/O, network, and/or other resource space. The LXC resource control mechanism is provided by namespaces and cgroups in the Linux kernel on the LXC host. Additional information regarding containers is found in “Docker Overview,” Docker, Inc., available at docs.docker.com/engine/understanding-docker, last accessed Jul. 9, 2016. Additional examples of containerization methods include OpenVZ, FreeBSD jail, AIX Workload partitions, and Solaris containers. Accordingly, as used herein, the term “containers” may encompass not only LXC-style containers but also any one or more of virtualization engines, virtual private servers, silos, or jails.

Servers12host virtual network endpoints for one or more virtual networks that operate over the physical network represented here by IP fabric20and switch fabric14. As depicted inFIG. 1and explained in further detail below, a subset of servers12host one or more overlay networks22that execute on top of an underlay network supported by an underlay segment.

One or more of servers12may each include a virtual router21that executes one or more routing instances for a corresponding overlay network22to provide virtual network interfaces and route packets among the virtual network endpoints of overlay network22. Each of the routing instances may be associated with a network forwarding table. Each of the routing instances may represent a virtual routing and forwarding instance (VRF) for an Internet Protocol-Virtual Private Network (IP-VPN). Packets received by virtual router21A of server12A, for instance, from the underlay segment, may include an outer header to allow devices within the underlay segment to tunnel the payload or “inner packet” to a physical network address for server12A that executes virtual router21A. The outer header may include not only the physical network address of server12A but also a virtual network identifier such as a VXLAN tag or Multiprotocol Label Switching (MPLS) label that identifies a specific underlay segment. An inner packet includes an inner header having a destination network address that conforms to the virtual network addressing space for overlay network22A identified by the virtual network identifier, e.g., a VXLAN identifier that identifies the corresponding routing instance executed by the virtual router21A.

Virtual routers21terminate virtual network overlay tunnels and determine virtual networks for received packets based on tunnel encapsulation headers for the packets, and forwards packets to the appropriate destination virtual network endpoints for the packets. For server12A, for example, for each of the packets outbound from virtual network endpoints hosted by server12A (e.g., overlay network22A), the virtual router21A attaches a tunnel encapsulation header indicating overlay network22A for the packet to generate an encapsulated or “tunnel” packet, and virtual router21A outputs the encapsulated packet via overlay tunnels for overlay network22A to a physical destination computing device, such as another one of servers12. As used herein, a virtual router21may execute the operations of a tunnel endpoint to encapsulate inner packets sourced by virtual network endpoints to generate tunnel packets and decapsulate tunnel packets to obtain inner packets for routing to other virtual network endpoints.

Network system8implements an automation platform for automating deployment, scaling, and operations of virtual execution elements across servers12to provide virtualized infrastructure for executing application workloads and services. In some examples, the platform may be a container orchestration platform that provides a container-centric infrastructure for automating deployment, scaling, and operations of containers to provide a container-centric infrastructure. “Orchestration,” in the context of a virtualized computing infrastructure generally refers to provisioning, scheduling, and managing virtual execution elements and/or applications and services executing on such virtual execution elements to the host servers available to the orchestration platform. Container orchestration, specifically, permits container coordination and refers to the deployment, management, scaling, and configuration, e.g., of containers to host servers by a container orchestration platform. Example instances of orchestration platforms include Kubernetes, Docker swarm, Mesos/Marathon, OpenShift, OpenStack, VMware, and Amazon ECS.

Underlay controller24implements a Software-defined Networking (SDN) controller for the computing infrastructure8. Underlay controller24may execute on one of servers12or another device or devices not depicted inFIG. 1. Underlay controller24may be a distributed application that executes on one or more computing devices. In general, underlay controller24controls the network configuration of EVPN23on switch fabric14to, e.g., establish one or more virtualized underlay segments for packetized communications among virtual network endpoints. Further, underlay controller24controls the deployment, scaling, and operations of virtual execution elements across EVPN23. Underlay controller24provides a logically and in some cases physically centralized controller for facilitating operation of EVPN23. In some examples, underlay controller24may operate in response to configuration input received from an administrator/operator. Additional information regarding the functioning of underlay controller24as a network controller operating in conjunction with other devices of data center10or other software-defined networks may be found in International Application Number PCT/US2013/044378, filed Jun. 5, 2013, and entitled “PHYSICAL PATH DETERMINATION FOR VIRTUAL NETWORK PACKET FLOWS;” and in U.S. patent application Ser. No. 14/226,509, filed Mar. 26, 2014, and entitled “Tunneled Packet Aggregation for Virtual Networks,” each which is incorporated by reference as if fully set forth herein. U.S. patent application Ser. No. 14/226,509 also includes further description of a virtual router, such as virtual router21A.

In accordance with the techniques of the disclosure, network8implements scalable virtualization of a plurality of tenant underlay networks supported by underlay network segments and a plurality of subtenant overlay networks24so as to enable multi-tenancy overlay networks supported by multi-tenancy underlay networks. For example, underlay controller24configures an underlay network on switch fabric14. Underlay controller24further configures a plurality of underlay segments. As described in more detail with respect toFIG. 2, each underlay segment is a virtualized segment implemented by chassis switches18and TOR switches16of switch fabric14. In some examples, the underlay network uses the VXLAN protocol and each underlay network segment is configured as a VXLAN segment of a plurality of VXLAN segments of the underlay network. Each underlay segment provides underlay network connectivity among a different subset of host computing devices (e.g., servers12) of network8.

Underlay controller24assigns a different tenant of a plurality of tenants to an underlay network that transports L2 communications through a respective underlay segment such that a different subset of host computing devices (e.g., servers12) supports the underlay segment to which the tenant is assigned. As such, chassis switches18and TOR switches16may receive customer traffic for the tenant respective servers12via VXLAN Tunnel Endpoints29A-29B (hereinafter, “VTEPs29”) corresponding to the underlay segment and forward the traffic to service provider network7via EVPN23. Similarly, chassis switches18and TOR switches16may receive L2 communications from EVPN23and forward the L2 communications for transport to servers12via VTEPs29corresponding to the underlay segment. In this way, VTEPs29for the underlay segments operate as gateways between EVPN23and subsets of servers12. That is, each underlay segment may include logically separate routing instances for servers12and each VTEP29operates to bridge traffic between the two distinct internal routing instances. For ease of illustration,FIG. 1depicts only VTEP29A of TOR switch16A as having connections to a first subset of servers12and VTEP29B of TOR switch16A as having connections to a second subset of servers12. However, VTEPs29A and29B of other TOR switches16B-16N typically are also connected to each subset of servers12. Thus, underlay controller24may enable orchestration of multiple tenants in EVPN23, each tenant having logically isolated underlay networks overlaid on chassis switches18and TOR switches16. That is, various customer networks provided within data centers5may be virtually isolated onto different underlay segments of EVPN23.

Each underlay segment may support a plurality of overlay networks22that execute on a subset of host computing devices. For example, as depicted in the example ofFIG. 1, each underlay segment supports overlay networks22A-22N that execute on a set of servers12A-12N. Further, a different overlay controller28is instantiated for each subset of host computing devices to control orchestration of overlay networks22A-22N. For example, with respect toFIG. 1, overlay controller28implements an SDN controller for overlay networks22A-22N of each underlay segment. Overlay controller28may execute on one of servers12or another device or devices not depicted inFIG. 1. Overlay controller28may be a distributed application that executes on one or more computing devices. In general, overlay controller28controls the network configuration of overlay networks22of the underlay segment for packetized communications among virtual network endpoints. Further, overlay controller28controls the deployment, scaling, and operations of virtual execution elements across overlay networks22. Overlay controller28provides a logically and in some cases physically centralized controller for facilitating operation of overlay networks22. In some examples, overlay networks22may operate in response to configuration input received from an administrator/operator. In some examples, overlay controller28operates as a network controller operating in conjunction with other devices of data center10or other software-defined networks as described by International Application Number PCT/US2013/044378 and U.S. patent application Ser. No. 14/226,509.

Overlay controller28assigns a subtenant of a plurality of subtenants of an underlay segment to a different overlay network22such that each subtenant may be virtually isolated from each other subtenant. That is, each overlay network22may include logically separate routing instances for servers12and each overlay network22operates to bridge traffic between the two distinct internal routing instances. Thus, overlay controller28may enable orchestration of multiple subtenants in overlay networks22, each subtenant having a dedicated virtual network, using, for instance, VXLAN, that is isolated from the virtual networks of other subtenants within the underlay segment. Thus, overlay controller28may enable orchestration of multiple subtenants in the first subset of the host computing devices supporting the underlay segment, each subtenant having a private network that is isolated from each other subtenant of the first subset of host computing devices.

The techniques of the disclosure may provide specific improvements to the computer-related field of virtualized computing infrastructure. Furthermore, the techniques of the disclosure may be integrated into numerous practical applications. For example, the techniques of the disclosure may allow for the robust and efficient scaling of tenants in an underlay network and subtenants in a plurality of overlay networks, thereby allowing for a much greater number of tenants and subtenants than supported by the use of a single layer of VXLAN alone. For example, the techniques of the disclosure may allow for the segmentation of each tenant of the underlay network to provide a private underlay environment or isolated hardware environment to each tenant. Additionally, the techniques of the disclosure may reduce the complexity of configuring overlay and underlay networks across the virtualized computing infrastructure and avoid the use of the inefficient spanning tree protocol to flood broadcast, unknown, and multicast (BUM) traffic. The techniques of the disclosure also allow for simplified management of access to isolated overlay networks from external networks as well as providing private access to hardware administration interfaces of servers (e.g., Intelligent Platform Management Interface (IPMI)). Furthermore, the techniques of the disclosure may enable the use of isolated environments for tenants and subtenants that allows for increased agility in the development and maintenance of application services within the virtualized computing infrastructure, as well as enabling service providers to offer improved white label services to other service providers and/or offer Bare Metal Server as a Service as a Service (BMSaaS) with agility and flexibility that is not currently possible with conventional techniques.

FIG. 2is a block diagram illustrating an example implementation of data center10ofFIG. 1in further detail. In the example ofFIG. 2, data center10includes underlay network segments26A-26B (hereinafter, “underlay network segments26” or “underlay segments26”) that extend switch fabric14from physical switches16,18to software or “virtual” switches30A-30N (collectively, “virtual routers21”). Virtual routers21dynamically create and manage one or more virtual overlay networks22usable for communication between application instances. In one example, virtual routers21execute the virtual network as an overlay network, which provides the capability to decouple an application's virtual address from a physical address (e.g., IP address) of the one of servers12A-12N (“servers12”) on which the application is executing. Each virtual overlay network22may use its own addressing and security scheme and may be viewed as orthogonal from underlay segment26A and its addressing scheme or from the physical network and its addressing scheme. Various techniques may be used to transport packets within and across overlay networks22over underlay segment26A and the physical network. In some examples, the techniques described in this disclosure provide multicast service within overlay networks22without requiring multicast support in underlay segment26A or the underlying physical network.

Each virtual router21may execute within a hypervisor, a host operating system or other component of each of servers12. Each of servers12may represent an x86 or other general-purpose or special-purpose server capable of executing virtual machines36. In the example ofFIG. 2, virtual router21A executes within hypervisor31, also often referred to as a virtual machine manager (VMM), which provides a virtualization platform that allows multiple operating systems to concurrently run on one of servers12. In the example ofFIG. 2, virtual router21A manages overlay networks22, each of which provides a network environment for execution of one or more virtual machines (VMs)36on top of the virtualization platform provided by hypervisor31. Each VM36is associated with one of the virtual networks VN0-VN1and may represent tenant VMs running customer applications such as Web servers, database servers, enterprise applications, or hosting virtualized services used to create service chains. In some cases, any one or more of servers12or another computing device may host customer applications directly, i.e., not as virtual machines. In some cases, some of VMs36may represent containers, another form of virtualized execution environment. That is, both virtual machines and containers are examples of virtualized execution environments for executing workloads.

In general, each VM36may be any type of software application and may be assigned a virtual address for use within a corresponding overlay network22, where each of the virtual networks may be a different virtual subnet provided by virtual router21A. A VM36may be assigned its own virtual layer three (L3) IP address, for example, for sending and receiving communications but may be unaware of, e.g., a VXLAN identifier for underlay segment26A or an IP address of the physical server12A on which the virtual machine is executing. In this way, a “virtual address” is an address for an application that differs from the logical address for the underlying, physical computer system, e.g., server12A in the example ofFIG. 1 or 2.

In one implementation, each of servers12includes a corresponding one of virtual network (VN) agents35A-35N (collectively, “VN agents35”) that controls overlay networks22and that coordinates the routing of data packets within server12. In general, each VN agent35communicates with overlay controller28, which generates commands to control routing of packets between overlay networks22and VTEP29A of underlay segment26A. For ease of illustration,FIG. 2depicts only VTEP29A of TOR switch16A as having connections to servers12. However, VTEPs29A of other TOR switches16B-16N typically are likewise connected to servers12. VN agents35may operate as a proxy for control plane messages between virtual machines36and overlay controller28. For example, a VM36may request to send a message using its virtual address via the VN agent35A, and VN agent35A may in turn send the message and request that a response to the message be received for the virtual address of the VM36that originated the first message. In some cases, a VM36may invoke a procedure or function call presented by an application programming interface of VN agent35A, and the VN agent35A may handle encapsulation of the message as well, including addressing.

In some example implementations, each server12further includes overlay controller28that communicates directly with underlay controller24. For example, responsive to instructions from underlay controller24, overlay controller28communicates attributes of the particular overlay networks22executing on the respective server12, and may create or terminate overlay networks22.

In one example, network packets, e.g., layer three (L3) IP packets or layer two (L2) Ethernet packets generated or consumed by the instances of applications executed by virtual machines36within the virtual network domain may be encapsulated in another packet (e.g., another IP or Ethernet packet) that is transported by the physical network. The packet transported in a virtual network may be referred to herein as an “inner packet” while the physical network packet may be referred to herein as an “outer packet” or a “tunnel packet.” Encapsulation and/or de-capsulation of virtual network packets within physical network packets may be performed within virtual routers21, e.g., within the hypervisor or the host operating system running on each of servers12. As another example, encapsulation and de-capsulation functions may be performed at the edge of switch fabric14at a first-hop TOR switch16that is one hop removed from the application instance that originated the packet. This functionality is referred to herein as tunneling and may be used within data center10to create one or more overlay networks22within an underlay segment26A. Besides IPinIP, other example tunneling protocols that may be used include IP over GRE, VxLAN, MPLS over GRE, MPLS over UDP, etc.

As noted above, overlay controller28provides a logically centralized controller for facilitating operation of one or more overlay networks22within underlay segment26A. Overlay controller28may, for example, maintain a routing information base, e.g., one or more routing tables that store routing information for overlay networks22of underlay segment26A. Further, underlay controller24may, for example, maintain a routing information base, e.g., one or more routing tables that store routing information for switches16,18of underlay segment26A. Similarly, switches16,18and virtual routers21maintain routing information, such as one or more routing and/or forwarding tables. In one example implementation, virtual router21A of hypervisor31implements a network forwarding table (NFT)32for each overlay network22. In general, each NFT32stores forwarding information for the corresponding overlay network22and identifies where data packets are to be forwarded and whether the packets are to be encapsulated in a tunneling protocol, such as with a tunnel header that may include one or more headers for different layers of the virtual network protocol stack.

In accordance with the techniques of the disclosure, data center10may perform datacenter slicing, which allows for the creation of network-isolated groups of servers12by connecting them to switch fabric14and configuring underlay segments26(e.g., via VXLAN or another overlay protocol) on the switch port. Further, data center10may make servers12and networking devices16,18available to their users.

As depicted inFIG. 2, switch fabric14is a physical underlay network that provides unicast IP connectivity amongst chassis switches18and TOR switches16. The underlay network may provide uniform low-latency, non-blocking, high-bandwidth connectivity from any point in the network to any other point in the network. Further, a virtualized underlay network is overlaid upon switch fabric14and logically separated into underlay segments26. In general, underlay controller24controls the network configuration of EVPN23on switch fabric14to, e.g., establish virtualized underlay segments26for packetized communications among virtual network endpoints (e.g., VTEPs29). Thus, underlay controller24may logically isolate each tenant to a separate underlay segment26supported by chassis switches18and TOR switches16.

Servers12attached to each isolated environment of an underlay segment26can be used to run separate cloud management systems, such as overlay networks22, that can belong to multiple providers, multiple groups within an organization, or can be multiple deployments of the same cloud management system (e.g., development, testing, or production deployments). For example, the use of VXLAN networking with an EVPN control plane may avoid several issues that arise with a system that uses only VLANs. For example, the data center ofFIG. 2may be easier to configure correctly on multiple devices, is not bound by the 4096 scaling limit applicable to VLAN systems, may avoid the use of the inefficient spanning tree protocol, and may avoid flooding BUM traffic throughout each overlay network22. Additionally, data center10may manage access to the isolated environments of underlay segments26for their users from external networks, and can provide private access to hardware administration interfaces of servers12(e.g. IPMI), by, e.g., configuring VRFs on gateway devices.

In one example data center10provides isolated environments via underlay segments26for executing multiple instances of a virtualized computing infrastructure. This may reduce the complexity of setting up multiple virtualized computing infrastructures within the same switching infrastructure, thereby simplifying development and testing of such infrastructures as well as improving the use of networking resources. Furthermore, data center10may reduce the burden to users that manage multiple cloud environments with different software versions and in different stages of deployment. Additionally, the use of data center10may an administrator of data center10to easily offer BMSaaS to multiple external organizations.

Cloud providers, whether private or public, desire to segment their infrastructure so that to each tenant, it appears that they have their own private networking environment. Conventionally, tenants are isolated at the VM or container level, and overlay networking provides connectivity between workloads according to network policy. A conventional orchestrator is used to ensure that each tenant can only create policies for connectivity between their own workloads. However, another level of segmentation is required if segmentation is required for physical hardware. For example, additional segmentation may be required at the server level to allow multiple “subtenants” to provide cloud services to their customers. The segmentation at the server level may require a segmented underlay network. Furthermore, the use of a segmented underlay network may be useful to prevent networking conflicts where different versions or sizes of an application stack need to be tested with identical configurations.

Conventionally, a segmented network may be implemented by configuring a VLAN on interfaces of network switches to which servers of each subtenant are connected. However, the configuration of VLANs on such switches may be complex. Further, such a system implements L2 networks with spanning trees between network segments, which may be inefficient, and such a system is limited to 4096 VLANs on a set of connected switches.

In accordance with the techniques of the disclosure, data center10automates the process, described herein as “datacenter slicing,” of segmenting the physical network of data center10using overlay networks22for use by subtenants to implement cloud or other services in an isolated environment. For example, the techniques of the disclosure may use an EVPN control plane which may be more robust and scalable than the use of VLANs alone. Data center10implements VXLAN (or some other encapsulation) to connect interfaces of physical servers12according to a network policy to create underlay network segments26between subsets of servers12. Each tenant of data center10is allocated a pool of servers12(which need not be physical neighbors) that are encapsulated within a VXLAN segment26of a VXLAN network. Each tenant may then install an overlay controller28within the VXLAN segment and configure servers12as compute nodes for that overlay controller28. Networking in a cloud is typically done using overlay networks22between virtual networking software components that run on each compute node12.

Further, each underlay network segment26between switch ports is used to support an overlay network22for a subtenant. The two levels of encapsulation may run completely independently of each other and have no interaction. Thus, data center10may be “sliced” into multiple underlay segments26as described above. Further, the endpoints of underlay segments26may attach to overlay networks22to segment the subtenants within each underlay network segment26. The endpoints of underlay segments26may be virtual switches or routers21attached to workload interfaces such as VMs36that run on compute nodes12. Further, the subtenant environments are segmented by overlay networks22where the end points are encapsulation endpoints in switches attached to interfaces of servers12.

Such subtenants may be, e.g., separate organizations from a tenant that manages the physical network (e.g., on one of underlay segments26), different groups or departments within the owner of data center10, or the administrators of data center10. In addition to creating isolated subtenant environments, the techniques of the disclosure may provide access into those environments by configuring gateway devices. Through similar means to creating subtenant environments, the techniques of the disclosure may provide isolated subtenant access to server hardware administration interfaces.

In some examples, a gateway may be used to allow subtenant administrators to install and access orchestrator28within each underlay segment26. This gateway may be configured to provide access from the subtenant environment to the IPMI (or other hardware administration interface) of servers12. In this example, each IPMI address is accessible by both the subtenant administrator and the tenant administrator.

Segmented IPMI access may be provided to each subtenant by creating a VXLAN network that the IPMI interface of a server12is configured into when the server12is made available to a subtenant. In this case, each IPMI interfaces may be given from the tenant administrator to the subtenant administrator.

A single underlay orchestrator24may manage the tenant environments (e.g., underlay network segments26) if appropriate gateway access is provided. For example, each tenant may be associated with an OpenStack project, wherein each OpenStack projects is allocated servers12in a dedicated availability zone. The availability zones can have an aggregate (tag) set such that only users in the associated project can manage these servers12and put workloads on them.

In some examples, data center10implements bare metal server (BMS) to VM/Container connectivity for a subtenant using slicing connectivity between physical servers12and virtual machines36. In this example, the underlay segment26in which BMSs12are located may be configured to route traffic in switch fabric14without encapsulation. Network traffic from VM36within overlay network22is passed to underlay segment26and ARP requests from both VMs36and BMSs12is flooded into overlay network22. In some examples, a virtual forwarder routes traffic directly between, e.g., overlay network22A containing VMs36and underlay segment26A containing BMSs12.

In some examples, VMs36may use floating IP (FIP) addresses. For example, underlay controller25may assign, to each underlay segment26, a plurality of FIP addresses. Overlay controller28may assign, to each overlay network22, a subset of the plurality of FIP addresses assigned to an underlay segment26to which overlay controller28is instantiated. Furthermore, overlay controller28may assign, to each virtual router21within an overlay network22, an FIP address of the subset of the plurality of FIP addresses.

In some examples, Source Network Address Translation (SNAT) may be used to exchange traffic between underlay segment26A and overlay networks22. For example, underlay controller24may provision, for underlay segment26A, an SNAT gateway between a VTEP for underlay segment26A and overlay networks22of underlay segment26A. The SNAT gateway may perform address translation for VTEP29A, servers12, and virtual routers21. For example, the SNAT gateway may serve to forward traffic received from virtual routers21of overlay networks22to VTEP29A. Further, the SNAT gateway may serve to forward traffic received from VTEP29A to virtual routers21of overlay networks22. In one example, the SNAT function can be performed in the virtual router21on a server12where a virtual machine sending traffic is running.

FIG. 3is a block diagram illustrating an example implementation of underlay controller24ofFIG. 1in further detail. In the example ofFIG. 3, underlay controller24includes orchestration engine300and SDN controller302.

SDN controller302includes one or more analytic nodes350A-350X (collectively, “analytic nodes350”), one or more configuration nodes352A-352X (collectively, “configuration nodes352”) and control nodes354A-354X (collectively, “control nodes354”). In general, each of the nodes350,352, and352may be implemented as a separate software process, and the nodes may be distributed across multiple hardware computing platforms that provide an environment for execution of the software. Moreover, each of the nodes maintains state data356, which may be stored within a centralized or distributed database. In some examples, state database356is a NoSQL database. In some examples, state database356is a database cluster.

In general, analytic nodes350are tasked with collecting, storing, correlating, and analyzing information from virtual and physical network elements within, e.g., EVPN23of data center10ofFIG. 1. This information may include statistics, logs, events, and errors for use in managing the routing and network configuration of EVPN23. Analytic nodes350store this information in state database356.

Configuration nodes352translate the high-level data model of orchestration engine300into lower level models suitable for interacting with network elements, such as physical switches16,18. Configuration nodes352keep a persistent copy of the configuration state of SDN controller302within state database56.

Control nodes354implement a logically centralized control plane responsible for maintaining ephemeral network state. Control nodes354interact with each other and with network elements, such as physical switches16,18, to ensure that the network state is eventually consistent with desired state as specified by orchestration engine300. In general, control nodes354receive configuration state of SDN controller302from configuration nodes352, and exchange routes with each other via IBGP to ensure that all control nodes354have the same network state. Further, control nodes354exchange routes with physical switches16,18via BGP or Netconf. Control nodes354communicate the configuration state information, such as routing instances and forwarding policy, to physical switches16,18, e.g., via BGP or Netconf, for installation within physical switches16,18. Further, control nodes354exchange routes with physical switches16,18via BGP, and exchange the configuration state of SDN controller302with physical switches16,18via Netconf.

Configuration nodes352provide a discovery service that tenants of data center10may use to locate various services available within an underlay segment26. For example, if a physical switch16,18attempts a connection with control node354A, it uses a discovery service provided by configuration nodes352to discover the IP address of control node354A. Physical switches16,18may use local configuration, DHCP or DNS to locate the service discovery server within configuration nodes352.

In some examples, configuration nodes352present northbound API that interfaces with orchestration engine300. Orchestration engine300uses this interface to install configuration state using the high-level data model. Configuration nodes352further include a message bus to facilitate communications amongst internal components. Configuration nodes352further include a transformer that discovers changes in the high-level model of orchestration engine300and transforms these changes into corresponding changes in the low level data model managed by SDN controller302. In some examples, configuration nodes352further include a server that provides a southbound API to push computed low-level configuration down to control nodes354. Furthermore, configuration nodes352include a distributed applications manager used to allocate unique object identifiers and to implement transactions across EVPN23.

In accordance with the techniques of the disclosure, one or more control nodes354configures an underlay network on switches16,18. The one or more control nodes354further configure a plurality of underlay segments26A-24N (hereinafter, “underlay segments26”). As shown inFIG. 3, each underlay segment26includes different chassis switches18and TOR switches16of switch fabric14ofFIG. 1that support different subsets of servers12ofFIG. 1. In some examples, the underlay network is a VXLAN and each underlay network segment26is configured as a VXLAN segment of a plurality of VXLAN segments of the VXLAN. Underlay segments26provide underlay network connectivity among a different subset of host computing devices (e.g., servers12) ofFIG. 1.

The one or more control nodes354assign a different tenant of a plurality of tenants to an underlay network that transports L2 communications through a respective underlay segment26such that a different subset of host computing devices (e.g., servers12ofFIG. 1) supports the underlay segment26to which the tenant is assigned. As such, chassis switches18and TOR switches16of an underlay segment26may receive customer traffic for the tenant from servers12ofFIG. 1and forward the traffic through service provider network7ofFIG. 1the underlay network. Similarly, chassis switches18and TOR switches16may receive L2 communications from the underlay network and forward the L2 communications for transport to servers12via underlay segment26. In this way, underlay segments26operate as gateways between the underlay network and the subsets of servers12. Thus, the one or more control nodes354may enable orchestration of multiple tenants across switch fabric14, each tenant being logically isolated from each other tenant on the switch fabric14. That is, various customer networks provided within data centers5may be virtually isolated onto different underlay segments26of EVPN23.

FIG. 4is a block diagram illustrating an example implementation of overlay controller28ofFIG. 1in further detail. In the example ofFIG. 4, overlay controller28includes orchestration engine400and SDN controller402. Orchestration engine400and SDN controller402may operate in a substantially similar fashion to orchestration engine300and SDN controller302of underlay controller24ofFIG. 3, respectively.

SDN controller402includes one or more analytic nodes450A-450X (collectively, “analytic nodes450”), one or more configuration nodes452A-452X (collectively, “configuration nodes452”) and control nodes454A-454X (collectively, “control nodes454”). In general, each of the nodes450,452, and452may operate in a substantially similar fashion to the like nodes350,352, and352. However, in contrast to underlay controller24, configuration nodes452may serve to configure VR agents35and overlay network22, while control nodes454may serve to implement a logically centralized control plane responsible for maintaining ephemeral network state of VR agents30and overlay network22. For example, control nodes454communicate the configuration state information, such as routing instances and forwarding policy, to VR agents35, e.g., via XMPP, for installation within respective virtual routers30.

In accordance with the techniques of the disclosure, one or more control nodes454control orchestration of overlay networks22A-22N. For example, with respect toFIG. 4, the one or more control nodes454control the network configuration of overlay networks22of, e.g., underlay segment26A ofFIG. 1for packetized communications among virtual network endpoints. Further, the one or more control nodes454control the deployment, scaling, and operations of virtual execution elements across overlay networks22.

In some examples, the one or more control nodes454assign a subtenant of a plurality of subtenants of underlay segment26A to a different overlay network22such that each subtenant may be virtually isolated from each other subtenant. That is, each overlay network22may include logically separate routing instances for servers12and each overlay network22operates to bridge traffic between the two distinct internal routing instances. Thus, the one or more control nodes454may enable orchestration of multiple subtenants in overlay networks22, each subtenant having a dedicated virtual network, using, for instance, VXLAN, that is isolated from the virtual networks of other subtenants within underlay segment26A. Thus, the one or more control nodes454may enable orchestration of multiple subtenants in a subset of the host computing devices12supporting, e.g., underlay segment26A, each subtenant having a private network that is isolated from each other subtenant of the subset of host computing devices12.

FIG. 5is a flowchart illustrating an example operation in accordance with the techniques of the disclosure. For convenience,FIG. 5is described with respect toFIG. 1.

With respect to the example ofFIG. 5, underlay controller24configures a plurality of underlay network segments26to enable orchestration of multiple tenants (500). Each underlay segment is a virtualized segment implemented by chassis switches18and TOR switches16of switch fabric14. In some examples, the underlay network is a VXLAN and each underlay network segment is configured as a VXLAN segment of a plurality of VXLAN segments of the VXLAN. Each underlay segment provides underlay network connectivity among a different subset of host computing devices (e.g., servers12) of network8.

Underlay controller24assigns a different tenant to each underlay network segment26(502). As such, chassis switches18and TOR switches16may receive customer traffic for the tenant respective servers12via VXLAN Tunnel Endpoints29A-29B (hereinafter, “VTEPs29”) corresponding to the underlay segment and forward the traffic to service provider network7via EVPN23. Similarly, chassis switches18and TOR switches16may receive L2 communications from EVPN23and forward the L2 communications for transport to servers12via VTEPs29corresponding to the underlay segment. In this way, VTEPs29for the underlay segments operate as gateways between EVPN23and subsets of servers12. That is, each underlay segment may include logically separate routing instances for servers12and each VTEP29operates to bridge traffic between the two distinct internal routing instances. Underlay controller24controls forwarding of network traffic for the tenant to the corresponding underlay network segment (504). Thus, underlay controller24may enable orchestration of multiple tenants in EVPN23, each tenant having logically isolated underlay networks overlaid on chassis switches18and TOR switches16. That is, various customer networks provided within data centers5may be virtually isolated onto different underlay segments of EVPN23.

Overlay controller28configures a plurality of overlay networks22in a first subset of host computing devices (e.g., servers12) to enable orchestration of multiple subtenants (506). In general, overlay controller28controls the network configuration of overlay networks22of the underlay segment for packetized communications among virtual network endpoints. Further, overlay controller28controls the deployment, scaling, and operations of virtual execution elements across overlay networks22.

Overlay controller28assigns a subtenant to each overlay network22(508). That is, each overlay network22may include logically separate routing instances for servers12and each overlay network22operates to bridge traffic between the two distinct internal routing instances. Thus, overlay controller28may enable orchestration of multiple subtenants in overlay networks22, each subtenant having one or more dedicated virtual networks, using, for instance, VXLAN, that is isolated from the virtual networks of other subtenants within the underlay segment. Further, overlay controller28controls forwarding of network traffic for the subtenant to the corresponding overlay network22(510). Thus, overlay controller28may enable orchestration of multiple subtenants in the first subset of the host computing devices supporting the underlay segment, each subtenant having a private network that is isolated from each other subtenant of the first subset of host computing devices.

FIG. 6is a block diagram illustrating an example of a segmented underlay network of a data center using VLANs. Specifically, VLANs are configured on each switch interface of the segmented underlay network. The segmented underlay network ofFIG. 6may implement a Layer-2 hierarchy or Layer-3 Clos. One disadvantage of the segmented underlay network ofFIG. 6is that the use of spanning trees and BUM flooding in VLANs may be required.

FIG. 7is a block diagram illustrating an example of a segmented underlay network of a data center using VXLAN in accordance with the techniques of the disclosure. The segmented underlay network ofFIG. 7may configure VTEPs on the switch interfaces. The use of VXLAN may allow the segmented underlay network ofFIG. 7to implement an EVPN control plane with limited BUM traffic flooding.

FIG. 8is a block diagram illustrating an example of a plurality of isolated overlay networks22in accordance with the techniques of the disclosure. Isolated overlay networks22may be, e.g., isolated cloud environments for subproviders. A subprovider administrator may access servers12via a gateway or VPN. The data center may allocate servers to subproviders and attach the servers their VXLANs

FIG. 9is a block diagram illustrating another example of a plurality of isolated overlay networks22in accordance with the techniques of the disclosure. Isolated overlay networks22may be, e.g., isolated cloud environments for multiple lab deployments. The data center may perform multiple deployments with identical tenant networking. Overlay networks22may avoid conflicts with underlay routing or SR-IOV.

FIG. 10is a block diagram illustrating an example of a sub-provider hardware administration network in accordance with the techniques of the disclosure. The network ofFIG. 10may provide subprovider administrators with access to their own hardware administrative network. Further, administrators of the data center may have their own IPMI VXLAN.

FIG. 11s a block diagram illustrating an example of single-orchestrator cluster management of data center slicing and multiple isolated clouds, in accordance with the techniques of the disclosure. The system ofFIG. 11may allow a single orchestrator to access multiple subclouds via a gateway. An administrator may use availability zones with a tenant ID aggregate filter to perform isolation between projects.

FIG. 12is a block diagram illustrating an example of a data path between virtual machines (VMs) in a sub-provider network in accordance with the techniques of the disclosure. As depicted inFIG. 12, the subprovider orchestration is performed within the subprovider VXLAN. Further, the VXLAN is managed by the data center, while a tenant overlay allows tunneling between VMs of the subprovider. A BGP session between the orchestrator and VMs is inside the subprovider VXLAN.

FIG. 13is a block diagram illustrating an example of a data path for provisioning bare metal servers (BMSs) in accordance with the techniques of the disclosure. A subprovider admin may cause the orchestrator to provision systems in an BMS. The orchestrator may use a VXLAN to connect to the BMS via switch fabric managed by the data center. In some examples, the orchestrator uses Ironic, PXE, TFTP, etc. in the subprovider fabric.

FIG. 14is a block diagram illustrating an example of a data path between VMs and BMSs in accordance with the techniques of the disclosure. As depicted byFIG. 14, the VM and BMS are in the same network. The BMS may flood the local fabric with BUM traffic. The VMs may flood the fabric with BUM traffic as well. The BMS network uses fabric routing. A vRouter may directly connect a VM to a BMS in a different network.

FIG. 15is a block diagram illustrating an example of a floating IP address (FIP) data path in accordance with the techniques of the disclosure. The subprovider admin may cause the orchestrator to request a tenant VRF in the gateway from the data center admin. The orchestrator may use XMPP to manage the VM. For example, the orchestrator may establish a VRF for an FIP pool network. Further, the orchestrator may use BGP to access a gateway of the datacenter.

FIG. 16is a block diagram illustrating an example use of VMs and BMS in different networks for a Layer-2 forwarder in accordance with the techniques of the disclosure. Both the BMS and the VM may flood a local fabric with BUM traffic. The orchestrator may use Ironic, etc. to manage the BMS. Further, each VM may implement an L2 forwarder. Further, the system may perform routing between networks in a VRF with IRBs for each network in a virtualized router.

FIG. 17is a block diagram illustrating an example of a source network address translation (SNAT) gateway for sub-providers in non-overlapping sub-provider fabrics in accordance with the techniques of the disclosure. An enterprise network may provide a router that is the default gateway for the subprovider fabric with forwarding via inet.0. The fabric subnet is advertised into the enterprise network.

FIG. 18s a block diagram illustrating an example of an SNAT data path in accordance with the techniques of the disclosure. The orchestrator may use XMPP to configure a gateway of a VM. The orchestrator may further configure a gateway of the datacenter via BGP.

FIG. 19is a block diagram illustrating an example of an SNAT gateway for sub-providers in overlapping sub-provider fabrics in accordance with the techniques of the disclosure. In the example ofFIG. 19, the system uses a separate gateway for each subprovider. Further, each subprovider is allocated overlapping subprovider fabrics. The router is the default gateway for the subprovider fabric with forwarding using NAT.

FIG. 20is a block diagram illustrating an example use of FIP for sub-providers using non-overlapping FIP pools in accordance with the techniques of the disclosure. In the example ofFIG. 20, the system uses a separate gateway for each subprovider. Further, each subprovider is allocated non-overlapping FIP pools. The router is default gateway for the VXLAN. IRB connects to each VTEP. The VTEP has IRB with default gateway for tenant network. The system uses forwarding in inet.0 to advertise the FIP network.

FIG. 21is a block diagram illustrating an example of a shared gateway using SNAT for sub-providers in non-overlapping sub-provider fabrics and FIP with non-overlapping tenant networks. SNAT may be used for non-overlapping subprovider fabrics. BGP may be used to communicate the fabric pool. FIP may be used for non-overlapping tenant networks. BGP may be used to communicate the provider FIP pool.

FIG. 22is a block diagram illustrating an example of a shared gateway using SNAT for sub-providers in overlapping sub-provider fabrics and FIP with overlapping tenant networks. SNAT may be used for overlapping subprovider fabrics. BGP may be used to communicate the fabric pool. FIP may be used for overlapping tenant FIP networks. BGP may be used to communicate the tenant FIP pools.