Patent ID: 12218800

DETAILED DESCRIPTION

In the following detailed description of the invention, numerous details, examples, and embodiments of the invention are set forth and described. However, it will be clear and apparent to one skilled in the art that the invention is not limited to the embodiments set forth and that the invention may be practiced without some of the specific details and examples discussed.

Some embodiments provide a method of selecting data links for an application in a network. The method receives, from a machine implementing the application, a set of identifiers of required link characteristics. In some embodiments, the applications hosted on the machines leverage the Geneve protocol to specify custom messages in the TLV format, encapsulating each of the application packets, which can carry a specification of link characteristics towards edge forwarding nodes. Based on at least one of the identifiers, the method selects a transport group that includes a set of optional links matching the identifiers. From the selected transport group, the method selects a link matching the set of identifiers.

The method of various embodiments selects the transport group based on different characteristics. For example, the method may select the transport group based on one or more identifiers of: (1) a security and/or encryption characteristic of links in the transport group, (2) a filtering capability of links in the transport group, (3) an identifier of a presence of proxies on the links in the transport group, (4) an identifier of a lowest maximum transmission unit (MTU) size of links of the transport group, (5) a minimum throughput of the links of the transport group, (6) a maximum rate of packet drops of links of the transport group, and/or (7) a maximum rate of interface errors of links of the transport group.

The method of some embodiments also sends an identifier of the application and the selected link for the application to an edge node of the logical network. The edge node may then identify packets of the application and route the packets of the application to the selected link.

As used in this document, data messages refer to a collection of bits in a particular format sent across a network. One of ordinary skill in the art will recognize that the term data message may be used herein to refer to various formatted collections of bits that may be sent across a network, such as Ethernet frames, IP packets, TCP segments, UDP datagrams, etc. Also, as used in this document, references to L2, L3, L4, and L7 layers (or layer 2, layer 3, layer 4, layer 7) are references, respectively, to the second data link layer, the third network layer, the fourth transport layer, and the seventh application layer of the OSI (Open System Interconnection) layer model.

FIG.1illustrates an example of a virtual network100that is created for a particular entity using SD-WAN forwarding elements deployed at branch sites (sometimes called “branches” or “sites”), datacenters, and public clouds. Examples of entities include a company (e.g., corporation, partnership, etc.), an organization (e.g., a school, a non-profit, a government entity, etc.), etc. In some embodiments, an SD-WAN is the application of software based network technologies that virtualize WAN connections. An SD-WAN decouples network software services from underlying hardware to create a virtualized network overlay. An SD-WAN in some embodiments connects different sites (e.g., different buildings, or locations in different neighborhood, cities, states, countries, continents, etc.) through software tools that deploy forwarding elements in the cloud (e.g., one or more public clouds, such as public clouds provided by Amazon Web Services (AWS), Microsoft, Google, and/or one or more private clouds) and connect these forwarding elements through route records.

The SD-WANs of some embodiments employ a hub and spoke architecture, in which the hubs serve as focal/intermediary points for connecting edge forwarding elements (in some embodiments, the edge forwarding elements could be routers, gateways, or other routing devices) at branch sites that serve as the spokes of the SD-WAN architecture. The branches themselves may be implemented as sites to support manufacturing, Points of Sale (POS), medical facilities such as hospitals and clinics, or other scenarios. In some embodiments, hubs act as a central point of management for some or all connected branch sites. Hubs of some embodiments are set up by a centralized management plane orchestrator. The orchestrator notifies all the edge forwarding elements on the branches about the hubs, and the edge forwarding elements build secure overlay (in some embodiments, multi-path) tunnels to the hubs. The hubs themselves include edge forwarding elements, typically deployed in datacenters to allow branches to access the datacenters' resources and to route traffic within and outside the SD-WAN.

The edge forwarding elements connect to each other either directly or through a hub (meaning traffic from one branch site would go through that site's edge forwarding element to a hub forwarding element at a datacenter, and this hub forwarding element would then relay the traffic to another branch site through that site's edge forwarding element). Similarly, in some embodiments, traffic from branches passes through a hub, then out of the SD-WAN, over an external network to an external (outside the SD-WAN) machine.

InFIG.1, the SD-WAN forwarding elements include cloud gateway105and SD-WAN forwarding elements130,132,134, and136. The cloud gateway (CGW) in some embodiments is a forwarding element that is in a private or public datacenter110. The CGW105in some embodiments has secure connection links (e.g., tunnels) with edge forwarding elements (e.g., SD-WAN edge forwarding elements (FEs)130,132,134, and136) at the particular entity's multi-machine sites (e.g., SD-WAN edge sites120,122, and124), such as branch offices, datacenters, etc. These multi-machine sites are often at different physical locations (e.g., different buildings, different cities, different states, etc.) and are referred to below as multi-machine sites or nodes.

Four multi-machine sites120-126are illustrated inFIG.1, with three of them being branch sites120-124, and one being a datacenter126. Each branch site is shown to include an edge forwarding node130-134, while the datacenter site126is shown to include a hub forwarding node136. Each branch site120-124and the datacenter126includes resources150-156respectively. These resources150-156may include servers, hosts, routers, switches, and/or other physical or logical elements (e.g., virtual machines, containers, etc.). The resources150-156may communicate with resources of other branches and/or other resources outside of their own site through the forwarding elements130-136, respectively. The datacenter SD-WAN forwarding node136is also referred to as a hub node136because in some embodiments this forwarding node can be used to connect (e.g., through a VPN tunnel) to other edge forwarding nodes of the branch sites120-124. The hub node136in some embodiments provides services (e.g., middlebox services) for packets that it forwards from one branch site to another branch site. The hub node136also provides access to the datacenter resources156, as further described below.

Each edge forwarding element (e.g., SD-WAN edge FEs130-134) exchanges data messages with one or more cloud gateways105through one or more connection links115(e.g., multiple connection links available at the edge forwarding element). In some embodiments, these connection links include secure and unsecure connection links, while in other embodiments they only include secure connection links. As shown by edge node134and gateway105, multiple secure connection links (e.g., multiple secure tunnels that are established over multiple physical links) can be established between one edge node and a gateway.

When multiple such links are defined between an edge node and a gateway, each secure connection link in some embodiments is associated with a different physical network link between the edge node and an external network. For instance, to access external networks, an edge node in some embodiments has one or more commercial broadband Internet links (e.g., a cable modem, a fiber optic link) to access the Internet, an MPLS (multiprotocol label switching) link to access external networks through an MPLS provider's network, and/or a wireless cellular link (e.g., a 5G LTE network). In some embodiments, the different physical links between the edge node134and the cloud gateway105are the same type of links (e.g., are different MPLS links).

In some embodiments, one edge forwarding node130-134can also have multiple direct links115(e.g., secure connection links established through multiple physical links) to another edge forwarding node130-134, and/or to a datacenter hub node136. Again, the different links in some embodiments can use different types of physical links or the same type of physical links. Also, in some embodiments, a first edge forwarding node of a first branch site can connect to a second edge forwarding node of a second branch site (1) directly through one or more links115, or (2) through a cloud gateway or datacenter hub to which the first edge forwarding node connects through two or more links115. Hence, in some embodiments, a first edge forwarding node (e.g.,134) of a first branch site (e.g.,124) can use multiple SD-WAN links115to reach a second edge forwarding node (e.g.,130) of a second branch site (e.g.,120), or a hub forwarding node136of a datacenter site126.

The cloud gateway105in some embodiments is used to connect two SD-WAN forwarding nodes130-136through at least two secure connection links115between the gateway105and the two forwarding elements at the two SD-WAN sites (e.g., branch sites120-124or datacenter site126). In some embodiments, the cloud gateway105also provides network data from one multi-machine site to another multi-machine site (e.g., provides the accessible subnets of one site to another site). Like the cloud gateway105, the hub forwarding element136of the datacenter126in some embodiments can be used to connect two SD-WAN forwarding nodes130-134of two branch sites through at least two secure connection links115between the hub136and the two forwarding elements at the two branch sites120-124.

In some embodiments, each secure connection link between two SD-WAN forwarding nodes (i.e., CGW105and edge forwarding nodes130-136) is formed as a VPN (virtual private network) tunnel between the two forwarding nodes. In this example, the collection of the SD-WAN forwarding nodes (e.g., forwarding elements130-136and cloud gateway105) and the secure connections between the forwarding nodes forms the virtual network100for the particular entity that spans at least public or private cloud datacenter110to connect the branch and datacenter sites120-126.

In some embodiments, secure connection links are defined between gateways in different public cloud datacenters to allow paths through the virtual network to traverse from one public cloud datacenter to another, while no such links are defined in other embodiments. Also, in some embodiments, the cloud gateway105is a multi-tenant gateway that is used to define other virtual networks for other entities (e.g., other companies, organizations, etc.). Some such embodiments use tenant identifiers to create tunnels between a gateway and edge forwarding element of a particular entity, and then use tunnel identifiers of the created tunnels to allow the gateway to differentiate data message flows that it receives from edge forwarding elements of one entity from data message flows that it receives along other tunnels of other entities. In other embodiments, gateways are single-tenant and are specifically deployed to be used by just one entity.

FIG.1illustrates a cluster of controllers140that serves as a central point for managing (e.g., defining and modifying) configuration data that is provided to the edge nodes130-0136and/or gateways to configure some or all of the operations. In some embodiments, this controller cluster140is in one or more public cloud datacenters, while in other embodiments it is in one or more private datacenters. In some embodiments, the controller cluster140has a set of manager servers that define and modify the configuration data, and a set of controller servers that distribute the configuration data to the edge forwarding elements (FEs), hubs and/or gateways. In some embodiments, the controller cluster140directs edge forwarding elements and hubs to use certain gateways (i.e., assigns a gateway to the edge forwarding elements and hubs). The controller cluster140also provides next hop forwarding rules in some embodiments.

FIG.2illustrates a datacenter200with multiple links230to one or more external networks235. The datacenter200includes host computers202A and202B and an edge node215. The edge node215connects the datacenter200to external networks235through multiple links230. Each of the links230may have some or all of their characteristics (e.g., MTU bandwidth, error rate, presence of proxies on the link, etc.) different from the characteristics of the other links230. The host computers202A and202B implement machines (e.g., VMs)205A-205C. The machines205A-205C in turn implement applications210A-210C, respectively. The applications210A-210C communicate with external networks235(e.g., to machines on the external networks) by accessing one or more of the links230through the SD-WAN edge FE220of the edge node215. The applications210A-210C may further communicatively connect to applications or other software or hardware of machines of another datacenter250(or other network connected computer or set of computers) through the external networks235.

As further described with respect toFIG.3, the characteristics of available links may change over time. To respond to these changes, in some embodiments, the SD-WAN edge FE220, ofFIG.2, updates the transport groups dynamically. In some such embodiments, the configuration of the transport groups is facilitated by the network controller240. Further, in some embodiments, reports detailing the results of the link monitoring and/or transport group monitoring are provided to an end user or administrator so that the end user or administrator can react to changes in the link characteristics or transport groups if necessary.

The host computers202A and202B send link requirements212A-212C for each app210A-210C to the SD-WAN edge FE220, which in turn sends the link requirements to a link policy analyzer225. The link policy analyzer225also receives transport group identifiers245from a network controller240. The process of some embodiments for generating the transport group identifiers245is described with respect toFIG.3. The process of selecting a link for a particular application, performed by the link policy analyzer225is described with respect toFIG.4. Characteristics of the link requirements for applications of some embodiments are described with respect toFIG.6. Characteristics of the transport group identifiers of some embodiments are described with respect toFIG.7.

FIG.3conceptually illustrates a process300of some embodiments for providing transport group identifiers to link policy analyzers of edge nodes. In some embodiments, the process300is performed by a network controller such as network controller240ofFIG.2. The process300, ofFIG.3, identifies (at305) characteristics of available links. For example, the process300may identify characteristics such as throughput, error rate, security features, etc. Characteristics of links are further described with respect toFIGS.6and7, below. The process300then groups (at310) the available links (e.g., links230ofFIG.2that SD-WAN FE220uses to communicate with external networks) into multiple transport groups.

In the process300ofFIG.3, links are assigned to a particular transport group based on the links having the characteristics that define that transport group. In some embodiments, the defining characteristics of each transport group include at least one characteristic that is not based on the physical nature of the links (e.g., not based on whether the link is wireless, wired, or using a particular type of physical interface). The transport groups of some embodiments are not exclusive, so any particular link may be assigned to multiple transport groups so long as the link has the defining characteristic of each transport group to which it is assigned. A defining characteristic of a transport group could be a threshold value (i.e., a minimum or maximum value of some characteristic of the links in the group). A defining characteristic could also be the presence or absence of some feature, such as the presence of a particular security protocol that is applied to some links or the absence of proxies on some links.

Once the links are assigned to the transport groups, the process300provides (at315) the transport group identifiers to link policy analyzers of edge nodes. In some embodiments, the transport group identifiers specify the defining characteristic(s) of each transport group, as well as identifiers of the links in each transport group. The identifier of each link of a transport group, in some embodiments, is accompanied by identifiers of additional characteristics of the link. In other embodiments, the link is identified as being a link of that transport group, but no identifiers of additional characteristics of the link are sent with the transport group identifier. The process300then ends.

Although for the sake of description, the process300is shown as a linear flow chart with a start and an end, in some embodiments, the process300is repeated as long as applications need links. That is, the state of the available links is dynamic as various link characteristics may change over time (e.g., throughput may improve or degrade, features may be added or removed in response to network conditions, etc.). These changes may qualify or disqualify the links for inclusion in different transport groups over time. Therefore the links in some embodiments are repeatedly or continuously monitored and the transport group members are adjusted in accord with changed characteristics of the monitored links. Thus, such embodiments produce more resilient matches between the required link characteristics for the applications and their assigned links at any given point of time.

In some embodiments, the active link monitoring continuously keeps track of various link characteristics and metrics. In other embodiments, this mechanism is extended to actively probe and monitor link state/attributes related to various links and transport groups.

Once the process300is complete, a link policy analyzer (e.g., link policy analyzer225ofFIG.2) uses the transport group identifiers (e.g., transport group identifiers245ofFIG.2) and other data to select links for each application using the SD-WAN edge FE served by the link policy analyzer.FIG.4conceptually illustrates a process400of some embodiments for selecting a link for an application. The process400in some embodiments is provided by a link policy analyzer that acts as a separate module from the SD-WAN edge FE. However, one of ordinary skill in the art will understand that in other embodiments, the link policy analyzer may be a subsystem of an SD-WAN edge FE, or may be a subsystem of some other component present in a datacenter.

The process400receives (at405) the transport group identifiers (e.g., from a network controller or other component implementing process300ofFIG.3). The process400, ofFIG.4, also receives (at410) a set of identifiers of required link characteristics for applications operating on VMs of host computers. Identifiers of required link characteristics of some embodiments are further described with respect toFIG.6. In some embodiments, the applications themselves supply the required link characteristics (e.g., pre-programed required link characteristics or required link characteristics derived from an analysis of network conditions and/or other data by the applications). In other embodiments, the VM or the host computer determines the required link characteristics from a database of required link characteristics for specific applications or by analyzing the application and/or traffic generated by the application and/or in response to the application.

The process400, ofFIG.4, selects (at415), based on at least one required link characteristic, a transport group. For example, the defining characteristic of a transport group may be a specified minimum MTU of the links. An MTU is the size of the largest packet that can be sent on a network path without fragmenting the packet. The required link characteristic for an application could be a minimum MTU. In that example, a transport group with a minimum MTU that was at or greater than the minimum MTU of the required link characteristic would be selected by the process400.

Once a particular transport group was selected, the process400would select (at420), from the particular transport group, a link matching the rest of the required link characteristics for that application (if any). In some embodiments, when more than one link in a transport group satisfies all required link characteristics of an application, some default characteristic of the links is used to determine which of multiple satisfactory links is used. That is, in some embodiments, the network controller implements an attribute/characteristic hierarchy for a set of links, based on the set of required or preferred link characteristics of the incoming data and/or the application. The link analyzer of such embodiments ranks the importance of at least a subset of those characteristics when determining which transport group/link to assign the data from a particular application to. In some embodiments, the ranking may be based on the specific requirements for a particular application and in order of decreasing importance of the characteristics to that application. For example, an application that requires high throughput but is tolerant of a high error rate would use a link ranking list that placed the throughput ranking of the links in a transport group above the error rate ranking. The node then sends the data to the most appropriate transport group link based on the progressive ranking of each characteristic. The link analyzers of some embodiments track changes to link characteristics (e.g., determining when throughput of a link drops, when error rates increase, etc.).

In some embodiment, an application may be assigned to a different link if the characteristics of the link it had been using and/or the characteristics of the new link have changed so as to make the new link a better match for the characteristics required/preferred for the app. Additionally, in some embodiments, the rules for determining the hierarchy of links for an application could be adjusted dynamically (e.g., by the network controller or the link analyzer) throughout a period of time depending on particular circumstances or external pressures going on. In still other embodiments, one of multiple satisfactory links may be chosen at random.

In some embodiments, the required link characteristics may include preferences as well as absolute requirements. For example, a set of required link characteristics may include a requirement that a link include a particular security protocol, and a preference for a link with an MTU of at least 1300. In that example, the link policy analyzer would provide a link with that security protocol and minimum MTU if such a link were available, but would still provide a link with that security protocol and a lower MTU if a link with both the required security protocol and preferred minimum MTU were not available.

In some embodiments, when no link with all required link characteristics is available, the process400will provide an error message informing a user or network administrator that no satisfactory links are available for a particular application. In other embodiments when no link with all required link characteristics is available, the process400will select (at420) a link that meets as many of the required link characteristics as possible. The process then ends.

The following are examples of transport groups and applications that may select particular transport groups. In the example case, there are 6 links/interfaces (Link 1, Link 2, Link 3, Link 4, Link 5, and Link 6). Links 1-3 provide larger MTUs (1450 bytes or above) while links 4-6 provide smaller MTUs (300-400 bytes). Real-time transport protocol (RTP) is a kind of data traffic which typically has small to medium sized packets (e.g., at or under 300 bytes). An application which primarily sends such RTP data traffic could use any of the links in a transport group called TG1 (which includes links with a minimum MTU of 300 and thus includes Link 1, Link 2, Link 3, Link 4, Link 5, and Link 6). In contrast, a bulk transaction type TCP application typically sends large sized packets (e.g., 1450 bytes). An application that sends such TCP traffic could require links with large MTUs. Such an application could use a transport group called TG2 (which includes links with a minimum MTU of 1450 and thus includes only Link 1, Link 2, and Link 3, which provide larger MTUs) so that the application can access maximum throughput. Within the transport group, some embodiments make use of an Adaptive Path MTU which would be the minimum native MTU within the transport group.

These link characteristic-based groups define a set of links that are acceptable for use with an application which has requirements matching the transport group's defining characteristics. This alerts the link policy analyzer that no links outside the transport group should be used for a particular application. However, within a particular transport group, the link policy analyzer of some embodiments may quickly switch the selected link for an application based on changing network conditions. For example, if the defining characteristic of a transport group is minimum path MTU for all links within a particular group, for an application that requires at least that minimum value, any link in the group is appropriate to switch the application to in the event that the originally assigned link becomes inoperative or otherwise undesirable. Similarly, in some embodiments, any TCP based applications are subject to maximum segment size (MSS) adjustments based on an adaptive path MTU within the transport group, which effectively improves and maximizes throughput utilization for the application traffic.

FIG.5illustrates the multiple applications using links selected by a link policy analyzer. Application210A has been assigned to link510. Applications210B and210C have been assigned to link520. Each of the applications210A-210C connects to the external network through SD-WAN edge FE220. SD-WAN edge FE220identifies packets sent from each of the application210A-210C and sends the packets to one or more external networks235through the link assigned to that application. In this embodiment, more than one application can be assigned to the same link, such as applications210B and210C being assigned to link520. Such an assignment could be because both application210B and application210C have required link characteristics that resulted in both applications being assigned to the same transport group or because link520is included in multiple transport groups, with application210B assigned to one transport group and application210C assigned to another transport group.

FIG.6illustrates a link requirement set600of some embodiments. The link requirements set600is one example of a structure of a link requirements data set. The link requirements set600includes an application identifier601, minimum bandwidth602, minimum MTU604, maximum allowed number of proxies606, security protocols608and610, and other features612and614. Any or all of these values may be required for a link for the application associated with link requirements set600.

Although the link requirement set600is shown as an ordered data structure with the specific type of requirements identified by their location in the data structure, one of ordinary skill in the art will understand that other orders are possible within the scope of the invention as well as alternate data structures such as using a code to identify a type of data requirement and a number to represent a particular magnitude of that requirement (e.g., an identifier such as “02” to identify a requirement as a MTU requirement followed by the number1450to indicate the minimum required value for the MTU).

FIG.7illustrates a set of transport group identifiers700of some embodiments. The transport group identifiers set700is one example of a structure of a transport group identifiers data set. Transport group identifiers set700includes a group identifier701, minimum bandwidth702, minimum MTU704, maximum allowed number of proxies706, security protocols708and710, and other features712and714. Any or all of these values may be used as the defining characteristics of the transport group defined by the transport group identifiers set700. In addition to the specified defining characteristics of the transport group, the transport group identifiers set700, of some embodiments, includes, for the links, link identifiers721A-721B, link minimum bandwidth722A-722B, a link minimum MTU724A-724B, a link maximum allowed number of proxies726A-726B, link security protocols728A-728B and730A-730B, and other link features732A-732B and734A-734B.

InFIG.7, the transport group identifiers set700includes identifiers of multiple characteristics of the available links, such as characteristics722A-734A of link1, in some embodiments, the transport group identifiers set700only contains characteristics of the group (e.g., characteristics701-714) and identifiers of the individual links (e.g.,721A) rather than the characteristics of each link. In some such embodiments, the link characteristics are still provided by the network controller, but in separate data sets from the transport group identifiers. Such a separation of the link characteristics from the transport group identifiers would be more efficient in cases where multiple transport groups included the same links. Rather than sending all link characteristics in each transport group that includes the link, the network controller would send a set of link characteristics once, and the link policy analyzer would use the link identifiers (e.g.,721A-721B) to retrieve the link characteristics for the identified link from the set of link characteristics.

Although the transport group identifiers set700is shown as an ordered data structure with the specific type of transport group characteristics identified by their location in the data structure, one of ordinary skill in the art will understand that other orders are possible within the scope of the invention as well as alternate data structures such as using a code to identify a type of data characteristic and a number to represent a particular magnitude of that characteristic (e.g., an identifier such as “02” to identify a requirement as a MTU requirement followed by the number1450to indicate the minimum required value for the MTU).

This specification refers throughout to computational and network environments that include virtual machines (VMs). However, virtual machines are merely one example of data compute nodes (DCNs) or data compute end nodes, also referred to as addressable nodes. DCNs may include non-virtualized physical hosts, virtual machines, containers that run on top of a host operating system without the need for a hypervisor or separate operating system, and hypervisor kernel network interface modules. Therefore, it should be understood that where the specification refers to VMs, the examples given could be any type of DCNs, including physical hosts, VMs, non-VM containers, and hypervisor kernel network interface modules. In fact, the example networks could include combinations of different types of DCNs in some embodiments.

Although the above figures show the transport group identifiers being generated by a network controller and sent to a link policy analyzer, in other embodiments, the link policy analyzer itself or the SD-WAN edge FE generates the transport croup identifiers based on the links connected to the SD-WAN edge FE. In other embodiments, the network controller identifies the link characteristics to the link policy analyzer and then the link policy analyzer generates the transport group identifiers. In still other embodiments, the network controller identifies the link characteristics to the SD-WAN edge FE, which then generates the transport group identifiers or forwards the link characteristics to the link policy analyzer (which then generates the transport group identifiers).

VMs, in some embodiments, operate with their own guest operating systems on a host using resources of the host virtualized by virtualization software (e.g., a hypervisor, virtual machine monitor, etc.). The tenant (i.e., the owner of the VM) can choose which applications to operate on top of the guest operating system. Some containers, on the other hand, are constructs that run on top of a host operating system without the need for a hypervisor or separate guest operating system. In some embodiments, the host operating system uses name spaces to isolate the containers from each other and therefore provides operating-system level segregation of the different groups of applications that operate within different containers. This segregation is akin to the VM segregation that is offered in hypervisor-virtualized environments that virtualize system hardware, and thus can be viewed as a form of virtualization that isolates different groups of applications that operate in different containers. Such containers are more lightweight than VMs.

Hypervisor kernel network interface modules, in some embodiments, are non-VM DCNs that include a network stack with a hypervisor kernel network interface and receive/transmit threads. One example of a hypervisor kernel network interface module is the vmknic module that is part of the ESXi™ hypervisor of VMware, Inc.

Many of the above-described features and applications are implemented as software processes that are specified as a set of instructions recorded on a computer-readable storage medium (also referred to as computer-readable medium). When these instructions are executed by one or more processing unit(s) (e.g., one or more processors, cores of processors, or other processing units), they cause the processing unit(s) to perform the actions indicated in the instructions. Examples of computer-readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard drives, EPROMs, etc. The computer-readable media does not include carrier waves and electronic signals passing wirelessly or over wired connections.

In this specification, the term “software” is meant to include firmware residing in read-only memory or applications stored in magnetic storage, which can be read into memory for processing by a processor. Also, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program while remaining distinct software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Finally, any combination of separate programs that together implement a software invention described here is within the scope of the invention. In some embodiments, the software programs, when installed to operate on one or more electronic systems, define one or more specific machine implementations that execute and perform the operations of the software programs.

FIG.8conceptually illustrates a computer system800with which some embodiments of the invention are implemented. The computer system800can be used to implement any of the above-described hosts, controllers, gateway and edge forwarding elements. As such, it can be used to execute any of the above-described processes. This computer system800includes various types of non-transitory machine-readable media and interfaces for various other types of machine-readable media. Computer system800includes a bus805, processing unit(s)810, a system memory825, a read-only memory830, a permanent storage device835, input devices840, and output devices845.

The bus805collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of the computer system800. For instance, the bus805communicatively connects the processing unit(s)810with the read-only memory830, the system memory825, and the permanent storage device835.

From these various memory units, the processing unit(s)810retrieve instructions to execute and data to process in order to execute the processes of the invention. The processing unit(s) may be a single processor or a multi-core processor in different embodiments. The read-only-memory (ROM)830stores static data and instructions that are needed by the processing unit(s)810and other modules of the computer system. The permanent storage device835, on the other hand, is a read-and-write memory device. This device is a non-volatile memory unit that stores instructions and data even when the computer system800is off. Some embodiments of the invention use a mass-storage device (such as a magnetic or optical disk and its corresponding disk drive) as the permanent storage device835.

Other embodiments use a removable storage device (such as a floppy disk, flash drive, etc.) as the permanent storage device835. Like the permanent storage device835, the system memory825is a read-and-write memory device. However, unlike storage device835, the system memory825is a volatile read-and-write memory, such as random access memory. The system memory825stores some of the instructions and data that the processor needs at runtime. In some embodiments, the invention's processes are stored in the system memory825, the permanent storage device835, and/or the read-only memory830. From these various memory units, the processing unit(s)810retrieve instructions to execute and data to process in order to execute the processes of some embodiments.

The bus805also connects to the input and output devices840and845. The input devices840enable the user to communicate information and select commands to the computer system800. The input devices840include alphanumeric keyboards and pointing devices (also called “cursor control devices”). The output devices845display images generated by the computer system800. The output devices845include printers and display devices, such as cathode ray tubes (CRT) or liquid crystal displays (LCD). Some embodiments include devices such as touchscreens that function as both input and output devices840and845.

Finally, as shown inFIG.8, bus805also couples computer system800to a network865through a network adapter (not shown). In this manner, the computer800can be a part of a network of computers (such as a local area network (“LAN”), a wide area network (“WAN”), or an Intranet), or a network of networks (such as the Internet). Any or all components of computer system800may be used in conjunction with the invention.

Some embodiments include electronic components, such as microprocessors, storage and memory that store computer program instructions in a machine-readable or computer-readable medium (alternatively referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, read-only compact discs (CD-ROM), recordable compact discs (CD-R), rewritable compact discs (CD-RW), read-only digital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a variety of recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.), magnetic and/or solid state hard drives, read-only and recordable Blu-Ray® discs, ultra-density optical discs, any other optical or magnetic media, and floppy disks. The computer-readable media may store a computer program that is executable by at least one processing unit and includes sets of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as is produced by a compiler, and files including higher-level code that are executed by a computer, an electronic component, or a microprocessor using an interpreter.

While the above discussion primarily refers to microprocessors or multi-core processors that execute software, some embodiments are performed by one or more integrated circuits, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions that are stored on the circuit itself.

As used in this specification, the terms “computer”, “server”, “processor”, and “memory” all refer to electronic or other technological devices. These terms exclude people or groups of people. For the purposes of the specification, the terms “display” or “displaying” mean displaying on an electronic device. As used in this specification, the terms “computer-readable medium,” “computer-readable media,” and “machine-readable medium” are entirely restricted to tangible, physical objects that store information in a form that is readable by a computer. These terms exclude any wireless signals, wired download signals, and any other ephemeral or transitory signals.

While the invention has been described with reference to numerous specific details, one of ordinary skill in the art will recognize that the invention can be embodied in other specific forms without departing from the spirit of the invention. For instance, several of the above-described embodiments deploy gateways in public cloud datacenters. However, in other embodiments, the gateways are deployed in a third-party's private cloud datacenters (e.g., datacenters that the third-party uses to deploy cloud gateways for different entities in order to deploy virtual networks for these entities). Thus, one of ordinary skill in the art would understand that the invention is not to be limited by the foregoing illustrative details, but rather is to be defined by the appended claims.