NETWORK PATH PERFORMANCE MEASUREMENTS BY UTILIZING MULTI-LAYER TUNNELING TECHNIQUES

Techniques for making network path performance measurements by utilizing multi-layer tunneling are described. In a distributed environment that includes one or more nodes configured to inject network traffic (compute nodes) and one or more nodes that are not configured to inject network traffic (router nodes), techniques are disclosed that allow for the measurement of performance metrics across network segments that include at least one router node. In certain implementations, with one or more router nodes configured with a tunnel termination endpoint and/or a locally-relevant label-to-port mapping, performance metrics between router nodes or between router nodes and compute nodes can be measured. Performance metrics that may be measured using the techniques disclosed herein include network latency, packet loss, and jitter. In addition, the techniques may be used for fault isolation.

BACKGROUND

Various network monitoring techniques are currently used to measure the performance of a network. For example, synthetic network traffic including probe packets can be generated to calculate metric values (e.g., network latency) for the path taken by the probe packet. Other metrics may also be computed such as packet loss and jitter, which may require sending collections of probe packets.

Existing techniques are capable of determining the performance of network paths between nodes that are configured to inject network traffic and take timing measurements for the packets. A network path between such nodes can traverse multiple network segments of a communication network. While existing techniques are able to measure the performance for end-to-end paths between source and destination compute nodes (e.g., from source data centers to the destination data centers), they are not suited for measuring the performances of individual segments of the path by the packet, where a segment is bounded by one or more router nodes. Organizations with complete control over the architecture of a communication network may require not only the end-to-end network performance between two data centers, but also the performance of segments within the communication network that provides connectivity between the data centers.

BRIEF SUMMARY

The present disclosure relates to monitoring network performance. More specifically, techniques are described that enable performance measurements to be made for segments of a communication network by utilizing multi-layer tunneling techniques.

For example, in a cloud infrastructure including a communication network that provides network connectivity between multiple data centers and where the communication network includes one or more router nodes and the data centers include compute nodes, techniques are disclosed that use tunneling and packet header manipulation techniques to enable measurement of a performance metric for one or more network segments in the communication network, where the network segment is a segment between two router nodes or is between a router node and a compute node. Examples of performance metrics that may be measured using the innovative techniques disclosed herein include network latency, packet loss, jitter, and others. The techniques described herein may also be used for fault isolation in the communication network.

In certain embodiments, techniques are described for determining performance metrics for segments of a communication network. For example, a method involves enabling a first packet header to be configured for a first packet to cause the first packet to traverse a path from a source compute node to a destination compute node, the path traversing a communication network comprising a set of one or more router nodes, the path comprising a plurality of segments including a first segment, the set of router nodes having a first router node; and wherein the first packet header comprises a plurality of header sections including a header section for each segment in the path, the plurality of header sections comprising a first header section corresponding to the first segment, the first header section storing information indicative of a manner for routing the first packet for the first segment. A first time is measured for the first packet at the source compute node, the first time indicative of a time when the first packet is communicated from the source compute node. A second time is measured for the first packet at the destination compute node, the second time indicative of a time when the first packet is received at the destination compute node after traversing the path starting at the source compute node and traversing the plurality of segments as per the first packet header. A performance metric is calculated for the first segment using the first time and the second time.

In some examples, the compute nodes may be nodes at data centers provided by a cloud service provider (CSP). For a path traversed by a packet, the source compute node may be a node in a first data center and the destination compute node may be node in a different data center where the data centers are communicatively coupled by the communication network comprising one or more router nodes. For a particular path traversed by a packet, the source compute node and the destination compute node may be the same compute node in a data center.

In certain embodiments, enabling the first packet header to be configured for the first packet comprises configuring a router node from the set of router nodes as a tunnel termination endpoint. Enabling the first packet header to be configured for the first packet may also include, for a router node from the set of router nodes, creating a mapping between a label and an interface of the router node.

In certain embodiments, a first packet header is configured for the first packet, wherein due to the configuring, the first packet header comprises the plurality of header sections including a header section for each segment in the path, the plurality of header sections comprising the first header section corresponding to the first segment, the first header section storing information indicative of a manner for routing the first packet for the first segment.

In another example embodiment, the first segment is between a start node and an end node. In this example, the first router node from set of router nodes is the end node for the first segment and the first header section includes information identifying a first tunnel and information identifying the first router node as an endpoint for the first tunnel.

In another example embodiment, the information identifying the first tunnel identifies a first tunneling protocol and the information identifying the first router node as the endpoint for the first tunnel comprises an address associated with the first router node. In some embodiments, the first tunneling protocol is Generic Routing Encapsulation (GRE) and the address associated with the first router node is an Internet Protocol (IP) address associated with the first router node.

In another example embodiment, the source compute node is the start node of the first segment.

In another example embodiment, another router node from the set of router nodes is the start node of the first segment.

In another example embodiment, the start node of the first segment is the source compute node or another router node from the set of router nodes. The computing device may determine, by the start node of the first segment and from the first packet header, the information identifying the first tunnel and the information identifying the first router node as the endpoint for the first tunnel. The computing device may encapsulate, by the start node, the first packet to generate a first encapsulated packet and communicate the first encapsulated packet from the start node to the end node via the first tunnel.

In another example embodiment, the first segment is between a start node and an end node. The first router node from set of router nodes is the end node for the first segment and the first header section includes information identifying a first label.

In another example embodiment, the start node of the first segment is the source compute node or another router node from the set of router nodes.

In another example embodiment, the start node of the first segment is the source compute node or another router node from the set of router nodes. The start node of the first segment may determine, from the first packet header, a first label. The start node may access, label-to-port mapping information configured for the start node, the label-to-port mapping information mapping the first label to a first egress port of the start node. The start node may communicate the first packet from the start node using the first egress port.

In another example embodiment, the destination compute node is same as the source compute node, the plurality of segments includes a second segment, the first segment is between the source compute node and the first router node, and the second segment is between the first router node and the source compute node. The computing device may calculate the performance metric for the first segment using the first time and the second time by computing a latency metric for the first segment using the first time and the second time. In yet another example embodiment, computing the latency metric includes subtracting the first time from the second time to generate a subtraction result and dividing the subtraction result by two.

DETAILED DESCRIPTION

The present disclosure relates to monitoring network performance. More specifically, techniques are described that enable performance measurements to be made for segments of a communication network by utilizing multi-layer tunneling techniques. For example, in a cloud infrastructure including a communication network that provides network connectivity between multiple data centers and where the communication network includes one or more router nodes and the data centers include compute nodes, techniques are disclosed that use tunneling and packet header manipulation techniques to enable measurement of a performance metric for one or more network segments in the communication network, where the network segment is a segment between two router nodes or is between a router node and a compute node. Examples of performance metrics that may be measured using the innovative techniques disclosed herein include network latency, packet loss, jitter, and others. The techniques described herein may also be used for fault isolation in the communication network.

A cloud services provider (CSP) may offer one or more cloud services to subscribing customers on demand (e.g., via a subscription model) using infrastructure provided by the CSP. The CSP-provided infrastructure is sometimes referred to as cloud infrastructure or cloud services provider infrastructure (CSPI). The CSPI provided by a CSP typically includes one or more data centers communicatively coupled with each other via a communication network (also sometimes referred to as a backbone network). The data centers are generally located in different geographical locations, where one data center can be separated from another data center by vast geographical distances. The communication network enables communication of network traffic between the data centers, where the network traffic can comprise data packets. Various communication protocols may be used to facilitate communications between the data centers over the communication network.

A data center can include one or more computer systems (referred to herein as compute nodes). In certain embodiments, compute nodes are referred to as “bouncers.” A compute node in a data center can be the source of network traffic where the destination of the network traffic is another compute node, possibly in a different data center, or even outside the CSPI. A compute node in a data center can be the destination of network traffic that originates from another compute node, possibly in a different data center, or from outside the CSPI. The communication network provided as part of the CSPI facilitates communication of network traffic between source and destination compute nodes. The network traffic can include one or more one or more data packets, one or more control packets, and other types of packets.

The communication network can include one or more networks and can comprise a heterogenous mix of components for reliably routing information across the network. In certain implementations, the network includes a collection of interconnected nodes. For purposes of this disclosure, the nodes within a communication network that are responsible for routing traffic to facilitate the communication of that traffic from a source compute node to a destination compute node are referred to as router nodes. Examples of router nodes include routers, forwarders, virtual networking entities such as different types of gateways, and the like. In certain embodiments, router nodes are referred to as “bb-core routers.” Unlike compute nodes, a router node is not configured for taking time measurements for packets that are routed by the router node.

Various network monitoring techniques are currently used to measure the performance of a network. For example, synthetic network traffic comprising probe packets can be used to measure the performance of a network path between two compute nodes that traverses the communication network. This can be done, for example, by injecting a probe packet at a source compute node and sending the packet to a destination compute node. A first time measurement is measured at the source compute node when the probe packet is communicated from the source compute node and a second time measurement is taken at the destination compute node when the probe packet arrives at the destination compute node. The values for the first time and the second time can then be used to measure a metric value (e.g., network latency) for the path taken by the probe packet from the source compute node to the destination compute node. Network latency refers to the time the network traffic (e.g., the probe packets) takes to traverse a network path from the source compute node to the destination compute node.

Other metrics may also be computed such as packet loss and jitter. Packet loss refers to a measurement of network traffic that is lost due to network congestion or network errors. For example, packet loss over a network path may estimated by calculating the ratio of packets (e.g., probe packets) lost to packets sent, for a given collection of packets sent over that network path. Jitter may be defined a measurement of the variability in network latency across a particular network segment. For example, jitter over a network path between two compute nodes may be estimated by measuring the network latency over that network path over a period of time and then calculating the variance of those latency measurements. A collection of probe packets may be communicated between a source compute node and a destination compute node to enable computations of packet loss and jitter.

As described above, existing techniques are capable of determining the performance of network paths between source and destination compute nodes. A network path between a source compute node and a destination compute node can traverse over multiple network segments (also referred to as sub-hops) of a communication network. For example, the network path can include a first segment from a source compute node in a source data center to a first router node in the communication network, another segment from the first router node to a second router node, and so on, and finally a segment from the “n”th router node in the communication network to the destination compute node in a destination data center. While existing techniques are able to measure the performance for end-to-end path from the source compute node in a source data center to the destination compute node in a destination data center, they are not suited for measuring the performances of individual segments of the path by the packet, for example, the performance of a segment from the source compute node to the first router node, for a segment between the first router node and a second router node, a segment between the “n”th router node and the destination compute node, and other segments. One of the reasons for this is because the router nodes are not configured to take time measurements for the packets routed by the router nodes that allow the measuring of metrics (e.g., latencies) for the segments.

For most users, measuring just the end-to-end performance as done by existing network performance techniques, is sufficient because the users do not have any control over the communication network that is used to transport packets between source compute nodes and destination compute nodes. However, a CSP has complete control over the architecture of a communication network provided by the CSP as part of the CSPI and that enables communications between data centers provided by the CSP. It is thus very desirable for a CSP to know, not only the end-to-end network performance between two data centers, but also the performance of segments of the communication network that provides connectivity between the data centers, where a segment is between a compute node and a router node or between two router nodes. It is important for the CSP to be able to probe certain segments of the CSP's communication network and measure network performance for the probed segments and compute performance metrics. The CSP may use the performance metrics measured or computed for the communication segments to make decisions regarding the communication network, such as, whether to change the communication network topology, use different router nodes, isolate certain segments of the communication network, and the like. This is not possible using existing techniques.

The present disclosure provides solutions that can be used for measuring network performance and metrics for segments of a communication network. For a network path traversed by a packet from a source compute node to a destination compute node, the teachings described herein enable performance metrics (e.g., latency, jitter, packet loss, etc.) to be measured for segments of the network path, where the segment can be between two router nodes or between a compute node and a router node.

In certain implementations, the measurement of one or more performance metrics for a segment is facilitated by enabling router nodes in a communication network to be configured in a special manner, and further by enabling the physical path traversed by a packet (e.g., a probe packet), from a source compute node to a destination compute node, to be controlled and specified in the header information of the packet before the probe packet is injected into network at the source compute node. The network path may be specified such that the path includes or traverses one or more network segments whose performance is to be measured. As the packet is routed from the source compute node to the destination compute node over the communication network, this packet header information is used by the router nodes of the communication network to decide how to route the packet. The header information of a packet can thus be manipulated and controlled to cause the packet to traverse a specified path through the communication network traversing one or more router nodes. Timing measurements taken for the probe packet at the source compute node when the packet is injected and at the destination compute node when the packet is received by the destination compute node can then be used to measure one or more performance metrics for one or more individual segments of the network path traversed by the packet, the segment is bounded by at least one router node, i.e., the segment is between two router nodes or between a compute node and a router node. In certain use cases, the source compute node and the destination compute node can be the same compute node.

As indicated above, novel techniques are described for configuring router nodes of a communication network to enable performance measurements for individual segments of a network path traversed by a packet. In certain implementations, one or more router nodes of a communication network can be configured as tunnel termination endpoints. A tunnel can be configured between a router node and a compute node (e.g., either the source compute node or the destination compute node) or between two router nodes. This enables the end-to-end path traversed by a packet from a source compute node to the destination compute node to be broken down and specified using one or more tunnels that make up that path. A tunnel can correspond to a segment of the end-to-end network path traversed by a packet from a source compute node to a destination compute node.

As part of configuring a router node, a tunnel is configured between two nodes, where at least one of the two nodes is a router node, and the other node is either a compute node or another router node. A router node that is configured as a tunnel termination endpoint for a tunnel is now capable of receiving a packet (e.g., an IP packet) via the tunnel using a tunneling protocol such as the Generic Routing Encapsulation (GRE) protocol. The packet is encapsulated (with encapsulation information) at the tunnel entry point and decapsulated to the tunnel termination endpoint. For example, the GRE protocol is a tunneling protocol that allows for the encapsulation of information that can be sent over an Internet Protocol (IP) network. Different tunneling protocols may be used in conjunction with the embodiments described in this disclosure.

When a system administrator or network performance engineer wants to probe a particular segment of the communication network, they can configure a probe packet and author header information for the packet to cause the packet to follow a specific path from a source compute node to a destination compute node that traverses the particular network segment. The header information can be authored to identify one or more tunnels that the packet is to traverse. The packet is then injected into the network via the source compute node. As the packet is communicated from the source compute node to the destination compute node, the packet gets routed by the router nodes based upon the header information in the packet's header. For example, if the header information of the packet specifies one or more tunnels in a particular order, the packet is routed through the communication network via the specified tunnels and in the order specified in the packet header, wherein each tunnel is terminated by a router node. At the end of a tunnel, the router node that terminates that tunnel uses the header information in the packet to determine how to further route the packet. In this manner, the packet traverses the network path specified in the header of the packet, where the network path can include one or more tunnels over specific network segments.

Router nodes are additionally configured with label-to-port mapping information. The label-to-port mapping for a router node identifies a list of one or more labels that are locally-relevant (i.e., locally unique to the router node), and each label is mapped to an egress port of the router node. This label-to-port mapping information provides another mechanism that can be used to specify a specific path to be traversed by a packet from the source compute node to the destination compute node. In this case, information identifying a label can be added to the header of a packet. When the packet reaches a router node to which the label applies, the packet is forwarded from that router node over the specific egress port corresponding to the label specified in the header of the packet. In this manner, by specifying labels in a packet's header, the specific egress port used to forward the packet from a router node can be controlled.

In certain implementations, Multiprotocol Label Switching (MPLS) labels may be used. MPLS is a packet routing technology that causes routers to forward packets across a physical link according to a specified label. An egress port corresponds to a specific physical connection between the router node and another node. For example, a particular router node may contain a label-to-port mapping that specifies that label “700002” maps to egress port “xe-0/0/1,” which is physically coupled with another router node or compute node. A router node may have one or multiple egress ports or interfaces. In certain network topologies, the same router node may have two different egress ports, representing two different physical paths, linking to the same next hop node, which may be another router node or a compute node.

When a probe packet is injected into the network at a source compute node, the header information of the packet can be authored to identify a particular label to be used at a tunnel termination endpoint. When the packet arrives at a router node corresponding to the tunnel termination endpoint, the router node reads the label information from the packet's header and then forwards the packet using the label-to-port mapping information configured for that router node. Based upon the label read from the packet's header, the router node uses the label-to-port mapping information to determine an egress port of the router node that the label maps to. The router node then uses that egress port to forward the packet from the router node. In this manner, when the header of a probe packet is configured, the header information in the packet can be specified to include a label that it used to control which egress port of a router node is used to forward the packet from the router node. This provides further control over the path taken by a probe packet from the source compute node to the destination compute node. This is particularly useful for testing and monitoring the performance of particular egress interfaces of router nodes in the communication network.

Once the router nodes have been configured, in order to probe a particular network segment of a communication network, a probe packet is configured to travel a network path from a source compute node to a destination compute node where the path traversed by the packet includes the particular network segment of the communication network whose performance it to be measured. The path to be taken by the packet is configured in the header of the packet before the packet is communicated from the source compute node. The path from the source compute node to the destination compute node traverses one or more router nodes of the communication network. The path comprises multiple network segments, each segment bounded by at least one router node. Each segment is characterized by a start node that represents the start of the network segment and an end node that represents the end of the segment. For a segment, both the start and end nodes can be router nodes, or the start node can be a compute node and the end node is a router node, or the start node is a router node and the end node is a compute node.

In certain implementations, when the header of a packet is configured to cause the packet to traverse a particular path comprising multiple segments, the header of the packet is configured to comprise multiple sections corresponding to the multiple segments of the network path to be traversed by the packet, one section for each segment. For each segment, the corresponding packet header section can be configured to store information identifying how the packet is to be routed for that segment. For a segment, this can be done by specifying a tunnel, a label, or IP information to be used for routing the packet in that segment. For a particular segment, the start node of the segment uses information specified/stored in the corresponding packet header section to determine how the packet is to be routed for that segment.

Additionally, an ordering is specified for the header sections of a packet to correspond to the order in which the segments are to be traversed as the packet is communicated from the source compute node to the destination compute node. In certain implementations, a stack data structure may be used to impose this ordering, with the top section in the stack represents the first network segment, the second section in the stack represents the second network segment, and so on.

In a simple example, let's assume that a probe packet is to traverse a path from a source compute node to a router node and then from the router node back to the source compute node. In this example, the same compute node is both the source compute node and the destination compute node. The network path includes two segments: a first segment from the source compute node to the router node; and a second segment from the router node to the source compute node. For this example, the header of the packet is configured to include two sections: a first section in which how the packet is to be routed from the source compute node to the router node is specified or configured; and a second section in which how the packet is to be routed from the router node to source compute node is specified. The packet header may be configured at the source compute node. The packet with the configured header is then injected at the source compute node. The source compute node, which is the start node of the first segment of the path to be traversed, examines the first section of the packet header and routes the packet over that segment based upon the information contained in the first section. For example, if the first section of the packet header indicates that the packet is to be routed to the router node using a tunneling protocol where the router node is the endpoint of the tunnel, the source compute node encapsulates the packet according to the tunnelling protocol and then tunnels the packet from the source compute node to the router node. The packet is received by the router node via the tunnel. The router node decapsulates the packet. The router node is the start node of the second segment. Upon receiving the packet, the router node examines the second section of the packet header corresponding to the second segment and routes the packet from the router node to source compute node based upon the information contained in the second section of the packet. In this manner, the packet is routed from the source compute node to the destination compute node based upon header information configured for the packet prior to the communication. Several additional examples of how the header of a packet can be configured to cause the packet to traverse a specific network path from the source compute node to the destination compute node are provided below.

In certain implementations, a first time measurement is taken for a probe packet at the source compute node at or approximately around the time when the packet is communicated from the source compute node. The first time may represent a time when the packet was sent from the source compute node. A second time measurement is taken at the destination compute node, for example, when the probe packet arrives at the destination compute node. The second time represent a time the packet is received by the destination compute node. Based upon the two time measurements, a metric (e.g., latency) can be computed for a segment of the network path traversed by the packet, where the segment is between a compute node (e.g., the source compute node or the destination compute node) and a router node or is between two router nodes. For example, in the simple example above, the difference between the second time and the first time divided by two indicates the network latency for the segment between the source compute node and the router node.

Other metrics may also be computed for a network segment using the techniques described herein. For example, in order to determine a packet loss metric or jitter for a network segment, multiple packets may be sent from a source compute node to a destination compute node that traverse the network segment. The number of packets sent from the source compute node may be compared to the number of packets received by the destination compute node to determine a packet loss metric.

Depending upon the complexity of the path and the numbers of segments in the path from the source compute node to the destination compute node, one or multiple packets and associated calculations may be used for measuring a performance metric for a particular segment of the path. For example, for the source compute node-to-router node-to-source compute node path example described above, a single probe packet is sufficient to calculate the latency for the segment between the source compute node and the router node. In some other network path configurations, multiple packets may have to be sent in order to determine a metric for a particular network segment. A metric that requires “n” packets to be sent is referred to as an nth-order metric.

For example, in order to calculate a second-order metric for a network segment, two probe packets may be needed. For example, consider a path from a source compute node to a destination compute node that includes three network segments: a first segment from the source compute node to a first router node; a second segment from the first router node to a second router node; and a third segment from the second router node to the destination compute node. A latency metric (L1) for the segment between the source compute node and the first router node can be calculated by sending a packet from the source compute node to the first router node and back to the source compute node and measuring the packet send and packet receipt times. Likewise, a latency metric (L2) for the segment between the second router node and the destination compute node can be calculated by sending a packet from the destination compute node to the second router node and back to the destination compute node. In order to measure the latency for the segment between the first router node and the second router node (L3), an additional packet is sent with the following path: source compute node to the first router node; first router node to second router node, second router node to first router node, and first router node to source compute node. The packet sent and packet received times are noted at the source compute node. This is used to measure the latency (L4) for the source compute node to second router node segment. The latency (L3) for the first router node to the second router node can then be determined by (L4-L1). Since a minimum of two packets were used to determine this latency value, this metric is referred to as a second-order metric.

The techniques described in this disclosure present several technical improvements over existing techniques. Any physical network segment in a communication network can be probed and a performance metric calculated for that segment. Additionally, individual egress ports or interfaces of the router nodes can be probed. The ability to directly probe physical links between pairs of router nodes and between router nodes and compute nodes allows for measurements of metrics such as network latency, packet loss, and jitter, and also facilitates fault isolation. The ability to isolate faults to a specific component (e.g., a specific router node or egress port) allows for speedy triaging by network reliability engineering teams.

A network performance analysis system (NPAS) is provided that is configured to perform the various functions described in the disclosure. In certain implementations, the NPAS is a component of the CSPI provided by a CSP. The NPAS enables various functions including setting up of tunnels and configuring router nodes as tunnel termination endpoints. The NPAS also enables label-to-port mapping information to be configured for one or more router nodes. The NPAS also provides tools (e.g., GUIs) for configuring headers of probe packets. The NPAS also is capable of measuring or getting information regarding probe packet send and packet receipt times. The NPAS then uses the time information to calculate one or more metrics for network segments. In certain implementations, the NPAS may generate reports showing various network segments and one or more performance metrics computed and associated with the network segments.

FIGS.1-5and the associated description provided in the “Example Virtual Networking Architecture” section below describes networking concepts including virtualization, underlay networks, regions, and availability domains, and provides examples of environments in which nodes implementing the improved techniques disclosed in this disclosure may be used.FIGS.6-13and19describe examples and embodiments related to the improved techniques described in this disclosure.FIGS.14-17depict examples of architectures for implementing cloud infrastructures for providing one or more cloud services, where the infrastructures may incorporate teachings described herein.FIG.18depicts a block diagram illustrating an example computer system, according to at least one embodiment.

Example Virtual Networking Architecture

The term cloud service is generally used to refer to a service that is made available by a cloud services provider (CSP) to users or customers on demand (e.g., via a subscription model) using systems and infrastructure (cloud infrastructure) provided by the CSP. Typically, the servers and systems that make up the CSP's infrastructure are separate from the customer's own on-premise servers and systems. Customers can thus avail themselves of cloud services provided by the CSP without having to purchase separate hardware and software resources for the services. Cloud services are designed to provide a subscribing customer easy, scalable access to applications and computing resources without the customer having to invest in procuring the infrastructure that is used for providing the services.

There are several cloud service providers that offer various types of cloud services. There are various different types or models of cloud services including Software-as-a-Service (SaaS), Platform-as-a-Service (PaaS), Infrastructure-as-a-Service (IaaS), and others.

A customer can subscribe to one or more cloud services provided by a CSP. The customer can be any entity such as an individual, an organization, an enterprise, and the like. When a customer subscribes to or registers for a service provided by a CSP, a tenancy or an account is created for that customer. The customer can then, via this account, access the subscribed-to one or more cloud resources associated with the account.

As noted above, infrastructure as a service (IaaS) is one particular type of cloud computing service. In an IaaS model, the CSP provides infrastructure (referred to as cloud services provider infrastructure or CSPI) that can be used by customers to build their own customizable networks and deploy customer resources. The customer's resources and networks are thus hosted in a distributed environment by infrastructure provided by a CSP. This is different from traditional computing, where the customer's resources and networks are hosted by infrastructure provided by the customer.

The CSPI may comprise interconnected high-performance compute resources including various host machines, memory resources, and network resources that form a physical network, which is also referred to as a substrate network or an underlay network. The resources in CSPI may be spread across one or more data centers that may be geographically spread across one or more geographical regions. Virtualization software may be executed by these physical resources to provide a virtualized distributed environment. The virtualization creates an overlay network (also known as a software-based network, a software-defined network, or a virtual network) over the physical network. The CSPI physical network provides the underlying basis for creating one or more overlay or virtual networks on top of the physical network. The physical network (or substrate network or underlay network) comprises physical network devices such as physical switches, routers, computers and host machines, and the like. An overlay network is a logical (or virtual) network that runs on top of a physical substrate network. A given physical network can support one or multiple overlay networks. Overlay networks typically use encapsulation techniques to differentiate between traffic belonging to different overlay networks. A virtual or overlay network is also referred to as a virtual cloud network (VCN). The virtual networks are implemented using software virtualization technologies (e.g., hypervisors, virtualization functions implemented by network virtualization devices (NVDs) (e.g., smartNICs), top-of-rack (TOR) switches, smart TORs that implement one or more functions performed by an NVD, and other mechanisms) to create layers of network abstraction that can be run on top of the physical network. Virtual networks can take on many forms, including peer-to-peer networks, IP networks, and others. Virtual networks are typically either Layer-3 IP networks or Layer-2 VLANs. This method of virtual or overlay networking is often referred to as virtual or overlay Layer-3 networking. Examples of protocols developed for virtual networks include IP-in-IP (or Generic Routing Encapsulation (GRE)), Virtual Extensible LAN (VXLAN-IETF RFC 7348), Virtual Private Networks (VPNs) (e.g., MPLS Layer-3 Virtual Private Networks (RFC 4364)), VMware's NSX, GENEVE (Generic Network Virtualization Encapsulation), and others.

For IaaS, the infrastructure (CSPI) provided by a CSP can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In an IaaS model, a cloud computing services provider can host the infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., a hypervisor layer), or the like). In some cases, an IaaS provider may also supply a variety of services to accompany those infrastructure components (e.g., billing, monitoring, logging, security, load balancing and clustering, etc.). Thus, as these services may be policy-driven, IaaS users may be able to implement policies to drive load balancing to maintain application availability and performance. CSPI provides infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services in a highly available hosted distributed environment. CSPI offers high-performance compute resources and capabilities and storage capacity in a flexible virtual network that is securely accessible from various networked locations such as from a customer's on-premises network. When a customer subscribes to or registers for an IaaS service provided by a CSP, the tenancy created for that customer is a secure and isolated partition within the CSPI where the customer can create, organize, and administer their cloud resources.

Customers can build their own virtual networks using compute, memory, and networking resources provided by CSPI. One or more customer resources or workloads, such as compute instances, can be deployed on these virtual networks. For example, a customer can use resources provided by CSPI to build one or multiple customizable and private virtual network(s) referred to as virtual cloud networks (VCNs). A customer can deploy one or more customer resources, such as compute instances, on a customer VCN. Compute instances can take the form of virtual machines, bare metal instances, and the like. The CSPI thus provides infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services in a highly available virtual hosted environment. The customer does not manage or control the underlying physical resources provided by CSPI but has control over operating systems, storage, and deployed applications; and possibly limited control of select networking components (e.g., firewalls).

The CSP may provide a console that enables customers and network administrators to configure, access, and manage resources deployed in the cloud using CSPI resources. In certain embodiments, the console provides a web-based user interface that can be used to access and manage CSPI. In some implementations, the console is a web-based application provided by the CSP.

CSPI may support single-tenancy or multi-tenancy architectures. In a single tenancy architecture, a software (e.g., an application, a database) or a hardware component (e.g., a host machine or a server) serves a single customer or tenant. In a multi-tenancy architecture, a software or a hardware component serves multiple customers or tenants. Thus, in a multi-tenancy architecture, CSPI resources are shared between multiple customers or tenants. In a multi-tenancy situation, precautions are taken and safeguards put in place within CSPI to ensure that each tenant's data is isolated and remains invisible to other tenants.

In a physical network, a network endpoint (“endpoint”) refers to a computing device or system that is connected to a physical network and communicates back and forth with the network to which it is connected. A network endpoint in the physical network may be connected to a Local Area Network (LAN), a Wide Area Network (WAN), or other type of physical network. Examples of traditional endpoints in a physical network include modems, hubs, bridges, switches, routers, and other networking devices, physical computers (or host machines), and the like. Each physical device in the physical network has a fixed network address that can be used to communicate with the device. This fixed network address can be a Layer-2 address (e.g., a MAC address), a fixed Layer-3 address (e.g., an IP address), and the like. In a virtualized environment or in a virtual network, the endpoints can include various virtual endpoints such as virtual machines that are hosted by components of the physical network (e.g., hosted by physical host machines). These endpoints in the virtual network are addressed by overlay addresses such as overlay Layer-2 addresses (e.g., overlay MAC addresses) and overlay Layer-3 addresses (e.g., overlay IP addresses). Network overlays enable flexibility by allowing network managers to move around the overlay addresses associated with network endpoints using software management (e.g., via software implementing a control plane for the virtual network). Accordingly, unlike in a physical network, in a virtual network, an overlay address (e.g., an overlay IP address) can be moved from one endpoint to another using network management software. Since the virtual network is built on top of a physical network, communications between components in the virtual network involves both the virtual network and the underlying physical network. In order to facilitate such communications, the components of CSPI are configured to learn and store mappings that map overlay addresses in the virtual network to actual physical addresses in the substrate network, and vice versa. These mappings are then used to facilitate the communications. Customer traffic is encapsulated to facilitate routing in the virtual network.

Accordingly, physical addresses (e.g., physical IP addresses) are associated with components in physical networks and overlay addresses (e.g., overlay IP addresses) are associated with entities in virtual or overlay networks. A physical IP address is an IP address associated with a physical device (e.g., a network device) in the substrate or physical network. For example, each NVD has an associated physical IP address. An overlay IP address is an overlay address associated with an entity in an overlay network, such as with a compute instance in a customer's virtual cloud network (VCN). Two different customers or tenants, each with their own private VCNs can potentially use the same overlay IP address in their VCNs without any knowledge of each other. Both the physical IP addresses and overlay IP addresses are types of real IP addresses. These are separate from virtual IP addresses. A virtual IP address is typically a single IP address that is represents or maps to multiple real IP addresses. A virtual IP address provides a 1-to-many mapping between the virtual IP address and multiple real IP addresses. For example, a load balancer may use a VIP to map to or represent multiple servers, each server having its own real IP address.

The cloud infrastructure or CSPI is physically hosted in one or more data centers in one or more regions around the world. The CSPI may include components in the physical or substrate network and virtualized components (e.g., virtual networks, compute instances, virtual machines, etc.) that are in an virtual network built on top of the physical network components. In certain embodiments, the CSPI is organized and hosted in realms, regions and availability domains. A region is typically a localized geographic area that contains one or more data centers. Regions are generally independent of each other and can be separated by vast distances, for example, across countries or even continents. For example, a first region may be in Australia, another one in Japan, yet another one in India, and the like. CSPI resources are divided among regions such that each region has its own independent subset of CSPI resources. Each region may provide a set of core infrastructure services and resources, such as, compute resources (e.g., bare metal servers, virtual machine, containers and related infrastructure, etc.); storage resources (e.g., block volume storage, file storage, object storage, archive storage); networking resources (e.g., virtual cloud networks (VCNs), load balancing resources, connections to on-premise networks), database resources; edge networking resources (e.g., DNS); and access management and monitoring resources, and others. Each region generally has multiple paths connecting it to other regions in the realm.

Generally, an application is deployed in a region (i.e., deployed on infrastructure associated with that region) where it is most heavily used, because using nearby resources is faster than using distant resources. Applications can also be deployed in different regions for various reasons, such as redundancy to mitigate the risk of region-wide events such as large weather systems or earthquakes, to meet varying requirements for legal jurisdictions, tax domains, and other business or social criteria, and the like.

The data centers within a region can be further organized and subdivided into availability domains (ADs). An availability domain may correspond to one or more data centers located within a region. A region can be composed of one or more availability domains. In such a distributed environment, CSPI resources are either region-specific, such as a virtual cloud network (VCN), or availability domain-specific, such as a compute instance.

ADs within a region are isolated from each other, fault tolerant, and are configured such that they are very unlikely to fail simultaneously. This is achieved by the ADs not sharing critical infrastructure resources such as networking, physical cables, cable paths, cable entry points, etc., such that a failure at one AD within a region is unlikely to impact the availability of the other ADs within the same region. The ADs within the same region may be connected to each other by a low latency, high bandwidth network, which makes it possible to provide high-availability connectivity to other networks (e.g., the Internet, customers' on-premise networks, etc.) and to build replicated systems in multiple ADs for both high-availability and disaster recovery. Cloud services use multiple ADs to ensure high availability and to protect against resource failure. As the infrastructure provided by the IaaS provider grows, more regions and ADs may be added with additional capacity. Traffic between availability domains is usually encrypted.

In certain embodiments, regions are grouped into realms. A realm is a logical collection of regions. Realms are isolated from each other and do not share any data. Regions in the same realm may communicate with each other, but regions in different realms cannot. A customer's tenancy or account with the CSP exists in a single realm and can be spread across one or more regions that belong to that realm. Typically, when a customer subscribes to an IaaS service, a tenancy or account is created for that customer in the customer-specified region (referred to as the “home” region) within a realm. A customer can extend the customer's tenancy across one or more other regions within the realm. A customer cannot access regions that are not in the realm where the customer's tenancy exists.

An IaaS provider can provide multiple realms, each realm catered to a particular set of customers or users. For example, a commercial realm may be provided for commercial customers. As another example, a realm may be provided for a specific country for customers within that country. As yet another example, a government realm may be provided for a government, and the like. For example, the government realm may be catered for a specific government and may have a heightened level of security than a commercial realm. For example, Oracle Cloud Infrastructure (OCI) currently offers a realm for commercial regions and two realms (e.g., FedRAMP authorized and IL5 authorized) for government cloud regions.

In certain embodiments, an AD can be subdivided into one or more fault domains. A fault domain is a grouping of infrastructure resources within an AD to provide anti-affinity. Fault domains allow for the distribution of compute instances such that the instances are not on the same physical hardware within a single AD. This is known as anti-affinity. A fault domain refers to a set of hardware components (computers, switches, and more) that share a single point of failure. A compute pool is logically divided up into fault domains. Due to this, a hardware failure or compute hardware maintenance event that affects one fault domain does not affect instances in other fault domains. Depending on the embodiment, the number of fault domains for each AD may vary. For instance, in certain embodiments each AD contains three fault domains. A fault domain acts as a logical data center within an AD.

When a customer subscribes to an IaaS service, resources from CSPI are provisioned for the customer and associated with the customer's tenancy. The customer can use these provisioned resources to build private networks and deploy resources on these networks. The customer networks that are hosted in the cloud by the CSPI are referred to as virtual cloud networks (VCNs). A customer can set up one or more virtual cloud networks (VCNs) using CSPI resources allocated for the customer. A VCN is a virtual or software defined private network. The customer resources that are deployed in the customer's VCN can include compute instances (e.g., virtual machines, bare-metal instances) and other resources. These compute instances may represent various customer workloads such as applications, load balancers, databases, and the like. A compute instance deployed on a VCN can communicate with public accessible endpoints (“public endpoints”) over a public network such as the Internet, with other instances in the same VCN or other VCNs (e.g., the customer's other VCNs, or VCNs not belonging to the customer), with the customer's on-premise data centers or networks, and with service endpoints, and other types of endpoints.

The CSP may provide various services using the CSPI. In some instances, customers of CSPI may themselves act like service providers and provide services using CSPI resources. A service provider may expose a service endpoint, which is characterized by identification information (e.g., an IP Address, a DNS name and port). A customer's resource (e.g., a compute instance) can consume a particular service by accessing a service endpoint exposed by the service for that particular service. These service endpoints are generally endpoints that are publicly accessible by users using public IP addresses associated with the endpoints via a public communication network such as the Internet. Network endpoints that are publicly accessible are also sometimes referred to as public endpoints.

In certain embodiments, a service provider may expose a service via an endpoint (sometimes referred to as a service endpoint) for the service. Customers of the service can then use this service endpoint to access the service. In certain implementations, a service endpoint provided for a service can be accessed by multiple customers that intend to consume that service. In other implementations, a dedicated service endpoint may be provided for a customer such that only that customer can access the service using that dedicated service endpoint.

In certain embodiments, when a VCN is created, it is associated with a private overlay Classless Inter-Domain Routing (CIDR) address space, which is a range of private overlay IP addresses that are assigned to the VCN (e.g., 10.0/16). A VCN includes associated subnets, route tables, and gateways. A VCN resides within a single region but can span one or more or all of the region's availability domains. A gateway is a virtual interface that is configured for a VCN and enables communication of traffic to and from the VCN to one or more endpoints outside the VCN. One or more different types of gateways may be configured for a VCN to enable communication to and from different types of endpoints.

A VCN can be subdivided into one or more sub-networks such as one or more subnets. A subnet is thus a unit of configuration or a subdivision that can be created within a VCN. A VCN can have one or multiple subnets. Each subnet within a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0/24 and 10.0.1.0/24) that do not overlap with other subnets in that VCN and which represent an address space subset within the address space of the VCN.

Each compute instance is associated with a virtual network interface card (VNIC), that enables the compute instance to participate in a subnet of a VCN. A VNIC is a logical representation of physical Network Interface Card (NIC). In general. a VNIC is an interface between an entity (e.g., a compute instance, a service) and a virtual network. A VNIC exists in a subnet, has one or more associated IP addresses, and associated security rules or policies. A VNIC is equivalent to a Layer-2 port on a switch. A VNIC is attached to a compute instance and to a subnet within a VCN. A VNIC associated with a compute instance enables the compute instance to be a part of a subnet of a VCN and enables the compute instance to communicate (e.g., send and receive packets) with endpoints that are on the same subnet as the compute instance, with endpoints in different subnets in the VCN, or with endpoints outside the VCN. The VNIC associated with a compute instance thus determines how the compute instance connects with endpoints inside and outside the VCN. A VNIC for a compute instance is created and associated with that compute instance when the compute instance is created and added to a subnet within a VCN. For a subnet comprising a set of compute instances, the subnet contains the VNICs corresponding to the set of compute instances, each VNIC attached to a compute instance within the set of computer instances.

Each compute instance is assigned a private overlay IP address via the VNIC associated with the compute instance. This private overlay IP address is assigned to the VNIC that is associated with the compute instance when the compute instance is created and used for routing traffic to and from the compute instance. All VNICs in a given subnet use the same route table, security lists, and DHCP options. As described above, each subnet within a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0/24 and 10.0.1.0/24) that do not overlap with other subnets in that VCN and which represent an address space subset within the address space of the VCN. For a VNIC on a particular subnet of a VCN, the private overlay IP address that is assigned to the VNIC is an address from the contiguous range of overlay IP addresses allocated for the subnet.

In certain embodiments, a compute instance may optionally be assigned additional overlay IP addresses in addition to the private overlay IP address, such as, for example, one or more public IP addresses if in a public subnet. These multiple addresses are assigned either on the same VNIC or over multiple VNICs that are associated with the compute instance. Each instance however has a primary VNIC that is created during instance launch and is associated with the overlay private IP address assigned to the instance—this primary VNIC cannot be removed. Additional VNICs, referred to as secondary VNICs, can be added to an existing instance in the same availability domain as the primary VNIC. All the VNICs are in the same availability domain as the instance. A secondary VNIC can be in a subnet in the same VCN as the primary VNIC, or in a different subnet that is either in the same VCN or a different one.

A compute instance may optionally be assigned a public IP address if it is in a public subnet. A subnet can be designated as either a public subnet or a private subnet at the time the subnet is created. A private subnet means that the resources (e.g., compute instances) and associated VNICs in the subnet cannot have public overlay IP addresses. A public subnet means that the resources and associated VNICs in the subnet can have public IP addresses. A customer can designate a subnet to exist either in a single availability domain or across multiple availability domains in a region or realm.

As described above, a VCN may be subdivided into one or more subnets. In certain embodiments, a Virtual Router (VR) configured for the VCN (referred to as the VCN VR or just VR) enables communications between the subnets of the VCN. For a subnet within a VCN, the VR represents a logical gateway for that subnet that enables the subnet (i.e., the compute instances on that subnet) to communicate with endpoints on other subnets within the VCN, and with other endpoints outside the VCN. The VCN VR is a logical entity that is configured to route traffic between VNICs in the VCN and virtual gateways (“gateways”) associated with the VCN. Gateways are further described below with respect toFIG.1. A VCN VR is a Layer-3/IP Layer concept. In one embodiment, there is one VCN VR for a VCN where the VCN VR has potentially an unlimited number of ports addressed by IP addresses, with one port for each subnet of the VCN. In this manner, the VCN VR has a different IP address for each subnet in the VCN that the VCN VR is attached to. The VR is also connected to the various gateways configured for a VCN. In certain embodiments, a particular overlay IP address from the overlay IP address range for a subnet is reserved for a port of the VCN VR for that subnet. For example, consider a VCN having two subnets with associated address ranges 10.0/16 and 10.1/16, respectively. For the first subnet within the VCN with address range 10.0/16, an address from this range is reserved for a port of the VCN VR for that subnet. In some instances, the first IP address from the range may be reserved for the VCN VR. For example, for the subnet with overlay IP address range 10.0/16, IP address 10.0.0.1 may be reserved for a port of the VCN VR for that subnet. For the second subnet within the same VCN with address range 10.1/16, the VCN VR may have a port for that second subnet with IP address 10.1.0.1. The VCN VR has a different IP address for each of the subnets in the VCN.

In some other embodiments, each subnet within a VCN may have its own associated VR that is addressable by the subnet using a reserved or default IP address associated with the VR. The reserved or default IP address may, for example, be the first IP address from the range of IP addresses associated with that subnet. The VNICs in the subnet can communicate (e.g., send and receive packets) with the VR associated with the subnet using this default or reserved IP address. In such an embodiment, the VR is the ingress/egress point for that subnet. The VR associated with a subnet within the VCN can communicate with other VRs associated with other subnets within the VCN. The VRs can also communicate with gateways associated with the VCN. The VR function for a subnet is running on or executed by one or more NVDs executing VNICs functionality for VNICs in the subnet.

Route tables, security rules, and DHCP options may be configured for a VCN. Route tables are virtual route tables for the VCN and include rules to route traffic from subnets within the VCN to destinations outside the VCN by way of gateways or specially configured instances. A VCN's route tables can be customized to control how packets are forwarded/routed to and from the VCN. DHCP options refers to configuration information that is automatically provided to the instances when they boot up.

Security rules configured for a VCN represent overlay firewall rules for the VCN. The security rules can include ingress and egress rules, and specify the types of traffic (e.g., based upon protocol and port) that is allowed in and out of the instances within the VCN. The customer can choose whether a given rule is stateful or stateless. For instance, the customer can allow incoming SSH traffic from anywhere to a set of instances by setting up a stateful ingress rule with source CIDR 0.0.0.0/0, and destination TCP port 22. Security rules can be implemented using network security groups or security lists. A network security group consists of a set of security rules that apply only to the resources in that group. A security list, on the other hand, includes rules that apply to all the resources in any subnet that uses the security list. A VCN may be provided with a default security list with default security rules. DHCP options configured for a VCN provide configuration information that is automatically provided to the instances in the VCN when the instances boot up.

In certain embodiments, the configuration information for a VCN is determined and stored by a VCN Control Plane. The configuration information for a VCN may include, for example, information about: the address range associated with the VCN, subnets within the VCN and associated information, one or more VRs associated with the VCN, compute instances in the VCN and associated VNICs, NVDs executing the various virtualization network functions (e.g., VNICs, VRs, gateways) associated with the VCN, state information for the VCN, and other VCN-related information. In certain embodiments, a VCN Distribution Service publishes the configuration information stored by the VCN Control Plane, or portions thereof, to the NVDs. The distributed information may be used to update information (e.g., forwarding tables, routing tables, etc.) stored and used by the NVDs to forward packets to and from the compute instances in the VCN.

In certain embodiments, the creation of VCNs and subnets are handled by a VCN Control Plane (CP) and the launching of compute instances is handled by a Compute Control Plane. The Compute Control Plane is responsible for allocating the physical resources for the compute instance and then calls the VCN Control Plane to create and attach VNICs to the compute instance. The VCN CP also sends VCN data mappings to the VCN data plane that is configured to perform packet forwarding and routing functions. In certain embodiments, the VCN CP provides a distribution service that is responsible for providing updates to the VCN data plane. Examples of a VCN Control Plane are also depicted inFIGS.14,15,16, and17(see references1416,1516,1616, and1716) and described below.

A customer may create one or more VCNs using resources hosted by CSPI. A compute instance deployed on a customer VCN may communicate with different endpoints. These endpoints can include endpoints that are hosted by CSPI and endpoints outside CSPI.

Various different architectures for implementing cloud-based service using CSPI are depicted inFIGS.1,2,3,4,5,14,15,16, and18, and are described below.FIG.1is a high-level diagram of a distributed environment100showing an overlay or customer VCN hosted by CSPI according to certain embodiments. The distributed environment depicted inFIG.1includes multiple components in the overlay network. Distributed environment100depicted inFIG.1is merely an example and is not intended to unduly limit the scope of claimed embodiments. Many variations, alternatives, and modifications are possible. For example, in some implementations, the distributed environment depicted inFIG.1may have more or fewer systems or components than those shown inFIG.1, may combine two or more systems, or may have a different configuration or arrangement of systems.

As shown in the example depicted inFIG.1, distributed environment100comprises CSPI101that provides services and resources that customers can subscribe to and use to build their virtual cloud networks (VCNs). In certain embodiments, CSPI101offers IaaS services to subscribing customers. The data centers within CSPI101may be organized into one or more regions. One example region “Region US”102is shown inFIG.1. A customer has configured a customer VCN104for region102. The customer may deploy various compute instances on VCN104, where the compute instances may include virtual machines or bare metal instances. Examples of instances include applications, database, load balancers, and the like.

In the embodiment depicted inFIG.1, customer VCN104comprises two subnets, namely, “Subnet-1” and “Subnet-2”, each subnet with its own CIDR IP address range. InFIG.1, the overlay IP address range for Subnet-1 is 10.0/16 and the address range for Subnet-2 is 10.1/16. A VCN Virtual Router105represents a logical gateway for the VCN that enables communications between subnets of the VCN104, and with other endpoints outside the VCN. VCN VR105is configured to route traffic between VNICs in VCN104and gateways associated with VCN104. VCN VR105provides a port for each subnet of VCN104. For example, VR105may provide a port with IP address 10.0.0.1 for Subnet-1 and a port with IP address 10.1.0.1 for Subnet-2.

Multiple compute instances may be deployed on each subnet, where the compute instances can be virtual machine instances, and/or bare metal instances. The compute instances in a subnet may be hosted by one or more host machines within CSPI101. A compute instance participates in a subnet via a VNIC associated with the compute instance. For example, as shown inFIG.1, a compute instance C1 is part of Subnet-1 via a VNIC associated with the compute instance. Likewise, compute instance C2 is part of Subnet-1 via a VNIC associated with C2. In a similar manner, multiple compute instances, which may be virtual machine instances or bare metal instances, may be part of Subnet-1. Via its associated VNIC, each compute instance is assigned a private overlay IP address and a MAC address. For example, inFIG.1, compute instance C1 has an overlay IP address of 10.0.0.2 and a MAC address of M1, while compute instance C2 has a private overlay IP address of 10.0.0.3 and a MAC address of M2. Each compute instance in Subnet-1, including compute instances C1 and C2, has a default route to VCN VR105using IP address 10.0.0.1, which is the IP address for a port of VCN VR105for Subnet-1.

Subnet-2 can have multiple compute instances deployed on it, including virtual machine instances and/or bare metal instances. For example, as shown inFIG.1, compute instances D1 and D2 are part of Subnet-2 via VNICs associated with the respective compute instances. In the embodiment depicted inFIG.1, compute instance D1 has an overlay IP address of 10.1.0.2 and a MAC address of MM1, while compute instance D2 has a private overlay IP address of 10.1.0.3 and a MAC address of MM2. Each compute instance in Subnet-2, including compute instances D1 and D2, has a default route to VCN VR105using IP address 10.1.0.1, which is the IP address for a port of VCN VR105for Subnet-2.

VCN A104may also include one or more load balancers. For example, a load balancer may be provided for a subnet and may be configured to load balance traffic across multiple compute instances on the subnet. A load balancer may also be provided to load balance traffic across subnets in the VCN.

A particular compute instance deployed on VCN104can communicate with various different endpoints. These endpoints may include endpoints that are hosted by CSPI200and endpoints outside CSPI200. Endpoints that are hosted by CSPI101may include: an endpoint on the same subnet as the particular compute instance (e.g., communications between two compute instances in Subnet-1); an endpoint on a different subnet but within the same VCN (e.g., communication between a compute instance in Subnet-1 and a compute instance in Subnet-2); an endpoint in a different VCN in the same region (e.g., communications between a compute instance in Subnet-1 and an endpoint in a VCN in the same region106or110, communications between a compute instance in Subnet-1 and an endpoint in service network110in the same region); or an endpoint in a VCN in a different region (e.g., communications between a compute instance in Subnet-1 and an endpoint in a VCN in a different region108). A compute instance in a subnet hosted by CSPI101may also communicate with endpoints that are not hosted by CSPI101(i.e., are outside CSPI101). These outside endpoints include endpoints in the customer's on-premise network116, endpoints within other remote cloud hosted networks118, public endpoints114accessible via a public network such as the Internet, and other endpoints.

Communications between compute instances on the same subnet are facilitated using VNICs associated with the source compute instance and the destination compute instance. For example, compute instance C1 in Subnet-1 may want to send packets to compute instance C2 in Subnet-1. For a packet originating at a source compute instance and whose destination is another compute instance in the same subnet, the packet is first processed by the VNIC associated with the source compute instance. Processing performed by the VNIC associated with the source compute instance can include determining destination information for the packet from the packet headers, identifying any policies (e.g., security lists) configured for the VNIC associated with the source compute instance, determining a next hop for the packet, performing any packet encapsulation/decapsulation functions as needed, and then forwarding/routing the packet to the next hop with the goal of facilitating communication of the packet to its intended destination. When the destination compute instance is in the same subnet as the source compute instance, the VNIC associated with the source compute instance is configured to identify the VNIC associated with the destination compute instance and forward the packet to that VNIC for processing. The VNIC associated with the destination compute instance is then executed and forwards the packet to the destination compute instance.

For a packet to be communicated from a compute instance in a subnet to an endpoint in a different subnet in the same VCN, the communication is facilitated by the VNICs associated with the source and destination compute instances and the VCN VR. For example, if compute instance C1 in Subnet-1 inFIG.1wants to send a packet to compute instance D1 in Subnet-2, the packet is first processed by the VNIC associated with compute instance C1. The VNIC associated with compute instance C1 is configured to route the packet to the VCN VR105using default route or port 10.0.0.1 of the VCN VR. VCN VR105is configured to route the packet to Subnet-2 using port 10.1.0.1. The packet is then received and processed by the VNIC associated with D1 and the VNIC forwards the packet to compute instance D1.

For a packet to be communicated from a compute instance in VCN104to an endpoint that is outside VCN104, the communication is facilitated by the VNIC associated with the source compute instance, VCN VR105, and gateways associated with VCN104. One or more types of gateways may be associated with VCN104. A gateway is an interface between a VCN and another endpoint, where the another endpoint is outside the VCN. A gateway is a Layer-3/IP layer concept and enables a VCN to communicate with endpoints outside the VCN. A gateway thus facilitates traffic flow between a VCN and other VCNs or networks. Various different types of gateways may be configured for a VCN to facilitate different types of communications with different types of endpoints. Depending upon the gateway, the communications may be over public networks (e.g., the Internet) or over private networks. Various communication protocols may be used for these communications.

For example, compute instance C1 may want to communicate with an endpoint outside VCN104. The packet may be first processed by the VNIC associated with source compute instance C1. The VNIC processing determines that the destination for the packet is outside the Subnet-1 of C1. The VNIC associated with C1 may forward the packet to VCN VR105for VCN104. VCN VR105then processes the packet and as part of the processing, based upon the destination for the packet, determines a particular gateway associated with VCN104as the next hop for the packet. VCN VR105may then forward the packet to the particular identified gateway. For example, if the destination is an endpoint within the customer's on-premise network, then the packet may be forwarded by VCN VR105to Dynamic Routing Gateway (DRG) gateway122configured for VCN104. The packet may then be forwarded from the gateway to a next hop to facilitate communication of the packet to it final intended destination.

Various different types of gateways may be configured for a VCN. Examples of gateways that may be configured for a VCN are depicted inFIG.1and described below. Examples of gateways associated with a VCN are also depicted inFIGS.14,15,16, and17(for example, gateways referenced by reference numbers1434,1436,1438,1534,1536,1538,1634,1636,1638,1734,1736, and1738) and described below. As shown in the embodiment depicted inFIG.1, a Dynamic Routing Gateway (DRG)122may be added to or be associated with customer VCN104and provides a path for private network traffic communication between customer VCN104and another endpoint, where the another endpoint can be the customer's on-premise network116, a VCN108in a different region of CSPI101, or other remote cloud networks118not hosted by CSPI101. Customer on-premise network116may be a customer network or a customer data center built using the customer's resources. Access to customer on-premise network116is generally very restricted. For a customer that has both a customer on-premise network116and one or more VCNs104deployed or hosted in the cloud by CSPI101, the customer may want their on-premise network116and their cloud-based VCN104to be able to communicate with each other. This enables a customer to build an extended hybrid environment encompassing the customer's VCN104hosted by CSPI101and their on-premises network116. DRG122enables this communication. To enable such communications, a communication channel124is set up where one endpoint of the channel is in customer on-premise network116and the other endpoint is in CSPI101and connected to customer VCN104. Communication channel124can be over public communication networks such as the Internet or private communication networks. Various different communication protocols may be used such as IPsec VPN technology over a public communication network such as the Internet, Oracle's FastConnect technology that uses a private network instead of a public network, and others. The device or equipment in customer on-premise network116that forms one end point for communication channel124is referred to as the customer premise equipment (CPE), such as CPE126depicted inFIG.1. On the CSPI101side, the endpoint may be a host machine executing DRG122.

In certain embodiments, a Remote Peering Connection (RPC) can be added to a DRG, which allows a customer to peer one VCN with another VCN in a different region. Using such an RPC, customer VCN104can use DRG122to connect with a VCN108in another region. DRG122may also be used to communicate with other remote cloud networks118, not hosted by CSPI101such as a Microsoft Azure cloud, Amazon AWS cloud, and others.

As shown inFIG.1, an Internet Gateway (IGW)120may be configured for customer VCN104the enables a compute instance on VCN104to communicate with public endpoints114accessible over a public network such as the Internet. IGW120is a gateway that connects a VCN to a public network such as the Internet. IGW120enables a public subnet (where the resources in the public subnet have public overlay IP addresses) within a VCN, such as VCN104, direct access to public endpoints112on a public network114such as the Internet. Using IGW120, connections can be initiated from a subnet within VCN104or from the Internet.

A Network Address Translation (NAT) gateway128can be configured for customer's VCN104and enables cloud resources in the customer's VCN, which do not have dedicated public overlay IP addresses, access to the Internet and it does so without exposing those resources to direct incoming Internet connections (e.g., L4-L7 connections). This enables a private subnet within a VCN, such as private Subnet-1 in VCN104, with private access to public endpoints on the Internet. In NAT gateways, connections can be initiated only from the private subnet to the public Internet and not from the Internet to the private subnet.

In certain embodiments, a Service Gateway (SGW)126can be configured for customer VCN104and provides a path for private network traffic between VCN104and supported services endpoints in a service network110. In certain embodiments, service network110may be provided by the CSP and may provide various services. An example of such a service network is Oracle's Services Network, which provides various services that can be used by customers. For example, a compute instance (e.g., a database system) in a private subnet of customer VCN104can back up data to a service endpoint (e.g., Object Storage) without needing public IP addresses or access to the Internet. In certain embodiments, a VCN can have only one SGW, and connections can only be initiated from a subnet within the VCN and not from service network110. If a VCN is peered with another, resources in the other VCN typically cannot access the SGW. Resources in on-premises networks that are connected to a VCN with FastConnect or VPN Connect can also use the service gateway configured for that VCN.

In certain implementations, SGW126uses the concept of a service Classless Inter-Domain Routing (CIDR) label, which is a string that represents all the regional public IP address ranges for the service or group of services of interest. The customer uses the service CIDR label when they configure the SGW and related route rules to control traffic to the service. The customer can optionally utilize it when configuring security rules without needing to adjust them if the service's public IP addresses change in the future.

A Local Peering Gateway (LPG)132is a gateway that can be added to customer VCN104and enables VCN104to peer with another VCN in the same region. Peering means that the VCNs communicate using private IP addresses, without the traffic traversing a public network such as the Internet or without routing the traffic through the customer's on-premises network116. In preferred embodiments, a VCN has a separate LPG for each peering it establishes. Local Peering or VCN Peering is a common practice used to establish network connectivity between different applications or infrastructure management functions.

Service providers, such as providers of services in service network110, may provide access to services using different access models. According to a public access model, services may be exposed as public endpoints that are publicly accessible by compute instance in a customer VCN via a public network such as the Internet and or may be privately accessible via SGW126. According to a specific private access model, services are made accessible as private IP endpoints in a private subnet in the customer's VCN. This is referred to as a Private Endpoint (PE) access and enables a service provider to expose their service as an instance in the customer's private network. A Private Endpoint resource represents a service within the customer's VCN. Each PE manifests as a VNIC (referred to as a PE-VNIC, with one or more private IPs) in a subnet chosen by the customer in the customer's VCN. A PE thus provides a way to present a service within a private customer VCN subnet using a VNIC. Since the endpoint is exposed as a VNIC, all the features associated with a VNIC such as routing rules, security lists, etc., are now available for the PE VNIC.

A service provider can register their service to enable access through a PE. The provider can associate policies with the service that restricts the service's visibility to the customer tenancies. A provider can register multiple services under a single virtual IP address (VIP), especially for multi-tenant services. There may be multiple such private endpoints (in multiple VCNs) that represent the same service.

Compute instances in the private subnet can then use the PE VNIC's private IP address or the service DNS name to access the service. Compute instances in the customer VCN can access the service by sending traffic to the private IP address of the PE in the customer VCN. A Private Access Gateway (PAGW)130is a gateway resource that can be attached to a service provider VCN (e.g., a VCN in service network110) that acts as an ingress/egress point for all traffic from/to customer subnet private endpoints. PAGW130enables a provider to scale the number of PE connections without utilizing its internal IP address resources. A provider needs only configure one PAGW for any number of services registered in a single VCN. Providers can represent a service as a private endpoint in multiple VCNs of one or more customers. From the customer's perspective, the PE VNIC, which, instead of being attached to a customer's instance, appears attached to the service with which the customer wishes to interact. The traffic destined to the private endpoint is routed via PAGW130to the service. These are referred to as customer-to-service private connections (C2S connections).

The PE concept can also be used to extend the private access for the service to customer's on-premises networks and data centers, by allowing the traffic to flow through FastConnect/IPsec links and the private endpoint in the customer VCN. Private access for the service can also be extended to the customer's peered VCNs, by allowing the traffic to flow between LPG132and the PE in the customer's VCN.

A customer can control routing in a VCN at the subnet level, so the customer can specify which subnets in the customer's VCN, such as VCN104, use each gateway. A VCN's route tables are used to decide if traffic is allowed out of a VCN through a particular gateway. For example, in a particular instance, a route table for a public subnet within customer VCN104may send non-local traffic through IGW120. The route table for a private subnet within the same customer VCN104may send traffic destined for CSP services through SGW126. All remaining traffic may be sent via the NAT gateway128. Route tables only control traffic going out of a VCN.

Security lists associated with a VCN are used to control traffic that comes into a VCN via a gateway via inbound connections. All resources in a subnet use the same route table and security lists. Security lists may be used to control specific types of traffic allowed in and out of instances in a subnet of a VCN. Security list rules may comprise ingress (inbound) and egress (outbound) rules. For example, an ingress rule may specify an allowed source address range, while an egress rule may specify an allowed destination address range. Security rules may specify a particular protocol (e.g., TCP, ICMP), a particular port (e.g., 22 for SSH, 3389 for Windows RDP), etc. In certain implementations, an instance's operating system may enforce its own firewall rules that are aligned with the security list rules. Rules may be stateful (e.g., a connection is tracked, and the response is automatically allowed without an explicit security list rule for the response traffic) or stateless.

Access from a customer VCN (i.e., by a resource or compute instance deployed on VCN104) can be categorized as public access, private access, or dedicated access. Public access refers to an access model where a public IP address or a NAT is used to access a public endpoint. Private access enables customer workloads in VCN104with private IP addresses (e.g., resources in a private subnet) to access services without traversing a public network such as the Internet. In certain embodiments, CSPI101enables customer VCN workloads with private IP addresses to access the (public service endpoints of) services using a service gateway. A service gateway thus offers a private access model by establishing a virtual link between the customer's VCN and the service's public endpoint residing outside the customer's private network.

Additionally, CSPI may offer dedicated public access using technologies such as FastConnect public peering where customer on-premises instances can access one or more services in a customer VCN using a FastConnect connection and without traversing a public network such as the Internet. CSPI also may also offer dedicated private access using FastConnect private peering where customer on-premises instances with private IP addresses can access the customer's VCN workloads using a FastConnect connection. FastConnect is a network connectivity alternative to using the public Internet to connect a customer's on-premise network to CSPI and its services. FastConnect provides an easy, elastic, and economical way to create a dedicated and private connection with higher bandwidth options and a more reliable and consistent networking experience when compared to Internet-based connections.

FIG.1and the accompanying description above describes various virtualized components in an example virtual network. As described above, the virtual network is built on the underlying physical or substrate network.FIG.2depicts a simplified architectural diagram of the physical components in the physical network within CSPI200that provide the underlay for the virtual network according to certain embodiments. As shown, CSPI200provides a distributed environment comprising components and resources (e.g., compute, memory, and networking resources) provided by a cloud service provider (CSP). These components and resources are used to provide cloud services (e.g., IaaS services) to subscribing customers, i.e., customers that have subscribed to one or more services provided by the CSP. Based upon the services subscribed to by a customer, a subset of resources (e.g., compute, memory, and networking resources) of CSPI200are provisioned for the customer. Customers can then build their own cloud-based (i.e., CSPI-hosted) customizable and private virtual networks using physical compute, memory, and networking resources provided by CSPI200. As previously indicated, these customer networks are referred to as virtual cloud networks (VCNs). A customer can deploy one or more customer resources, such as compute instances, on these customer VCNs. Compute instances can be in the form of virtual machines, bare metal instances, and the like. CSPI200provides infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services in a highly available hosted environment.

In the example embodiment depicted inFIG.2, the physical components of CSPI200include one or more physical host machines or physical servers (e.g.,202,206,208), network virtualization devices (NVDs) (e.g.,210,212), top-of-rack (TOR) switches (e.g.,214,216), and a physical network (e.g.,218), and switches in physical network218. The physical host machines or servers may host and execute various compute instances that participate in one or more subnets of a VCN. The compute instances may include virtual machine instances, and bare metal instances. For example, the various compute instances depicted inFIG.1may be hosted by the physical host machines depicted inFIG.2. The virtual machine compute instances in a VCN may be executed by one host machine or by multiple different host machines. The physical host machines may also host virtual host machines, container-based hosts or functions, and the like. The VNICs and VCN VR depicted inFIG.1may be executed by the NVDs depicted inFIG.2. The gateways depicted inFIG.1may be executed by the host machines and/or by the NVDs depicted inFIG.2.

The host machines or servers may execute a hypervisor (also referred to as a virtual machine monitor or VMM) that creates and enables a virtualized environment on the host machines. The virtualization or virtualized environment facilitates cloud-based computing. One or more compute instances may be created, executed, and managed on a host machine by a hypervisor on that host machine. The hypervisor on a host machine enables the physical computing resources of the host machine (e.g., compute, memory, and networking resources) to be shared between the various compute instances executed by the host machine.

For example, as depicted inFIG.2, host machines202and208execute hypervisors260and266, respectively. These hypervisors may be implemented using software, firmware, or hardware, or combinations thereof. Typically, a hypervisor is a process or a software layer that sits on top of the host machine's operating system (OS), which in turn executes on the hardware processors of the host machine. The hypervisor provides a virtualized environment by enabling the physical computing resources (e.g., processing resources such as processors/cores, memory resources, networking resources) of the host machine to be shared among the various virtual machine compute instances executed by the host machine. For example, inFIG.2, hypervisor260may sit on top of the OS of host machine202and enables the computing resources (e.g., processing, memory, and networking resources) of host machine202to be shared between compute instances (e.g., virtual machines) executed by host machine202. A virtual machine can have its own operating system (referred to as a guest operating system), which may be the same as or different from the OS of the host machine. The operating system of a virtual machine executed by a host machine may be the same as or different from the operating system of another virtual machine executed by the same host machine. A hypervisor thus enables multiple operating systems to be executed alongside each other while sharing the same computing resources of the host machine. The host machines depicted inFIG.2may have the same or different types of hypervisors.

A compute instance can be a virtual machine instance or a bare metal instance. InFIG.2, compute instances268on host machine202and274on host machine208are examples of virtual machine instances. Host machine206is an example of a bare metal instance that is provided to a customer.

In certain instances, an entire host machine may be provisioned to a single customer, and all of the one or more compute instances (either virtual machines or bare metal instance) hosted by that host machine belong to that same customer. In other instances, a host machine may be shared between multiple customers (i.e., multiple tenants). In such a multi-tenancy scenario, a host machine may host virtual machine compute instances belonging to different customers. These compute instances may be members of different VCNs of different customers. In certain embodiments, a bare metal compute instance is hosted by a bare metal server without a hypervisor. When a bare metal compute instance is provisioned, a single customer or tenant maintains control of the physical CPU, memory, and network interfaces of the host machine hosting the bare metal instance and the host machine is not shared with other customers or tenants.

As previously described, each compute instance that is part of a VCN is associated with a VNIC that enables the compute instance to become a member of a subnet of the VCN. The VNIC associated with a compute instance facilitates the communication of packets or frames to and from the compute instance. A VNIC is associated with a compute instance when the compute instance is created. In certain embodiments, for a compute instance executed by a host machine, the VNIC associated with that compute instance is executed by an NVD connected to the host machine. For example, inFIG.2, host machine202executes a virtual machine compute instance268that is associated with VNIC276, and VNIC276is executed by NVD210connected to host machine202. As another example, bare metal instance272hosted by host machine206is associated with VNIC280that is executed by NVD212connected to host machine206. As yet another example, VNIC284is associated with compute instance274executed by host machine208, and VNIC284is executed by NVD212connected to host machine208.

For compute instances hosted by a host machine, an NVD connected to that host machine also executes VCN VRs corresponding to VCNs of which the compute instances are members. For example, in the embodiment depicted inFIG.2, NVD210executes VCN VR277corresponding to the VCN of which compute instance268is a member. NVD212may also execute one or more VCN VRs283corresponding to VCNs corresponding to the compute instances hosted by host machines206and208.

A host machine may include one or more network interface cards (NIC) that enable the host machine to be connected to other devices. A NIC on a host machine may provide one or more ports (or interfaces) that enable the host machine to be communicatively connected to another device. For example, a host machine may be connected to an NVD using one or more ports (or interfaces) provided on the host machine and on the NVD. A host machine may also be connected to other devices such as another host machine.

For example, inFIG.2, host machine202is connected to NVD210using link220that extends between a port234provided by a NIC232of host machine202and between a port236of NVD210. Host machine206is connected to NVD212using link224that extends between a port246provided by a NIC244of host machine206and between a port248of NVD212. Host machine208is connected to NVD212using link226that extends between a port252provided by a NIC250of host machine208and between a port254of NVD212.

The NVDs are in turn connected via communication links to top-of-the-rack (TOR) switches, which are connected to physical network218(also referred to as the switch fabric). In certain embodiments, the links between a host machine and an NVD, and between an NVD and a TOR switch are Ethernet links. For example, inFIG.2, NVDs210and212are connected to TOR switches214and216, respectively, using links228and230. In certain embodiments, the links220,224,226,228, and230are Ethernet links. The collection of host machines and NVDs that are connected to a TOR is sometimes referred to as a rack.

Physical network218provides a communication fabric that enables TOR switches to communicate with each other. Physical network218can be a multi-tiered network. In certain implementations, physical network218is a multi-tiered Clos network of switches, with TOR switches214and216representing the leaf level nodes of the multi-tiered and multi-node physical switching network218. Different Clos network configurations are possible including but not limited to a 2-tier network, a 3-tier network, a 4-tier network, a 5-tier network, and in general a “n”-tiered network. An example of a Clos network is depicted inFIG.5and described below.

Various different connection configurations are possible between host machines and NVDs such as one-to-one configuration, many-to-one configuration, one-to-many configuration, and others. In a one-to-one configuration implementation, each host machine is connected to its own separate NVD. For example, inFIG.2, host machine202is connected to NVD210via NIC232of host machine202. In a many-to-one configuration, multiple host machines are connected to one NVD. For example, inFIG.2, host machines206and208are connected to the same NVD212via NICs244and250, respectively.

In a one-to-many configuration, one host machine is connected to multiple NVDs.FIG.3shows an example within CSPI300where a host machine is connected to multiple NVDs. As shown inFIG.3, host machine302comprises a network interface card (NIC)304that includes multiple ports306and308. Host machine300is connected to a first NVD310via port306and link320and connected to a second NVD312via port308and link322. Ports306and308may be Ethernet ports and the links320and322between host machine302and NVDs310and312may be Ethernet links. NVD310is in turn connected to a first TOR switch314and NVD312is connected to a second TOR switch316. The links between NVDs310and312, and TOR switches314and316may be Ethernet links. TOR switches314and316represent the Tier-0 switching devices in multi-tiered physical network318.

The arrangement depicted inFIG.3provides two separate physical network paths to and from physical switch network318to host machine302: a first path traversing TOR switch314to NVD310to host machine302, and a second path traversing TOR switch316to NVD312to host machine302. The separate paths provide for enhanced availability (referred to as high availability) of host machine302. If there are problems in one of the paths (e.g., a link in one of the paths goes down) or devices (e.g., a particular NVD is not functioning), then the other path may be used for communications to/from host machine302.

In the configuration depicted inFIG.3, the host machine is connected to two different NVDs using two different ports provided by a NIC of the host machine. In other embodiments, a host machine may include multiple NICs that enable connectivity of the host machine to multiple NVDs.

Referring back toFIG.2, an NVD is a physical device or component that performs one or more network and/or storage virtualization functions. An NVD may be any device with one or more processing units (e.g., CPUs, Network Processing Units (NPUs), FPGAs, packet processing pipelines, etc.), memory including cache, and ports. The various virtualization functions may be performed by software/firmware executed by the one or more processing units of the NVD.

An NVD may be implemented in various different forms. For example, in certain embodiments, an NVD is implemented as an interface card referred to as a smartNIC or an intelligent NIC with an embedded processor onboard. A smartNIC is a separate device from the NICs on the host machines. InFIG.2, the NVDs210and212may be implemented as smartNICs that are connected to host machines202, and host machines206and208, respectively.

A smartNIC is however just one example of an NVD implementation. Various other implementations are possible. For example, in some other implementations, an NVD or one or more functions performed by the NVD may be incorporated into or performed by one or more host machines, one or more TOR switches, and other components of CSPI200. For example, an NVD may be embodied in a host machine where the functions performed by an NVD are performed by the host machine. As another example, an NVD may be part of a TOR switch, or a TOR switch may be configured to perform functions performed by an NVD that enables the TOR switch to perform various complex packet transformations that are used for a public cloud. A TOR that performs the functions of an NVD is sometimes referred to as a smart TOR. In yet other implementations, where virtual machines (VMs) instances, but not bare metal (BM) instances, are offered to customers, functions performed by an NVD may be implemented inside a hypervisor of the host machine. In some other implementations, some of the functions of the NVD may be offloaded to a centralized service running on a fleet of host machines.

In certain embodiments, such as when implemented as a smartNIC as shown inFIG.2, an NVD may comprise multiple physical ports that enable it to be connected to one or more host machines and to one or more TOR switches. A port on an NVD can be classified as a host-facing port (also referred to as a “south port”) or a network-facing or TOR-facing port (also referred to as a “north port”). A host-facing port of an NVD is a port that is used to connect the NVD to a host machine. Examples of host-facing ports inFIG.2include port236on NVD210, and ports248and254on NVD212. A network-facing port of an NVD is a port that is used to connect the NVD to a TOR switch. Examples of network-facing ports inFIG.2include port256on NVD210, and port258on NVD212. As shown inFIG.2, NVD210is connected to TOR switch214using link228that extends from port256of NVD210to the TOR switch214. Likewise, NVD212is connected to TOR switch216using link230that extends from port258of NVD212to the TOR switch216.

An NVD receives packets and frames from a host machine (e.g., packets and frames generated by a compute instance hosted by the host machine) via a host-facing port and, after performing the necessary packet processing, may forward the packets and frames to a TOR switch via a network-facing port of the NVD. An NVD may receive packets and frames from a TOR switch via a network-facing port of the NVD and, after performing the necessary packet processing, may forward the packets and frames to a host machine via a host-facing port of the NVD.

In certain embodiments, there may be multiple ports and associated links between an NVD and a TOR switch. These ports and links may be aggregated to form a link aggregator group of multiple ports or links (referred to as a LAG). Link aggregation allows multiple physical links between two end-points (e.g., between an NVD and a TOR switch) to be treated as a single logical link. All the physical links in a given LAG may operate in full-duplex mode at the same speed. LAGs help increase the bandwidth and reliability of the connection between two endpoints. If one of the physical links in the LAG goes down, traffic is dynamically and transparently reassigned to one of the other physical links in the LAG. The aggregated physical links deliver higher bandwidth than each individual link. The multiple ports associated with a LAG are treated as a single logical port. Traffic can be load-balanced across the multiple physical links of a LAG. One or more LAGs may be configured between two endpoints. The two endpoints may be between an NVD and a TOR switch, between a host machine and an NVD, and the like.

An NVD implements or performs network virtualization functions. These functions are performed by software/firmware executed by the NVD. Examples of network virtualization functions include without limitation: packet encapsulation and de-capsulation functions; functions for creating a VCN network; functions for implementing network policies such as VCN security list (firewall) functionality; functions that facilitate the routing and forwarding of packets to and from compute instances in a VCN; and the like. In certain embodiments, upon receiving a packet, an NVD is configured to execute a packet processing pipeline for processing the packet and determining how the packet is to be forwarded or routed. As part of this packet processing pipeline, the NVD may execute one or more virtual functions associated with the overlay network such as executing VNICs associated with compute instances in the VCN, executing a Virtual Router (VR) associated with the VCN, the encapsulation and decapsulation of packets to facilitate forwarding or routing in the virtual network, execution of certain gateways (e.g., the Local Peering Gateway), the implementation of Security Lists, Network Security Groups, network address translation (NAT) functionality (e.g., the translation of Public IP to Private IP on a host by host basis), throttling functions, and other functions.

In certain embodiments, the packet processing data path in an NVD may comprise multiple packet pipelines, each composed of a series of packet transformation stages. In certain implementations, upon receiving a packet, the packet is parsed and classified to a single pipeline. The packet is then processed in a linear fashion, one stage after another, until the packet is either dropped or sent out over an interface of the NVD. These stages provide basic functional packet processing building blocks (e.g., validating headers, enforcing throttle, inserting new Layer-2 headers, enforcing L4 firewall, VCN encapsulation/decapsulation, etc.) so that new pipelines can be constructed by composing existing stages, and new functionality can be added by creating new stages and inserting them into existing pipelines.

An NVD may perform both control plane and data plane functions corresponding to a control plane and a data plane of a VCN. Examples of a VCN Control Plane are also depicted inFIGS.14,15,16, and17(see references1416,1516,1616, and1716) and described below. Examples of a VCN Data Plane are depicted inFIGS.14,15,16, and17(see references1418,1518,1618, and1718) and described below. The control plane functions include functions used for configuring a network (e.g., setting up routes and route tables, configuring VNICs, etc.) that controls how data is to be forwarded. In certain embodiments, a VCN Control Plane is provided that computes all the overlay-to-substrate mappings centrally and publishes them to the NVDs and to the virtual network edge devices such as various gateways such as the DRG, the SGW, the IGW, etc. Firewall rules may also be published using the same mechanism. In certain embodiments, an NVD only gets the mappings that are relevant for that NVD. The data plane functions include functions for the actual routing/forwarding of a packet based upon configuration set up using control plane. A VCN data plane is implemented by encapsulating the customer's network packets before they traverse the substrate network. The encapsulation/decapsulation functionality is implemented on the NVDs. In certain embodiments, an NVD is configured to intercept all network packets in and out of host machines and perform network virtualization functions.

As indicated above, an NVD executes various virtualization functions including VNICs and VCN VRs. An NVD may execute VNICs associated with the compute instances hosted by one or more host machines connected to the VNIC. For example, as depicted inFIG.2, NVD210executes the functionality for VNIC276that is associated with compute instance268hosted by host machine202connected to NVD210. As another example, NVD212executes VNIC280that is associated with bare metal compute instance272hosted by host machine206and executes VNIC284that is associated with compute instance274hosted by host machine208. A host machine may host compute instances belonging to different VCNs, which belong to different customers, and the NVD connected to the host machine may execute the VNICs (i.e., execute VNICs-relate functionality) corresponding to the compute instances.

An NVD also executes VCN Virtual Routers corresponding to the VCNs of the compute instances. For example, in the embodiment depicted inFIG.2, NVD210executes VCN VR277corresponding to the VCN to which compute instance268belongs. NVD212executes one or more VCN VRs283corresponding to one or more VCNs to which compute instances hosted by host machines206and208belong. In certain embodiments, the VCN VR corresponding to that VCN is executed by all the NVDs connected to host machines that host at least one compute instance belonging to that VCN. If a host machine hosts compute instances belonging to different VCNs, an NVD connected to that host machine may execute VCN VRs corresponding to those different VCNs.

In addition to VNICs and VCN VRs, an NVD may execute various software (e.g., daemons) and include one or more hardware components that facilitate the various network virtualization functions performed by the NVD. For purposes of simplicity, these various components are grouped together as “packet processing components” shown inFIG.2. For example, NVD210comprises packet processing components286and NVD212comprises packet processing components288. For example, the packet processing components for an NVD may include a packet processor that is configured to interact with the NVD's ports and hardware interfaces to monitor all packets received by and communicated using the NVD and store network information. The network information may, for example, include network flow information identifying different network flows handled by the NVD and per flow information (e.g., per flow statistics). In certain embodiments, network flows information may be stored on a per VNIC basis. The packet processor may perform packet-by-packet manipulations as well as implement stateful NAT and L4 firewall (FW). As another example, the packet processing components may include a replication agent that is configured to replicate information stored by the NVD to one or more different replication target stores. As yet another example, the packet processing components may include a logging agent that is configured to perform logging functions for the NVD. The packet processing components may also include software for monitoring the performance and health of the NVD and, also possibly of monitoring the state and health of other components connected to the NVD.

FIG.1shows the components of an example virtual or overlay network including a VCN, subnets within the VCN, compute instances deployed on subnets, VNICs associated with the compute instances, a VR for a VCN, and a set of gateways configured for the VCN. The overlay components depicted inFIG.1may be executed or hosted by one or more of the physical components depicted inFIG.2. For example, the compute instances in a VCN may be executed or hosted by one or more host machines depicted inFIG.2. For a compute instance hosted by a host machine, the VNIC associated with that compute instance is typically executed by an NVD connected to that host machine (i.e., the VNIC functionality is provided by the NVD connected to that host machine). The VCN VR function for a VCN is executed by all the NVDs that are connected to host machines hosting or executing the compute instances that are part of that VCN. The gateways associated with a VCN may be executed by one or more different types of NVDs. For example, certain gateways may be executed by smartNICs, while others may be executed by one or more host machines or other implementations of NVDs.

As described above, a compute instance in a customer VCN may communicate with various different endpoints, where the endpoints can be within the same subnet as the source compute instance, in a different subnet but within the same VCN as the source compute instance, or with an endpoint that is outside the VCN of the source compute instance. These communications are facilitated using VNICs associated with the compute instances, the VCN VRs, and the gateways associated with the VCNs.

For communications between two compute instances on the same subnet in a VCN, the communication is facilitated using VNICs associated with the source and destination compute instances. The source and destination compute instances may be hosted by the same host machine or by different host machines. A packet originating from a source compute instance may be forwarded from a host machine hosting the source compute instance to an NVD connected to that host machine. On the NVD, the packet is processed using a packet processing pipeline, which can include execution of the VNIC associated with the source compute instance. Since the destination endpoint for the packet is within the same subnet, execution of the VNIC associated with the source compute instance results in the packet being forwarded to an NVD executing the VNIC associated with the destination compute instance, which then processes and forwards the packet to the destination compute instance. The VNICs associated with the source and destination compute instances may be executed on the same NVD (e.g., when both the source and destination compute instances are hosted by the same host machine) or on different NVDs (e.g., when the source and destination compute instances are hosted by different host machines connected to different NVDs). The VNICs may use routing/forwarding tables stored by the NVD to determine the next hop for the packet.

For a packet to be communicated from a compute instance in a subnet to an endpoint in a different subnet in the same VCN, the packet originating from the source compute instance is communicated from the host machine hosting the source compute instance to the NVD connected to that host machine. On the NVD, the packet is processed using a packet processing pipeline, which can include execution of one or more VNICs, and the VR associated with the VCN. For example, as part of the packet processing pipeline, the NVD executes or invokes functionality corresponding to the VNIC (also referred to as executes the VNIC) associated with source compute instance. The functionality performed by the VNIC may include looking at the VLAN tag on the packet. Since the packet's destination is outside the subnet, the VCN VR functionality is next invoked and executed by the NVD. The VCN VR then routes the packet to the NVD executing the VNIC associated with the destination compute instance. The VNIC associated with the destination compute instance then processes the packet and forwards the packet to the destination compute instance. The VNICs associated with the source and destination compute instances may be executed on the same NVD (e.g., when both the source and destination compute instances are hosted by the same host machine) or on different NVDs (e.g., when the source and destination compute instances are hosted by different host machines connected to different NVDs).

If the destination for the packet is outside the VCN of the source compute instance, then the packet originating from the source compute instance is communicated from the host machine hosting the source compute instance to the NVD connected to that host machine. The NVD executes the VNIC associated with the source compute instance. Since the destination end point of the packet is outside the VCN, the packet is then processed by the VCN VR for that VCN. The NVD invokes the VCN VR functionality, which may result in the packet being forwarded to an NVD executing the appropriate gateway associated with the VCN. For example, if the destination is an endpoint within the customer's on-premise network, then the packet may be forwarded by the VCN VR to the NVD executing the DRG gateway configured for the VCN. The VCN VR may be executed on the same NVD as the NVD executing the VNIC associated with the source compute instance or by a different NVD. The gateway may be executed by an NVD, which may be a smartNIC, a host machine, or other NVD implementation. The packet is then processed by the gateway and forwarded to a next hop that facilitates communication of the packet to its intended destination endpoint. For example, in the embodiment depicted inFIG.2, a packet originating from compute instance268may be communicated from host machine202to NVD210over link220(using NIC232). On NVD210, VNIC276is invoked since it is the VNIC associated with source compute instance268. VNIC276is configured to examine the encapsulated information in the packet and determine a next hop for forwarding the packet with the goal of facilitating communication of the packet to its intended destination endpoint, and then forward the packet to the determined next hop.

A compute instance deployed on a VCN can communicate with various different endpoints. These endpoints may include endpoints that are hosted by CSPI200and endpoints outside CSPI200. Endpoints hosted by CSPI200may include instances in the same VCN or other VCNs, which may be the customer's VCNs, or VCNs not belonging to the customer. Communications between endpoints hosted by CSPI200may be performed over physical network218. A compute instance may also communicate with endpoints that are not hosted by CSPI200, or are outside CSPI200. Examples of these endpoints include endpoints within a customer's on-premise network or data center, or public endpoints accessible over a public network such as the Internet. Communications with endpoints outside CSPI200may be performed over public networks (e.g., the Internet) (not shown inFIG.2) or private networks (not shown inFIG.2) using various communication protocols.

The architecture of CSPI200depicted inFIG.2is merely an example and is not intended to be limiting. Variations, alternatives, and modifications are possible in alternative embodiments. For example, in some implementations, CSPI200may have more or fewer systems or components than those shown inFIG.2, may combine two or more systems, or may have a different configuration or arrangement of systems. The systems, subsystems, and other components depicted inFIG.2may be implemented in software (e.g., code, instructions, program) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or combinations thereof. The software may be stored on a non-transitory storage medium (e.g., on a memory device).

FIG.4depicts connectivity between a host machine and an NVD for providing I/O virtualization for supporting multitenancy according to certain embodiments. As depicted inFIG.4, host machine402executes a hypervisor404that provides a virtualized environment. Host machine402executes two virtual machine instances, VM1406belonging to customer/tenant #1 and VM2408belonging to customer/tenant #2. Host machine402comprises a physical NIC410that is connected to an NVD412via link414. Each of the compute instances is attached to a VNIC that is executed by NVD412. In the embodiment inFIG.4, VM1406is attached to VNIC-VM1420and VM2408is attached to VNIC-VM2422.

As shown inFIG.4, NIC410comprises two logical NICs, logical NIC A416and logical NIC B418. Each virtual machine is attached to and configured to work with its own logical NIC. For example, VM1406is attached to logical NIC A416and VM2408is attached to logical NIC B418. Even though host machine402comprises only one physical NIC410that is shared by the multiple tenants, due to the logical NICs, each tenant's virtual machine believes they have their own host machine and NIC.

In certain embodiments, each logical NIC is assigned its own VLAN ID. Thus, a specific VLAN ID is assigned to logical NIC A416for Tenant #1 and a separate VLAN ID is assigned to logical NIC B418for Tenant #2. When a packet is communicated from VM1406, a tag assigned to Tenant #1 is attached to the packet by the hypervisor and the packet is then communicated from host machine402to NVD412over link414. In a similar manner, when a packet is communicated from VM2408, a tag assigned to Tenant #2 is attached to the packet by the hypervisor and the packet is then communicated from host machine402to NVD412over link414. Accordingly, a packet424communicated from host machine402to NVD412has an associated tag426that identifies a specific tenant and associated VM. On the NVD, for a packet424received from host machine402, the tag426associated with the packet is used to determine whether the packet is to be processed by VNIC-VM1420or by VNIC-VM2422. The packet is then processed by the corresponding VNIC. The configuration depicted inFIG.4enables each tenant's compute instance to believe that they own their own host machine and NIC. The setup depicted inFIG.4provides for I/O virtualization for supporting multi-tenancy.

FIG.5depicts a simplified block diagram of a physical network500according to certain embodiments. The embodiment depicted inFIG.5is structured as a Clos network. A Clos network is a particular type of network topology designed to provide connection redundancy while maintaining high bisection bandwidth and maximum resource utilization. A Clos network is a type of non-blocking, multistage or multi-tiered switching network, where the number of stages or tiers can be two, three, four, five, etc. The embodiment depicted inFIG.5is a 3-tiered network comprising tiers 1, 2, and 3. The TOR switches504represent Tier-0 switches in the Clos network. One or more NVDs are connected to the TOR switches. Tier-0 switches are also referred to as edge devices of the physical network. The Tier-0 switches are connected to Tier-1 switches, which are also referred to as leaf switches. In the embodiment depicted inFIG.5, a set of “n” Tier-0 TOR switches are connected to a set of “n” Tier-1 switches and together form a pod. Each Tier-0 switch in a pod is interconnected to all the Tier-1 switches in the pod, but there is no connectivity of switches between pods. In certain implementations, two pods are referred to as a block. Each block is served by or connected to a set of “n” Tier-2 switches (sometimes referred to as spine switches). There can be several blocks in the physical network topology. The Tier-2 switches are in turn connected to “n” Tier-3 switches (sometimes referred to as super-spine switches). Communication of packets over physical network500is typically performed using one or more Layer-3 communication protocols. Typically, all the layers of the physical network, except for the TORs layer are n-ways redundant thus allowing for high availability. Policies may be specified for pods and blocks to control the visibility of switches to each other in the physical network so as to enable scaling of the physical network.

A feature of a Clos network is that the maximum hop count to reach from one Tier-0 switch to another Tier-0 switch (or from an NVD connected to a Tier-0-switch to another NVD connected to a Tier-0 switch) is fixed. For example, in a 3-Tiered Clos network at most seven hops are needed for a packet to reach from one NVD to another NVD, where the source and target NVDs are connected to the leaf tier of the Clos network. Likewise, in a 4-tiered Clos network, at most nine hops are needed for a packet to reach from one NVD to another NVD, where the source and target NVDs are connected to the leaf tier of the Clos network. Thus, a Clos network architecture maintains consistent latency throughout the network, which is important for communication within and between data centers. A Clos topology scales horizontally and is cost effective. The bandwidth/throughput capacity of the network can be easily increased by adding more switches at the various tiers (e.g., more leaf and spine switches) and by increasing the number of links between the switches at adjacent tiers.

In certain embodiments, each resource within CSPI is assigned a unique identifier called a Cloud Identifier (CID). This identifier is included as part of the resource's information and can be used to manage the resource, for example, via a Console or through APIs. An example syntax for a CID is:ocid1.<RESOURCE TYPE>.<REALM>. [REGION][.FUTURE USE].<UNIQUE ID> where,ocid1: The literal string indicating the version of the CID;resource type: The type of resource (for example, instance, volume, VCN, subnet, user, group, and so on);realm: The realm the resource is in. Example values are “c1” for the commercial realm, “c2” for the Government Cloud realm, or “c3” for the Federal Government Cloud realm, etc. Each realm may have its own domain name;region: The region the resource is in. If the region is not applicable to the resource, this part might be blank;future use: Reserved for future use.unique ID: The unique portion of the ID. The format may vary depending on the type of resource or service.

Network Path Performance Measurements Utilizing Multi-Layer Tunneling Techniques

FIG.6Ais a simplified diagram of an example distributed environment600incorporating a network performance analysis system according to certain embodiments. In the embodiment depicted inFIG.6A, distributed environment600includes a number of networked components including compute nodes601,603,605,607,609, and611, indicated with shaded circles. Distributed environment600also includes router nodes602,604, and606, indicated with unshaded circles. InFIG.6A, nodes are connected by lines representing physical or logical connections between network nodes. A portion of the network between two nodes may be referred to as a network segment. A network path is a sequence of one or more network segments connecting two nodes. For instance, one possible network path between nodes601and604includes the network segments connecting compute node601and compute node603, and compute node603and router node604. There may be one or more network paths between two nodes in a network connected by at least one network segment. For instance, another path between nodes601and604is the single network segment connecting compute node601and router node604.

The nodes depicted inFIG.6Acan be located at different geographical locations. InFIG.6A, each node is labeled with a three-lettered identifier identifying a physical geographic location of the node. For example, compute node603is labeled with “PHX” which may correspond to Phoenix, Arizona. Similarly, compute node607is labeled with “LHR” which may correspond to London, England. The geographic dispersion applies to router nodes as well. Router node604is labeled with “ORD” which may correspond to Chicago, Illinois and router node606is labeled with LGA which may correspond to New York, New York. These examples serve to illustrate network nodes that are physically connected and separated by hundreds or thousands of miles.

In certain implementations, the compute nodes and router nodes depicted inFIG.6Amay be as part of cloud infrastructure (CSPI) provided by a CSP for providing one or more cloud services. The compute nodes may represent computer systems at various data centers provided by the CSP. For example, compute node601may correspond to a data center located in Seattle, compute node603may correspond to a data center located in Phoenix, compute node605may correspond to a data center in Washington DC, compute node607may correspond to a data center in London, compute node609may correspond to a data center in Amsterdam, and compute node611may correspond to a data center in France. Compute nodes601,603,605,607,609, and611may be referred to, in certain embodiments, using names like “bouncer1” and “bouncer2.” Compute nodes601,603,605,607,609, and611may represent computer systems with processors, memory devices, storage devices, caches, I/O devices, etc. Compute nodes601,603,605,607,609, and611may include components that are implemented by hardware, software, or a combination of both. In some examples, compute nodes601,603,605,607,609, and611are general-purpose computing devices.

Compute nodes601,603,605,607,609, and611may execute arbitrary program code including program code written in scripting languages like bash, batch, or PowerShell or programming languages like C, Python, and Java. Among other capabilities, compute nodes601,603,605,607,609, and611may create packets, inject packets into the network, perform operations on packet payloads, determine times, perform arbitrary calculations, etc. A compute node in a data center can be the source or origin of network traffic (e.g., data packets or probe packets) that is destined for another compute node, potentially in a different data center. Compute nodes601,603,605,607,609, and611may be configured to perform packet routing and forwarding functions to facilitate network traffic originating with compute nodes to compute nodes that are the intended destinations of the traffic.

Router nodes602,604, and606depicted inFIG.6Amay be part of a communication network that provides connectivity between the different data centers represented by the compute nodes. The communication network may be part of the CSPI provided by a CSP. The router nodes are configured to perform packet routing and forwarding functions to facilitate network traffic originating with compute nodes to compute nodes that are the intended destinations of the traffic. In contrast to the compute nodes, router nodes602,604, and606, the router nodes are not configured to be the sources or destinations of network traffic. For example, a router node is not configured to inject packets, determine times associated with the packets, or perform latency or other metrics calculations based upon the packets.

Router nodes602,604, and606may be implemented by hardware, software, or a combination of both. Examples of router nodes include routers, forwarders, virtual networking entities such as different types of gateways, and the like. Router nodes602,604, and606may be referred to, in certain embodiments, using names like “bb-core1” and “bb-core2.”

FIG.6Aalso depicts a network performance analysis system (NPAS)620that is configured to perform the various functions and methods described in the disclosure. As shown inFIG.6A, NPAS620may be provided by the CSP as part of the CSPI. NPAS620enables network performance functions to be performed. For example, as described herein, NPAS620may be configured to monitor the compute nodes and router nodes depicted inFIG.6Aand computes metrics for the various network segments depicted inFIG.6Ausing the teachings described herein. For example, NPAS620may enable setting up of tunnels with router node terminations endpoints and enable the configuration of label-to-port mapping information for individual router nodes. NPAS620may also provide tools (e.g., GUIs) that enable users of NPAS620to configure headers of probe packets to cause the probe packets to traverse specific paths in the network, measure or acquire timing information regarding when a probe packet is sent from a source compute node and received by a destination compute node. NPAS620may use these and other probe packets-related timing measurements to calculate one or more metrics for one or more network segments, where a network segment can be between two router nodes or between a compute node and a router node. In certain implementations, the NPAS may generate reports showing various network segments and one or more performance metrics computed and associated with the network segments.

NPAS620may be implemented using hardware, software, or combinations thereof. The software may be in the form of code or instructions that are executable by one or more processors. The code or instructions may be stored on a non-transitory computer readable storage medium. In certain implementations, NPAS620may have a distributed architecture with components located in the various data centers.

FIG.6Bis a simplified diagram of the distributed environment600ofFIG.6Awith additional details included for a portion of the distributed environment600. Detail is added to the portion of the distributed environment600included in the bounded area represented by the line613. However, selection of this portion of the distributed environment600for discussion is for illustrative purposes only and similar principles apply to any portion of the distributed environment600that may be selected. The term “portion” in this example refers only to a subset of the compute and node nodes depicted in distributed environment600and not to network configuration details. The portion613of the distributed environment600includes compute nodes603and607and router nodes604and606. All nodes in the portion613may have at least one assigned IP address. For example, compute nodes603and607have assigned IP addresses615and621, respectively. Likewise, router nodes604and606have assigned IP addresses617and619, respectively. Assigned IP addresses may be any suitable assignment consistent with the IP protocol. In this example, IP addresses615and621are part of a logical private subnet and IP addresses617and619are part of a different logical private subnet. However, these assignments are merely illustrative. Network components may be on different subnets or the same subset, or have public or private IP addresses, provided a suitable route exists between the nodes according to the network topology.

Router nodes604and606also include locally-relevant label-to-port mappings between labels and egress ports. For example, router node604includes label-to-port mappings623and625, and router node606includes label-to-port mapping627. Each network segment connecting a router node to another node, whether a compute node or another router node, may have one or more such label-to-port mappings. There may be at most one such label-to-port mapping between a given label and a particular egress port. In other words, each label-to-port mapping must be locally unique. Two labels cannot map to the same egress port and one label cannot map to two egress ports. Locally-relevant means that label-to-port mappings have no relevance outside the router node to which they are assigned. In this example, the label-to-port mappings are labeled according to the node that is physically connected to the given egress port. For example, in label-to-port mapping623, “LABEL C” maps to “PORT PHX,” which refers to the physical connection between router node604and compute node603. Both of these identifiers used in label-to-port mapping623are merely illustrative. Both the label and corresponding egress port may be any suitable identifier according to the implementation of the label routing scheme. For example, an example commercial router configuration that maps a label to a port using MPLS is:

In this example, the label “700002” maps to the port “xe-0/0/1.”

In some examples, the labels may be allocated from a globally-designated static label range maintained by an NPAS. For example, the distributed environment600could be configured by the NPAS with assigned labels from the range 700000-799999. Maintenance of a network map including globally-designated static labels is needed to determine network paths for performance measurements, and to ensure complete coverage of the network by performance measuring. Additionally, compute nodes may store an index of the allocated labels so that the desired network path through the router nodes can be defined in examples in which the probe packet is created by the compute nodes.

FIG.6Cis a simplified diagram of the distributed environment600ofFIG.6Awith additional details included for a portion613of the distributed environment600.FIG.6Cillustrates a router node604with multiple physical connections to compute node603. Router node604has label-to-port mappings629that include 2 labels and 2 egress ports. 2 egress ports/physical connections may be provided, for example, to ensure network redundancy in the event of a failure affecting only one egress port or physical connection. When measuring network performance, separate probe packets may be constructed that include both label-to-port mappings629so that faults can be isolated to a particular egress port or physical connection. For example, two probe packets may be constructed that define the path from compute node603to router node604and back to compute node603. The two probe packets may be otherwise identical except for the inclusion of label631in the first packet and label633in the second packet. A high network latency or packet loss measurement by one probe packet but not the other may isolate the fault to a particular egress port or physical connection.

FIG.7depicts a simplified flowchart700showing a method performed for controlling the path taken by a packet through a communication network by utilizing multi-layer tunneling techniques and computing one or more metrics for one or more segments of the path according to certain embodiments. The method depicted inFIG.7may be implemented in software (e.g., code, instructions, program) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or combinations thereof. The software may be stored on a non-transitory storage medium (e.g., on a memory device). The method presented inFIG.7and described below is intended to be illustrative and non-limiting. AlthoughFIG.7depicts the various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, the processing may be performed in some different order, or some steps may also be performed in parallel. One or more of the processing steps depicted inFIG.7and described below may be performed or facilitated by NPAS620depicted inFIG.6A.

Conceptually, the processing depicted in flowchart700inFIG.7is divided into two phases, a configuration phase702and a probing phase704. In the configuration phase702, the various nodes in the communication network are identified and are configured in a manner that enables probing to be performed in the probing phase704. In the embodiment depicted inFIG.7, configuration phase702includes processing performed in706,708, and710, and probing phase704includes processing performed in712,714, and716.

In certain implementations, the processing performed in the configuration phase702and probing phase704is facilitated by a network performance analysis system (NPAS), such as NPAS620depicted inFIG.6A. The NPAS can include one or more computer systems and comprise one or more processors that are capable of executing computer instructions. The NPAS may provide and execute one or more programs (e.g., execute computer-readable instructions that are executable by one or more processors of the NPAS) that enable the processing in702and704to be performed. For example, the NPAS may execute computer instructions that cause one or more graphical user interfaces (GUIs) to be displayed by the NPAS. In certain implementations, the NPAS may be co-located with one or more compute nodes. In other implementations, the NPAS may be implemented using one or more servers that may be distributed and communicatively coupled to the network. For example, the NPAS may include cloud computing servers provided by a cloud services provider (CSP) as part of infrastructure provided by the CSP for offering one or more cloud services to subscribing customers. A user of the NPAS may use the tools (e.g., GUIs, command line interfaces) provided by the NPAS to initiate and control the processing performed in702and704.

In702, the configuration phase is depicted. As discussed above, router nodes are not configured to construct probe packets, inject probe packets into the network, or perform calculations necessary for performance metric measurements. In contrast, compute nodes may be, for example, co-located with a datacenter and may therefore perform operations such as constructing probe packets, injecting probe packets into the network, or making calculations necessary for performance metric measurements.

In706, one or more compute nodes and one or more router nodes may be identified for a network to be probed and for performance metrics are to be computed. The NPAS may facilitate the identification of these compute and router nodes. For example, the NPAS may provide various user-selectable options (e.g., commands) to identify the scope of the network to be probed, and to identify the compute and router nodes associated with that network. Further, as part of the processing performed in706, and based upon the compute and router nodes identified in706, the NPAS may also determine the various network segments in the network to be probed, where a network segment is bounded by a compute node and a router node, by two router nodes, or by two compute nodes. These segments can then be probed, and performance metrics computed for the network segments during the probing phase704.

In certain implementations, the NPAS may generate a map of the network, or a representation thereof, based upon the various nodes and segments determined in706. The NPAS may then determine a performance measurement plan (or probing plan) whereby probe packets are injected into the network at particular nodes and the packets are configured to traverse a specific path terminated by compute nodes, where the path includes specific segments of the network for which performance metrics are to be computed. A plan may be devised by NPAS that, during the probing phase704, results in metrics being computed for all the segments in the network. In addition to measurement of network performance, the map of the network can be used to perform fault isolation. In that case, the NPAS may devise a plan that selectively computes metrics over certain network segments in response to indications of network errors or performance degradation in order to identify a malfunctioning router, network segment, configuration, etc.

In708, one or more of the router nodes determined in706may be configured to enable the router nodes to become tunnel termination endpoints. For example, a router node identified in706may be configured such that it becomes a termination endpoint for a tunnel. Various different tunneling protocols may be supported, for example, GRE (e.g., the router node is configured to become a GRE tunnel termination endpoint), MPLS over GRE (MPLSoGRE) (e.g., the router node is configured to be a MPLSoGRE tunnel termination endpoint), and others.

Multiple router nodes may be configured as tunnel termination endpoints in708corresponding to endpoints for multiple tunnels. A tunnel can be configured between a router node and a compute node. For example, a tunnel may be configured between a source compute node, where a probe packet is injected into the network, and a router node. As another example, a tunnel may be configured between a router node and destination compute node, which is the end destination of a probe packet. A tunnel may also be configured between two router nodes. A tunnel can traverse one or more segments of the communication network.

The tunnel termination endpoints configured in706enable the network and end-to-end paths traversed by a packet between compute nodes (e.g., from a source compute node to a destination compute node) to be broken down into smaller components. A probe packet can be configured at the point of ingestion (i.e., at the source compute node) to traverse a specific path from the source compute node to the destination compute node over one or more specific tunnels. The tunnels thus enable the path traversed by a probe packet during the probing phase to be controlled as desired. A tunnel can correspond to a segment of the end-to-end network path traversed by a packet from a source compute node to a destination compute node. By configuring the router nodes as tunnel termination endpoints, the router nodes can now be made the end points of specific tunnels.

In710, one or more router nodes may be configured such that, for a probe packet received by a router node, the outgoing network interface or port used by the router node for communicating the packet from the router node can be controlled. For a router node, this is done by creating a mapping or association between a label and an outgoing interface or port of the router node. In certain implementations, the label is a locally-relevant to the router node, i.e., the label is unique to the router node for which the mapping is created. This allows the same label to be used for a label-to-interface/port mapping in other router nodes. For a router node, one or multiple such label-to-port mappings may be configured. As described below in detail, these labels can then be used for controlling which outgoing port of a router node is used to communicate a probe packet from the router node. This enables the path taken by a probe packet through the network to be precisely controlled and specified when the probe packet is injected into the network at the source compute node.

In certain implementations, the NPAS facilitates the configuration of label-to-port mappings for one or more router nodes. For example, via a GUI displayed by the NPAS, a user can select a router node from the multiple router nodes identified in706or select or input information identifying a label and an interface or port of the selected router node to be associated or mapped to the label. One or multiple such mappings may be configured for a router node. Information related to the mappings may be stored as part of label-to-port mapping information, which may be stored by the router node being configured and also stored by the NPAS. In certain implementations, the label-to-port mapping information may be stored and maintained by a control plane associated with the network.

The NPAS may have access to the label-to-port mapping information for the multiple router nodes that are configured. This information, may for example, be stored in a memory store (e.g., a database) accessible to the NPAS. When a particular router node is selected to be configured in710, the NPAS may determine if any label-to-port mappings have previously already been configured for the router node. This information may be displayed to a user using the NPAS. The user may then determine if a new label-to-port mapping is to be created for the router node. If so, the NPAS may allow the user to select a locally-relevant label and a port of the router node to the mapped to the label. The label-to-port mapping information for the router node may then be updated to include this new mapping. The NPAS may then communicate the label-to-port mappings to the router node being configured such that the information is locally available on the router node. In some examples, configuration of locally-relevant label-to-port mappings on router nodes may be facilitated with one or more protocols including, for example, the Label Distribution Protocol (LDP) or the Tag Distribution Protocol (TDP). The NPAS may store information for the network including information identifying the compute nodes and router nodes in the network, and for each router node, the label-to-port mapping information, if any, for the router node.

After the network and the router nodes have been configured according to the processing performed in702, in probing phase704, based upon the configuration, probe packets can made to traverse a path between two compute nodes (from a source compute node to a destination compute node) where the path traverses multiple segments of the network and the path passes through at least one router node. The path traversed by a probe packet can be controlled (by, for example, a network engineer) by configuring a header of the probe packet. In certain implementations, the header is configured such that it comprises a section for each segment traversed by the path. Each header section corresponding to a network segment can be configured such that it causes the probe packet to traverse a specific segment of the network. In this manner, the entire path traversed by the probe packet from the source compute node to the destination compute node can be controlled via information configured in the header of the probe packet before the probe packet is injected into the network at the source compute node. Time measurements taken when the packet is injected into the network at the source compute node and when the packet is received at the destination compute node can then be used to compute performance metrics for one or more segments of the network paths traversed by a probe packet. In certain use cases, the source compute node and the destination compute node can be the same compute node.

At712, based upon the configurations performed in702(e.g., based upon the tunnel termination endpoints and the label-to-port mappings), a probe packet is configured having a header where the header is configured to cause the probe packet to traverse a specific path in the network from a specific source compute node to a specific destination compute node and the path traverses specific segments of the network. A segment can be between a source compute node and a router node, between two router nodes, or between a router node and a destination compute node. The path traversed by the probe packet traverses multiple segments and one or more router nodes. The node at the start of a segment is referred to as the segment start node and the node at the end of the segment is referred to as the segment end node. Each segment can be characterized by its segment start node and segment end node.

In certain implementations, the probe packet header is configured such that it includes a section for each segment of the path traversed from the source compute node to the destination compute node, where the source compute node is the start node for the first segment and the destination compute node is the end node for the last segment. In certain cases, a segment can correspond to a tunnel and the end node for the segment is a node configured as the tunnel termination endpoint for the tunnel. The header section of the probe packet corresponding to such a tunnel segment can include information identifying the tunnel termination endpoint. In some other use cases, for a segment where a router node is a start node of the segment, the probe packet header may include a section for that segment where the section includes information identifying a label. In such a case, when the probe packet arrives at a router node that is the start node of the segment, the packet is forwarded from the router node using an outbound interface of the router node that is locally mapped to the label identified in the header section. In this manner, the tunnel termination endpoints and label-to-port mappings configured in702can be used to control the path traversed by a probe packet during the probing phase702. Further examples of how a packet header can be configured to cause the probe packet to traverse specific paths are depicted inFIGS.8A,8B,8C,8D,10,11,13A,13B,13C,13D, and13E, and the accompanying description.

In714, as a result of the header configured for the probe packet, the probe packet traverses a path from the source compute node to the destination compute node, in which the path includes multiple segments and at least one router node. In some cases, the source compute node and the destination compute node can be the same compute node. The probe packet is made to traverse specific network segments corresponding to information configured in the probe packet header.

Different communication protocols may be used for communicating probe packets. These include, for example, without limitations, various IP protocols, various tunneling protocols (Internet Protocol Security (IPsec), IP-in-IP, Point-to-Point Tunneling Protocol (PPTP), Layer 2 Tunneling Protocol (L2TP), among others), GRE protocol, MPLS, and others.

In716, based upon timing measurements taken when the probe packet is injected into the network at the source compute node and when the packet is received at the destination compute node, one or more performance metrics can be computed for one or more segments of the network path traversed by the probe packet. In this manner, a performance metric can be measured for a segment of the path, where the segment is between a compute node and a router node (e.g., the segment has a router node as its start node or its end node) or is a segment between two router nodes. Examples of performance metrics that may be computed for a network segment include network latency, packet loss, and jitter, among others.

In addition, the NPAS facilitates processing performed during the probing phase704. For example, the NPAS may provide tools (e.g., GUIs, command-line interfaces) that enable a user of the NPAS to configure the contents of headers of probe packets. For example, the tool may enable the user to select the source and destination compute nodes for the path to be traversed by the probe packet and show the various segments for the path. The NPAS may then enable the user to configure (e.g., specify) a packet header with sections corresponding to the various segments, and for each segment, specify information in the header section for that segment that controls how the packet is to be routed for that segment.

The NPAS may also cause the source compute node and the destination compute node to take timing measurements when the packet in injected into the network at the source compute node and when the packet is received by the destination compute node. The timing measurements may be communicated by the source and destination compute nodes to the NPAS and the NPAS may then compute one or more performance metric values for one or more of the segments based upon the timing measurements.

FIG.8Ais a simplified diagram of an example communications network800similar to the portion of the distributed environment600bounded by the line613in distributed environment600ofFIGS.6A-C. The communications network800is chosen for illustrative purposes and the principles and concepts discussed herein apply equally to all portions of the distributed environment600. InFIG.8A, a probe packet traversing a round-trip network path802including a compute node and a router node is depicted.

The round-trip network path802includes the network segment804. A probe packet may be constructed at compute node603that defines the round-trip network path802. In some examples, the probe packet may be constructed by an NPAS and sent to the compute node along with instructions to cause the probe packet to be sent. In some examples, the NPAS may send instructions to compute node603to cause the construction of a probe packet that will be used to make a particular performance measurement. Determination of times the probe packet is injected into the communications network and the time of receipt of the probe packet back at compute node603can allow for direct, first-order measurement of the network latency on network segment804. Likewise, the use of multiple probe packets along round-trip network path802can allow for measurements of packet loss by calculating the ratio of the number of lost probe packets to the total number of sent probe packets. Or the use of multiple probe packets along round-trip network path802can allow for measurements of jitter by storing the values of network latency for multiple packets and calculating the variance or other suitable statistics to measure network jitter.

The compute node603, or NPAS, can generate a probe packet, for example, packet806. Packet806includes both packet headers807and a probe payload809. Packet headers807include header information that define the network path the probe packet will follow. Header information may include one or more header sections. The packet headers807inFIG.8Aand subsequent drawings are a schematic of the information contained in an example packet on a packet-switched network and not intended to be an authentic representation of the data contained in a network packet.

Packet headers807may also include tunneling headers illustrating, for example, an IP tunnel. For example, an IP packet can include an outer IP header, a tunnelling header, and an inner IP header. The inner IP header may be the header of an inner IP packet that includes a data section which includes additional transport-layer information. In some examples, the tunnelling header may be a GRE header, but other tunnelling protocols may be used.

In packet806, the ETH frame header includes an IP packet. An IP packet is a packet configured for routing using the Internet Protocol version 4 (IPv4). The IP packet contains a header and a data section. The IP packet is represented in packet806by an IP header including an IP address indicating the destination of the packet which is included in the header of the IP packet. For example, an IP packet may be represented as an IP header labeled “IP: 198.19.248.30.” The IP packet is followed by a GRE tunnel header, represented in packet806by the letters “GRE.” The GRE header is followed by another inner IP header, represented inFIG.8Aby a different IP address. The inner IP packet also has a header and a data section which includes additional routing information. In this case, the inner IP packet includes a Universal Datagram Protocol (UDP) datagram. The UDP datagram data section also includes the probe payload809. The probe payload809may include information needed to perform performance measurements.

InFIG.8A, the source compute node is compute node603and the destination compute node is likewise compute node603. The network path802begins at compute node603, traverses network segment804to router node604, and then returns to destination compute node603. There is one network segment804in network path802that is traversed twice, once on the outgoing leg and once on the returning leg. The start node of network segment804on the outgoing leg is compute node603and the end node is router node604. Likewise, on the return leg of network path804, the start node is router node604and the end node is compute node603.

The compute node603can inject the probe packet806into the network. In some examples, the probe packet806may be instantiated and configured by an NPAS and send to the compute node603for injection. Based on the outer IP packet in packet header807, the packet806will traverse the network segment804and be received by the router node604. The router node604may remove the outer IP header and the GRE header, using the GRE tunnel termination endpoint, and examine the information contained therein. In this case, the GRE tunnel header precedes another IP packet, the inner IP packet discussed above. Router node604may then route the inner IP packet, now represented by packet808. As with packet806, packet808includes an ETH frame header that includes the inner IP packet. The inner IP packet specifies the IP address of the compute node603. In its data section, the inner IP packet includes the same UDP datagram and probe payload discussed with respect to packet806. Router node604routes the packet808back to the compute node603. Compute node603may then use the information contained in the probe payload809to calculate one or more performance metrics, as will be discussed in detail inFIG.9.

FIG.8Adepicts a round-trip network path802performance measurement made without the use of labels at the router node604. However, the measurement is only possible because of the tunnel termination endpoint configured at router node604. Thus, this illustration shows that the innovative performance measurements over network segments containing a router node and compute node do not require the use of locally-relevant labels at the router node in some cases. However, the use of locally-relevant labels does allow for specification of a particular physical path, which may lead to more accurate or reproducible performance measurements.

FIGS.8B and8Cillustrate performance measurements over the same network segment804utilizing both the tunnel termination endpoint configuration and label-to-port mapping information at the router node. Performance measurements made using the technique ofFIG.8AandFIGS.8B-Cmay yield different results. Unlike the use of label-to-port mapping information inFIGS.8B-C, which correspond to a particular physical connection between the compute node603and the router node604, the IP-switched packets806and808need only traverse the logical network segment between compute node603and router node604. The logical network segment may include any physical network path between the compute node603and the router node604with a suitable IP route according to factors such as the configuration of the routers in the network, network traffic, and other factors. However, as routing protocols are often optimized for efficiency, in practice, performance measurements across single-hop or short network paths may be quite similar using either technique.

FIG.8B, a probe packet traversing a round-trip network path802including a compute node603and a router node604is depicted. InFIG.8B, the source compute node is compute node603and the destination compute node is likewise compute node603. The network path802begins at compute node603, traverses network segment804to router node604, and then returns to destination compute node603. There is one network segment804in network path802that is traversed twice, once on the outgoing leg and once on the returning leg. The start node of network segment804on the outgoing leg is compute node603and the end node is router node604. Likewise, on the return leg of network path804, the start node is router node604and the end node is compute node603. The compute node603, or other suitable device, may generate the packet810, which follows the same conventions of packet806inFIG.8A. However, the GRE header of packet810precedes MPLS routing information. This is represented by the letters “MPLS” followed by a label name in a box in packet810depicted inFIG.8B. The label contained in packet810corresponds to the locally-relevant label-to-port mapping814configured in router node604. The label-to-port mapping814illustrates a label-to-port mapping from “LABEL C” to egress port “PORT PHX,” referring to the physical link between router node604and compute node603. The compute node603may inject the packet810into the network. The tunnel termination endpoint configured at router node604may remove the GRE header and route the resulting GRE payload, packet812, back to the compute node603. In this example, the ETH frame header contains an MPLS routing header that specifies “LABEL C,” which corresponds to the label-to-port mapping814. The packet812will traverse the network segment804along the physical link connecting the egress port designated by “PORT PHX” to compute node603. Upon receipt of compute node603of packet812, the compute node603may direct the probe payload to a suitable handler according to the port specified in the UDP datagram and calculate a performance metric according to the information in the probe payload.

FIG.8C, a probe packet traversing a round-trip network path802including a compute node603and a router node604is depicted. InFIG.8C, the source compute node is compute node603and the destination compute node is likewise compute node603. The network path802begins at compute node603, traverses network segment804to router node604, and then returns to destination compute node603. There is one network segment804in network path802that is traversed twice, once on the outgoing leg and once on the returning leg. The start node of network segment804on the outgoing leg is compute node603and the end node is router node604. Likewise, on the return leg of network path804, the start node is router node604and the end node is compute node603.FIG.8Cdepicts a network traversal similar to the packet810inFIG.8Cand is routed by router node804using an MPLS routing header using decapsulated packet818. However, the locally-relevant label-to-port mappings820of label to egress port contain two possible labels/egress ports. In this example, the route defined in packet816specifies the label “LABEL D” which corresponds to the egress port designated by “PORT PHX2.”FIG.8Cthus illustrates that two physical paths along the same network segment804can be probed by specifying the physical path using GRE-encapsulated label-based routing. In the event of a network failure along network segment804, packets could be generated defining routes including both egress ports “PORT PHX1” and “PORT PHX2” that could then be injected to isolate the fault to a particular egress port or physical path. Such packets would be identical except for the MPLS label contained in packets816,818.

FIG.8Dis a simplified diagram of the communications network800comprising the portion of distributed environment600bounded by the line613ofFIGS.6A-C.FIG.8Dillustrates the same routing principles illustrated inFIG.8A, specifically a round-trip network path822performance measurement over network segment824made without the use of labels at the router node606. InFIG.8D, the source compute node is compute node607and the destination compute node is likewise compute node607. The network path822begins at compute node607, traverses network segment824to router node606, and then returns to destination compute node607. There is one network segment824in network path822that is traversed twice, once on the outgoing leg and once on the returning leg. The start node of network segment824on the outgoing leg is compute node607and the end node is router node606. Likewise, on the return leg of network path824, the start node is router node606and the end node is compute node607. The compute node607injects packet828, which is decapsulated to packet826at router node606. Router node606routes packet826back to compute node607. Compute node607, or another suitable computing device, may use the contents of the probe payload or other stored data to calculate performance metrics associated with network segment824. In addition to illustrating the universality of the concepts fromFIG.8A,FIG.8Dincludes a network path that will be used to illustrate an indirect, third-order performance measurement calculation inFIG.10.

FIG.9depicts a simplified flowchart900showing methods for making performance measurements to be made for segments of a communication network by utilizing multi-layer tunneling techniques in communications networks similar to the example communications networks800inFIGS.8A-D. The methods depicted inFIG.9may be implemented in software (e.g., code, instructions, program) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or combinations thereof. The software may be stored on a non-transitory storage medium (e.g., on a memory device). The method presented inFIG.9and described below is intended to be illustrative and non-limiting. AlthoughFIG.9depicts the various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, the processing may be performed in some different order, or some steps may also be performed in parallel. One or more of the processing steps depicted inFIG.9and described below may be performed or facilitated by NPAS620depicted inFIG.6A.

At902, the NPAS620or the source compute node can enable a first packet header to be configured for a first packet to cause the first packet to traverse a path from a source compute node to a destination compute node. The path may traverse a communication network, e.g., example communications network800, comprising a set of one or more router nodes, the path comprising a plurality of network segments including a first segment, the set of router nodes having a first router node. The first packet header may comprise a plurality of header sections including a header section for each segment in the path, the plurality of header sections comprising a first header section corresponding to the first segment, the first header section storing information indicative of a manner for routing the packet for the first segment. For example, the NPAS620can configure a packet including a header, similar to the packet806. The source compute node may be triggered by an NPAS or by locally-executing program code. For example, the source compute node may contain program code that executes periodically according to a pre-determined schedule. Or the NPAS may send instructions to the source compute node to construct and inject packets as needed to conduct performance measurements of the network. Such instructions may specify which network segments to probe, particular network paths, or which performance measurements to conduct, among other possible instructions. In this example, the flowchart900describes construction of a packet for traversing a network path similar to the one depicted inFIGS.8A-Dincluding at least one compute node and one router node.

At904, the source compute node measures a first timestamp indicative of the time the packet was or is to be injected into the network. The timestamp may be sent to the NPAS620for short-term storage in a memory device or stored locally at the source compute node. The NPAS620may contain program code describing the metric to be calculated. Performance over network segments can be measured directly or indirectly. A direct measurement includes defining a path that includes a network segment including at least one router node and calculating the time the probe packet takes to traverse that network segment or measuring packet loss across that segment. The measurement is made directly in the sense that the metric determined corresponds to the network property under measurement and is not a derived quantity requiring, for instance, additional measurements or data to compute. For example, the time a probe packet takes to traverse a network segment including one compute node and one router node may be directly measured. In this example, the probe packet contains header information that specifies that the probe packet traverse the segment from the compute node to the router node, and then immediately return to the compute node. The examples considered inFIGS.8A-8D and9are examples of direct, or first-order, performance measurements.

In contrast, an indirect measurement includes defining a first network path that includes a network segment containing at least one router node and calculating the time a probe packet takes to traverse the first network path or packet loss along the first network path. Additional probe packets may be constructed and injected that include a portion of that first network path, and metrics for the probe packets traversing the portion of the first network path can be calculated. A performance metric for a network segment included in the first network path can be calculated, for example, by subtracting the traversal times for the network path portion from the traversal time for the first network path. An indirect measurement including two probe packets is called a second-order measurement, an indirect measurement including three probe packets is called a third-order measurement, and so on. For example, for a network path including a compute node and two router nodes, a second-order, indirect performance metric for the segment including the two router nodes can be calculated by measuring the round-trip traversal time for the entire network path and subtracting from it the round-trip traversal time for the portion of the path including the compute node and the first router node. In general, an “n”th-order metric refers to the number of packets that are required to calculate the metric. Examples of indirect second-order performance measurements will be discussed below inFIGS.11and12. Examples of indirect third-order performance measurements will be discussed below inFIG.10.

For example, a first-order metric is directly measured and may only require calculating the difference between the time of receipt of the packet at a destination compute node and the timestamp, whereas a second- or higher-order metric may require the receipt of additional information from other network locations to calculate the metric. In some examples, probe packets may be used to calculate a plurality of metrics. For example, a probe packet may be used for calculating network latency and may also contribute to the calculation of packet loss and jitter measurements, by belonging to a collection of probe packets.

The timestamp is indicative of the time the packet is to be injected into the network. Since the timestamp cannot be added after the packet is communicated to the network at906, the timestamp is necessarily indicative of a time immediately before injection into the network. Since network latency measurements are likely to be on the order of milliseconds, the timestamp may be added immediately before network injection so as to minimize the introduction of measurement error due to delays injecting the packet. In some examples, packets are injected using an agent program running on the compute nodes. In some examples, router nodes may not be used for probe injection because they are not configured to inject packets, determine times associated with the packets, or perform latency or other metrics calculations based upon the packets.

Dashed box908includes the portions of the method whereby the packet traverses a path from the source compute node to the destination compute node. At910, the packet traverses at least a network segment from the source compute node to the router node based on the first section of the header. In some examples, traversal of this first segment from compute node to router node will made be according to the IP protocol. For example, packet header807illustrates the first network segment being traversed according to the IP protocol in the first section of the header. In some embodiments, other network-layer protocols may be used, such as Internetwork Packet Exchange (IPX) or Internet Protocol Security (IPSec). The chosen network-layer protocol may be followed by a tunnelling protocol header, such as GRE, that may be removed by the router node if it is configured as a tunnel termination endpoint.

At912, the router node receives the packet and examines the header to determine the second section of the header. For example, the first section of the header may be an IP protocol header that precedes a GRE header. The router node receives the packet from the source compute node. The router node may proceed to examine the sections of the header following the outer IP protocol header. The outer IP header may be followed by a GRE header and may contain a GRE tunnel payload. Since the router node has been configured to be a GRE tunnel endpoint, in some examples, the GRE packet payload may be decapsulated and extracted. The GRE payload may itself be, for example, another IP packet or a label-routed packet. This capability effectively makes the router nodes act as probe packet relay agents, allowing certain packet loss and round-trip time metrics to be measured using such tunneled probes.

At914, the packet traverses the segment from the router node to a destination compute node based on the section of the packet header decapsulated in912. InFIGS.8A-D, the source compute node and the destination compute node were the same. InFIGS.8A and8D, the GRE payload was another IP packet which was subsequently routed back to the source compute node.

InFIGS.8B and8C, the GRE payload was an MPLS routing header which was subsequently routed back to the source compute node. Use of MPLS routing when routing packets away from router nodes ensures that the packet traverses a specific physical path, whereas use of other protocols, like the IP protocol, does not. In the case where the destination compute node is not the same as the source node, a second-order metric may be needed to measure network performance along network segments making up even very simple network paths.

At916, the packet is received by the destination compute node, which may, as mentioned previously, be the same as the source compute node. The destination compute node may extract the probe payload from the packet. At918, the destination compute node may determine the time of receipt of the packet at the destination compute node. As with the timestamp, the time of receipt determined should be as close as possible to the actual time of receipt, to minimize the introduction of measurement error.

The destination compute node or the NPAS may calculate a direct, first-order metric at920. In some examples, the first, first-order metric may include the difference between the time of receipt of the packet at the destination compute node and the timestamp. That difference represents the round-trip time from the source compute node to the destination compute node. In the examples ofFIGS.8A-D, that difference divided by 2 may be indicative of the one-way transit time between the source/destination compute node and the router node. Alternatively, packet loss may be estimated by counting the packets sent over a particular time or belonging to a particular collection of packets and dividing the number of packets lost (packets that never arrived) by the total number of packets sent.

However, such measurements are only an estimate of performance, since difference physical paths may have been traversed in either direction and because the state of the network may have changed even during the short time between the traversal from the source compute node to the router node, and the traversal from the router node back to the source compute node. Thus, performance measurements across the entire network may include performance measurements of network segments using multiple methods. For example, performance across a network segment may be measured using both a first- and second-order metric to determine the accuracy of those measurements and to confirm the location of network problems. Certain embodiments may calculate the average or other comparable statistic of metric measurements across network segments made using various network paths or calculations to estimate the performance of the network. The innovations of the present disclosure serve to, in part, minimize measurement errors by specifying particular physical paths between network segments.

Turning now toFIGS.10-12, examples of indirect, higher-order metric calculations will be given.FIG.10is a simplified diagram of the communications network800included in the portion of distributed environment600bounded by the line613FIGS.6A-C. The communications network800is chosen for illustrative purposes and the principles and concepts discussed herein apply equally to all portions of the distributed environment600.

InFIG.10, a probe packet traversing a network path1002from source compute node603, through two router nodes604,606, to destination compute node607is depicted. Measurements along network path1002may be used with other metrics to calculate second- and higher-order performance metrics. InFIG.10, the source compute node is compute node603and the destination compute node is compute node607. The network path1002includes network segments1012,1014, and1016. The start node of network segment1012is compute node603and the end node is router node604. The start node of network segment1014is router node604and the end node is router node606. The start node of network segment1016is router node606and end node is compute node607.

Source compute node603constructs or receives from the NPAS a packet1004to define network path1002, including adding a timestamp to the probe payload indicative of the time of injecting packet1004into the network. The packet1004traverses network segment1012according to the header of packet1004that specifies IP protocol routing between source compute node603and router node604. Router node604remove the GRE header preceded by the outer IP header of IP packet1004, which includes a GRE payload including a packet1006with an MPLS routing header. The packet1006is routed by router node604according to the MPLS label in packet1006header using the locally-relevant label-to-port mapping information1010. The label-to-port mapping information1010specifies a physical path between router node604and router node606along network segment1014. Router node606examines the header of packet1006and removes the MPLS label, exposing IP packet1008. Router node606routes packet1008to destination compute node607along network segment1016using IP routing, completing the traversal of network path1002.

Upon receipt of the packet1008, destination compute node607determines the time of receipt of the packet1008and can calculate a metric using the timestamp and the time of receipt. The metric represents the network traversal time, or network latency, between source compute node603and destination compute node607. Other metrics may be calculated including, for example, packet loss. To calculate packet loss, a plurality of probe packets can be sent from source compute node603to destination compute node607. Packet loss equals the ratio of the number of lost probe packets that never arrive at destination compute node607to the total number of sent probe packets. The plurality of probe packets may be sent as a collection including, for example, a collection identifier in the probe payload. Or a specified number of probe packets may be sent by compute node603during a first specified window of time and counted by destination compute node607for a second specified window of time to calculate packet loss. Similar techniques may be used to calculate network jitter.

The metric representing network path traversal time between source compute node603and destination compute node607can be used to calculate an indirect, third-order metric representing the network latency between router nodes604and606. Upon calculating the overall metric associated with network path1002, the destination compute node607or NPAS may receive calculated metrics associated with network paths802and822fromFIGS.8A-D. The calculated metrics may be received from the NPAS, from another compute node, or may be stored at destination compute node607from previous operations. In some examples, the destination compute node607may send the data necessary for metric calculations to, for example, an NPAS and perform no calculations locally.

The third-order metric representing the network latency between router nodes604and606can be calculated by first dividing the traversal times of network paths802and822by two, since the calculated metric for network segment1014represents the one-way traversal time of network segment1014, whereas the calculated metrics associated with the traversal time of network paths802and822are round-trip times. The halved times associated with the traversal time of network paths802and822may be subtracted from the network path1002traversal time between source compute node603and destination compute node607to obtain an estimate of the network segment1014traversal time between router nodes604and606. This calculation represents the packet travel time along the physical path specified by the label-to-port mappings1010, including the egress port “PORT LGA.” The estimate may include some uncertainty as a result of the inability to specify a physical path between the router nodes and compute nodes, where IP routing is used. In some cases, the traversal time along the network segment1014may differ depending on the direction of traffic or depending on the particular physical connection specified in probe packets1004,1006,1008.

A third-order metric representing packet loss between router nodes604and606can be similarly calculated by first dividing the packet losses associated with network paths802and822by two, since the calculated packet loss metric for network path1002stems from the one-way traversal of network path1002, whereas the calculated packet loss metrics associated with the traversal of network paths802and822are round-trip traversals. The halved packet losses associated with the traversal of network paths802and822may be subtracted from the packet loss over the network path between source compute node603and destination compute node607to obtain an estimate of the network segment1014packet loss between router nodes604and606. In essence, packet loss along network segment1014is equal to overall packet loss along network path1002minus contributions to this packet loss from network segments1012and1016. This calculation represents the packet loss along the physical path specified by the label-to-port mappings1010, including the egress port “PORT LGA.” The estimate may include some uncertainty as a result of the inability to specify a physical path between the router nodes and compute nodes, where IP routing is used.

FIG.11is a simplified diagram of the communications network800included in the portion of distributed environment600bounded by the line613ofFIGS.6A-C. The communications network800is chosen for illustrative purposes and the principles and concepts discussed herein apply equally to all portions of the example network600. InFIG.11, a probe packet traversing a round-trip network path1102from source compute node603, through two router nodes604,606, and back to source compute node603is depicted. InFIG.11, the source compute node is compute node603and the destination compute node is likewise compute node603. The network path1102includes network segments1116and1118, each traversed twice on a round-trip path. The start node of network segment1116on the outgoing leg is compute node603and the end node is router node604. The start node of network segment1118on the outgoing leg is router node604and the end node is router node606. The start node of network segment1118on the return path is router node606and end node is router node604. The start node of the network segment1116on the return path is router node604and the end node is compute node603.

Source compute node603constructs a packet1104to define network path1102, including adding a timestamp to the probe payload indicative of the time of injecting packet1104into the network. The packet1104traverses network segment1116according to the header of packet1104that specifies IP protocol routing between source compute node603and router node604. Router node604removes the GRE header following the outer IP header of IP packet1104, which includes a GRE payload including a packet1106with an MPLS routing header. The packet1106is routed by router node604according to the MPLS label in packet1106header using the locally-relevant label-to-egress port mappings1112. The label-to-port mappings1112specify a physical path between router node604and router node606along network segment1118. Router node606examines the header of packet1106and removes or “pops” the first MPLS label, resulting in packet1108. Router node606routes packet1108back to router node604along network segment1118according to the MPLS label in packet1108header using the locally-relevant label-to-port mapping information1114. Router node604examines the header of packet1108and removes the MPLS label, exposing IP packet1110. Router node604routes packet1110to source compute node603along network segment1116using IP routing, completing the traversal of network path1102.

As with the network path1102depicted inFIG.10, calculation of direct, first-order metrics relating to the network path1102is possible. For example, the network latency and packet loss along network path1102may be calculated by source compute node603or elsewhere at, for example, the NPAS. The computation of second-order metrics may be possible as well. The source compute node603may receive calculated first-order metrics associated with the network latency or packet loss of round-trip network path802fromFIGS.8A-C. The calculation of network latency or packet loss between router nodes604and606along network segment1118may be accomplished by subtracting the traversal time or packet loss of measured from network path802from the metrics calculated along network path1102. In this example, the calculated first-order metrics associated with the network latency or packet loss of round-trip network path802do not need to be halved because network path1102is itself a round-trip. However, in some cases, the physical path from router node604to router node606may differ from the reverse physical path from router node606to router node604. In that case, additional measurements may be needed to accurately measure network metrics in both directions. In order for the second-order metrics associated with network segment1118calculated using the methods described to be most accurate, the same physical path along network segment1118should be used.

FIGS.12A-Bdepict a simplified flowchart1200showing methods for making performance measurements to be made for segments of a communication network by utilizing multi-layer tunneling techniques in a communications network similar to the communication network800inFIG.10. The methods depicted inFIGS.12A-Bmay be implemented in software (e.g., code, instructions, program) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or combinations thereof. The software may be stored on a non-transitory storage medium (e.g., on a memory device). The method presented inFIGS.12A-Band described below is intended to be illustrative and non-limiting. AlthoughFIGS.12A-Bdepicts the various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, the processing may be performed in some different order, or some steps may also be performed in parallel. One or more of the processing steps depicted inFIGS.12A-Band described below may be performed or facilitated by NPAS620depicted inFIG.6A.

FIGS.12A-Billustrates a simple example of calculating a third-order, indirect performance measurement, but the methods are equally applicable to second- or other higher-order calculations. At1202, the NPAS620may enable a packet header to be configured, wherein the including a header with at least three sections. As with method900, the header sections correspond to network segments defining a path. In this example, the network path proceeds from the source compute node to a first router node, from the first router node to a second router node, and then to the destination compute node. This network path will enable calculation of one portion of the metrics needed for the third-order calculation. As calculation of a higher-order metric requires inputs from other metric calculations, triggering of the methods inFIGS.12A-Bmay require additional coordination and may be initiated by the NPAS620. Alternatively, in some examples, the calculation of higher-order metrics may be initiated and coordinated by one or more compute nodes, by or among themselves.

The header is similar to the header of packet1004fromFIG.10. The first header section may include a path from a source compute node to a first router node. The first header section may use, for example, the IP protocol or any other protocol for causing the packet to reach the first router node. Some network-layer protocols, such as the IP protocol, may not guarantee a particular physical path between the source compute node and the first router node and may thus be a source of error in performance measurements. In some examples, the source compute node may be chosen with a minimum number of network segments between it and the first router node to minimize this error. In some examples, a network layer protocol may be chosen to guarantee a specific physical path. For example, MPLS routing may be used between the source compute node and the first router node to guarantee a specific physical path. The second section of the header may be encapsulated inside a GRE header and include a path definition from the first router node to a second router node. The path definition may use label-based routing to require a specific physical path from the first router node to the second router node. The third section may define a path from the second router node to the destination compute node. As with the first section, any network-layer protocol may be used including, for example, the IP protocol.

At1204and1206, the source compute node measures a timestamp prior to injecting the packet configured in1202into the network. In some examples, packets are injected using an agent program running on the compute hosts. In some examples, router nodes may not be used for probe injection because they do not have the compute resources to run the probe injection agent. The traversal of the network by the path defined in the probe packet header in1202is included in dashed box1208. At1210, the packet traverses the first network segment from the source compute node to the first router node. At1212, the first router node examines the header to determine the second section of the header. For example, the second section may be GRE header that includes a GRE payload to be decapsulated by the first router node. At1214, the first router node routes the packet to the second router node according to the second section. In this example, the second section may utilize label-based routing to ensure a specific physical path from the first router node to the second router node is traversed for the purposes of accurate performance measurements. At1216, the second router node examines the header to determine the third section of the header and at1218the second router node routes the packet to the destination compute node.

Upon receipt of the packet at the destination compute node at1220, the destination compute node may extract the probe payload from the packet. At1222, the destination compute node may determine the time of receipt of the packet at the destination compute node. As with the timestamp, the time of receipt determined should be as close as possible to the actual time of receipt, to minimize the introduction of measurement error. The destination compute node can send the packet receipt timestamp to the NPAS for metric calculation. In some embodiments, the calculation of performance metrics may occur at the destination compute node. The NPAS may then, at1224, receive metrics determined from other packet traversals. In this example, for the calculation of the third-order metrics involving the network segment between the first router node and the second router node, metrics for the first and third network segments, between the source compute node and the first router node, and the second router node and the destination compute node, respectively, may be required. The metrics may be received by or from an NPAS, from other compute nodes, or other suitable location for calculation and storage of performance metrics.

At1226, the third-order metric is calculated using first the timestamp and time of receipt of the packet at the destination compute node, along with the two metrics received for other network segment traversals, as is discussed in the description ofFIG.10. Alternatively, third-order packet loss may be calculated using the packet loss for the network path from method1200along with packet loss for the other two network segments, as is also discussed in the description ofFIG.10. Calculation of other metrics may be possible. For example, in certain embodiments, network jitter may be calculated. In the context of networking, jitter may be defined as variations in network latency across a particular network segment. Calculation of jitter may be affected using the method1200or other methods of this disclosure by calculating multiple latency measurements in series and comparing the variation.

The destination compute node, or other suitable compute resource for metric calculation, may send the calculated metrics to the NPAS for processing. In certain embodiments, the NPAS may receive metric calculations from a variety of compute sources and prepare one or more visualizations or reports. For example, using the innovations of the present disclosure, network latency or packet loss between router nodes may be added to charts or heatmaps that allow network administrators to quickly see where network failures may be occurring.

Turning now toFIGS.19A-B,FIGS.19A-Bdepict a simplified flowchart1900depicting processing performed for controlling the path taken by a packet through a communication network and computing one or more metrics for one or more segments of the path according to certain embodiments. The method depicted inFIGS.19A-Bmay be implemented in software (e.g., code, instructions, program) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or combinations thereof. The software may be stored on a non-transitory storage medium (e.g., on a memory device). The method presented inFIGS.19A-Band described below is intended to be illustrative and non-limiting. AlthoughFIGS.19A-Bdepicts the various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, the processing may be performed in some different order, or some steps may also be performed in parallel. One or more of the processing steps depicted inFIGS.19A-Band described below may be performed or facilitated by NPAS620depicted inFIG.6A.

At1902, the NPAS may enable a probe packet to be configured, including enabling configuration of a packet header for the packet to cause the packet to traverse a path from a source compute node to a destination compute node over a path that includes multiple segments and one or more router nodes, and where the configured packet header comprises a header section corresponding to each of the multiple segments, and where the header section corresponding to a segment stores information indicative of a manner in which the packet is to be routed for that segment. For example, the NPAS620may perform periodic monitoring of all network segments or generate a map of the network, or a representation thereof, based upon the various nodes and segments determined in706. The NPAS may then determine a performance measurement plan (or probing plan) whereby probe packets are injected into the network at particular nodes and the packets are configured to traverse a specific path terminated by compute nodes, where the path includes specific segments of the network for which performance metrics are to be computed.

The NPAS enables configuration of the router nodes to become tunnel termination endpoints. For example, a router node may be configured such that it becomes a termination endpoint for a tunnel. In addition, one or more router nodes may be configured such that, for a probe packet received by a router node, the outgoing network interface or port used by the router node for communicating the packet from the router node can be controlled. For a router node, this is done by creating a locally-relevant mapping or association between a label and an outgoing interface or port of the router node.

Based upon these configurations (e.g., based upon the tunnel termination endpoints and the label-to-port mappings), a probe packet is configured having a header where the header is configured to cause the probe packet to traverse a specific path in the network from a specific source compute node to a specific destination compute node and the path traverses specific segments of the network, in which each segment corresponds to a header section. A segment can be between a source compute node and a router node, between two router nodes, or between a router node and a destination compute node. The path traversed by the probe packet traverses multiple segments and one or more router nodes.

At1904, a first time is measured for the packet at the source compute node, where the first time is indicative of a time when the packet is communicated from the source compute node. The time may correspond to the time immediately before the packet is injected into the network at the source compute node. The first time may be communicated to the NPAS620, which may perform metric calculations, as described below.

In dashed box1906, the packet is communicated from the source compute node to the destination compute node and over the multiple segments based upon information included in the packet header, wherein each segment corresponds to a header section. At1908, the packet is received by start node of a segment, among the set of multiple segments. At1910, the start node for the segment reads information from a section of the header corresponding to the segment. At1912, the packet is routed from the start node to the end node of the segment based upon the information read in1910, where the end node can be the destination compute node or a start node of the next segment. The process described in boxes1908,1910, and1912may be repeated for each of the multiple segments as the packet traverses all of the segments between the source compute node and the destination compute node.

At1914, the packet is received at the destination compute node and at1916, a second time is measured for the packet at the destination compute node, where the second time is indicative of a time when the packet is received by the destination compute node. The second time may be communicated to the NPAS620, which may perform metric calculations, as described below. The NPAS620may store the times and other metric calculation information in a memory device pending calculation of metrics. For example, the NPAS620may store the first time and the second time in RAM or in a database, or the like.

At1918, a performance metric is calculated for a segment from the multiple segments based upon the first time and the second time. A performance metric can be measured for a segment of the path, where the segment is between a compute node and a router node (e.g., the segment has a router node as its start node or its end node) or is a segment between two router nodes. Examples of performance metrics that may be computed for a network segment include network latency, packet loss, and jitter, among others. For example, the NPAS620may use the first time and the second time to calculate a performance metric. In some examples, for nth-order metrics, additional metric calculation information may be needed to calculate performance metrics in addition to the first time and the second time.

At1920, optionally, one or more actions are performed further to the metric calculation in1918upon the metric. The one or more actions can include outputting information associated with the metric to a user. For example, information regarding the network segment for which the metric is calculated may be output to a user via a GUI provided by NPAS620. The GUI may provide a visual representation of metric so that network issues can be quickly diagnosed by network engineers. For example, metrics may be presented in numerical or chart form, or in the form of lines graphs, heat maps, etc. In certain use cases, NPAS620may generate a report that includes information regarding the network segment and the metric.

The one or more actions can include communicating information regarding the network segment and the metric to a consumer of the metric, where the consumer could be a user such as a system administrator or a network performance engineer. The communicating of information may occur by way of the NPAS620. The consumer could be a computer system or software component that uses the metrics information. For example, the computer system or software component may be configured to provide a visual, audible, or electronic notification in the event certain network errors or failures are detected. Network errors or failures may be detected, for instance, by assigning a threshold value to the calculated metrics. Network latency, for instance, may have a maximum value above which certain notifications can be triggered. Likewise, packet loss may have a maximum threshold value and jitter may have a maximum threshold value. Other actions may also be triggered based upon the value of the metric computed in1920. For example, if the metric measure network latency and is calculated to be over a threshold, automatic corrective actions may be triggered or initiated to reduce the network latency.

Turning next toFIGS.13A-E,FIGS.13A-Eare simplified diagrams of an example network1300. The example network1300depict a network including several connected router nodes, which may serve to further illustrate one or more of the innovative techniques described in this disclosure. Example network1300includes two compute nodes1301and1313, using the node shading conventions ofFIGS.6A-C. Example network1300depicts compute nodes1301and1313that may be geographically separated. Geographically separated networks may be connected by way of a subnetwork of intervening nodes1315that may be router nodes. Compute nodes1301,1313can be connected to subnetwork1315by network fabric1302,1312. Network fabric1302,1312may represent a plurality of routers, switches, and other networking devices used to connect one network to another network. In some examples, the subnetwork1315may be referred to as a backbone. The backbone may, for example, provide inter-region connectivity configured to provided suitable performance for bandwidth, latency, and jitter for high-performance cloud operations.

FIG.13Adepicts a round-trip network path1317defined by probe packet1321. Packet1321defines a round-trip network path1317from source compute node1301to router node1303and back to source/destination compute node1301by way of network fabric1302. LikeFIG.8A, the return path is defined using the IP protocol, but could have equally been defined using label-based routing methods including, for example, MPLS. Network path1317may be of limited value in measuring performance, however, because the path through the networking devices making up network fabric1302may introduce sources of delay or packet loss, rendering performance measurements difficult to interpret. In addition, the use of IP-based routing guarantees only traversal of the logical network path, in which the physical network path is indeterminate. The performance measurements of the present disclosure may be most accurate when used to determine latency or packet loss, or to conduct component fault isolation between two router nodes because the physical path between the two devices may be exactly specified in that case.

FIG.13Bdepicts another round-trip network path1323defined by probe packet1327. Packet1327defines a round-trip network path1323from source compute node1301to router nodes1303,1307,1305and back to source/destination compute node1301by way of network fabric1302. In this example, the network path1323defined by the probe packet1327includes MPLS routing information encapsulated by the GRE tunneling protocol. MPLS labels are used to route the decapsulated packet from router node1303to router node1307, and from router node1307to router node1305, according to locally-relevant mappings1306,1308. As withFIG.13A, the final portion of the network path1323is routed using the IP protocol. Metrics calculated using the network path1323may be used to determine third-order or higher-order metrics. For example, the time for traversal of the network path1317fromFIG.13A, along with the time for traversal of the network path from compute node1301to router node1305to router node1307and back to compute node1301(not shown inFIG.13B) could be used together along with the time for traversal of the network path1323fromFIG.13Bto determine performance metrics associated with the network segment between router node1303and router node1307. In some examples, such a calculation may need to account for the double-inclusion of the network segment between compute node1301and network fabric1302in the third-order metric calculation.

FIG.13Cdepicts a network path1329from compute node1301to compute node1313across subnetwork1315. In this example, the network path1329defined by probe packet1333includes MPLS routing information encapsulated by the GRE tunneling protocol. MPLS labels are used to route the decapsulated packet from router node1303to router node1307, and router node1307to router node1309, in accordance with locally-relevant label-to-port mappings1306and1310, respectively. As withFIG.13A, the final portion of the network path1329is routed using the IP protocol. The network path1329may be used for calculation of third-order metrics between, for example, router nodes1303and1307or router nodes1307and1309. For example, a metric may be calculated associated the time for traversal of the network path1329. Metrics associated with the time for traversal of round-trips paths between compute node1301and router node1303and between compute node1313and router node1307can be calculated, in analogy toFIGS.8A and8D. The round-trip metrics may be halved and subtracted from the time associated with traversing network path1329to estimate the latency across the network segment defined by the physical path between router node1303and router node1307. An analogous procedure could be used to estimate the latency across the network segment defined by the physical path between router node1307and router node1309.

FIG.13Ddepicts a round-trip network path1331from compute node1301, through several router nodes, and back to compute node1301. In this example, the network path1335defined by probe packet1337includes MPLS routing information encapsulated by the GRE tunneling protocol.FIG.13Dillustrates a path similar the round-trip ofFIG.13A, but with a reversal in direction utilizing MPLS labels rather than the IP protocol. A time measurement of the round-trip network path1335may be used along with measurements of shortened paths for second-order metric calculations. For example, another network path along the same network path1335except stopping at router node1307may be used to estimate performance measurements between router nodes1305and1307.

FIG.13Edepicts a round-trip network path1339from compute node1301, through several router nodes, to compute node1313, and back to compute node1301, through several more router nodes. In this example, the network path1339defined by probe packet1341includes MPLS routing information encapsulated by the GRE tunneling protocol.FIG.13Eillustrates a packet traversing all of the nodes making up the network at least once. Metrics calculated using the network path1339may be used in conjunction with other network paths (not shown) to calculate second- and higher-order metrics along any segment between two router nodes within subnetwork1315.

The teachings described in this disclosure can be implemented in an underlay network containing one or more compute nodes and one or more router nodes. For example, the teachings can be implemented in an underlay network that is part of the infrastructure used by a cloud services provider (CSP) to provide one or more cloud services to one or more subscribing customers. Performance monitoring of such a network that incorporates the various techniques described in this disclosure can enable rapid and localized fault isolation to the components making up the network. This translates to providing more reliable and fault-resilient delivery of cloud services and data communications to the subscribing customers, which in turn translates to a better customer experience. The following section describes examples of CSP infrastructure setups that may be used to provide cloud services to subscribing customers, where the infrastructure can include a network comprising one or more compute nodes and one or more router nodes that implement the various features described in this disclosure.

Example Architectures for Providing a Cloud Service

As noted above, infrastructure as a service (IaaS) is one particular type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In an IaaS model, a cloud computing provider can host the infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., a hypervisor layer), or the like). In some cases, an IaaS provider may also supply a variety of services to accompany those infrastructure components (example services include billing software, monitoring software, logging software, load balancing software, clustering software, etc.). Thus, as these services may be policy-driven, IaaS users may be able to implement policies to drive load balancing to maintain application availability and performance.

The VCN1406can include a local peering gateway (LPG)1410that can be communicatively coupled to a secure shell (SSH) VCN1412via an LPG1410contained in the SSH VCN1412. The SSH VCN1412can include an SSH subnet1414, and the SSH VCN1412can be communicatively coupled to a control plane VCN1416via the LPG1410contained in the control plane VCN1416. Also, the SSH VCN1412can be communicatively coupled to a data plane VCN1418via an LPG1410. The control plane VCN1416and the data plane VCN1418can be contained in a service tenancy1419that can be owned and/or operated by the IaaS provider.

The control plane VCN1416can include a control plane demilitarized zone (DMZ) tier1420that acts as a perimeter network (e.g., portions of a corporate network between the corporate intranet and external networks). The DMZ-based servers may have restricted responsibilities and help keep breaches contained. Additionally, the DMZ tier1420can include one or more load balancer (LB) subnet(s)1422, a control plane app tier1424that can include app subnet(s)1426, a control plane data tier1428that can include database (DB) subnet(s)1430(e.g., frontend DB subnet(s) and/or backend DB subnet(s)). The LB subnet(s)1422contained in the control plane DMZ tier1420can be communicatively coupled to the app subnet(s)1426contained in the control plane app tier1424and an Internet gateway1434that can be contained in the control plane VCN1416, and the app subnet(s)1426can be communicatively coupled to the DB subnet(s)1430contained in the control plane data tier1428and a service gateway1436and a network address translation (NAT) gateway1438. The control plane VCN1416can include the service gateway1436and the NAT gateway1438.

The control plane VCN1416can include a data plane mirror app tier1440that can include app subnet(s)1426. The app subnet(s)1426contained in the data plane mirror app tier1440can include a virtual network interface controller (VNIC)1442that can execute a compute instance1444. The compute instance1444can communicatively couple the app subnet(s)1426of the data plane mirror app tier1440to app subnet(s)1426that can be contained in a data plane app tier1446.

The data plane VCN1418can include the data plane app tier1446, a data plane DMZ tier1448, and a data plane data tier1450. The data plane DMZ tier1448can include LB subnet(s)1422that can be communicatively coupled to the app subnet(s)1426of the data plane app tier1446and the Internet gateway1434of the data plane VCN1418. The app subnet(s)1426can be communicatively coupled to the service gateway1436of the data plane VCN1418and the NAT gateway1438of the data plane VCN1418. The data plane data tier1450can also include the DB subnet(s)1430that can be communicatively coupled to the app subnet(s)1426of the data plane app tier1446.

The Internet gateway1434of the control plane VCN1416and of the data plane VCN1418can be communicatively coupled to a metadata management service1452that can be communicatively coupled to public Internet1454. Public Internet1454can be communicatively coupled to the NAT gateway1438of the control plane VCN1416and of the data plane VCN1418. The service gateway1436of the control plane VCN1416and of the data plane VCN1418can be communicatively couple to cloud services1456.

In some examples, the service gateway1436of the control plane VCN1416or of the data plane VCN1418can make application programming interface (API) calls to cloud services1456without going through public Internet1454. The API calls to cloud services1456from the service gateway1436can be one-way: the service gateway1436can make API calls to cloud services1456, and cloud services1456can send requested data to the service gateway1436. But, cloud services1456may not initiate API calls to the service gateway1436.

In some examples, the secure host tenancy1404can be directly connected to the service tenancy1419, which may be otherwise isolated. The secure host subnet1408can communicate with the SSH subnet1414through an LPG1410that may enable two-way communication over an otherwise isolated system. Connecting the secure host subnet1408to the SSH subnet1414may give the secure host subnet1408access to other entities within the service tenancy1419.

The control plane VCN1416may allow users of the service tenancy1419to set up or otherwise provision desired resources. Desired resources provisioned in the control plane VCN1416may be deployed or otherwise used in the data plane VCN1418. In some examples, the control plane VCN1416can be isolated from the data plane VCN1418, and the data plane mirror app tier1440of the control plane VCN1416can communicate with the data plane app tier1446of the data plane VCN1418via VNICs1442that can be contained in the data plane mirror app tier1440and the data plane app tier1446.

In some examples, users of the system, or customers, can make requests, for example create, read, update, or delete (CRUD) operations, through public Internet1454that can communicate the requests to the metadata management service1452. The metadata management service1452can communicate the request to the control plane VCN1416through the Internet gateway1434. The request can be received by the LB subnet(s)1422contained in the control plane DMZ tier1420. The LB subnet(s)1422may determine that the request is valid, and in response to this determination, the LB subnet(s)1422can transmit the request to app subnet(s)1426contained in the control plane app tier1424. If the request is validated and requires a call to public Internet1454, the call to public Internet1454may be transmitted to the NAT gateway1438that can make the call to public Internet1454. Metadata that may be desired to be stored by the request can be stored in the DB subnet(s)1430.

In some examples, the data plane mirror app tier1440can facilitate direct communication between the control plane VCN1416and the data plane VCN1418. For example, changes, updates, or other suitable modifications to configuration may be desired to be applied to the resources contained in the data plane VCN1418. Via a VNIC1442, the control plane VCN1416can directly communicate with, and can thereby execute the changes, updates, or other suitable modifications to configuration to, resources contained in the data plane VCN1418.

In some embodiments, the control plane VCN1416and the data plane VCN1418can be contained in the service tenancy1419. In this case, the user, or the customer, of the system may not own or operate either the control plane VCN1416or the data plane VCN1418. Instead, the IaaS provider may own or operate the control plane VCN1416and the data plane VCN1418, both of which may be contained in the service tenancy1419. This embodiment can enable isolation of networks that may prevent users or customers from interacting with other users', or other customers', resources. Also, this embodiment may allow users or customers of the system to store databases privately without needing to rely on public Internet1454, which may not have a desired level of threat prevention, for storage.

In other embodiments, the LB subnet(s)1422contained in the control plane VCN1416can be configured to receive a signal from the service gateway1436. In this embodiment, the control plane VCN1416and the data plane VCN1418may be configured to be called by a customer of the IaaS provider without calling public Internet1454. Customers of the IaaS provider may desire this embodiment since database(s) that the customers use may be controlled by the IaaS provider and may be stored on the service tenancy1419, which may be isolated from public Internet1454.

FIG.15is a block diagram1500illustrating another example pattern of an IaaS architecture, according to at least one embodiment. Service operators1502(e.g., service operators1402ofFIG.14) can be communicatively coupled to a secure host tenancy1504(e.g., the secure host tenancy1404ofFIG.14) that can include a virtual cloud network (VCN)1506(e.g., the VCN1406ofFIG.14) and a secure host subnet1508(e.g., the secure host subnet1408ofFIG.14). The VCN1506can include a local peering gateway (LPG)1510(e.g., the LPG1410ofFIG.14) that can be communicatively coupled to a secure shell (SSH) VCN1512(e.g., the SSH VCN1412ofFIG.14) via an LPG1410contained in the SSH VCN1512. The SSH VCN1512can include an SSH subnet1514(e.g., the SSH subnet1414ofFIG.14), and the SSH VCN1512can be communicatively coupled to a control plane VCN1516(e.g., the control plane VCN1416ofFIG.14) via an LPG1510contained in the control plane VCN1516. The control plane VCN1516can be contained in a service tenancy1519(e.g., the service tenancy1419ofFIG.14), and the data plane VCN1518(e.g., the data plane VCN1418ofFIG.14) can be contained in a customer tenancy1521that may be owned or operated by users, or customers, of the system.

The control plane VCN1516can include a control plane DMZ tier1520(e.g., the control plane DMZ tier1420ofFIG.14) that can include LB subnet(s)1522(e.g., LB subnet(s)1422ofFIG.14), a control plane app tier1524(e.g., the control plane app tier1424ofFIG.14) that can include app subnet(s)1526(e.g., app subnet(s)1426ofFIG.14), a control plane data tier1528(e.g., the control plane data tier1428ofFIG.14) that can include database (DB) subnet(s)1530(e.g., similar to DB subnet(s)1430ofFIG.14). The LB subnet(s)1522contained in the control plane DMZ tier1520can be communicatively coupled to the app subnet(s)1526contained in the control plane app tier1524and an Internet gateway1534(e.g., the Internet gateway1434ofFIG.14) that can be contained in the control plane VCN1516, and the app subnet(s)1526can be communicatively coupled to the DB subnet(s)1530contained in the control plane data tier1528and a service gateway1536(e.g., the service gateway1436ofFIG.14) and a network address translation (NAT) gateway1538(e.g., the NAT gateway1438ofFIG.14). The control plane VCN1516can include the service gateway1536and the NAT gateway1538.

The control plane VCN1516can include a data plane mirror app tier1540(e.g., the data plane mirror app tier1440ofFIG.14) that can include app subnet(s)1526. The app subnet(s)1526contained in the data plane mirror app tier1540can include a virtual network interface controller (VNIC)1542(e.g., the VNIC of1442) that can execute a compute instance1544(e.g., similar to the compute instance1444ofFIG.14). The compute instance1544can facilitate communication between the app subnet(s)1526of the data plane mirror app tier1540and the app subnet(s)1526that can be contained in a data plane app tier1546(e.g., the data plane app tier1446ofFIG.14) via the VNIC1542contained in the data plane mirror app tier1540and the VNIC1542contained in the data plane app tier1546.

The Internet gateway1534contained in the control plane VCN1516can be communicatively coupled to a metadata management service1552(e.g., the metadata management service1452ofFIG.14) that can be communicatively coupled to public Internet1554(e.g., public Internet1454ofFIG.14). Public Internet1554can be communicatively coupled to the NAT gateway1538contained in the control plane VCN1516. The service gateway1536contained in the control plane VCN1516can be communicatively couple to cloud services1556(e.g., cloud services1456ofFIG.14).

In some examples, the data plane VCN1518can be contained in the customer tenancy1521. In this case, the IaaS provider may provide the control plane VCN1516for each customer, and the IaaS provider may, for each customer, set up a unique compute instance1544that is contained in the service tenancy1519. Each compute instance1544may allow communication between the control plane VCN1516, contained in the service tenancy1519, and the data plane VCN1518that is contained in the customer tenancy1521. The compute instance1544may allow resources, that are provisioned in the control plane VCN1516that is contained in the service tenancy1519, to be deployed or otherwise used in the data plane VCN1518that is contained in the customer tenancy1521.

In other examples, the customer of the IaaS provider may have databases that live in the customer tenancy1521. In this example, the control plane VCN1516can include the data plane mirror app tier1540that can include app subnet(s)1526. The data plane mirror app tier1540can reside in the data plane VCN1518, but the data plane mirror app tier1540may not live in the data plane VCN1518. That is, the data plane mirror app tier1540may have access to the customer tenancy1521, but the data plane mirror app tier1540may not exist in the data plane VCN1518or be owned or operated by the customer of the IaaS provider. The data plane mirror app tier1540may be configured to make calls to the data plane VCN1518but may not be configured to make calls to any entity contained in the control plane VCN1516. The customer may desire to deploy or otherwise use resources in the data plane VCN1518that are provisioned in the control plane VCN1516, and the data plane mirror app tier1540can facilitate the desired deployment, or other usage of resources, of the customer.

In some embodiments, the customer of the IaaS provider can apply filters to the data plane VCN1518. In this embodiment, the customer can determine what the data plane VCN1518can access, and the customer may restrict access to public Internet1554from the data plane VCN1518. The IaaS provider may not be able to apply filters or otherwise control access of the data plane VCN1518to any outside networks or databases. Applying filters and controls by the customer onto the data plane VCN1518, contained in the customer tenancy1521, can help isolate the data plane VCN1518from other customers and from public Internet1554.

In some embodiments, cloud services1556can be called by the service gateway1536to access services that may not exist on public Internet1554, on the control plane VCN1516, or on the data plane VCN1518. The connection between cloud services1556and the control plane VCN1516or the data plane VCN1518may not be live or continuous. Cloud services1556may exist on a different network owned or operated by the IaaS provider. Cloud services1556may be configured to receive calls from the service gateway1536and may be configured to not receive calls from public Internet1554. Some cloud services1556may be isolated from other cloud services1556, and the control plane VCN1516may be isolated from cloud services1556that may not be in the same region as the control plane VCN1516. For example, the control plane VCN1516may be located in “Region 1,” and cloud service “Deployment 14,” may be located in Region 1 and in “Region 2.” If a call to Deployment 14 is made by the service gateway1536contained in the control plane VCN1516located in Region 1, the call may be transmitted to Deployment 14 in Region 1. In this example, the control plane VCN1516, or Deployment 14 in Region 1, may not be communicatively coupled to, or otherwise in communication with, Deployment 14 in Region 2.

FIG.16is a block diagram1600illustrating another example pattern of an IaaS architecture, according to at least one embodiment. Service operators1602(e.g., service operators1402ofFIG.14) can be communicatively coupled to a secure host tenancy1604(e.g., the secure host tenancy1404ofFIG.14) that can include a virtual cloud network (VCN)1606(e.g., the VCN1406ofFIG.14) and a secure host subnet1608(e.g., the secure host subnet1408ofFIG.14). The VCN1606can include an LPG1610(e.g., the LPG1410ofFIG.14) that can be communicatively coupled to an SSH VCN1612(e.g., the SSH VCN1412ofFIG.14) via an LPG1610contained in the SSH VCN1612. The SSH VCN1612can include an SSH subnet1614(e.g., the SSH subnet1414ofFIG.14), and the SSH VCN1612can be communicatively coupled to a control plane VCN1616(e.g., the control plane VCN1416ofFIG.14) via an LPG1610contained in the control plane VCN1616and to a data plane VCN1618(e.g., the data plane1418ofFIG.14) via an LPG1610contained in the data plane VCN1618. The control plane VCN1616and the data plane VCN1618can be contained in a service tenancy1619(e.g., the service tenancy1419ofFIG.14).

The control plane VCN1616can include a control plane DMZ tier1620(e.g., the control plane DMZ tier1420ofFIG.14) that can include load balancer (LB) subnet(s)1622(e.g., LB subnet(s)1422ofFIG.14), a control plane app tier1624(e.g., the control plane app tier1424ofFIG.14) that can include app subnet(s)1626(e.g., similar to app subnet(s)1426ofFIG.14), a control plane data tier1628(e.g., the control plane data tier1428ofFIG.14) that can include DB subnet(s)1630. The LB subnet(s)1622contained in the control plane DMZ tier1620can be communicatively coupled to the app subnet(s)1626contained in the control plane app tier1624and to an Internet gateway1634(e.g., the Internet gateway1434ofFIG.14) that can be contained in the control plane VCN1616, and the app subnet(s)1626can be communicatively coupled to the DB subnet(s)1630contained in the control plane data tier1628and to a service gateway1636(e.g., the service gateway ofFIG.14) and a network address translation (NAT) gateway1638(e.g., the NAT gateway1438ofFIG.14). The control plane VCN1616can include the service gateway1636and the NAT gateway1638.

The data plane VCN1618can include a data plane app tier1646(e.g., the data plane app tier1446ofFIG.14), a data plane DMZ tier1648(e.g., the data plane DMZ tier1448ofFIG.14), and a data plane data tier1650(e.g., the data plane data tier1450ofFIG.14). The data plane DMZ tier1648can include LB subnet(s)1622that can be communicatively coupled to trusted app subnet(s)1660and untrusted app subnet(s)1662of the data plane app tier1646and the Internet gateway1634contained in the data plane VCN1618. The trusted app subnet(s)1660can be communicatively coupled to the service gateway1636contained in the data plane VCN1618, the NAT gateway1638contained in the data plane VCN1618, and DB subnet(s)1630contained in the data plane data tier1650. The untrusted app subnet(s)1662can be communicatively coupled to the service gateway1636contained in the data plane VCN1618and DB subnet(s)1630contained in the data plane data tier1650. The data plane data tier1650can include DB subnet(s)1630that can be communicatively coupled to the service gateway1636contained in the data plane VCN1618.

The untrusted app subnet(s)1662can include one or more primary VNICs1664(1)-(N) that can be communicatively coupled to tenant virtual machines (VMs)1666(1)-(N). Each tenant VM1666(1)-(N) can be communicatively coupled to a respective app subnet1667(1)-(N) that can be contained in respective container egress VCNs1668(1)-(N) that can be contained in respective customer tenancies1670(1)-(N). Respective secondary VNICs1672(1)-(N) can facilitate communication between the untrusted app subnet(s)1662contained in the data plane VCN1618and the app subnet contained in the container egress VCNs1668(1)-(N). Each container egress VCNs1668(1)-(N) can include a NAT gateway1638that can be communicatively coupled to public Internet1654(e.g., public Internet1454ofFIG.14).

The Internet gateway1634contained in the control plane VCN1616and contained in the data plane VCN1618can be communicatively coupled to a metadata management service1652(e.g., the metadata management system1452ofFIG.14) that can be communicatively coupled to public Internet1654. Public Internet1654can be communicatively coupled to the NAT gateway1638contained in the control plane VCN1616and contained in the data plane VCN1618. The service gateway1636contained in the control plane VCN1616and contained in the data plane VCN1618can be communicatively couple to cloud services1656.

In some examples, the customer of the IaaS provider may grant temporary network access to the IaaS provider and request a function to be attached to the data plane app tier1646. Code to run the function may be executed in the VMs1666(1)-(N), and the code may not be configured to run anywhere else on the data plane VCN1618. Each VM1666(1)-(N) may be connected to one customer tenancy1670. Respective containers1671(1)-(N) contained in the VMs1666(1)-(N) may be configured to run the code. In this case, there can be a dual isolation (e.g., the containers1671(1)-(N) running code, where the containers1671(1)-(N) may be contained in at least the VM1666(1)-(N) that are contained in the untrusted app subnet(s)1662), which may help prevent incorrect or otherwise undesirable code from damaging the network of the IaaS provider or from damaging a network of a different customer. The containers1671(1)-(N) may be communicatively coupled to the customer tenancy1670and may be configured to transmit or receive data from the customer tenancy1670. The containers1671(1)-(N) may not be configured to transmit or receive data from any other entity in the data plane VCN1618. Upon completion of running the code, the IaaS provider may kill or otherwise dispose of the containers1671(1)-(N).

In some embodiments, the trusted app subnet(s)1660may run code that may be owned or operated by the IaaS provider. In this embodiment, the trusted app subnet(s)1660may be communicatively coupled to the DB subnet(s)1630and be configured to execute CRUD operations in the DB subnet(s)1630. The untrusted app subnet(s)1662may be communicatively coupled to the DB subnet(s)1630, but in this embodiment, the untrusted app subnet(s) may be configured to execute read operations in the DB subnet(s)1630. The containers1671(1)-(N) that can be contained in the VM1666(1)-(N) of each customer and that may run code from the customer may not be communicatively coupled with the DB subnet(s)1630.

In other embodiments, the control plane VCN1616and the data plane VCN1618may not be directly communicatively coupled. In this embodiment, there may be no direct communication between the control plane VCN1616and the data plane VCN1618. However, communication can occur indirectly through at least one method. An LPG1610may be established by the IaaS provider that can facilitate communication between the control plane VCN1616and the data plane VCN1618. In another example, the control plane VCN1616or the data plane VCN1618can make a call to cloud services1656via the service gateway1636. For example, a call to cloud services1656from the control plane VCN1616can include a request for a service that can communicate with the data plane VCN1618.

FIG.17is a block diagram1700illustrating another example pattern of an IaaS architecture, according to at least one embodiment. Service operators1702(e.g., service operators1402ofFIG.14) can be communicatively coupled to a secure host tenancy1704(e.g., the secure host tenancy1404ofFIG.14) that can include a virtual cloud network (VCN)1706(e.g., the VCN1406ofFIG.14) and a secure host subnet1708(e.g., the secure host subnet1408ofFIG.14). The VCN1706can include an LPG1710(e.g., the LPG1410ofFIG.14) that can be communicatively coupled to an SSH VCN1712(e.g., the SSH VCN1412ofFIG.14) via an LPG1710contained in the SSH VCN1712. The SSH VCN1712can include an SSH subnet1714(e.g., the SSH subnet1414ofFIG.14), and the SSH VCN1712can be communicatively coupled to a control plane VCN1716(e.g., the control plane VCN1416ofFIG.14) via an LPG1710contained in the control plane VCN1716and to a data plane VCN1718(e.g., the data plane1418ofFIG.14) via an LPG1710contained in the data plane VCN1718. The control plane VCN1716and the data plane VCN1718can be contained in a service tenancy1719(e.g., the service tenancy1419ofFIG.14).

The control plane VCN1716can include a control plane DMZ tier1720(e.g., the control plane DMZ tier1420ofFIG.14) that can include LB subnet(s)1722(e.g., LB subnet(s)1422ofFIG.14), a control plane app tier1724(e.g., the control plane app tier1424ofFIG.14) that can include app subnet(s)1726(e.g., app subnet(s)1426ofFIG.14), a control plane data tier1728(e.g., the control plane data tier1428ofFIG.14) that can include DB subnet(s)1730(e.g., DB subnet(s)1630ofFIG.16). The LB subnet(s)1722contained in the control plane DMZ tier1720can be communicatively coupled to the app subnet(s)1726contained in the control plane app tier1724and to an Internet gateway1734(e.g., the Internet gateway1434ofFIG.14) that can be contained in the control plane VCN1716, and the app subnet(s)1726can be communicatively coupled to the DB subnet(s)1730contained in the control plane data tier1728and to a service gateway1736(e.g., the service gateway ofFIG.14) and a network address translation (NAT) gateway1738(e.g., the NAT gateway1438ofFIG.14). The control plane VCN1716can include the service gateway1736and the NAT gateway1738.

The data plane VCN1718can include a data plane app tier1746(e.g., the data plane app tier1446ofFIG.14), a data plane DMZ tier1748(e.g., the data plane DMZ tier1448ofFIG.14), and a data plane data tier1750(e.g., the data plane data tier1450ofFIG.14). The data plane DMZ tier1748can include LB subnet(s)1722that can be communicatively coupled to trusted app subnet(s)1760(e.g., trusted app subnet(s)1660ofFIG.16) and untrusted app subnet(s)1762(e.g., untrusted app subnet(s)1662ofFIG.16) of the data plane app tier1746and the Internet gateway1734contained in the data plane VCN1718. The trusted app subnet(s)1760can be communicatively coupled to the service gateway1736contained in the data plane VCN1718, the NAT gateway1738contained in the data plane VCN1718, and DB subnet(s)1730contained in the data plane data tier1750. The untrusted app subnet(s)1762can be communicatively coupled to the service gateway1736contained in the data plane VCN1718and DB subnet(s)1730contained in the data plane data tier1750. The data plane data tier1750can include DB subnet(s)1730that can be communicatively coupled to the service gateway1736contained in the data plane VCN1718.

The untrusted app subnet(s)1762can include primary VNICs1764(1)-(N) that can be communicatively coupled to tenant virtual machines (VMs)1766(1)-(N) residing within the untrusted app subnet(s)1762. Each tenant VM1766(1)-(N) can run code in a respective container1767(1)-(N), and be communicatively coupled to an app subnet1726that can be contained in a data plane app tier1746that can be contained in a container egress VCN1768. Respective secondary VNICs1772(1)-(N) can facilitate communication between the untrusted app subnet(s)1762contained in the data plane VCN1718and the app subnet contained in the container egress VCN1768. The container egress VCN can include a NAT gateway1738that can be communicatively coupled to public Internet1754(e.g., public Internet1454ofFIG.14).

The Internet gateway1734contained in the control plane VCN1716and contained in the data plane VCN1718can be communicatively coupled to a metadata management service1752(e.g., the metadata management system1452ofFIG.14) that can be communicatively coupled to public Internet1754. Public Internet1754can be communicatively coupled to the NAT gateway1738contained in the control plane VCN1716and contained in the data plane VCN1718. The service gateway1736contained in the control plane VCN1716and contained in the data plane VCN1718can be communicatively couple to cloud services1756.

In some examples, the pattern illustrated by the architecture of block diagram1700ofFIG.17may be considered an exception to the pattern illustrated by the architecture of block diagram1600ofFIG.16and may be desirable for a customer of the IaaS provider if the IaaS provider cannot directly communicate with the customer (e.g., a disconnected region). The respective containers1767(1)-(N) that are contained in the VMs1766(1)-(N) for each customer can be accessed in real-time by the customer. The containers1767(1)-(N) may be configured to make calls to respective secondary VNICs1772(1)-(N) contained in app subnet(s)1726of the data plane app tier1746that can be contained in the container egress VCN1768. The secondary VNICs1772(1)-(N) can transmit the calls to the NAT gateway1738that may transmit the calls to public Internet1754. In this example, the containers1767(1)-(N) that can be accessed in real-time by the customer can be isolated from the control plane VCN1716and can be isolated from other entities contained in the data plane VCN1718. The containers1767(1)-(N) may also be isolated from resources from other customers.

In other examples, the customer can use the containers1767(1)-(N) to call cloud services1756. In this example, the customer may run code in the containers1767(1)-(N) that requests a service from cloud services1756. The containers1767(1)-(N) can transmit this request to the secondary VNICs1772(1)-(N) that can transmit the request to the NAT gateway that can transmit the request to public Internet1754. Public Internet1754can transmit the request to LB subnet(s)1722contained in the control plane VCN1716via the Internet gateway1734. In response to determining the request is valid, the LB subnet(s) can transmit the request to app subnet(s)1726that can transmit the request to cloud services1756via the service gateway1736.

FIG.18illustrates an example computer system1800, in which various embodiments may be implemented. The system1800may be used to implement any of the computer systems described above. As shown in the figure, computer system1800includes a processing unit1804that communicates with a number of peripheral subsystems via a bus subsystem1802. These peripheral subsystems may include a processing acceleration unit1806, an I/O subsystem1808, a storage subsystem1818and a communications subsystem1824. Storage subsystem1818includes tangible computer-readable storage media1822and a system memory1810.

Processing unit1804, which can be implemented as one or more integrated circuits (e.g., a conventional microprocessor or microcontroller), controls the operation of computer system1800. One or more processors may be included in processing unit1804. These processors may include single core or multicore processors. In certain embodiments, processing unit1804may be implemented as one or more independent processing units1832and/or1834with single or multicore processors included in each processing unit. In other embodiments, processing unit1804may also be implemented as a quad-core processing unit formed by integrating two dual-core processors into a single chip.

In various embodiments, processing unit1804can execute a variety of programs in response to program code and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed can be resident in processor(s)1804and/or in storage subsystem1818. Through suitable programming, processor(s)1804can provide various functionalities described above. Computer system1800may additionally include a processing acceleration unit1806, which can include a digital signal processor (DSP), a special-purpose processor, and/or the like.

Computer system1800may comprise a storage subsystem1818that provides a tangible non-transitory computer-readable storage medium for storing software and data constructs that provide the functionality of the embodiments described in this disclosure. The software can include programs, code modules, instructions, scripts, etc., that when executed by one or more cores or processors of processing unit1804provide the functionality described above. Storage subsystem1818may also provide a repository for storing data used in accordance with the present disclosure.

As depicted in the example inFIG.18, storage subsystem1818can include various components including a system memory1810, computer-readable storage media1822, and a computer readable storage media reader1820. System memory1810may store program instructions that are loadable and executable by processing unit1804. System memory1810may also store data that is used during the execution of the instructions and/or data that is generated during the execution of the program instructions. Various different kinds of programs may be loaded into system memory1810including but not limited to client applications, Web browsers, mid-tier applications, relational database management systems (RDBMS), virtual machines, containers, etc.

System memory1810may also store an operating system1816. Examples of operating system1816may include various versions of Microsoft Windows®, Apple Macintosh®, and/or Linux operating systems, a variety of commercially-available UNIX® or UNIX-like operating systems (including without limitation the variety of GNU/Linux operating systems, the Google Chrome® OS, and the like) and/or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, and Palm® OS operating systems. In certain implementations where computer system1800executes one or more virtual machines, the virtual machines along with their guest operating systems (GOSs) may be loaded into system memory1810and executed by one or more processors or cores of processing unit1804.

System memory1810can come in different configurations depending upon the type of computer system1800. For example, system memory1810may be volatile memory (such as random access memory (RAM)) and/or non-volatile memory (such as read-only memory (ROM), flash memory, etc.) Different types of RAM configurations may be provided including a static random access memory (SRAM), a dynamic random access memory (DRAM), and others. In some implementations, system memory1810may include a basic input/output system (BIOS) containing basic routines that help to transfer information between elements within computer system1800, such as during start-up.

Computer-readable storage media1822may represent remote, local, fixed, and/or removable storage devices plus storage media for temporarily and/or more permanently containing, storing, computer-readable information for use by computer system1800including instructions executable by processing unit1804of computer system1800.

Computer-readable storage media1822can include any appropriate media known or used in the art, including storage media and communication media, such as but not limited to, volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage and/or transmission of information. This can include tangible computer-readable storage media such as RAM, ROM, electronically erasable programmable ROM (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or other tangible computer readable media.

Machine-readable instructions executable by one or more processors or cores of processing unit1804may be stored on a non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium can include physically tangible memory or storage devices that include volatile memory storage devices and/or non-volatile storage devices. Examples of non-transitory computer-readable storage medium include magnetic storage media (e.g., disk or tapes), optical storage media (e.g., DVDs, CDs), various types of RAM, ROM, or flash memory, hard drives, floppy drives, detachable memory drives (e.g., USB drives), or other type of storage device.

Communications subsystem1824provides an interface to other computer systems and networks. Communications subsystem1824serves as an interface for receiving data from and transmitting data to other systems from computer system1800. For example, communications subsystem1824may enable computer system1800to connect to one or more devices via the Internet. In some embodiments communications subsystem1824can include radio frequency (RF) transceiver components for accessing wireless voice and/or data networks (e.g., using cellular telephone technology, advanced data network technology, such as 3G, 4G or EDGE (enhanced data rates for global evolution), WiFi (IEEE 802.11 family standards, or other mobile communication technologies, or any combination thereof), global positioning system (GPS) receiver components, and/or other components. In some embodiments communications subsystem1824can provide wired network connectivity (e.g., Ethernet) in addition to or instead of a wireless interface.

In some embodiments, communications subsystem1824may also receive input communication in the form of structured and/or unstructured data feeds1826, event streams1828, event updates1830, and the like on behalf of one or more users who may use computer system1800.

By way of example, communications subsystem1824may be configured to receive data feeds1826in real-time from users of social networks and/or other communication services such as Twitter® feeds, Facebook® updates, web feeds such as Rich Site Summary (RSS) feeds, and/or real-time updates from one or more third party information sources.

Additionally, communications subsystem1824may also be configured to receive data in the form of continuous data streams, which may include event streams1828of real-time events and/or event updates1830, that may be continuous or unbounded in nature with no explicit end. Examples of applications that generate continuous data may include, for example, sensor data applications, financial tickers, network performance measuring tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, and the like.

Communications subsystem1824may also be configured to output the structured and/or unstructured data feeds1826, event streams1828, event updates1830, and the like to one or more databases that may be in communication with one or more streaming data source computers coupled to computer system1800.