Patent Description:
The disclosure relates to computer networks and, more specifically, to Virtual Network Functions provided on computer networks.

Virtualized data centers are becoming a core foundation of the modem information technology (IT) infrastructure. In particular, modern data centers have extensively utilized virtualized environments in which virtual hosts, also referred to herein as virtual execution elements, such virtual machines or containers, are deployed and executed on an underlying compute platform of physical computing devices. Virtualization within a data center can provide several advantages. One advantage is that virtualization can provide significant improvements to efficiency. As the underlying physical computing devices (i.e., servers) have become increasingly powerful with the advent of multicore microprocessor architectures with a large number of cores per physical CPU, the benefits of virtualization have increased substantially. As another advantage, virtualized network functions may be executed by servers, switches, storage devices, and cloud computing infrastructure, instead of having custom hardware appliances for each network function.

A network operator may provide a Network Functions Virtualization (NFV) architecture or infrastructure (NFVI) that is capable of orchestration and management of multiple virtualized network functions (VNFs) or physical devices that apply network functions (or "services") to flows of packets in an ordered sequence. An example VNF instance may provide firewall, routing/switching, carrier grade network address translation (CG-NAT), performance enhancement proxies for video, transport control protocol (TCP) optimization and header enrichment, caching, load balancing, or other network functions. To implement a service chain, VNF instances are interconnected using one or more virtual networks, by which packet flows are forwarded along the ordered sequence for application of the network functions that make up the service chain. A service in a service chain that is applied by one or more VNF instances may be scaled up, responsive to increased load, by spawning more VNF instances. Likewise, the service may be scaled down, responsive to decreased load, by deleting one or more of the VNF instances spawned for the service. The one or more VNF instances for a single service may be hosted by separate computing devices, e.g., compute nodes or servers, but a computing device may host multiple VNF instances for one or more services.

In general, this disclosure describes techniques in which a network service orchestrator for network function virtualization infrastructure (NFVI) dynamically allocates compute resources among virtual network functions assigned to tenants based on tenant-flow statistics. For example, an NFVI provider may deploy NFVI that hosts VNFs that are assigned to multiple different customers tenants, also referred to more simply as tenants or customers, of the NFVI provider. Each VNF executes its corresponding network function to process packet flows that are associated with the one of the tenants of the NFVI provider to which the VNF is assigned. As described herein, a network service orchestrator for the NFVI dynamically distributes physical resources, such as a number of processing cores in a server of the NFVI, among the VNFs executing on the server based on actual utilization of the VNFs for processing flows associated with tenants assigned to the VNFs.

The described techniques may provide one or more technical advantages that present at least one practical application. For example, techniques described herein may improve compute resource utilization by enabling a server's compute resources to execute more VNFs through on-demand resource allocation while still meeting tenant service level agreements (SLAs) for flow processing. For instance, when a tenant's actual bandwidth utilization allocated to a VNF, as determined using tenant-flow statistics, does not reach the allocated bandwidth for the VNF (and corresponding processing resource allocation under a static allocation scheme), there is an envelope of unused processing resources that is dynamically re-assigned to one or more other VNFs. As such, implementing these techniques described herein may increase the maximum number of VNFs executing at one time on a server.

As additional advantages, these techniques allow for tenant-driven dynamic resource allocation by an orchestration system in a data center rather than from a component within the same server as the tenant's VNF. The orchestration system provides a number of benefits to resource allocation, including access to tenant-flow statistics for multiple VNFs spread across the NFVI and assigned to multiple different tenant. For at least this reason, such dynamic resource allocation may be performed even when the tenant's customer devices are not in the same data center environment (e.g., an interconnection facility) as the NFVI on which the tenant VNF is executing.

According to one aspect, there is provided an orchestration system operated by a data center provider for a data center, the orchestration system comprising: processing circuitry coupled to a memory; and software instructions stored in the memory and configured for execution by the processing circuitry to: map packet flow virtual network identifiers to a virtual network identifier space of a plurality of tenants of the data center, wherein a tenant of the data center is associated with a first virtual network identifier of the virtual network identifier space and a second virtual network identifier of the virtual network identifier space; identify flow statistics corresponding to the tenant based upon the first virtual network identifier and the second virtual network identifier in the mapping; compute, based on the flow statistics indicating a first packet flow having the first virtual network identifier associated with the tenant and a second packet flow having the second virtual network identifier associated with the tenant, an aggregate bandwidth for a plurality of packet flows associated with the tenant and processed by a virtual network function that is assigned to the tenant and executes on physical compute resources of a server of the data center; and modify, based on the aggregate bandwidth, an allocation of the physical compute resources of the server to re-assign a portion of the physical compute resources of the server between the virtual network function assigned to the tenant and one or more virtual network functions associated with a different tenant.

According to a second aspect, there is provided a method of an orchestration system operated by a data center provider for a data center, the method comprising: mapping packet flow virtual network identifiers to a virtual network identifier space of a plurality of tenants of the data center, wherein a tenant of the data center is associated with a first virtual network identifier of the virtual network identifier space and a second virtual network identifier of the virtual network identifier space; identifying flow statistics corresponding to the tenant based upon the first virtual network identifier and the second virtual network identifier in the mapping; computing, based on the flow statistics indicating a first packet flow having the first virtual network identifier associated with the tenant and a second packet flow having the second virtual network identifier associated with the tenant, an aggregate bandwidth for a plurality of packet flows associated with the tenant and processed by a virtual network function, that is assigned to the tenant and executes on physical compute resources of a server of the data center; and modifying, based on the aggregate bandwidth, an allocation of the physical compute resources of the server to reassign a portion of the physical compute resources of the server between the virtual network function assigned to the tenant and one or more virtual network functions associated with a different tenant.

In accordance with a third aspect, there is provided an interconnection system comprising: at least one interconnection facility and a programmable network platform, the at least one interconnection facility including a cluster comprising one or more computing devices to host virtual network functions for tenants, the programmable network platform being configured to: map packet flow virtual network identifiers to a virtual network identifier space of a plurality of tenants of a data center, wherein a tenant of the data center is associated with a first virtual network identifier of the virtual network identifier space and a second virtual network identifier of the virtual network identifier space; identify flow statistics corresponding to the tenant based upon the first virtual network identifier and the second virtual network identifier in the mapping; compute, based on the flow statistics indicating a first packet flow having the first virtual network identifier associated with the tenant and a second packet flow having the second virtual network identifier associated with the tenant, an aggregate bandwidth for a plurality of packet flows associated with the tenant and processed by a virtual network function that is assigned to the tenant and executes on physical compute resources of a server; and modify, based on the aggregate bandwidth, an allocation of the physical compute resources of the server to re-assign a portion of the physical compute resources of the server between the virtual network function assigned to the tenant and one or more virtual network functions associated with a different tenant.

<FIG> is a block diagram that illustrates a conceptual view of a network system having a metro-based cloud exchange that provides multiple cloud exchange points and with network function virtualization infrastructure, in accordance with techniques described herein. Each of cloud-based services exchange points 120A-120C (described hereinafter as "cloud exchange points" and collectively referred to as "cloud exchange points <NUM>") of cloud-based services exchange <NUM> ("cloud exchange <NUM>") may represent a different data center (or interconnection facility) geographically located within the same metropolitan area ("metro-based," e.g., in New York City, New York; Silicon Valley, California; Seattle-Tacoma, Washington; Minneapolis-St. Paul, Minnesota; London, UK; etc.) to provide resilient and independent cloud-based services exchange by which cloud-based services customers ("cloud customers") and cloud-based service providers ("cloud providers") connect to receive and provide, respectively, cloud services. In various examples, cloud exchange <NUM> may include more or fewer cloud exchange points <NUM>. In some instances, a cloud exchange <NUM> includes just one cloud exchange point <NUM>. As used herein, reference to a "cloud exchange" or "cloud-based services exchange" may refer to a cloud exchange point. A cloud exchange provider may deploy instances of cloud exchanges <NUM> in multiple different metropolitan areas, each instance of cloud exchange <NUM> having one or more cloud exchange points <NUM>.

Any one or more of cloud exchange points <NUM> may be used to implement, at least in part, NFVI <NUM>. Each of cloud exchange points <NUM> includes network infrastructure and an operating environment by which cloud customers 108A-108C (collectively, "cloud customers <NUM>") receive cloud services from multiple cloud service providers 110A-110N (collectively, "cloud service providers <NUM>"). The cloud service provider <NUM> may host one of more cloud services. As noted above, the cloud service providers <NUM> may be public or private cloud service providers.

Cloud exchange <NUM> provides customers of the exchange, e.g., enterprises, network carriers, network service providers, and SaaS customers, with secure, private, virtual connections to multiple cloud service providers (CSPs) globally. The multiple CSPs participate in the cloud exchange by virtue of their having at least one accessible port in the cloud exchange by which a customer may connect to the one or more cloud services offered by the CSPs, respectively. Cloud exchange <NUM> allows private networks of any customer to be directly cross-connected to any other customer at a common point, thereby allowing direct exchange of network traffic between the networks of the customers.

Cloud customers <NUM> may receive cloud-based services directly via a layer <NUM> peering and physical connection to one of cloud exchange points <NUM> or indirectly via one of network service providers 106A-106B (collectively, "NSPs <NUM>," or alternatively, "carriers <NUM>"). Cloud customers <NUM> may include customers associated with a VNF 306A as described above. For example, cloud customers <NUM> may include systems used by any or all of customer devices used by cloud client <NUM> to access cloud services via VNF executing in NFVI <NUM> of cloud exchange points <NUM>. NSPs <NUM> provide "cloud transit" by maintaining a physical presence within one or more of cloud exchange points <NUM> and aggregating layer <NUM> access from one or customers <NUM>. NSPs <NUM> may peer, at layer <NUM>, directly with one or more cloud exchange points <NUM> and in so doing offer indirect layer <NUM> connectivity and peering to one or more customers <NUM> by which customers <NUM> may obtain cloud services from the cloud exchange <NUM>. Each of cloud exchange points <NUM>, in the example of <FIG>, is assigned a different autonomous system number (ASN). For example, cloud exchange point 120A is assigned ASN <NUM>, cloud exchange point 120B is assigned ASN <NUM>, and so forth. Each cloud exchange point <NUM> is thus a next hop in a path vector routing protocol (e.g., BGP) path from cloud service providers <NUM> to customers <NUM>. As a result, each cloud exchange point <NUM> may, despite not being a transit network having one or more wide area network links and concomitant Internet access and transit policies, peer with multiple different autonomous systems via external BGP (eBGP) or other exterior gateway routing protocol in order to exchange, aggregate, and route service traffic from one or more cloud service providers <NUM> to customers. In other words, cloud exchange points <NUM> may internalize the eBGP peering relationships that cloud service providers <NUM> and customers <NUM> would maintain on a pair-wise basis. Instead, a customer <NUM> may configure a single eBGP peering relationship with a cloud exchange point <NUM> and receive, via the cloud exchange, multiple cloud services from one or more cloud service providers <NUM>. While described herein primarily with respect to eBGP or other layer <NUM> routing protocol peering between cloud exchange points and customer, NSP, or cloud service provider networks, the cloud exchange points may learn routes from these networks in other way, such as by static configuration, or via Routing Information Protocol (RIP), Open Shortest Path First (OSPF), Intermediate System-to-Intermediate System (IS-IS), or other route distribution protocol.

As examples of the above, customer 108C is illustrated as having contracted with a cloud exchange provider for cloud exchange <NUM> to directly access layer <NUM> cloud services via cloud exchange points 120C. In this way, customer 108C receives redundant layer <NUM> connectivity to cloud service provider 110A, for instance. Customer 108C, in contrast, is illustrated as having contracted with the cloud exchange provider for cloud exchange <NUM> to directly access layer <NUM> cloud services via cloud exchange point 120C and also to have contracted with NSP 106B to access layer <NUM> cloud services via a transit network of the NSP 106B. Customer 108B is illustrated as having contracted with multiple NSPs 106A, 106B to have redundant cloud access to cloud exchange points 120A, 120B via respective transit networks of the NSPs 106A, 106B. The contracts described above are instantiated in network infrastructure of the cloud exchange points <NUM> by L3 peering configurations within switching devices of NSPs <NUM> and cloud exchange points <NUM> and L3 connections, e.g., layer <NUM> virtual circuits, established within cloud exchange points <NUM> to interconnect cloud service provider <NUM> networks to NSPs <NUM> networks and customer <NUM> networks, all having at least one port offering connectivity within one or more of the cloud exchange points <NUM>.

In some examples, cloud exchange <NUM> allows a corresponding one of customer customers 108A, 108B of any network service providers (NSPs) or "carriers" 106A-106B (collectively, "carriers <NUM>") or other cloud customers including customers 108C to be directly connected, via a virtual layer <NUM> (L2) or layer <NUM> (L3) connection to any other customer network and/or to any of CSPs <NUM>, thereby allowing direct exchange of network traffic among the customer networks and CSPs <NUM>. The virtual L2 or L3 connection may be referred to as a "virtual circuit.

Carriers <NUM> may each represent a network service provider that is associated with a transit network by which network subscribers of the carrier <NUM> may access cloud services offered by CSPs <NUM> via the cloud exchange <NUM>. In general, customers of CSPs <NUM> may include network carriers, large enterprises, managed service providers (MSPs), as well as Software-as-a-Service (SaaS), Platform-aaS (PaaS), Infrastructure-aaS (IaaS), Virtualization-aaS (VaaS), and data Storage-aaS (dSaaS) customers for such cloud-based services as are offered by the CSPs <NUM> via the cloud exchange <NUM>.

In this way, cloud exchange <NUM> streamlines and simplifies the process of partnering CSPs <NUM> and customers (via carriers <NUM> or directly) in a transparent and neutral manner. One example application of cloud exchange <NUM> is a co-location and interconnection data center in which CSPs <NUM> and carriers <NUM> and/or customers <NUM> may already have network presence, such as by having one or more accessible ports available for interconnection within the data center, which may represent any of cloud exchange points <NUM>. This allows the participating carriers, customers, and CSPs to have a wide range of interconnectivity options within the same facility. A carrier/customer may in this way have options to create many-to-many interconnections with only a one-time hook up to one or more cloud exchange points <NUM>. In other words, instead of having to establish separate connections across transit networks to access different cloud service providers or different cloud services of one or more cloud service providers, cloud exchange <NUM> allows customers to interconnect to multiple CSPs and cloud services.

Cloud exchange <NUM> includes programmable network platform <NUM> for dynamically programming cloud exchange <NUM> to responsively and assuredly fulfill service requests that encapsulate business requirements for services provided by cloud exchange <NUM> and/or cloud service providers <NUM> coupled to the cloud exchange <NUM>. Programmable network platform <NUM> may include network service orchestrator <NUM> that handles tenant (e.g., cloud client) requests for deployment of VNFs. For example, network service orchestrator <NUM> may organize, direct and integrate underlying services through virtual machines (or containers), as well as other software and network subsystems, for managing various services (e.g., deployment of VNFs). The programmable network platform <NUM> may, as a result, orchestrate a business-level service across heterogeneous cloud service providers <NUM> according to well-defined service policies, quality of service policies, service level agreements, and costs, and further according to a service topology for the business-level service.

Hardware and/or software components for NFVI <NUM> implement network management and resource orchestration systems of which at least one, in general, performs at least one technique described herein. In one example, the network management and resource orchestration systems form an architecture having at least three functional blocks of which one example functional block, an orchestration system, is responsible for onboarding of new network services (NS) and virtual network function (VNF) packages; NS lifecycle management; global and local resource management; and validation and authorization of network functions virtualization infrastructure (NFVI) resource requests. In some examples, network service orchestrator <NUM> in programmable network platform <NUM> is an example of the orchestration system while, in other examples, the orchestration system instructs (e.g., by way of function call) network service orchestrator <NUM> to dynamically allocate resources (e.g., compute) resources. Other functional blocks (e.g., management blocks) oversee lifecycle management of VNF instances; fill the coordination and adaptation role for configuration and event reporting between NFV infrastructure (NFVI) and Element/Network Management Systems, and control and manage the NFVI compute, storage, and network resources. While shown separately as part of programmable network platform <NUM>, these management blocks may reside in NFVI <NUM> and cooperate with the orchestration system when deploying VNFs.

NFVI <NUM> includes one or more servers 123A-123N (servers <NUM>) for executing/hosting virtual network functions (VNFs) 124A-124N that apply network services to packet flows. Network service orchestrator <NUM> handles deployment and organization of these network services, for example, by instantiating VNFs on servers <NUM> to execute such network services.

The programmable network platform <NUM> enables the cloud service providers <NUM> that administer the cloud exchange <NUM> to dynamically configure and manage the cloud exchange <NUM> to, for instance, facilitate virtual connections for cloud-based services delivery from multiple cloud service providers <NUM> to one or more cloud customers <NUM>. The cloud exchange <NUM> may enable cloud customers <NUM> to bypass the public Internet to directly connect to cloud services providers <NUM> so as to improve performance, reduce costs, increase the security and privacy of the connections, and leverage cloud computing for additional applications. In this way, enterprises, network carriers, and SaaS customers, for instance, can at least in some aspects integrate cloud services with their internal applications as if such services are part of or otherwise directly coupled to their own data center network.

In other examples, programmable network platform <NUM> enables the cloud service provider to configure cloud exchange <NUM> with a L3 instance requested by a cloud customer <NUM>, as described herein. A customer <NUM> may request an L3 instance to link multiple cloud service providers by the L3 instance, for example (e.g., for transferring the customer's data between two cloud service providers, or for obtaining a mesh of services from multiple cloud service providers).

Programmable network platform <NUM> may represent an application executing within one or more data centers of the cloud exchange <NUM> or alternatively, off-site at a back office or branch of the cloud provider (for instance). Programmable network platform <NUM> may be distributed in whole or in part among the data centers, each data center associated with a different cloud exchange point <NUM> to make up the cloud exchange <NUM>. Although shown as administering a single cloud exchange <NUM>, programmable network platform <NUM> may control service provisioning for multiple different cloud exchanges. Alternatively or additionally, multiple separate instances of the programmable network platform <NUM> may control service provisioning for respective multiple different cloud exchanges.

In the illustrated example, programmable network platform <NUM> includes a service interface (or "service API") <NUM> that defines the methods, fields, and/or other software primitives by which applications <NUM>, such as a customer portal, may invoke the programmable network platform <NUM>. The service interface <NUM> may allow carriers <NUM>, customers <NUM>, cloud service providers <NUM>, and/or the cloud exchange provider programmable access to capabilities and assets of the cloud exchange <NUM> according to techniques described herein.

For example, the service interface <NUM> may facilitate machine-to-machine communication to enable dynamic provisioning of virtual circuits in the cloud exchange for interconnecting customer and/or cloud service provider networks. In this way, the programmable network platform <NUM> enables the automation of aspects of cloud services provisioning. For example, the service interface <NUM> may provide an automated and seamless way for customers to establish, de-install and manage interconnections among multiple, different cloud providers participating in the cloud exchange.

Further example details of a cloud-based services exchange can be found in <CIT> and entitled "CLOUD-BASED SERVICES EXCHANGE;" <CIT> and entitled "INTERCONNECTION PLATFORM FOR REAL-TIME CONFIGURATION AND MANAGEMENT OF A CLOUD-BASED SERVICES EXCHANGE;" and <CIT> and entitled "ORCHESTRATION ENGINE FOR REAL-TIME CONFIGURATION AND MANAGEMENT OF INTERCONNECTIONS WITHIN A CLOUD-BASED SERVICES EXCHANGE;".

Customer 108B represents (or includes) a tenant network for cloud exchange <NUM>. Customer 108B exchanges packetized data in packet flows with one or more other networks, e.g., via virtual circuits or other connection through cloud exchange point 120A. In some cases, NFVI <NUM> applies one or more network services to the packet flows on behalf of a tenant associated with customer <NUM>.

In the illustrated example, customer 108B exchanges packets for packet flow 125A with cloud service provider network 110A and for packet flow 125B with cloud service provider network 110N. A VNF of the one or more VNFs 124A executed by server 123A applies a network function, in some cases as part of a network service, to packets for the packet flows 125A-125B (collectively, "packet flows <NUM>"). This VNF may be referred to as the tenant VNF in that it is assigned to or otherwise associated with a tenant of the NFVI <NUM> provider (here, also the cloud exchange <NUM> provider). In other words, the tenant VNF processes packet flows <NUM> associated with the tenant. The tenant VNF may be leased or sold to the tenant by NFVI <NUM> provider, or the tenant may deploy the tenant VNF to server 123A or virtual execution elements thereof. Virtual execution elements may include virtual machines or containers. The tenant may invoke service interface <NUM> or otherwise request a service chain in NFVI <NUM> that includes the tenant VNF for packet flows destined to and/or sourced by customer 108B. Server 123A may execute multiple VNFs 124A associated with multiple different tenants, each VNF of the VNFs 124A processing one or more packet flows associated with the tenant for that VNF.

In accordance with techniques of this disclosure, network service orchestrator <NUM> for NFVI <NUM> dynamically distributes physical resources, such as a number of processing cores in each of servers <NUM>, among the VNFs <NUM> executing on any of servers <NUM> based on actual utilization of the VNFs <NUM> for processing flows associated with tenants associated with the VNFs <NUM>. For example, network service orchestrator <NUM> generates flow statistics for server 123A that indicates a bandwidth for each of flows <NUM> forwarded by server 123A. That is, server 123A performs flow monitoring of flows traversing server 123A and generates flow records <NUM> for the flows. Network service orchestrator <NUM> collects the flow records, which are usable for computing an estimated bandwidth for each of the flows as well as other flow properties, such as the virtual network interface of server 123A used by the flow and the source and destination network address for the flow. Using the collected flow records, network service orchestrator <NUM> computes estimated bandwidths for each of flows <NUM>. In addition, network service orchestrator <NUM> determines, based on the flow records, that flows <NUM> are associated with customer 108B that is a tenant of the NFVI provider. Network service orchestrator <NUM> may consolidate this computed data into tenant-flow statistics.

In an example where packet flows are encapsulated in frames that are transmitted under User Datagram Protocol (UDP), perform the following steps to compute the estimated bandwidth:.

In an example where packet flows are transmitted under Transmission Control Protocol (TCP), perform the following steps to compute the estimated bandwidth:.

Tenant-flow statistics generally includes measured or computed packet flow information (e.g., packet size, packets per second, estimated bandwidth, and/or the like) for flows associated with each of one or more tenants of the NFVI <NUM> provider. As described above, network service orchestrator <NUM> may map tenant information to flow records to determine which flows are associated with at least one specific tenant. As described herein, the tenant-flow statistics may indicate actual aggregate bandwidth processed by a tenant VNF for a tenant and that information can be compared with resource allocation information to dynamically allocate resources. For example, in response to determining that the aggregate bandwidth processed by a tenant VNF of VNFs 124A does not satisfy a bandwidth allocation/requirement, the techniques described herein may modify an allocation of computer resources of server 123A, such as a number of processor cores or an amount of memory, allocated for use by the tenant VNF.

In this way, by determining a rate at which a server 123A forwards each of packets flows <NUM> associated with a tenant, the rates correspondingly indicates the rates at which tenant VNF processes packet flows <NUM> (e.g., an aggregate bandwidth or throughput of the tenant VNF). These rates can be combined to obtain an aggregate bandwidth, and in response to determining the aggregate bandwidth for the packet flows <NUM> associated with the tenant and processed by the tenant VNF, network service orchestrator <NUM> may compare the aggregate bandwidth to a bandwidth requirement provided by resource allocation information for the VNFs and produce a comparison result. The comparison result may indicate an expected aggregate bandwidth for the tenant VNF that indicates more or fewer compute resources needed to process the expected aggregate bandwidth versus the currently allocated amount of compute resources for the tenant VNF. As a result and therefore based on the aggregate bandwidth, network service orchestrator <NUM> may output configuration data <NUM> to cause server <NUM> to modify an allocation of compute resources of server 123A to allocate accordingly more or fewer compute resources for executing the tenant VNF. In some examples, network service orchestrator <NUM> periodically repeats this comparison and maintains comparison results for determining a trend such that, over time, a correlation may be identified between a number of processor cores needed and actual bandwidth usage. This correlation may be a function or lookup table that network service orchestrator <NUM> uses to determine the appropriate number of processor cores to satisfy an expected bandwidth for executing the tenant VNF to continue processing flows <NUM> (and in some cases other flows associated with customer 108B). Although described herein as primarily being performed by a network service orchestrator, the techniques described in this disclosure may be performed by a separate computing system, such as a controller or application, that having determined a modification to compute resources for a tenant VNF, invokes network service orchestrator <NUM> to request the modification to the compute resources for the tenant VNF in NFVI <NUM>.

<FIG> is a block diagram illustrating an example data center that provides an operating environment for VNFs with tenant-driven resource allocation, in accordance with one or more aspects of the techniques described in this disclosure.

In this example data center, cloud exchange <NUM> allows a corresponding one of customer networks 202A, 202B and NSP networks 204A-204C (collectively, ‴NSP or 'carrier' networks <NUM>") of any NSPs 106A-106C or other customers to be directly cross-connected, via a layer <NUM> (L2) or layer <NUM> (L3) connection to any other customer network, thereby allowing exchange of service traffic among the customer networks and CSPs <NUM>. Data center <NUM> may be entirely located within a centralized area, such as a warehouse or localized data center complex, and provide power, cabling, security, and other services to NSPs, customers, and cloud service providers that locate their respective networks within the data center <NUM> (e.g., for colocation) and/or connect to the data center <NUM> by one or more external links.

Cloud exchange <NUM> includes network infrastructure <NUM> (e.g., for a virtual network) and an operating environment by which customer networks <NUM> may receive services from one or more CSPs <NUM> via interconnections. In the example of <FIG>, network infrastructure <NUM> represents the switching fabric of an interconnection facility of cloud exchange <NUM> and includes multiple ports that may be dynamically interconnected with virtual circuits by, e.g., invoking service interface <NUM> of the programmable network platform <NUM>. Each of the ports is associated with NSPs <NUM>, customers <NUM>, and CSPs <NUM>. This enables an NSP customer to have options to create many-to-many interconnections with only a one-time hook up to the switching network and underlying network infrastructure <NUM> that presents an interconnection platform for cloud exchange <NUM>. In other words, instead of having to establish separate connections across transit networks to access different CSPs <NUM>, cloud exchange <NUM> allows a customer to interconnect to multiple CSPs <NUM> using network infrastructure <NUM> within data center <NUM>.

An interconnection as described herein may refer to, e.g., a physical crossconnect, an Ethernet connection such as a Layer <NUM> VPN or virtual private LAN (e.g., E-LINE, E-LAN, E-TREE, or E-Access), an Internet exchange-based interconnection in which respective network devices (e.g., routers and/or switches) of interconnected customers directly peer and exchange layer <NUM> routes for service traffic exchanged via network infrastructure <NUM>, and a cloud exchange in which customer routers peer with network infrastructure <NUM> (or "provider") network devices rather than directly with other customers. Cloud exchange <NUM> may provide, to customers, interconnection services to network services provided by CSPs <NUM>. That is, an interconnection service by cloud exchange <NUM> provides access to a network service (e.g., VNF) provided by CSPs <NUM>.

For interconnections at layer <NUM> or above, customers <NUM> may receive services directly via a layer <NUM> peering and physical connection to one of colocation facility exchange points or indirectly via one of NSPs <NUM>. NSPs <NUM> provide "transit" by maintaining a physical presence within data center <NUM> and aggregating layer <NUM> access from one or more customers <NUM>. NSPs <NUM> may peer, at layer <NUM>, directly with data center <NUM> and in so doing offer indirect layer <NUM> connectivity and peering to one or more customers <NUM> by which customers <NUM> may obtain services from the cloud exchange <NUM>.

In instances in which cloud exchange <NUM> offers an internet exchange, network infrastructure <NUM> may be assigned a different autonomous system number (ASN). Network infrastructure <NUM> is thus a next hop in a path vector routing protocol (e.g., BGP) path from CSPs <NUM> to customers <NUM> and/or NSPs <NUM>. As a result, cloud exchange <NUM> may, despite not being a transit network having one or more wide area network links and concomitant Internet access and transit policies, peer with multiple different autonomous systems via external BGP (eBGP) or other exterior gateway routing protocol in order to exchange, aggregate, and route service traffic from one or more CSPs <NUM> to customers <NUM>. In other words, cloud exchange <NUM> may internalize the eBGP peering relationships that CSPs <NUM> and customers <NUM> would maintain on a pair-wise basis. Instead, a customer <NUM> may configure a single eBGP peering relationship with cloud exchange <NUM> and receive, via the cloud exchange, multiple services from one or more CSPs <NUM>. While described herein primarily with respect to eBGP or other layer <NUM> routing protocol peering between colocation facility points and customer, NSP, or service provider networks, the colocation facility points may learn routes from these networks in other way, such as by static configuration, or via Routing Information Protocol (RIP), Open Shortest Path First (OSPF), Intermediate System-to-Intermediate System (IS-IS), or other route distribution protocol.

As examples of the above for a cloud exchange deployment, customer network 202B in <FIG> is illustrated as having contracted with the cloud exchange provider for cloud exchange <NUM> to directly access layer <NUM> services via cloud exchange <NUM> and also to have contracted with NSP 106B to access layer <NUM> services via a transit network of NSP 106B. Customer network 202A is illustrated as having contracted with NSP 106B to access layer <NUM> services via a transit network of NSP 106B. The contracts described above may be instantiated in network infrastructure <NUM> of the cloud exchange <NUM> by L3 peering configurations within switching devices of NSPs <NUM> and cloud exchange <NUM> and L3 connections, e.g., layer <NUM> virtual circuits, established within cloud exchange <NUM> to interconnect CSPs <NUM> to NSPs <NUM> and customer networks <NUM>, all having at least one port offering connectivity within cloud exchange <NUM>.

In some examples, network infrastructure <NUM> includes one or more virtual machines or containers of NFVi <NUM> that is used to deploy Virtualized Network Functions. In these examples, network service orchestrator <NUM> may receive a request via service interface <NUM> to deploy one or more virtualized network functions (e.g., virtual router, load balancer, and/or the like) that are implemented in NFVi <NUM> of network infrastructure <NUM>. Network service orchestrator <NUM> may request a VNF distribution including one or more VNF images from a VNF provider, e.g., VNF provider <NUM>.

Further details regarding the example network system of <FIG> and example data center <NUM> of <FIG> may be found in <CIT>.

As described above with respect to <FIG>, orchestrator <NUM> may dynamically allocate compute resources for servers of NFVI <NUM> to tenant VNFs for tenants of the data center <NUM> / NFVI <NUM> provider, based on aggregate bandwidths for flows processed by the tenant VNFs.

<FIG> is a block diagram that illustrates an example architecture for Network Function Virtualization Infrastructure having tenant-driven dynamic resource allocation according to techniques described herein. In the example of <FIG>, system <NUM> refers to an exchange point (e.g., a cloud exchange point) having Network Functions Virtualization infrastructure (NFVi) <NUM> connecting customer devices, via gateway <NUM>, and cloud service providers running on cloud network <NUM>. In some examples, NFVi <NUM>, gateway <NUM>, and/or cloud network <NUM> may be provided in a data center environment.

NFVI <NUM> includes one or more servers 302A-302N (servers <NUM>) for executing/hosting network services including a virtual network device connecting customer devices with cloud services. The example architecture of NFVi <NUM> enables deployment one or more services, such as Virtualized Network Functions (VNFs), on servers <NUM>. NFVi <NUM> includes computing hardware, storage hardware, and network hardware for executing Virtual Network Functions (VNFs). NFV management <NUM> handles deployment and organization of these network services, for example, by instantiating VNFs on servers <NUM> to execute such services. As instructed by an orchestration system, NFV management <NUM> designates resources (e.g., resource capacities) in servers <NUM> to support execution of VNFs.

A VNF may provide similar functionality to hardware-based network devices such as dedicated network appliances, but VNFs provide such functionality in software. A VNF is primarily a software construct and thus may be decoupled from the underlying hardware. For example, VNF 306A can provide the same routing, switching firewall, intrusion detection or other services that have traditionally been provided by specialized hardware, but in software. VNF 306A can provide forwarding and network address translation services for network traffic to and from the VNF 306A. In some examples, VNF 306A-in a role as a routing VNF or virtual router to cloud network <NUM>-performs routing and forwarding operations on packets from customer devices.

In the example of <FIG>, NFVI <NUM> platform includes servers, e.g., server 302A, running virtualization software (e.g., hypervisors) in virtualization layers that enable virtual execution environments on which VNF images (including the network infrastructure software) are deployed. Server 302A may represent an example instance of any of servers <NUM> of <FIG>. Virtualization layer 308A may operate a platform for virtualizaing network infrastructure components (e.g., for data plane and control plane functionality) including networking protocols such as those using in routing/switching. Server 302A may provide via virtualization layer 308A one or more virtual machines (VMs) of which each VM emulates hardware for running software. In other words, an example VM (e.g., a LINUX kernel VM (KVM)) provides a virtualized operating system and application suite (e.g., to deploy VNFs) for customer access. Alternatively, or additionally, server 302A may provide containers (e.g., such as those provided by the open source Docker Container application), or other virtual execution environments in which VNFs are implemented. In some examples, NFVi <NUM> further includes a virtualization layer 308A over the hardware to offer virtual computing, virtual storage, and virtual network for executing VNFs. NFVi <NUM> may be executed by one or more computing devices in a centralized or distributed manner.

In the example of <FIG>, server 302A may be part of a computer cluster or pod whose physical resources are virtualized into network infrastructure such as NFVI <NUM>. The computer cluster may be labeled a network edge for cloud service providers. Each cloud service provider may be a data center tenant having one or more VNFs running in servers <NUM>, e.g., server 302A, to provide access to cloud services from devices in cloud network <NUM>. As the network edge, server 302A executes a VNF to perform network edge services such as routing and forwarding operations for packets directed to that VNF and intended for cloud service providers on cloud network <NUM> or received from cloud service providers on cloud network <NUM>. However, VNF may apply a network function to flows destined to or sourced by any network, the flows associated with one or more tenants of the NFVI <NUM> provider.

An orchestration system (e.g., network service orchestrator <NUM>) in control over resource orchestration in NFVi <NUM> may use NFV management <NUM> for NFVI <NUM> to instruct server 302A to allocate compute resources among VNFs according to techniques described herein. These techniques direct usage-based allocations of resources towards executing VNFs. In view of service level agreements and overall goals of the data center and/or the tenant, one example technique optimizes an resource allocation upon balancing of two tenant/data center conditions: <NUM>) a quality of service (QoS) expectation on behalf of the tenant (i.e., actual resource utilization) and <NUM>) a quality of service (QoS) obligation on behalf of the data center provider (i.e., resource requirement).

NFV management <NUM> distributes physical resources of resource pool <NUM> to VNFs running in servers <NUM> of NFVI <NUM>. Examples of the various physical resources include processing resources (e.g., processor cores), networking resources (e.g., physical network interface cards (NICs)), and memory resources. A plurality of processor cores 314A-314N ("processor cores <NUM>") may be examples of processing or compute resources. In one example, using virtualization layer 308A to generate an abstraction of various physical resources of resource pool 310A, NFV management <NUM> configures that abstraction into countable virtual resources use by VNFs running in servers <NUM> of NFVI <NUM>. Physical resource may be virtualized into virtual compute resources of which each resource (unit) includes one or more processor cores. Another compute resource may be a compute node such as a virtual machine. Other virtual resources include virtual storage resources and virtual network resources.

NFV management <NUM> may couple virtual network interfaces in VNI space such as VNIs 316A-N (or VNIs <NUM>) to virtual switch 318A. Virtual switch 318A is configured with VNIs <NUM>, which are logical interfaces where encapsulation and decapsulation for virtual networking in the NFVI <NUM> occurs. Each of VNIs <NUM> may be associated with a virtual network of the NFVI <NUM> that is assigned to a tenant. That is, a tenant may be assigned one or more virtual networks for packet flows. The virtual network may have a corresponding virtual network identifier, such as a VXLAN network identifier. Packet flows that are transported using these virtual networks are associated with the tenant, and packets of such a packet flow may include the virtual network identifier for the virtual network on which the packet flow is transported.

In one example, virtual switch 318A may be configured with VXLAN interfaces, each VXLAN interface being configured with a different VNI and corresponding VNI identifier of VNIs <NUM>. When physical network interfaces in server 302A (e.g., NIC <NUM>) receive network traffic in form of one or more packet flows of which packets include information identifying those VNIs <NUM> (e.g., having same VNI identifiers). Virtual switch 318A switches each packet flow to their correct VNF which may be the VNF to which the VNIs are assigned. sFlow agent 326A collects a packet flow data, including the VNI per flow, and sends the collected packet flow data to the collector.

Statistics and other data points associated with the transmitted packet flows may provide useful information, for example, for vector packet processing by VPP 318A and for tenant-driven dynamic resource allocation for VNFs by the orchestration system for NFVI <NUM> that may be running in the same data center environment as server 302A. sFlow agent 326A captures and then, provides these statistics and other data points to sflow collector <NUM>, which combines the provided statistics and other data points with information provided by other sFlow agents <NUM> in other servers <NUM>.

In some examples, sFlow Collector <NUM> leverages SNMP to communicate with sFlow Agent 326A in server 302A in order to configure sFlow monitoring on VNF 306A. sFlow Agent 326A uses two forms of sampling mechanisms: statistical packet-based sampling of switched or routed Packet Flows, and time-based sampling of counters. In general, Packet Flow Sampling and Counter Sampling is performed by sFlow Instances associated with individual data sources within sFlow Agent 326A. In order to perform Packet Flow Sampling, an sFlow Instance is configured with a Sampling Rate. The Packet Flow sampling process results in the generation of Packet Flow Records. In order to perform Counter Sampling, an sFlow Instance is configured with a Sampling Interval. The Counter Sampling process results in the generation of Counter Records. The sFlow Agent 326A collects Counter Records and Packet Flow Records and sends them in the form of sFlow Datagrams to sFlow Collector <NUM>.

Packet Flow Sampling, an example of Packet Flow Monitoring, is accomplished as follows: When a packet arrives on an interface (e.g., VNI 316A), VNF 306A makes a filtering decision to determines whether the packet should be dropped. If the packet is not filtered, a destination interface is assigned by VNF 306A's switching/routing function. At this point, sFlow Agent 326A determines whether or not to sample the packet. sFlow Agent 326A uses a counter that is decremented with each packet. When the counter reaches zero a sample is taken, whether or not a sample is taken, the counter Total_Packets is incremented and Total_Packets is a count of all the packets that could have been sampled. sFlow Agent 326A, using Counters such as the counter Total_Packets, generates a variety of information including flow statistics. Agents/component of the orchestration system described herein may use these flow statistics to instruct NFVI <NUM> regarding whether or not to modify an allocated resource capacity, such as an allocation of compute resources. In one example, the collected flow statistics may be used to add or subtract one or more compute resources (e.g., virtual and physical compute nodes such as processor cores in a multi-core environment).

To illustrate by way of example, the orchestration system may modify a (current) allocation of compute resources to a running VNF (e.g., VNF 306A) based upon actual bandwidth usage such that the running VNF may consume at most the modified allocation of compute resources. In one implementation, sFlow Agent 326A may compute a bandwidth utilization rate based upon the counter Total_Packets, one or more flow statistics, and time information (e.g., a time interval (in seconds)), sFlow Agent 326A computes a bandwidth utilization rate (e.g., packet processing rate or throughput over the given time interval). In one example, sFlow Agent 326A computes a flow statistic to assist in the bandwidth utilization computation, one example flow statistic includes a number of packets per second (or simply Packets per second (PPS)). In another example, sFlow Agent 326A computes a flow statistic known as an average packet size per individual flow, which may be aggregated into the average packet size for a tenant's flows. By multiplying the average packet size and the packets per second, sFlow agent 326A computes the bandwidth utilization rate. As an alternative, sFlow agent 326A may compute the bandwidth utilization rate using a different counter, such as a counter that is incremented for each byte in a packet flow. This counter may determine a total number of bytes in the packet flow. Using this counter and timestamp data, sFlow agent 326A may compute the bandwidth utilization rate by dividing the total number of bytes by a time interval (in seconds).

Taking a sample involves either copying the packet's header or extracting features from the packet and storing the sampled information in a sFlow datagram. Example flow attributes of the sampled information include: a source address SRC, a destination address DEST, a virtual network identifier (e.g., virtual network identifier such as a VXLAN network identifier or one of VNIs <NUM>), and a packet size. Based upon an average packet size-assuming that the packet size is consistent across the packets-and the counter Total_Packets average (e.g., per flow), sFlow Agent 326A computes an average bandwidth utilization rate.

An sFlow Datagram contains lists of Packet Flow Records and Counter Records. The format of each record is identified by a data_format value. The data_format name space is extensible, allowing for the addition of standard record types as well as vendor specific extensions. A number of standard record types have been defined. However, an sFlow Agent is not required to support all the different record types, only those applicable to its treatment of the particular packet being reporting on. For example, if VNF 306A implements a layer <NUM>/<NUM> switch, VNF 306A reports to sFlow agent 326A layer <NUM> information for packets it switches, and layer <NUM> and <NUM> information for packets it routes. The data_format uniquely identifies the format of an opaque structure in the sFlow specification. A data_format is constructed to uniquely identify the format of the structure (e.g., a standard structure). An example data_format could identify a set of flow attributes when used to describe flow_data.

Every time a sample is taken, the counter Total_Samples, is incremented. Total_Samples is a count of the number of samples generated. Samples are sent by the sFlow Instance to the sFlow Agent 326A for processing. The sample includes the packet information, and the values of the Total_Packets and Total_Samples counters. The sFlow Agent 326A may then use the samples to obtain additional information about the packet's trajectory through NFVI <NUM>. Such information depends on the forwarding functions of VNF 306A. Examples of trajectory information provided are source and destination interface, source and destination address, source and destination VLAN, next hop subnet, full AS path. Details of the trajectory information are stored in the sFlow Datagram Format along with an average bandwidth rate. Virtual switch 318A assumes that the trajectory information applies to each packet.

Virtual Switch 318A refers to a vector packet processing application built on a software platform (e.g., proprietary and open source versions of CISCO® Vector Packet Processing technology). Virtual switch 318A, in general, provides a data plane functionality (e.g., in a virtual router or virtual switch) including packet forwarding operations. Virtual switch 318A includes a set of forwarding nodes arranged in a directed graph and a supporting framework. The framework has all the basic data structures, timers, drivers (and interfaces to driver software development kits (e.g., data plane development kit (DPDK)), a scheduler which allocates the CPU time between the graph nodes, performance and debugging tools, like counters and built-in packet trace. The latter enables capturing the trajectory information or the paths taken by the packets within the graph with high timestamp granularity, giving full insight into the processing on a per-packet level. Virtual switch 318A may process trajectory information such as the trajectory information determined by the framework and any trajectory information generated via flow monitoring. Virtual switch 318A may couple to VNI interfaces and process packets that arrive through physical network hardware on server 302A. Using the trajectory information, virtual switch 318A assembles those packets into a vector, e.g. Virtual switch 318A sorts packets by protocol or format and when software nodes in Virtual switch 318A are scheduled, Virtual switch 318A takes its vector of packets and processes them in a tight dual loop (or quad-loop) with prefetching to the CPU cache to achieve optimal performance.

Cloud network <NUM> may communicatively couple VNF 306A to one or more cloud services <NUM>). Cloud network <NUM> may be generally hidden from or otherwise unavailable to devices on a public network. For example, cloud network <NUM> may receive packet flows from the VNF 306A that are communicated to a cloud service from a customer device or another cloud service. Examples of cloud services include Google Cloud, Azure, Oracle Cloud, Amazon Web Services (AWS), IBM Cloud, Alibaba Cloud, and Salesforce. In some aspects, cloud network <NUM> can be an Equinix Cloud Exchange Fabric provided by Equinix Inc. of Redwood, California. VNF 306A may be a vendor-neutral VNF that combines two or more cloud services into a hybrid cloud service.

Gateway <NUM> may communicatively couple VNF 306A to a public network and/or a private network of customer devices. A public network may be a network that is publicly available with few or no restrictions. For example, the public network may be a network that is part of the Internet. A private network may be a network that is part of an enterprise network and only accessible to authorized users. Customer devices are clients of VNF 306A and, as an example, may be computing devices located in a branch office of the tenant or otherwise associated with the tenant or customer. Public gateway may receive traffic having a destination address of the server 302A hosting the VNF 306A within the data center from the public network. VNF 306A may receive network traffic from gateway <NUM>.

When the above-mentioned orchestration system (e.g., orchestrator <NUM> of <FIG>), as an example, performs an initial resource allocation for VNF 306A, a particular tenant's VNF, and in response, NFV management <NUM> deploys VNF 306A on server 302A as a virtual point of presence VNF (e.g., a routing VNF connecting customer devices with the particular tenant's cloud service in cloud network <NUM>). NFV management <NUM> also assigns a portion of VNI space to that particular tenant to use for receiving/transmitting packet flows to/from customer devices/cloud services. The VNI space may refer to a range of virtual network identifiers for corresponding virtual network interfaces 316A-316N (VNIs <NUM>). NFV management <NUM> distributes virtual networks in the physical network for NFVI <NUM> among a plurality of tenants. Each of the virtual networks may represent an overlay network, e.g., a VXLAN, and be associated with a virtual network identifier, e. g, a VXLAN network identifier. The virtual network identifier may included in packets to identify the virtual network for the virtual switch 318A to facilitate forwarding by virtual switch 318A to one of VNFs <NUM> and to a next-hop device.

In one example, NFV management <NUM> assigns VNI 316A and a corresponding address to the particular tenant's VNF, VNF 306A, in server 302A. The corresponding address may be a Layer <NUM> or a Layer <NUM> construct to uniquely identify VNI 316A as a destination for packet flows directed to the particular tenant's cloud service. For at least this reason, the particular tenant's assigned network address may be stored in packets of the packet flows (e.g., in packet headers)-possibly along with the assigned VNI 316A. When the packets arrive at gateway <NUM> (or in another network device), the assigned address is extracted and translated into a VNI identifier (e.g., number) matching the assigned VNI identifier for VNI 316A. Hence, the corresponding VNI identifier for VNI 316A uniquely identifies the particular tenant's assigned VNI such that packets delivered to that VNI are forwarded (in part) by Virtual switch 318A to the VNF <NUM>, the particular tenant's VNF.

VNF 306A manages network traffic to and from cloud services communicatively coupled to server 302A via cloud network <NUM>. VNF 306A may process network traffic that includes the network address (e.g., an IP address for server 302A) as a source or destination address when communicating over gateway <NUM> or cloud network <NUM>. In some examples, multiple VNFs on server 302A can each have a distinct public IP address that shares a single physical network interface. In the example of <FIG>, servers <NUM> and VNFs <NUM> are described as the current NFVi infrastructure. It is noted that a current NFVi infrastructure may include only one server and only one VNF or, alternatively, more than one server including more than one VNF.

In accordance with techniques described in this disclosure, in some aspects, VNF 306A may form part of the particular tenant's network service. The tenant may have purchased resources for executing the network service from the data center provider such that the purchased resources are a requirement that the data center provider is obligated to provide. As described herein, in a number of instances, the tenant's network services does not actually utilize all the purchased resource, leaving a substantial resource capacity unused and wasted.

In some examples, an amount of network traffic directed to the particular tenant's VNF fluctuates in such a manner that the particular tenant's VNF may not fully utilize the initial allocation of resources (e.g., compute resources) under certain circumstances. The particular tenant, the cloud service provider, and/or the data center provider may have over-estimated an expected amount of network traffic and thus, purchased unnecessary resource capacities including extraneous bandwidth capacity (e.g., in a service level agreement (SLA)). Over time, the orchestration system of NFVI <NUM> may determine that the particular tenant's VNF consistently utilizes only a portion of the initial allocation of physical resources (bare metal servers, communication infrastructure etc.) provided by the data center and for at least reason, may modify the particular tenant's VNF's (current) resource allocation, for example, to match the particular tenant's VNF's actual resource usage (e.g., bandwidth utilization). In some examples, the orchestration system of NFVI <NUM> also modifies the SLA to update a resource requirement/obligation to the particular tenant for executing that tenant's VNF; in this manner, the orchestration system of NFVI <NUM> does not violate the SLA by modifying the resource allocation while maintaining an expected quality of service. In some examples, the orchestration system of NFVI <NUM> achieves an optimal resource allocation in which a maximum capacity of compute resources are available for executing a new VNF and/or a maximum number of VNFs are executing at a same time.

As described herein, flow information captured by sFlow agent 326A may be mined for flow statistics associated with tenants and tenant VNFs including the particular tenant and the particular tenant VNF, VNF 306A. Over a time period, each tenant to NFVI <NUM> and server 302A receives a plurality of packet flows. In some examples, the orchestration system of NFVI <NUM> may use NFV management <NUM> to map each plurality of packet flows' VNI to the VNI space corresponding to VNIs <NUM> by correlating the flow statistics with tenant information. In some examples, the orchestration system uses sflow agent 326A to identify VNI(s) that is/are assigned to a particular tenant from the tenant information and using that assigned VNI, to identify corresponding flow statistics having the same VNI(s).

The orchestration system uses the corresponding flow statistics to compute various statistics and other informational data points. The corresponding flow statistics may indicate values for various variables describing a packet flow including a total number of packets, a total amount in bytes, packets per second, an average packet size or distribution of packet size(s) and/or the like. In some examples, the orchestration system uses these flow statistics to compute an aggregate bandwidth for the plurality of flows associated with the particular tenant of the data center provider and processed by the tenant's VNF executing on server 302A and then, modifies, based on the aggregate bandwidth, an allocation of compute resources of the server 302A executing the VNF. In some examples, the allocation of compute resources refers to an allocation of physical (hardware) processor resources (e.g., processing circuitry) towards executing the VNF.

In some examples, the orchestration system is configured to compute an average bandwidth for (packet) flows through one VNI based upon the corresponding flow statistics. The orchestration system may determine the average packet size from the corresponding flow statistics, for example, by aggregating packet sizes and dividing by a total number of packets. As an alternative, the orchestration system may determine an average packet size from the corresponding flow statistics, for example, by extracting that average from stored sflow datagrams. Once the average packet size is known, the orchestration system may then compute the average bandwidth by multiplying a number of packets per second (PPS) by the average packet size and compute the aggregated bandwidth as a summation of each average bandwidth of a plurality of packet flows. In another example, the orchestration system may compute the average bandwidth by dividing a total number of bytes by an amount of time in seconds and compute the aggregated bandwidth as a summation of each average bandwidth of a plurality of packet flows.

In some examples, the orchestration system compares an allocation of bandwidth to the aggregate bandwidth to produce a comparison result and (possibly) store the comparison result for trending. In some examples, to comply with an SLA obligating resources of server 302A towards executing the VNF, the orchestration system of NFVI <NUM> may provide an allocation of compute resources that satisfies at least a minimum or sufficient resource requirement for executing the VNF. The orchestration system of NFVI <NUM> may analyze the comparison result to determine whether (or not) to modify the allocation of compute resources (e.g., while maintaining compliance with the SLA). If the comparison result indicates resource under-utilization (e.g., of the compute resource or another resource (e.g., bandwidth)) by VNF, the orchestration system of NFVI <NUM> may modify the allocation by reducing the allocation of compute resources to a capacity that is optimal for the aggregated bandwidth. In some instances, the orchestration system of NFVI <NUM> may break the SLA and allocate fewer then the allocation of compute resources.

In some examples, the orchestration system of NFVI <NUM> (periodically) repeats the comparison (e.g., after a time interval) and stores each comparison result for trending. Over time, the comparison results develop into a relationship between the VNF's performance (e.g., in terms of bandwidth utilization) and the VNF's allocation of compute resources. For example, the orchestration system of NFVI <NUM> may combine (a sequence of) aggregate bandwidth measurements into an aggregate bandwidth trend. The orchestration system of NFVI <NUM> may modify, based on the aggregate bandwidth trend, the allocation of compute resources to the execution of the VNF. Assuming steady network traffic, the aggregate bandwidth trend eventually converges into a well-defined mapping that can be used when deploying the tenant VNF in another virtual execution environment in another NFVI. When the aggregate bandwidth changes (e.g., due to changes in level of network traffic), the orchestration system of NFVI <NUM> performs a comparison between the current allocation of bandwidth to an updated aggregate bandwidth to produce an updated comparison result for trending. In some instances, the updated comparison result may indicate an inability to meet a sufficient bandwidth and an opportunity to modify the resource allocation by adding one or more compute resource (e.g., processor cores).

The orchestration system may perform the abovementioned comparison with different resource configurations to determine maximum achievable (aggregated) bandwidth trend(s) for (a progression of) allocations of compute resources. The orchestration system may store several mappings between different allocations of compute resources and a corresponding maximum achievable (aggregated) bandwidth trend in a resource allocation table. The orchestration system may modify the allocation of compute resources based upon the resource allocation table. In other examples, the orchestration system may use the resource allocation table for an initial or modified allocation of compute resources toward executing the same or similar VNF in another NFVI or in another data center.

<FIG> is a flowchart illustrating example operations for tenant-driven resource allocation based upon actual resource usage according to techniques described herein.

The (following) example operations may be performed by an orchestration system-a computing system in control over NFVI in a data center-such as when the orchestration system recognizes that a tenant's VNF bandwidth consumption (rate) fails to meet the tenant's VNF bandwidth allocation. In some aspects, the tenant's VNF may be implemented on a physical or virtual machine in the data center. The data center may include a gateway (e.g., a public or private gateway) that receives network packets from a network (e.g., a customer network or a cloud service network) that have a destination address for the tenant's VNF (e.g., a public IP address). If the tenant's customers rarely (if ever) require the total VNF bandwidth allocation, the orchestration system may perform the following example operations, for example, instructing a server hosting the tenant's VNF to increase or decrease the VNF's allocated resource capacity.

The following operations are described with respect to orchestrator <NUM> of <FIG> as an example orchestration system. In some examples, orchestrator <NUM> determines a resource usage context by mapping flow VNI to tenant VNI (<NUM>). Tenant information <NUM> stores information associated with each tenant of a data center provider that is assigned one or more VNFs in the data center. Tenant information <NUM> also stores, for each tenant, the virtual network identifiers for virtual networks configured for use for flows associated with the tenant. Some of these virtual network identifiers for the tenant may be configured on servers of NFVI to cause the virtual switch of the server to switch packets having those virtual network identifiers to a tenant VNF. This set of virtual network identifiers is referred to as the VNF VNI space. Flow information <NUM> stores information describing a plurality of flows associated with each tenant of the data center provider, each flow referring to packets being forwarded through the NFVI between endpoints (e.g., between tenant customers and service providers). Each of tenant information <NUM> and flow information <NUM> stores VNIs that can be used as an index for mapping each tenant to their corresponding flows. Tenant information <NUM> and flow information <NUM> are combined such that each tenant VNI in tenant information is mapped to each flow VNI in flow information <NUM>. Dynamic resource mapping database <NUM> may be created to store the mappings between the flow VNI and the tenant VNI.

If tenant information <NUM> is organized into a database, an example entry in that database may be structured as follows:
<IMG>.

If flow information <NUM> is organized into a database, example entries in that database may be structured as follows:
<IMG>.

To identify flows associated with a tenant, network service orchestrator <NUM> maps a VNI for a flow, e.g., VNI: <NUM> for FLOW1, to the VNF VNI space for a tenant, e.g., VNF VNI space: <NUM>-<NUM> of Tenant1. If the VNF VNI space for a tenant includes the VNI for the flow, then the flow is associated with the tenant. Multiple flows associated a tenant are then aggregated, based on the flow records, to compute an average packet size and average bandwidth for instance. When flow information <NUM> and tenant information <NUM> is combined into dynamic resource mapping database information <NUM>, example entries in that database may be structured as follows:
Tenant1 {
Actual Resource: {
Actual BW utilization: <NUM>
Core(s): <NUM>
Memory: <NUM> GB }
Bandwidth Usage: {
Flows : <NUM>
Avg Packet Size: <NUM>
Avg BW: <NUM> } }
Tenant <NUM>{
Actual Resource: {
Actual BW : <NUM>
Core : <NUM>
Memory: 4GB}
Bandwidth Usage: {
Flows : <NUM>
Avg Packet Size: <NUM>
Avg BW: <NUM> } }.

Orchestrator <NUM> determines an average packet size for all tenant flows (<NUM>). Flow information <NUM> stores packet flow statistics corresponding to an amount of data (in number of bytes) being received at a network edge of the data center over a period of time. The network edge may be an exchange point comprising a pod or cluster of servers that virtualize resources for NFVI. In some examples, dynamic resource mapping database <NUM> stores a rate (e.g., in packets or bytes) at which a particular tenant's data is received and forwarded through the NFVI. From this rate, orchestrator <NUM> computes an average packet size, which also may be stored in dynamic resource mapping database <NUM>. In some examples, dynamic resource mapping database <NUM> stores the average packet size for data at each VNI assigned to the particular tenant.

Orchestrator <NUM> determines an aggregated bandwidth (i.e., an aggregated average bandwidth) for all flows associated with the tenant (<NUM>). As mentioned above, dynamic resource mapping database <NUM> stores the rate (e.g., in packets or bytes) at which bandwidth is consumed by the particular tenant's VNF for data received at each VNI and over time, that rate may represent an average rate per packet flow; by combining each VNI's corresponding rate, orchestrator <NUM> determines an aggregated rate for data received at all the VNIs assigned to that particular tenant.

In some examples, orchestrator <NUM> creates a tenant bandwidth usage trend (<NUM>). In some examples, orchestrator <NUM> plots a data point representing the aggregated average bandwidth against an allocation of compute resources and stores that data point for trending. That data point may correlate the actual rate at which the particular tenant's packet flows are received and forwarded through NFVI <NUM> with a number of processor cores allocated to such packet processing. Over time, orchestrator <NUM> plots a number of data points representing a trend between the allocation of processor cores and an achievable aggregate bandwidth for the particular tenant's VNF. In this manner, orchestrator <NUM> may compute a maximum bandwidth rate that is possible for any VNF being executed in NFVI <NUM> based upon that VNF's allocation of processor cores (or another compute resource). Orchestrator <NUM> may store that maximum bandwidth rate and the VNF's allocation of processor cores in a resource allocation table such as resource allocation table <NUM> of <FIG>. Based on the trend of aggregate bandwidth for flows associated with a tenant of the data center provider and processed by a tenant VNF, orchestrator <NUM> may determine that additional compute resources (for trending higher aggregate bandwidth) or fewer compute resources (for trending lower aggregate bandwidth) should be assigned to the tenant VNF.

Orchestrator <NUM> determines a number of processor cores to add/remove to modify a resource allocation for the particular tenant (<NUM>). In some examples, orchestrator <NUM> modifies an allocation of compute resources towards executing the particular tenant's VNF. Orchestrator <NUM> may access resource allocation table <NUM> and determine the modified allocation of compute resources. If the tenant's VNF bandwidth utilization changes, as indicated by the aggregate bandwidth for tenant flows processed by the tenant VNF, orchestrator <NUM> may access resource allocation table <NUM> and, in accordance with that table, modify the allocation of compute resources to a server executing the particular tenant's VNF. For example, orchestrator <NUM> may use resource allocation table <NUM> to determine a number of processor cores to add or remove in order to meet the changed bandwidth utilization. In some examples, orchestrator <NUM> issues a function call through NFVI interface <NUM> to instruct NFVI management to add or remove the number of processor cores to the server executing the particular tenant's VNF.

<FIG> is a block diagram illustrating a conceptual view of bandwidth usage-based processor core management in accordance with one or more techniques of the present disclosure. In <FIG>, a plurality of data center tenants (or tenants) initiate VNFs in NFVI <NUM> running in compute node <NUM>. Each tenant is illustrated in <FIG> as an oval graphic with letter "T" and a numeral. Each tenant is further illustrated in <FIG> with one or two black circular graphics representing either one or two processor cores and overlapping that tenant's oval graphic. Compute node <NUM> may present a physical compute node or, in some cases, a virtual compute resource that is an abstraction of physical processing resources in a computer cluster (e.g., a pod). Virtual switch <NUM> is coupled to the virtual compute resource and uses the virtualized physical processing resources to process packet flows received via a virtual network resource for compute node <NUM>. In some examples, the virtual network resource may correspond to one or more VNIs.

Orchestration system <NUM> represents a functional block (e.g., a network service orchestrator) for NFVI <NUM>. Orchestration system <NUM>, in general, orchestrates resources (e.g., virtual resources) to instantiate VNFs to run network services for the plurality of tenants. In some examples, orchestration system <NUM> allocates to each VNF virtual compute resources such as one or more processors cores or other processing resources. In some examples, orchestration system <NUM> maintains core reserve <NUM> to include one or more processor cores that are unavailable for VNF allocation and utilization. In some examples, orchestration system <NUM> also maintains core pool <NUM> to include one or more processor cores that are available for VNF allocation and utilization. When compared to conventional NFVi implementations, <FIG> illustrates an improved NFVI with an increase in the maximum number of available cores for allocation to VNFs. In <FIG>, between <NUM> and <NUM> additional cores are allocated to the VNFs associated with the tenants of compute node <NUM>.

The following table illustrates some example realized benefits from transitioning from conventional resource orchestration techniques (i.e., a Present Mode of Operation (PMO)) to techniques described herein (i.e., Future Mode of Operation (FMO):.

The above table demonstrates that those skilled in NFVi may allocate the additional available cores to increase the total number of the VNFs being executed at a particular moment in time. In the above table, a "POD" refers to the cluster of servers (e.g., servers <NUM> of <FIG>) hosting the improved NFVI described herein.

<FIG> is a block diagram illustrating further details of one example of a computing device that operates in accordance with one or more techniques of the present disclosure. <FIG> may illustrate a particular example of a server or other computing device <NUM> that includes one or more processor(s) <NUM> for executing any one or more of any system, application, or module described herein. For example, the one or more processor(s) <NUM> may execute instructions of orchestration system <NUM> to instantiate and deploy VNFs to NFVI and apply techniques described herein to determine and configure compute resource allocations for VNFs executing on servers of the NFVI. As such, computing device <NUM> may represent an example instance of network service orchestrator <NUM> or other orchestration system or other system to configure resource allocations for VNFs executing on servers of the NFVI in accordance with techniques of this disclosure. Other examples of computing device <NUM> may be used in other instances. Although shown in <FIG> as a stand-alone computing device <NUM> for purposes of example, a computing device may be any component or system that includes one or more processors or other suitable computing environment for executing software instructions and, for example, need not necessarily include one or more elements shown in <FIG> (e.g., communication units <NUM>; and in some examples components such as storage device(s) <NUM> may not be co-located or in the same chassis as other components).

As shown in the specific example of <FIG>, computing device <NUM> includes one or more processors <NUM>, one or more input devices <NUM>, one or more communication units <NUM>, one or more output devices <NUM>, one or more storage devices <NUM>, and user interface (UI) device <NUM>, and communication unit <NUM>. Computing device <NUM>, in one example, further includes one or more applications <NUM>, programmable network platform application(s) <NUM>, and operating system <NUM> that are executable by computing device <NUM>. Each of components <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> are coupled (physically, communicatively, and/or operatively) for inter-component communications. In some examples, communication channels <NUM> may include a system bus, a network connection, an inter-process communication data structure, or any other method for communicating data. As one example, components <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> may be coupled by one or more communication channels <NUM>.

Processors <NUM>, in one example, are configured to implement functionality and/or process instructions for execution within computing device <NUM>. For example, processors <NUM> may be capable of processing instructions stored in storage device <NUM>. Examples of processors <NUM> may include, any one or more of a microprocessor, a controller, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or equivalent discrete or integrated logic circuitry.

One or more storage devices <NUM> may be configured to store information within computing device <NUM> during operation. Storage device <NUM>, in some examples, is described as a computer-readable storage medium. In some examples, storage device <NUM> is a temporary memory, meaning that a primary purpose of storage device <NUM> is not long-term storage. Storage device <NUM>, in some examples, is described as a volatile memory, meaning that storage device <NUM> does not maintain stored contents when the computer is turned off. Examples of volatile memories include random access memories (RAM), dynamic random access memories (DRAM), static random access memories (SRAM), and other forms of volatile memories known in the art. In some examples, storage device <NUM> is used to store program instructions for execution by processors <NUM>. Storage device <NUM>, in one example, is used by software or applications running on computing device <NUM> to temporarily store information during program execution.

Storage devices <NUM>, in some examples, also include one or more computer-readable storage media. Storage devices <NUM> may be configured to store larger amounts of information than volatile memory. Storage devices <NUM> may further be configured for long-term storage of information. In some examples, storage devices <NUM> include non-volatile storage elements. Examples of such non-volatile storage elements include magnetic hard discs, optical discs, floppy discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories.

Computing device <NUM>, in some examples, also includes one or more communication units <NUM>. Computing device <NUM>, in one example, utilizes communication units <NUM> to communicate with external devices via one or more networks, such as one or more wired/wireless/mobile networks. Communication units <NUM> may include a network interface card, such as an Ethernet card, an optical transceiver, a radio frequency transceiver, or any other type of device that can send and receive information. In some examples, computing device <NUM> uses communication unit <NUM> to communicate with an external device.

Computing device <NUM>, in one example, also includes one or more user interface devices <NUM>. User interface devices <NUM>, in some examples, are configured to receive input from a user through tactile, audio, or video feedback. Examples of user interface devices(s) <NUM> include a presence-sensitive display, a mouse, a keyboard, a voice responsive system, video camera, microphone or any other type of device for detecting a command from a user. In some examples, a presence-sensitive display includes a touch-sensitive screen.

One or more output devices <NUM> may also be included in computing device <NUM>. Output device <NUM>, in some examples, is configured to provide output to a user using tactile, audio, or video stimuli. Output device <NUM>, in one example, includes a presence-sensitive display, a sound card, a video graphics adapter card, or any other type of device for converting a signal into an appropriate form understandable to humans or machines. Additional examples of output device <NUM> include a speaker, a cathode ray tube (CRT) monitor, a liquid crystal display (LCD), or any other type of device that can generate intelligible output to a user.

Computing device <NUM> may include operating system <NUM>. Operating system <NUM>, in some examples, controls the operation of components of computing device <NUM>. For example, operating system <NUM>, in one example, facilitates the communication of one or more applications <NUM> with processors <NUM>, communication unit <NUM>, storage device <NUM>, input device <NUM>, user interface devices <NUM>, and output device <NUM>.

Application(s) <NUM> and Orchestration System <NUM> may also include program instructions and/or data that are executable by computing device <NUM>. Furthermore, orchestration system <NUM>, as an example, may include software to implement orchestrator <NUM> of <FIG> and may operate as illustrated and described herein. As instructed by orchestration system <NUM> running in a data center in which computing device <NUM> resides, a maximum number of VNFs are instantiated and executing at a same time. This may be accomplished in part by implementing techniques described herein. As one example, the orchestration system <NUM> may apply techniques described herein to determine an allocation of compute resources of servers for executing virtual network functions and configure NFVI (via a NFVI management system (e.g., NFVI management <NUM> of <FIG>) coupled to) with a number of processor cores for the servers in accordance with the allocation. In one example technique, computing device <NUM> uses information stored in memory (e.g., resource allocation table <NUM> of <FIG>) to determine the number of processor cores to allocate.

If implemented in hardware, this disclosure may be directed to an apparatus such as a processor or an integrated circuit device, such as an integrated circuit chip or chipset. Alternatively or additionally, if implemented in software or firmware, the techniques may be realized at least in part by a computer-readable data storage medium comprising instructions that, when executed, cause a processor to perform one or more of the methods described above. For example, the computer-readable data storage medium may store such instructions for execution by a processor.

A computer-readable medium may form part of a computer program product, which may include packaging materials. A computer-readable medium may comprise a computer data storage medium such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), Flash memory, magnetic or optical data storage media, and the like. In some examples, an article of manufacture may comprise one or more computer-readable storage media.

In some examples, the computer-readable storage media may comprise non-transitory media. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier wave or a propagated signal. In certain examples, a non-transitory storage medium may store data that can, over time, change (e.g., in RAM or cache).

Claim 1:
An orchestration system (<NUM>) operated by a data center provider (110A) for a data center (<NUM>), the orchestration system comprising:
processing circuitry (<NUM>) coupled to a memory (<NUM>); and
software instructions stored in the memory and configured for execution by the processing circuitry to:
map (<NUM>) packet flow virtual network identifiers (<NUM>) to a virtual network identifier space of a plurality of tenants (<NUM>) of the data center, wherein a tenant (108A) of the data center is associated with a first virtual network identifier (316A) of the virtual network identifier space and a second virtual network identifier (316N) of the virtual network identifier space;
identify (<NUM>, <NUM>) flow statistics corresponding to the tenant (108A) based upon the first virtual network identifier (316A) and the second virtual network identifier (316N) in the mapping;
compute (<NUM>), based on the flow statistics indicating a first packet flow having the first virtual network identifier (316A) associated with the tenant (108A) and a second packet flow having the second virtual network identifier (316N) associated with the tenant, an aggregate bandwidth for a plurality of packet flows associated with the tenant and processed by a virtual network function (124A) that is assigned to the tenant and executes on physical compute resources (<NUM>) of a server (302A) of the data center (<NUM>); and
modify (<NUM>), based on the aggregate bandwidth, an allocation (<NUM>) of the physical compute resources (<NUM>) of the server (302A) to re-assign a portion of the physical compute resources of the server between the virtual network function (<NUM>) assigned to the tenant (108A) and one or more virtual network functions (306A) associated with a different tenant (108B).