High availability and high utilization cloud data center architecture for supporting telecommunications services

The concepts and technologies disclosed herein provide high availability and high utilization cloud data center architecture for supporting telecommunications services. According to one aspect of the concepts and technologies disclosed herein, a 4-site model of application placement within the cloud computing environment provides 37.5% resource utilization with site availability of five 9s (99.999%) and virtual machine availability of five 9s. According to another aspect of the concepts and technologies disclosed herein, a 3-site model of application placement within the cloud computing environment provides 66% resource utilization with site availability of five 9s and virtual machine availability of five 9s. According to another aspect of the concepts and technologies disclosed herein, a 4-site model of application placement within the cloud computing environment provides 75% resource utilization with site availability of five 9s and virtual machine availability of five 9s.

BACKGROUND

Cloud computing allows dynamically scalable virtualized resources to host applications and services. Cloud computing enables appropriate levels of resources to power software applications when and where the resources are needed in response to demand. As a result, cloud computing allows entities to respond quickly, efficiently, and in an automated fashion to rapidly changing business environments.

Paramount to any computing platform is availability. Availability is typically measured as a percentage of time during which a given computing system is operational. While optimal availability is 100%, this is often not achievable. The current standard of “high availability” is often referred to as “five 9s” or 99.999% availability. Over a period of 1 year, a computing system operating at five 9s availability will experience only 5 minutes and 36 seconds of downtime.

Five 9s availability has long been the goal of system administrators, whether the target system is a telecom or computing system. With the recent trend of cloud computing being used as a replacement for hardware-based solutions, the concept of five 9s availability has taken a backseat to the ease of simply instantiating new instances of an application or service to counteract lower-than-average availability. However, as cloud computing becomes more ubiquitous, cloud providers will endeavor once again to reach five 9s availability.

Next generation applications, such as Internet of Things (“IoT”), connected cars, remote surgery, augmented reality, virtual reality, video streaming, 5G voice and data applications, and others, require real-time sensitive applications running on cloud resources to provide such services. These applications, in addition to stringent real-time performance requirements (e.g., latency, jitter, etc.), demand five 9s availability to provide such services.

In addition to availability and real-time performance requirements, a primary indicator of system performance is utilization. Utilization generally refers to the energy efficiency of a system. Many existing cloud computing models operate with low utilization. In other words, cloud computing resources are often left idle consuming electricity but not performing any tasks. As more and more applications and services are moved to the cloud, utilization will need to increase while maintaining high availability and real-time performance requirements.

SUMMARY

The concepts and technologies disclosed herein provide a high availability and high utilization cloud data center architecture for supporting telecommunications services. The high availability and high utilization cloud data center architecture can include a plurality of sites (also referred to as geo-sites or geo-regions) representing specific geographical locations in which computing resources are located. Each site can include one or more availability zones (“AZs”), each of which represents an isolated physical location within a site. Site resources of a given site are available to any AZ within that site. Zonal resources are available only in the associated AZ. Machines in different AZs have no single point of failure. Availability regions (“AR”) (also referred to as cloud instances) are isolated instances of a cloud controller and associated cloud resources within an AZ. AR resources are available only in the corresponding AR. In some embodiments, AZ resources also can be shared across multiple ARs within the corresponding AZ.

An instance of a machine, a virtual machine (“VM”), an application, a container POD, a container instance, or a container cluster can be instantiated in any site, AZ, or AR. A collection of VMs together can provide a service such as, for example, connected car, 5G voice and data service, and others. A local redundancy model for a service can spread VMs locally in a site across AZs and ARs to manage AZ and/or AR failures. Another level of redundancy, referred to herein as geo-redundancy, can spread a service across sites to manage site failures. In spreading VMs across AZs in different ARs, the real-time performance requirements (e.g., latency, jitter, etc.) of these services still need to be met. In general, an AR provides for resiliency within an AZ and enables high availability and higher cloud resource utilization while providing capabilities to meet the stringent real-time requirements of the services.

According to one aspect of the concepts and technologies disclosed herein, a 4-site local and geo-redundancy model for application placement within a cloud computing environment provides 37.5% cloud resource utilization with site availability of five 9s (99.999%) and virtual machine availability of five 9s. In particular, a plurality of sites operating as part of the cloud computing environment can include, for example, a first site, a second site, a third site, and a fourth site. The first site can include a first availability zone (“AZ”) that, in turn, includes a first availability region (“AR”) and a second AR. The first AR can include a first server and the second AR includes a second server. The first server can include a first virtual machine, and the second server can include a second virtual machine. The second site can include a second AZ that, in turn, includes a first duplicate of the first AR and a first duplicate of the second AR. The first duplicate of the first AR can include a first duplicate of the first server and the first duplicate of the second AR can include a first duplicate of the second server. The first duplicate of the first server can include a first duplicate of the first virtual machine. The first duplicate of the second server can include a first duplicate of the second virtual machine. The third site can include a third AZ that, in turn, includes a second duplicate of the first AR and a second duplicate of the second AR. The second duplicate of the first AR can include a second duplicate of the first server and the second duplicate of the second AR can include a second duplicate of the second server. The second duplicate of the first server can include a second duplicate of the first virtual machine. The second duplicate of the second server can include a second duplicate of the second virtual machine. The fourth site can include a fourth AZ that, in turn, includes a third duplicate of the first AR and a third duplicate of the second AR. The third duplicate of the first AR can include a third duplicate of the first server and the third duplicate of the second AR can include a third duplicate of the second server. The third duplicate of the first server can include a third duplicate of the first virtual machine. The third duplicate of the second server can include a third duplicate of the second virtual machine. The VMs across ARs and within an AZ can be connected via a layer 3 or layer 2 network. The plurality of sites also can be connected via a layer 3 or layer 2 network. The first server and the second server can be connected via a first layer 2 connection within the first AZ. The first duplicate of the first server and the first duplicate of the second server can be connected via a second layer 2 connection within the second AZ. The second duplicate of the first server and the second duplicate of the second server can be connected via a third layer 2 connection within the third AZ. The third duplicate of the first server and the third duplicate of the second server can be connected via a fourth layer 2 connection within the fourth AZ.

According to another aspect of the concepts and technologies disclosed herein, a 3-site local and geo-redundancy model for application placement within the cloud computing environment provides 66% cloud resource utilization with site availability of five 9s and virtual machine availability of five 9s. In particular, a plurality of sites operating as part of the cloud computing environment can include, for example, a first site, a second site, and a third site. The first site can include a first AZ that, in turn, includes a first AR, a second AR, and a third AR. The first AR can include a first server, the second AR can include a second server, and the third AR can include a third server. The first server can include a first virtual machine, the second server can include a second virtual machine, and the third server can include a third virtual machine. The second site can include a second AZ that, in turn, includes a first duplicate of the first AR, a first duplicate of the second AR, and a first duplicate of the third AR. The first duplicate of the first AR can include a first duplicate of the first server, the first duplicate of the second AR can include a first duplicate of the second server, and the first duplicate of the third AR can include a first duplicate of the third server. The first duplicate of the first server can include a first duplicate of the first virtual machine, the first duplicate of the second server can include a first duplicate of the second virtual machine, and the first duplicate of the third server can include a first duplicate of the third virtual machine. The third site can include a third AZ that, in turn, includes a second duplicate of the first AR, a second duplicate of the second AR, and a second duplicate of the third AR. The second duplicate of the first AR can include a second duplicate of the first server, the second duplicate of the second AR can include a second duplicate of the second server, and the second duplicate of the third AR can include a second duplicate of the third server. The second duplicate of the first server can include a second duplicate of the first virtual machine. The second duplicate of the second server can include a second duplicate of the second virtual machine, and the second duplicate of the third server can include a second duplicate of the third virtual machine. The VMs across ARs and within an AZ can be connected via a layer 3 or layer 2 network. The plurality of sites also can be connected via a layer 3 or layer 2 network. The first server, the second server, and the third server can be connected via a first layer 2 connection within the first AZ. The first duplicate of the first server, the first duplicate of the second server, and the first duplicate of the third server can be connected via a second layer 2 connection within the second AZ. The second duplicate of the first server, the second duplicate of the second server, and the second duplicate of the third server can be connected via a third layer 2 connection within the third AZ.

According to another aspect of the concepts and technologies disclosed herein, a 4-site local and geo-redundancy model for application placement within the cloud computing environment provides 75% cloud resource utilization with site availability of five 9s and virtual machine availability of five 9s. In particular, a plurality of sites operating as part of a cloud computing environment can include, for example, a first site, a second site, a third site, and a fourth site. The first site can include a first AZ that, in turn, includes a first AR, a second AR, a third AR, and a fourth AR. The first AR can include a first server, the second AR can include a second server, the third AR can include a third server, and the fourth AR can include a fourth server. The first server can include a first virtual machine, the second server can include a second virtual machine, the third server can include a third virtual machine, and the fourth server can include a fourth virtual machine. The second site can include a second AZ that, in turn, includes a first duplicate of the first AR, a first duplicate of the second AR, a first duplicate of the third AR, and a first duplicate of the fourth AR. The first duplicate of the first AR includes a first duplicate of the first server, the first duplicate of the second AR includes a first duplicate of the second server, the first duplicate of the third AR includes a first duplicate of the third server, and the first duplicate of the fourth AR includes a first duplicate of the fourth server. The first duplicate of the first server can include a first duplicate of the first virtual machine, the first duplicate of the second server can include a first duplicate of the second virtual machine, the first duplicate of the third server can include a first duplicate of the third virtual machine, and the first duplicate of the fourth server can include a first duplicate of the fourth virtual machine. The third site can include a third AZ that, in turn, includes a second duplicate of the first AR, a second duplicate of the second AR, a second duplicate of the third AR, and a second duplicate of the fourth AR. The second duplicate of the first AR can include a second duplicate of the first server, the second duplicate of the second AR can include a second duplicate of the second server, the second duplicate of the third AR can include a second duplicate of the third server, and the second duplicate of the fourth AR can include a second duplicate of the fourth server. The second duplicate of the first server can include a second duplicate of the first virtual machine, the second duplicate of the second server can include a second duplicate of the second virtual machine, the second duplicate of the third server can include a second duplicate of the third virtual machine, and the second duplicate of the fourth server can include a second duplicate of the fourth virtual machine. The fourth site can include a fourth AZ that, in turn, includes a third duplicate of the first AR, a third duplicate of the second AR, a third duplicate of the third AR, and a third duplicate of the fourth AR. The third duplicate of the first AR can include a third duplicate of the first server, the third duplicate of the second AR can include a third duplicate of the second server, the third duplicate of the third AR can include a third duplicate of the third server, and the third duplicate of the fourth AR can include a third duplicate of the fourth server. The third duplicate of the first server can include a third duplicate of the first virtual machine, the third duplicate of the second server can include a third duplicate of the second virtual machine, the third duplicate of the third server can include a third duplicate of the third virtual machine, and the third duplicate of the fourth server can include a third duplicate of the fourth virtual machine. The VMs across ARs within an AZ can be connected via a layer 3 or layer 2 network. The plurality of sites also can be connected via a layer 3 or layer 2 network. The first server, the second server, the third server, and the fourth server can be connected via a first layer 2 connection within the first AZ. The first duplicate of the first server, the first duplicate of the second server, the first duplicate of the third server, and the first duplicate of the fourth server can be connected via a second layer 2 connection within the second AZ. The second duplicate of the first server, the second duplicate of the second server, the second duplicate of the third server, and the second duplicate of the fourth server can be connected via a third layer 2 connection within the third AZ. The third duplicate of the first server, the third duplicate of the second server, the third duplicate of the third server, and the third duplicate of the fourth server can be connected via a fourth layer 2 connection within the fourth AZ.

In some embodiments, the cloud computing environment can detect an event within one of the plurality of sites. The event can include a planned event or an unplanned event. A planned event can include an upgrade to at least a portion of one of the plurality of sites. An unplanned event can include a failure of a least a portion of one of the plurality of sites. In response to the event, the cloud computing environment can redirect traffic from a first portion of the plurality of sites to a second portion of the plurality of sites.

In some embodiments, each of the virtual machines can provide, at least in part, a real-time service. Each of the virtual machines can be an instance of a virtual network function (“VNF”) that provides traditional or evolving mobility networking functions, such as access network elements, core network elements, transport network elements, and others, from purpose-built hardware to commercial-off-the-shelf (“COTS”) server-based platforms, such as those operating within the aforementioned servers. The real-time service can include, particularly, a voice service that can benefit greatly from the high availability and high utilization characteristics provided by the aforementioned models.

DETAILED DESCRIPTION

The practice standard for deployment of information technology (“IT”) applications in a cloud computing environment is to use availability zones to achieve resiliency. Many cloud service providers, such as AWS by Amazon Web Services, Inc., rely on availability zones for application deployment. These service providers can cope with increases in latency and jitter common with such practices. Telecommunications service providers may use cloud computing environments to deploy virtual network functions (“VNFs”) that provide various network functionality in support of real-time services, such as voice and data services. Most telecommunications VNFs cannot exploit availability zones for resiliency due to high latency and high jitter. Currently, the use of availability zones in a cloud computing environment results in very low cloud resource utilization (e.g., on the order of peak utilization of around 25% and average utilization of around 13%) to achieve high availability (i.e., five 9s of availability). This creates issues for telecommunications service providers, since deployment and maintenance of a cloud computing environment with low utilization and high availability significantly increases capital expenditures (“capex”) and operational expenditures (“opex”) to meet high availability requirements.

Referring now toFIG. 1A, a block diagram illustrating an example availability region (“AR”) and availability zone (“AZ”) distribution model in a high availability cloud data center architecture implemented in a cloud computing environment100to support real-time services will be described. The cloud computing environment100illustrates a data center site (“site”)102(also referred to as a geo-site or a geo-region) that is representative of a geographical location in which hardware resources of a data center operating as part of the cloud computing environment100are located. As used herein, a “data center” refers to a computing facility that includes one or more switches (e.g., core switches, top-of-rack switches, spine switches, leaf switches, and/or the like) and one or more server racks that, in turn, can include one or more servers upon which one or more virtual machines (“VMs”) can be executed. As used herein, a “VM” refers to a software-based computing system that runs one or more operating systems and/or one or more applications. Although VMs are illustrated and referenced throughout this disclosure, the cloud computing environment100alternatively can include other virtualized resources, such as virtual network functions (“VNFs”), virtual volumes, virtual networks, virtual containers, and/or other virtualized resources.

The site102can be identified, for example, by the city, such as San Diego, Houston, or New York, in which hardware resources operating in one or more data centers of the cloud computing environment100are located. The site102is not intended to encompass the entirety of a named location (e.g., San Diego), but instead a general area in which the hardware resources are located. Alternatively, the site102can identify general areas, such as, for example, North-United States, South-United States, East-United States, or West-United States. Although only a single site102is illustrated, the cloud computing environment100can include any number of sites102. An example multi-site configuration of the cloud computing environment100is shown inFIG. 1B.

A given site102can include any number of AZs104, each of which represents an isolated physical location within the site102, and each of which can be defined by a provider edge/gateway (“PE/GW”)106that designates a service demarcation for connectivity between resources in an AZ104and a backbone network (i.e., layer 3 “L3” network)108. The illustrated site102includes a first AZ (“AZ1”)104A and a second AZ (“AZ2”)104B defined by a first PE/GW (“PE/GW1”)106A and a second PE/GW (PE/GW2”)106B, respectively. A site102identified as “San Diego” might include multiple AZs, such as “San Diego 1,” “San Diego 2,” and so on.

In accordance with the concepts and technologies disclosed herein, and different from current availability zone distribution models used to achieve resiliency in real-time applications deployed in a cloud, a given AZ104can include any number of ARs110(alternatively referred to as “cloud instances”), each having a local control plane (“LCP”) that includes a server (not shown inFIG. 1) hosting a cloud controller (“CC”)112instance that manages a pool of tenant servers (shown as “tenant114”; single tenant server configurations are also contemplated) hosting one or more applications, such as one or more VNFs that support one or more real-time services.

In the illustrated embodiment, the AZ1104A includes a first AR (“AR1”)110A that, in turn, includes a first CC (“CC1”)112A that manages a first pool of tenant servers (“tenant1”)114A; a second AR (“AR2”)110B that, in turn, includes a second CC (“CC2”)112B that manages a second pool of tenant servers (“tenant2”)114B; and a third AR (“AR3”)110C that, in turn, includes a third CC (“CC3”)112C that manages a third pool of tenant servers (“tenant3”)114C. The AZ2104B duplicates the configuration of the AZ1104A for high availability, and as such, the elements in each AZ104of the site102are identified using the same numbering scheme. This numbering scheme is used throughout the description of the remaining FIGURES. Moreover, references to the AZs104, the PE/GWs106, the ARs110, the CCs112, and the tenants114hereinafter can be interpreted as an instance thereof. For example, both the AZ1104A and the AZ2104B shown in the configuration of the cloud computing environment100inFIG. 1Ainclude an instance of the AR1110A, the AR2110B, and the AR3110C to illustrate an embodiment of the redundancy that can provided by the cloud computing environment100in accordance with some of the concepts and technologies disclosed herein.

Each of the CCs112provides a set of cloud controller services116, such as compute services, networking services, storage services, orchestration services, and other services. In the illustrated embodiment, the cloud controller services116are OPENSTACK services, including Nova118, Neutron120, Cinder122, Swift124, Glance126, Heat127, and other services (not shown), each accessible via application programming interfaces (“APIs”; not shown) exposed by OPENSTACK. Nova118is an OPENSTACK service that allows the provisioning of compute instances (e.g., virtual machines, bare metal servers, and containers). Neutron120is an OPENSTACK service that provides network connectivity as-a-service between interface devices (e.g., virtual network interface controllers) managed by other OPENSTACK services, such as Nova118. Cinder122is an OPENSTACK service that provides tools to manage storage resources consumed by compute instances created by Nova118. Swift124is an OPENSTACK service that provides tools for data storage in the cloud. Glance126is an OPENSTACK service that facilitates discovery, registration, and retrieval of virtual machine images and associated metadata. Heat127is an OPENSTACK service that orchestrates cloud applications using a declarative template format through an OPENSTACK REST API. An example portion of a Heat127template for use in accordance with the concepts and technologies disclosed herein is described herein below with reference toFIG. 5. OPENSTACK is well-documented and understood by those skilled in the art. Therefore, additional details regarding OPENSTACK in general and the OPENSTACK services116particularly referenced herein are not provided, since those skilled in the art will readily understand the capabilities of OPENSTACK as pertinent to the concepts and technologies disclosed herein. It should be understand that the use of OPENSTACK herein is only an example software platform upon which the concepts and technologies disclosed herein can be implemented, and software platforms for cloud computing are contemplated, and the applicability of which would be understood by one skilled in the art.

Network functions virtualization (“NFV”) is a new technology initiative that aims to move traditional and evolving mobility networking functions, such as access network elements, core network elements, transport network elements, and others, from purpose-built hardware to commercial-off-the-shelf (“COTS”) server-based platforms, such as those operating within servers disclosed herein. This is achieved through virtualization of mobility networking functions to create VNFs that operate on COTS hardware. The VNFs can perform any telecommunications function in support of one or more real-time services, including, particularly, voice services that benefit greatly from high availability.

In some embodiments, the cloud computing environment100is or includes a software-defined network (“SDN”). SDN is an architectural framework that provides a software-centric cloud environment for creating intelligent networks that are programmable, application aware, and more open. SDN provides an agile and cost-effective communications platform for handling the dramatic increase in data traffic on carrier networks by providing a high degree of scalability, security, and flexibility. SDN provides several benefits over traditional networks. SDN allows for the creation of multiple virtual network control planes on common hardware. SDN helps extend service virtualization and software control into many existing network elements. SDN enables applications to request and manipulate services provided by the network and to allow the network to expose network states back to the applications. SDN exposes network capabilities through application programming interfaces (“APIs”), making the control of network equipment remotely accessible and modifiable via third-party software clients using open protocols such as OpenFlow, available from Open Network Forum (“ONF”).

Combining SDN and NFV functionality, such as in Domain 2.0, available from AT&T, provides a highly complex and dynamic set of relationships between virtual, logical, and physical resources. Networks, such as embodied in Domain 2.0 deployments, provide intelligent software systems and applications operating on general purpose commodity hardware (e.g., COTS). This not only drives down capital expenditures, ongoing operational costs, and helps to configure networks with less human intervention, but also creates significant opportunities to scale and monetize existing and new intelligent services.

Within service providers, such as AT&T, orchestration systems like control, orchestration, management, and policy (“ECOMP”) have been created to dramatically reduce monotonous tasks and monitoring required by human operators through data-based analytics. Current orchestration systems often incite frustration among operators due to over-complicated network status readouts, non-specific network manipulations automatically performed by the orchestration system, and the inability to quickly “revert” changes caused by such manipulations. AT&T's ECOMP has been combined with the Open Orchestrator Project (“OPEN-O”) to create the Open Network Automation Platform (“ONAP”) project supported by the Linux Foundation. ONAP is an open source software platform that delivers capabilities for the design, creation, orchestration, monitoring, and life cycle management of SDNs and the VNFs operating therein, as well as higher-level services that utilize the functionality of SDN/VNF. ONAP provides automatic, policy-driven interaction of these functions and services in a dynamic, real-time cloud environment, such as the cloud computing environment100.

In some embodiments, the cloud computing environment100provides, at least in part, Infrastructure-as-a-Service (“IaaS”), through which the tenants(s)114can interact with a front end (not shown) to provision processing, storage, networks, and other computing resources, whereby the tenants(s)114is/are able to deploy and run software, which can include, for example, VNFs to provide, at least in part, one or more telecommunications service(s) for the tenants114and/or others such as users or subscribers to the service(s). The tenant(s)114do not manage or control the underlying cloud infrastructure of the cloud computing environment100, but have control over operating systems, storage, and deployed applications, and in some implementations, limited control of select networking components (e.g., host firewalls and/or other security components).

In some embodiments, the cloud computing environment100is provided as part of a private cloud infrastructure. A private cloud infrastructure is a cloud infrastructure that is provisioned for exclusive use by a single organization, which can include multiple users. A private cloud infrastructure might be owned, managed, and operated by the organization, a third party, or some combination thereof. A private cloud infrastructure can exist on or off premises. The tenant114can access a private cloud infrastructure provided, at least in part, by the cloud computing environment100via a front end, which can be provided by and/or accessed through a client, such as a web client application, or a native client application, for example.

In some embodiments, the cloud computing environment100is provided as part of a community cloud infrastructure. A community cloud infrastructure is a cloud infrastructure that is provisioned for exclusive use by a specific community of users from organizations that have shared concerns (e.g., mission, security requirements, policy, and compliance considerations). A community cloud infrastructure might be owned, managed, and operated by one or more organizations in the community, a third party, or some combination thereof. A community cloud infrastructure may exist on or off premises. The tenant114can access a community cloud infrastructure provided, at least in part, by the cloud computing environment100via a front end, which can be provided by and/or accessed through a client, such as a web client application, or a native client application, for example.

In some embodiments, the cloud computing environment100is provided as part of a public cloud infrastructure. A public cloud infrastructure is a cloud infrastructure that is provisioned for open use by the general public. A public cloud infrastructure might be owned, managed, and operated by a business, academic, or government organization, or some combination thereof. A public cloud infrastructure exists on the premises of the cloud service provider. The tenants114can access a public cloud infrastructure provided, at least in part, by the cloud computing environment100via a front end, which can be provided by and/or accessed through a client, such as a web client application, or a native client application, for example.

In some embodiments, the cloud computing environment100is provided as part of a hybrid cloud infrastructure. A hybrid cloud infrastructure is a cloud infrastructure that is a composition of two or more distinct cloud infrastructures—private, community, or public—that remain unique entities, but are bound together by standardized or proprietary technology that enables data and application portability. The tenants114can access a hybrid cloud infrastructure provided, at least in part, by the cloud computing environment100via a front end, which can be provided by and/or accessed through a client, such as a web client application, or a native client application, for example.

Referring now toFIG. 1B, a block diagram illustrating another example AR and AZ distribution model in a high availability cloud data center architecture implemented in the cloud computing environment100to support real-time services will be described, according to an illustrative embodiment. The distribution model used by the cloud computing environment100inFIG. 1Billustrates a plurality of sites102A-102N, including a first site (“SITE1”)102A, a second site (“SITE”)102B, and an Nthsite (“SITEN”)102N. The SITE1102A includes a single AZ104—that is, the AZ1104A defined by the PE/GW1106A that designates a service demarcation for connectivity between cloud resources in the AZ1104A and the backbone network108. The SITE2102B includes three AZs104, including the AZ1104A, the AZ2104B, and the AZ3104C, defined, respectively, by the PE/GW1106A, the PE/GW2106B, and the PE/GW3106C that designate service demarcations for connectivity between cloud resources in a corresponding AZ104and the backbone network108. The SITEN102N includes two AZs104, including the AZ1104A and the AZ2104B, defined, respectively, by the PE/GW1106A and the PE/GW2106B that designate service demarcations for connectivity between cloud resources in a corresponding AZ104and the backbone network108. Each of the instances of the AZ1104A in the SITE1102A and the SITE2102B include the AR1110A, the AR2110B, and the AR3110C, as does the AZ2104B in the SITE2102B. The AZ3104C instance in the SITE2102B includes the AR1110A and the second AR2110B, as does the AZ1104A in SITEN102N. The AZ2104B instance in the SITEN102N includes the AR1110A. The SITEN102N is illustrative of three 9s (i.e., 99.9%) availability that can be achieved with a single AR110(see AZ2104B in the SITEN102N), and of five 9s (i.e., 99.999%) availability that can be achieved with two or more ARs110(see AZ1104A in the SITEN102N). The CCs112and the tenants114introduced inFIG. 1Aare not shown inFIG. 1B, but the ARs110should be interpreted as including at least one CC112and at least one tenant114such as in the configuration of the cloud computing environment100described above with reference toFIG. 1A.

Referring now toFIG. 1C, a block diagram illustrating a networking configuration for the aforementioned AR and AZ distribution models in a high availability cloud data center architecture implemented in the cloud computing environment100to support real-time services will be described, according to an illustrative embodiment. In the illustrated embodiment, the site102has one AZ104(AZ1104A) and two ARs (AR1110A and AR2110B), with each AR110having two servers128(i.e., representative of the tenant114) that, in turn, each include a VM130and a virtual router/virtual switch (“vR/vS”)132. In particular, the VM1130A hosted by the server1128A is communicatively connected to the VM2130B hosted by the server2128B via a layer 2 (“L2”) connection, and the VM3130C hosted by the server3128C is communicatively connected to the VM4130D in the server4128D via another L2 connection. The vR/vS1132A and the vR/vS2132B are communicatively connected to each other and to a software-defined network (“SDN”) controller1134A, which communicates, via a peer-to-peer connection, with the SDN controller2134B that serves the vR/vS3132C and the vR/vS4132D. The East-West communications between VMs130within a given AR110is typically layer 2, but alternatively can be layer 3.

Referring now toFIG. 2, a block diagram illustrating a contrast between a new cloud architecture200upon which the cloud computing environment100can be deployed and a conventional cloud architecture202, such as implemented, for example, in AWS by Amazon Web Services, Inc. will be described. In the illustrated example, the new cloud architecture200includes the SITE1102A and the SITE2102B, each having instances of the AZ1104A and the AZ2104B. The AZ1104A in both sites102A-102B includes instances of the AR1110A, the AR2110B, and the AR3110C. The AZ2104B in both sites102includes instances of the AR1110A and the AR2110B. In contrast to the new cloud architecture200, the conventional cloud architecture202includes two regions204(REGION1204A and REGION2204B), each of which includes two zones206(ZONE1206A and ZONE2206B).

Table 1 below shows the availability and latency achieved with the new cloud architecture200and the conventional cloud architecture202. The new cloud architecture200is capable of offering five 9s availability within the sites102A-102B, the AZs104A-104B, and the ARs110A-110C. The conventional cloud architecture202also is capable of offering five 9s availability within the regions204A-204B (as compared to the sites102), but fails to provide such high availability in the zones206A-206B (as compared to the AZs104). Moreover, the conventional cloud architecture202does not offer the additional distribution granularity provided by the ARs110A-110C in the new cloud architecture200, which also are capable of offering five 9s availability. Latency remains the same (>2 ms) for communications between AZs104A-104B in the new cloud architecture200and between zones206A-206B in the conventional cloud architecture202. Latency less than 2 ms (i.e., low latency) is achievable for communications between the ARs110A-110C. Since the conventional cloud architecture202fails to provide a demarcation similar such as provided by the ARs110A-110C in the new cloud architecture200, latency values associated with such a demarcation are not available for the conventional cloud architecture202.

Referring now toFIG. 3A, a block diagram illustrating an example VNF L2 stretch network300for the AR and AZ distribution models in the high availability cloud data center architecture implemented in the cloud computing environment100will be described, according to an illustrative embodiment. In the illustrated embodiment, the cloud computing environment100includes the site102having one AZ104(AZ1104A) and two ARs110(AR1110A and AR2110B). The AR1110A hosts the server1128A, which includes a VNF in an active state (“VNF-ACTIVE”)302and the vR/vS1132A. The AR2110B hosts the server2128B, which includes a VNF in a passive state (“VNF-PASSIVE”)304and the vR/vS2132B.

L2 networks within the AR1110A and the AR2110B are represented as the AR1L2 network306A and the AR2L2 network306B, respectively. The VNF L2 stretch network300utilizes Ethernet virtual private network (“EVPN”) routes to stretch the AR1L2 network306A and the AR2L2 network306B between the ARs110. East-to-west traffic between the ARs110A-110B (i.e., via vR/vS1132A to vR/vS2132B) traverses the VNF L2 stretch network300through spine switches310(to achieve latency<2 ms) without the traffic being routed through the PE/GW106.

The vR/vS1132A and the vR/vS2132B are communicatively connected to the SDN controller1134A and the SDN controller2134B, respectively. The SDN controllers134A-134B communicate, via a peer-to-peer connection, with a virtual route reflector (“vRR”)312. The vRR312advertises, to the SDN controllers134A-134B, IP addresses/routes and/or MAC addresses across the ARs110A-110B. The SDN controllers134A-134B instruct the vR/vSs132A-132B to forward tenant traffic through the routes/addresses/MACs advertised by the vRR312. Though the East-West communications between VMs130across ARs110is typically layer 2, and hence stretch L2 networks, it could be layer 3 as well. To meet the stringent real-time requirements, the L2/L3 traffic can be switched/routed within an AZ104.

Referring now toFIG. 3B, a block diagram illustrating a physical network topology for the AR and AZ distribution models in the high availability cloud data center architecture and implemented in the cloud computing environment100will be described, according to an illustrative embodiment. This topology scales to support the local cloud infrastructure and tenant traffic association with a tenant VNF layer 2 and layer 3 protocol adjacencies with each other and/or with the PE/GW106. The physical network topology illustrates a data center with a multi-AR configuration in a single site102using a full CLOS fabric network314. The site102has one AZ104defined by the PE/GWs106A,106B service demarcation that provides connectivity between resources in the AZ104and the backbone network108. The full CLOS fabric network314includes the PE/GW106communicatively connected to leaf switches316(border leaf 1 and border leaf 2) that, in turn, are communicatively connected to the spine switches310(introduced above with reference toFIG. 3A—illustrated in a hierarchy including two super spine switches and eight spine switches) that, in turn, are communicatively connected to additional leaf switches316, which provide connectivity to the ARs110A-110H and the tenants114A-114H.

Referring now toFIGS. 3C-3E, site architectures for a data center with a multi-AR configuration will be described, according to illustrative embodiments. Turning first toFIG. 3C, an example site architecture shows a data center with a multi-AR configuration in a single site using an implementation of the full CLOS fabric network314(introduced above with reference toFIG. 3B), according to an illustrative embodiment. The first site architecture includes the site102having one AZ104defined by the PE/GW106service demarcation that provides connectivity between resources in the AZ104and the backbone network108(not shown inFIG. 3C). The illustrated implementation of the full CLOS fabric network314includes the PE/GW106communicatively connected to the spine switches310(introduced above with reference toFIG. 3A) that, in turn, are communicatively connected to the leaf switches316(also introduced above with reference toFIG. 3B), which provide connectivity to the ARs110A-110B and the tenants114A-114B.

Turning now toFIG. 3D, another example site architecture shows a data center with a multi-AR configuration in the site102using one or more spine peer links318and/or one or more leaf peer links320, according to an illustrative embodiment. The illustrated example architecture includes the site102having one AZ104defined by the PE/GW106service demarcation that provides connectivity between resources in the AZ104and the backbone network108(not shown inFIG. 3D). In the illustrated embodiment, the CLOS fabric network314includes a first set of spine switches310A communicatively connected to a first set of leaf switches316A that provide connectivity to the ARs110A-110B, and a second set of spine switches310B communicatively connected to a second set of leaf switches316B that provide connectively to the tenants114A-114B. The first set of spine switches310A is communicatively connected to the second set of spine switches310B via the spine peer link(s)318. The first set of leaf switches316A is communicatively connected to the second set of leaf switches316B via the leaf peer link(s)320.

Turning now toFIG. 3E, another example site architecture shows a data center with a multi-AR configuration in the site102using the spine peer link(s)318and/or the leaf peer links320, according to an illustrative embodiment. The illustrated example architecture includes the site102having one AZ104defined by the PE/GW106service demarcation that provides connectivity between resources in the AZ104and the backbone network108(not shown inFIG. 3E). In the illustrated embodiment, the CLOS fabric network314includes the first set of spine switches310A communicatively connected to the first set of leaf switches316A that provide connectivity to the AR1110A and the tenant1114A, and the second set of spine switches310B communicatively connected to the second set of leaf switches316B that provide connectively to the AR2110B and the tenant2114B. The first set of spine switches310A is communicatively connected to the second set of spine switches310B via the spine peer link(s)318. The first set of leaf switches316A is communicatively connected to the second set of leaf switches316B via the leaf peer link(s)320.

Referring now toFIGS. 3F-3G, a VNF configured in an example active-passive distribution model (FIG. 3F) within the site102will be compared to a VNF in an example cluster distribution model (FIG. 3F) within the site102. The example active-passive distribution model shown inFIG. 3Fillustrates the site102, including the AZ1104A and the AZ2104B, each having instances of the AR1110A and the AR2110B. The AR1110A in the AZ1104A includes a first active VNF (“VNF1-ACTIVE”)302A and a second active VNF (“VNF2-ACTIVE”)302B. The VNF2-ACTIVE302B is communicatively connected via an L2 connection to a second passive VNF (“VNF2-PASSIVE”)304B in the AR2110B of the AZ1104A. The VNF1-ACTIVE302A is communicatively connected via an L2 connection to a first passive VNF (“VNF1-PASSIVE”)304A in the AR2110B of the AZ2104B. The AR1110A of the AZ2104B also includes a duplicate of the VNF2-ACTIVE302B along with a third active VNF (“VNF3-ACTIVE”)302C. The VNF2-ACTIVE302B in the AR1110A of the AZ2104B is communicatively connected via an L2 connection to a VNF2-PASSIVE304B in the AR2110B of the AZ2104B. The VNF3-ACTIVE302C in the AR1110A of the AZ2104B is communicatively connected via an L2 connection to a third passive VNF (“VNF3-PASSIVE”)304C in the AR2110B of the AZ2104B.

Most telecommunications carrier grade physical network functions have 1+1 redundancy for high availability. The concepts and technologies disclosed herein for the cloud computing environment100are able to achieve high availability and high utilization on par with these physical network functions using VNFs302/304. In the example illustrated inFIG. 3F, if the AZ1104A availability is 99%, the AZ2104B availability is 99%, and the standalone AZ1104A and AZ2104B are not telecommunications carrier grade, then: the availability of the VNF1302A/304A is 99.99% and the utilization of the VNF1302A/304A is 50%; the availability of the VNF2302B/304B is 99.99% and the utilization of the VNF2302B/304B is 25%; and the availability of the VNF3302C/304C is 99% and the utilization of the VNF3302C/304C is 50%. The availability and utilization of the VNF1302A/304A is in line with telecommunications carrier requirements. This results in significant Capex and Opex savings. The availability of the VNF2302B/304B is in line with telecommunications carrier requirements, but utilization is below the requirement, resulting in significant Capex and Opex costs. The availability of the VNF3302C/304C is below the requirement, but the utilization is in line with telecommunications carrier requirements, also resulting in significant Capex and Opex costs.

The example cluster distribution model shown inFIG. 3Gillustrates the site102, including the AZ1104A and the AZ2104B, each having instances of the AR1110A and the AR2110B. The AR1110A in the AZ1104A includes an instance of the VNF1-ACTIVE302A. The AR2110B in the AZ1104A includes two duplicate instances of the VNF1-ACTIVE302A. These three instances of the VNF1-ACTIVE302A are communicatively connected via L2 connections, thereby forming a cluster322A. The AR1110A in the AZ2104B includes two instances of the VNF1-ACTIVE302A. The AR2110B in the AZ2104B includes one instance of the VNF1-ACTIVE302A. These three instances of the VNF1-ACTIVE302A are communicatively connected via L2 connections, thereby forming a cluster322B. The clusters322A,322B are communicatively connected via L2 connection.

The concepts and technologies disclosed herein for the cloud computing environment100are able to achieve high availability and high utilization on par with physical network functions with cluster redundancy for high availability. In the example illustrated inFIG. 3G, if the AZ1104A availability is 99%, the AZ2104B availability is 99%, and the standalone AZ1104A and AZ2104B is not telecommunications carrier grade, then, with the cluster322A set as active and the cluster322B set as passive (stand-by), the availability is 99.99% and the utilization is 50%. The cluster availability and utilization are in line with telecommunications carrier requirements. This results in significant Capex and Opex savings.

Referring now toFIG. 4, a block diagram illustrating an end-to-end work flow400for VNF placement within the cloud computing environment100will be described, according to an illustrative embodiment. The illustrated embodiment shows an orchestrator402, a central placement decision system404, an inventory for active and available resources (“inventory”)406, and a target site408(e.g., one of the sites102described herein above).

The orchestrator402generates and sends a VNF homing request416to a conductor service410provided by the central placement decision system404. The conductor service410performs a capacity check for each candidate site of the sites102. The site102having capacity needed to accommodate the VNF homing request416is selected by the conductor service410as the target site408. The conductor service410responds to the orchestrator402by identifying the target site408to which VNFs should be homed.

The orchestrator402then generates and sends a VNF placement request418, including an OPENSTACK Heat orchestration template (“HOT”) (example shown inFIG. 5), to a valet service412. The valet service412determines VNF placements and returns, to the orchestrator402, the OPENSTACK HOT, including any modifications the valet service412made thereto. Based upon the placement decision, the valet service412schedules corresponding host-aggregates via OPENSTACK Nova REST APIs (generally shown as host-aggregates control420) of the OPENSTACK services116in the target site408.

The orchestrator402then instantiates VNF placements (generally shown as placement instantiation422). In particular, the orchestrator402communicates with OPENSTACK Heat127to instantiate VMs (e.g., the VMs130best shown inFIG. 1C) and receives results. The orchestrator402will notice rollback and retrial with the valet service412if the results indicate the VNF placement failed, or will confirm if the placement is successful. The OPENSTACK services116in the target site408will notify a resource orchestrator414of the resources consumed. The resource orchestrator414reports this placement result424to the inventory406, which can then update the active and available resources.

The orchestrator402creates new valet group declarations (i.e., valet affinity, diversity, and exclusivity groups), and updates the metadata associated therewith. The valet service412listens to OPENSTACK events (shown as VM and host update event428) from the inventory406. The valet service412also performs periodic resource status checks426of the target site408resources and caches via Nova REST APIs.

Referring now toFIG. 5, a block diagram illustrating an example snippet of a HOT template (requested HOT snippet500) sent by the orchestrator402to configure the target site408(shown inFIG. 4) within the cloud computing environment100will be described, according to an illustrative embodiment. The requested HOT snippet500shows the resources for each VM/VNF to be instantiated. The illustrated requested HOT snippet500provides resource definitions for three VMs130-VM1130A, VM2130B, and VM3130C—each of which are shown in the target site408. In particular, the requested HOT snippet500defines the instance types and properties for each of the VMs130such that the VM1130A is instantiated in a first host (“HOST1”)502A of hosts502A-502D in racks504A-504B in the AR1110A of the AZ1104A; the VM2130B is instantiated in a fifth host (HOST5″)502E of hosts502E-502H in racks504C-504D in the AR2110B of the AZ1104A; and the VM3130C is instantiated in a ninth host (“HOST9”)502I of hosts502I-502J in rack504E in the AR3110C of the AZ1104A. It should be noted that the “valet_availability_region” property in the requested HOT snippet500is a new property under OS::Nova::Server Type in the requested HOT snippet500.

Referring now toFIGS. 6A-6B, embodiments of an OPENSTACK neutron networking framework (“networking framework”)600for a data center602with a multi-AR configuration will be described, according to illustrative embodiments. Turning first toFIG. 6A, an embodiment of the networking framework600A for the data center602will be described. The data center602includes the AR1110A and the AR2110B served by the vR/vS132, which provides connectivity to the backbone network108. The AR1110A includes the server1128A that hosts the VM1130A and the VM2130B. The AR2110B includes the server2128B that hosts the VM3130C and the VM4130D. The VMs130A-130D are connected to the vR/vS132via a subnet604(e.g., the VNF L2 stretch network300shown inFIG. 3A) that spans both of the ARs110. In case of single root input/output virtualization (“SR-IOV”) implementations, a virtual Ethernet bridge (“VEB”) in a network interface card (“NIC”) can be used instead of the vR/vS132in the host kernel or in the user space.

Turning toFIG. 6B, another embodiment of the networking framework600B for the data center602will be described. The data center602includes the AR1110A served by the vR/vS1132A and the AR2110B served by the vR/vS2132B. The vR/vSs132A,132B provide connectivity to the backbone network108through the PE/GW106. The AR1110A includes the server1128A that hosts the VM1130A and the VM2130B. The AR2110B includes the server2128B that hosts the VM3130C and the VM4130D. The VMs130A-130B are connected to the vR/vS1132A via a constituent subnet (“subnet1”)606A (e.g., the AR1L2 network306A shown inFIG. 3A). The VMs130C-130D are connected to the vR/vS2132B via another constituent subnet (“subnet2”)606B (e.g., the AR2L2 network306B shown inFIG. 3A).

Referring now toFIG. 7, a block diagram illustrating an in-service sequential AR-by-AR upgrade within an AZ104followed by an AZ-by-AZ upgrade in a site102without impacting tenant services will be described, according to an illustrative embodiment. If each of the services in a site102are contained locally in an AZ104, then the AZ-by-AZ upgrade in the site102alternatively can occur in parallel, but the AR-by-AR upgrade within an AZ is preferred to be sequential.

FIG. 7illustrates the site102, including the AZ1104A and the AZ2104B, each having instances of the AR1110A and the AR2110B. The AR1110A of the AZ1104A includes the VNF1-ACTIVE302A and the VNF2-ACTIVE302B. The second VNF2-ACTIVE302B is communicatively connected via an L2 connection to the VNF2-PASSIVE304B in the AR2110B of the AZ1104A. The VNF1-ACTIVE302A in the AR1110A of the AZ1104A is communicatively connected via an L2 connection to VNF1-PASSIVE304A in the AR2110B of the AZ2104B. The AR2110B of the AZ2104B also includes a duplicate of the VNF2-PASSIVE304B that is communicatively connected via an L2 connection to a duplicate instance of the VNF2-ACTIVE302B in the AR1110A of the AZ2104B. To perform an OPENSTACK and/or VNF upgrade from an N version to an N+1 version, the AZ1104A is set to active mode while the AZ2104B is set to maintenance mode and upgraded. After the AZ2104B is upgraded, the AZ2104B returns to active mode and the AZ1104A is set to maintenance mode and upgraded.

Turning now toFIG. 8A, a block diagram illustrating layer 2 adjacency between tenant VNFs hosted in different ARs (generally shown as800) will be described, according to an illustrative embodiment. In the illustrated embodiment, the tenants114A,114B are hosted as VMs, Containers, or other virtual hosting solution on the servers128A,128B, respectively. The servers128A,128B are associated with the ARs110A,110B, respectively. The control functions—that is, the cloud controllers116A,116B and the SDN controllers134A,134B—for the ARs110A,110B manage the virtual hosting and the virtual network configuration details for the tenants114A,114B via the hosting agents and the SDN endpoints (e.g., the vR/vSs132A,132B). The SDN endpoints can be implemented by vRouter, vSwitch, Virtual Ethernet Bridge, or other network client in the servers128A,128B. The WAN network of the tenants114A,114B is configured as EVPN or VRF at the PE/GW106. An SDN Gateway at the PE/GW106also is managed by the SDN controllers134A,134B. Network adjacency between the tenants114A,114B and the WAN edge can be L2 and/or L3. Tenant network traffic can be forwarded over the leaf switches316A,316B and the spine switches310A,310B either as tagged or tunneled, depending on the specific SDN implementation model.

Turning now toFIG. 8B, a block diagram illustrating an SDN link between VNFs in different ARs (generally shown as802) will be described, according to an illustrative embodiment. The illustrated embodiment shows the L2-L4 protocol fields that are managed over an SDN logical link between the tenants and across the network cloud leaf and spine infrastructure when the SDN endpoints are in the servers. The SDN endpoints forward tenant traffic as tagged or tunneled, depending on the specific SDN implementation model. The SDN logical link provides L2 and/or L3 network adjacency between the tenants hosted in different ARs.

Turning now toFIG. 8C, a block diagram illustrating an SDN link between VNFs in different ARs in a partial SR-IOV implementation (generally shown at804) will be described, according to an illustrative embodiment. The illustrated embodiment shows how the L2-L4 protocol fields are managed over an SDN logical link between the tenants and across the network cloud leaf and spine infrastructure when one SDN endpoint is in a server and the other endpoint is at a leaf switch, such as when the SDN model uses SR-IOV at one server. The SDN endpoints forward tenant traffic as tagged or tunneled, depending on the specific SDN implementation model. The SDN logical link provides L2 and/or L3 network adjacency between the tenants hosted in different ARs.

Turning now toFIG. 8D, a block diagram illustrating an SDN link between VNFs in different ARs in a full SR-IOV implementation (generally shown at806) will be described, according to an illustrative embodiment. The illustrated embodiment shows how the L2-L4 protocol fields are managed over an SDN logical link between the tenants and across the network cloud leaf and spine infrastructure when both SDN endpoints are at leaf switches, such as when the SDN model uses SR-IOV at both servers. The SDN endpoints forward tenant traffic as tagged or tunneled, depending on the specific SDN implementation model. The SDN logical link provides L2 and/or L3 network adjacency between the tenants hosted in different ARs.

Referring now toFIG. 9A, a block diagram illustrating a 4-site model900A for configuring the cloud computing environment100to achieve 25% resource utilization will be described.FIG. 9Awill be described using the numbering scheme established above for ease of explanation.

The illustrated 4-site model900A includes sites102A-102D, each communicatively connected via the backbone network108. Each of the sites102includes one AZ104. In particular, the SITE′102A includes the AZ1104A; the SITE2102B includes the AZ2104B; the SITE3102C includes the AZ3104C; and the site4102D includes the AZ4104D.

Each of the AZs104in the 4-site model900A includes one AR110. In particular, the AZ1104A includes the AR1110A; the AZ2104B includes the AR2110B; the AZ3104C includes the AR3110C; and the AZ4104D includes the AR4110D. Each of the ARs110can include a pool of tenant servers114(shown as “tenant114”; single tenant server configurations are also contemplated) hosting one or more applications. In particular, the AR1110A includes the tenant1114A; the AR2110B includes the tenant2114B; the AR3110C includes tenant3114C; and the AR4110D includes the tenant4114D.

Each of the tenants114can host one or more of the VMs130. In particular, the tenant1114A hosts the VM1130A and the VM2130B; the tenant2114B hosts the VM3130C and the VM4130D; the tenant3114C hosts the VM5130E and the VM6130F; and the tenant4114D hosts the VM7130G and the VM8130H. Each pair of VMs130(e.g., the VM1130A and the VM2130B) can be implemented in an active-passive configuration.

The 4-site model900A provides a total 8 million (“8 M”) quota in the four sites102A-102D, with each site102providing a 2 M quota with 1 M each for active and passive (i.e., stand-by) VMs130to achieve 25% utilization. Site availability for the 4-site model900A is three 9s (99.9%); AR110(LCP) availability also is three 9s; VM availability for a given active-passive VM pair is three 9s; site availability is five 9s; and the storage design of the 4-site model900A provides a single point of failure. Each site102in the 4-site model900A has only one AR110. Each of the sites102carries 500 thousand (“500K”) active traffic for a total traffic of 2 M in the four sites102A-102D. An upgrade or failure within any of the sites102A-102D results in the upgraded/failed site going out-of-service. Thus, when one of the sites102A-102D (e.g., SITE1102A is upgraded during a planned event) and another one of the sites102A-102D (e.g., SITE2102B) fails as a result of an unplanned event (e.g., a dramatic increase in traffic), the remaining two sites102C,102D are required to manage the 2 M traffic.

Turning now toFIG. 9B, a block diagram illustrating a novel 4-site model900B for configuring the cloud computing environment100to achieve 50% resource utilization will be described, according to an illustrative embodiment. The illustrated novel 4-site model900B includes the sites102A-102D communicatively connected via the backbone network108. In the illustrated novel 4-site model900B, each of the sites102includes one AZ104defined by the PE/GW106service demarcation that provides connectivity between resources in the AZ104and the backbone network108. In particular, the site1102A includes the AZ1104A defined by the PE/GW1106A; the SITE2102B includes AZ2104B defined by the PE/GW2106B; the SITE3102C includes the AZ3104C defined by the PE/GW3106C; and the SITE4102D includes the AZ4104D defined by the PE/GW4106D.

Each of the AZs104A-104D in the illustrated novel 4-site model900B includes two ARs110. In particular, the AZ1104A includes the AR1110A and the AR2110B; the AZ2104B includes duplicate instances of the AR1110A and the AR2110B; the AZ3104C includes duplicate instances of the AR1110A and the AR2110B; and the AZ4104D includes duplicate instances of the AR1110A and the AR2110B. Each of the ARs110in the illustrated novel 4-site model900B includes one CC112and one tenant114. In particular, the AR1110A includes the CC1112A and the tenant1114A, and the AR2110B includes the CC2112B and the tenant2114B. Each of the tenants114in the illustrated novel 4-site model900B includes one VM130. In particular, the tenant1114A includes the VM1130A, and the tenant2114B includes the VM2130B. Each pair of the tenants114A,114B can communicate via an L2 connection.

The illustrated novel 4-site model900B provides a total 8 M quota in the four sites102A-102D, with each site102providing a 2 M quota with 1 M quota for active VMs130in one AR110(e.g., the AR1110A) and 1 M quota for standby VMs130in the other AR110(e.g., the AR2110B) to achieve 50% utilization (a 25% utilization improvement over the 4-site model900A described above with referenceFIG. 9A). Site102availability for the novel 4-site model900B is five 9s (99.999%), AR (LCP)110availability is three 9s, VM130availability in an active-passive VM pair (e.g., VM1130A, VM2130B) is five 9s, site102network availability is five 9s, and storage design provides redundancy with via the L2 connections between the VMs130A,130B. Each of the sites102carries 750K active traffic for a total traffic of 3 M in the four sites102A-102D (instead of 2 M in the 4 sites102A-102D of the 4-site model900A described above with reference toFIG. 9A). An upgrade or failure within any of the sites102A-102D is managed locally. For example, if the AR1110A is upgraded (or fails), the AR2110B manages any redirected traffic. If any of the sites102A-102D goes down, traffic is redirected to the remaining three sites102resulting in 3 M traffic to be handled by these sites102.

Turning now toFIG. 9C, a block diagram illustrating a novel 3-site model900C for application placement in the cloud computing environment100to achieve 66% resource utilization will be described, according to an illustrative embodiment. The illustrated novel 3-site model900C includes the sites102A-102C communicatively connected via the backbone network108. In the illustrated novel 3-site model900C, each of the sites102A-102C includes one AZ104defined by the PE/GW106service demarcation that provides connectivity between resources in the AZ104and the backbone network108. In particular, the site1102A includes the AZ1104A defined by the PE/GW1106A; the SITE2102B includes AZ2104B defined by the PE/GW2106B; and the SITES102C includes the AZ3104C defined by the PE/GW3106C.

Each of the AZs104A-104C in the illustrated novel 3-site model900C includes three ARs110A-110C. In particular, the AZ1104A includes the AR1110A, the AR2110B, and the AR3110C; the AZ2104B includes a duplicate of the AR1110A, the AR2110B, and the AR3110C; and the AZ3104C includes a duplicate of the AR1110A, the AR2110B, and the AR3110C. Each of the ARs110in the illustrated novel 3-site model900C includes one CC112and one tenant114. In particular, the AR1110A includes the CC1112A and the tenant1114A, the AR2110B includes the CC2112B and the tenant2114B, and the AR3110C includes the CC3112C and the tenant3114C. Each of the tenants114in the illustrated novel 3-site model900C includes one VM130. In particular, the tenant1114A includes the VM1130A, the tenant2114B includes the VM2130B, and the tenant3114C includes the VM3130C. The tenants114A-114C can communicate via an L2 connection.

The illustrated novel 3-site model900C provides a total 3 M quota in the three sites102A-102C, with each site102providing a 330K quota for each AR110for a total quota of 1 M per site102. Also, each site102carries only 666K traffic, thus providing 66% utilization (a 41% improvement over the 4-site model900A; seeFIG. 9A). Site102availability for the novel 3-site model900C is five 9s (99.999%), AR110(LCP) availability is three 9s, VM130availability in an active-passive VM130pair is five 9s, site102network availability is five 9s, and storage design provides redundancy with via the L2 connections between the VMs130A-130C. An upgrade or failure within any of the sites102is managed locally within the site102. For example, if an upgrade or failure occurs in the AR1110A, traffic is redirected to the other ARs in that site102(e.g., the AR2110B and the AR3110C). If any of the sites102experiences a total failure, traffic is redirected to spare VMs130executing on the other two sites102.

Referring now toFIG. 9D, a block diagram illustrating a second novel 4-site model900D for configuring a cloud computing environment, such as provided by the cloud computing environment100, to achieve 75% resource utilization will be described, according to an illustrative embodiment. The illustrated novel 4-site model900D includes the sites102A-102D communicatively connected via the backbone network108. In the illustrated novel 4-site model900D, each of the sites102A-102D includes one AZ104defined by a PE/GW106service demarcation that provides connectivity between resources in an AZ104and the backbone network108. In particular, the site1102A includes the AZ1104A defined by the PE/GW1106A; the SITE2102B includes AZ2104B defined by the PE/GW2106B; the SITE3102C includes the AZ3104C defined by the PE/GW3106C; and the SITE4102D includes the AZ4104D defined by the PE/GW4106D.

Each of the AZs104A-104D in the illustrated novel 4-site model900D includes four ARs110A-110D. In particular, the AZ1104A includes the AR1110A, the AR2110B, the AR3110C, and the AR4110D; the AZ2104B includes a duplicate of the AR1110A, the AR2110B, the AR3110C, and the AR4110D; the AZ3104C includes a duplicate of the AR1110A, the AR2110B, the AR3110C, and the AR4110D; and the AZ4104D includes a duplicate of the AR1110A, the AR2110B, the AR3110C, and the AR4110D. Each of the ARs110in the illustrated novel 4-site model900D includes one CC112and one tenant114. In particular, the AR1110A includes the CC1112A and the tenant1114A, the AR2110B includes the CC2112B and the tenant2114B, the AR3110C includes the CC3112C and the tenant3114C, and the AR4110D includes the CC4112D and the tenant4114D. Each of the tenants114in the illustrated novel 4-site model900D includes one VM130. In particular, the tenant1114A includes the VM1130A, the tenant2114B includes the VM2130B, the tenant3114C includes the VM3130C, and the tenant4114D includes the VM4130D. The tenants114A-114D can communicate via an L2 connection.

The illustrated novel 4-site model900D provides a total 3 M quota in four sites102A-102D, with each site102providing a 250K quota for each AR110for a total quota of 1 M per site102. Also, each site102carries only 750K traffic, thus providing 75% utilization (a 50% improvement over the 4-site model900A shown inFIG. 9A). Site102availability for the novel 4-site model900D is five 9s (99.999%), AR110(LCP) availability is three 9s, VM130availability in an active-passive VM pair is five 9s, site102network availability is five 9s, and storage design provides redundancy with via the L2 connections between the VMs130A-130D. Each site102in the novel 4-site model900D has a 250K active quota on each of the ARs110A-110D. An upgrade or failure within any of the sites102is managed locally within the site102. For example, if an upgrade or failure occurs in the AR1110A, traffic is redirected to the other ARs110in that site102(e.g., the AR2110B, the AR3110C, and the AR4110D). If any of the sites102experiences a total failure, traffic is redirected to spare VMs130executing on the other three sites102.

Turning now toFIG. 10A, a graph1000illustrating an example cluster size (x-axis) versus peak cloud utilization (y-axis) will be described, according to an illustrative embodiment. The cluster size refers to the local cluster and geo-cluster size. Five 9s of VM availability can be achieved using a combination of VM clusters spread across ARs110within the AZ104for local redundancy and replication of this across sites102for geo-redundancy. For example, a cluster size of 4 in the graph1000refers to a local cluster of four VMs130and a geo-cluster of four such sites102(with four VMs130each; as shown inFIG. 9D). The peak cloud utilization refers to effective quota utilization. Bigger cluster sizes increase the complexity for implementation. Cluster sizes of 3 to 6 offer better utilization and manageable complexity. Since telecommunications networks are typically engineered for 80% peak utilization, cluster sizes of 4 or 5 are optimal.

Turning now toFIG. 10B, a table1002illustrating example topologies and cloud resource utilization scenarios will be described, according to an illustrative embodiment. Conditions for defining utilization can be modified based upon the site physical topology and application requirements. In such cases, cloud resource utilization might vary accordingly but still be maintained above 50% that is typical of physical infrastructures for real-time services.

For purposes of illustrating and describing some of the concepts of the present disclosure, the methods disclosed herein are described as being performed, at least in part, by a VM placement system, such as the central placement decision system404(seeFIG. 4) executing instructions to perform operations disclosed herein. It should be understood that additional and/or alternative devices and/or network nodes can provide the functionality described herein via execution of one or more modules, applications, and/or other software. Thus, the illustrated embodiments are illustrative, and should not be viewed as being limiting in any way.

The method1100begins and proceeds to operation1102, where the central placement decision system404receives an application placement request from a request queue. From operation1102, the method1100proceeds to operation1104, where the central placement decision system404determines the availability and utilization requirements for application placement.

In some embodiments, the application placement request specifies availability and/or utilization requirements to be met for placement of the requested application. In this manner, the application placement request can identify any of the novel high availability and high utilization models disclosed herein to be used for application placement.

In other embodiments, the central placement decision system404determines the availability and utilization requirements under which the application is to be placed. The central placement decision system404can make such determinations based upon one or more policies created by or for the provider of at least a portion of the cloud computing environment.

The central placement decision system404also can consider the status of one or more cloud resources in this determination. The status can include current utilization metrics for one or more of the cloud resources available from the cloud computing environment. The status can identify any cloud resource failures based upon output received from one or more monitoring systems of one or more servers (e.g., server128). The status can include information regarding any planned event, including, for example, any planned upgrades to any of the sites102, the AZs104, the ARs110, the servers128associated with at least a portion of the cloud computing environment. From operation1104, the method1100proceeds to operation1106, where the central placement decision system404places the requested application in the cloud computing environment in accordance with the availability and utilization requirements determined at operation1104.

From operation1106, the method1100proceeds to operation1108, where the cloud computing environment detects a failure or a planned event. A failure can be detected via one or more monitoring systems that are deployed within the cloud computing environment at any level—that is, the site102, AZ104, AR110, server128, or VM130level. The planned event can be an upgrade or other modification to any hardware and/or software associated with at least a portion of the cloud computing environment in which the application was placed at operation1106.

From operation1108, the method1100proceeds to operation1110, where the cloud computing environment, in response to the failure or planned event detected at operation1106, redirects traffic associated with application from the portion of the cloud computing environment affected by the failure or planned event to one or more spare VMs operating elsewhere in the cloud computing environment. For example, the cloud computing environment can redirect traffic from one of the sites102to one or more other sites102that have available spare VMs130. From operation1110, the method1100proceeds to operation1112, where the method1100ends.

Turning now toFIG. 12, an illustrative functions virtualization platform1200capable of implementing aspects of the cloud computing environment100will be described, according to an illustrative embodiment. The functions virtualization platform1200includes a hardware resource layer1202, a hypervisor layer1204, a virtual resource layer1206, a virtual function layer1208, and a service layer1210. While no connections are shown between the layers illustrated inFIG. 12, it should be understood that some, none, or all of the components illustrated inFIG. 12can be configured to interact with one other to carry out various functions described herein. In some embodiments, the components are arranged so as to communicate via one or more networks. Thus, it should be understood thatFIG. 12and the remaining description are intended to provide a general understanding of a suitable environment in which various aspects of the embodiments described herein can be implemented and should not be construed as being limiting in any way.

The hardware resource layer1202provides hardware resources. In the illustrated embodiment, the hardware resource layer1202includes one or more compute resources1212, one or more memory resources1214, and one or more other resources1215. The compute resource(s)1212can include one or more hardware components that perform computations to process data and/or to execute computer-executable instructions of one or more application programs, one or more operating systems, and/or other software. In particular, the compute resources1212can include one or more central processing units (“CPUs”) configured with one or more processing cores. The compute resources1212can include one or more graphics processing unit (“GPU”) configured to accelerate operations performed by one or more CPUs, and/or to perform computations to process data, and/or to execute computer-executable instructions of one or more application programs, one or more operating systems, and/or other software that may or may not include instructions particular to graphics computations. In some embodiments, the compute resources1212can include one or more discrete GPUs. In some other embodiments, the compute resources1212can include CPU and GPU components that are configured in accordance with a co-processing CPU/GPU computing model, wherein the sequential part of an application executes on the CPU and the computationally-intensive part is accelerated by the GPU processing capabilities. The compute resources1212can include one or more system-on-chip (“SoC”) components along with one or more other components, including, for example, one or more of the memory resources1214, and/or one or more of the other resources1215. In some embodiments, the compute resources1212can be or can include one or more SNAPDRAGON SoCs, available from QUALCOMM of San Diego, Calif.; one or more TEGRA SoCs, available from NVIDIA of Santa Clara, Calif.; one or more HUMMINGBIRD SoCs, available from SAMSUNG of Seoul, South Korea; one or more Open Multimedia Application Platform (“OMAP”) SoCs, available from TEXAS INSTRUMENTS of Dallas, Tex.; one or more customized versions of any of the above SoCs; and/or one or more proprietary SoCs. The compute resources1212can be or can include one or more hardware components architected in accordance with an ARM architecture, available for license from ARM HOLDINGS of Cambridge, United Kingdom. Alternatively, the compute resources1212can be or can include one or more hardware components architected in accordance with an x86 architecture, such an architecture available from INTEL CORPORATION of Mountain View, Calif., and others. Those skilled in the art will appreciate the implementation of the compute resources1212can utilize various computation architectures, and as such, the compute resources1212should not be construed as being limited to any particular computation architecture or combination of computation architectures, including those explicitly disclosed herein.

The memory resource(s)1214can include one or more hardware components that perform storage/memory operations, including temporary or permanent storage operations. In some embodiments, the memory resource(s)1214include volatile and/or non-volatile memory implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data disclosed herein. Computer storage media includes, but is not limited to, random access memory (“RAM”), read-only memory (“ROM”), Erasable Programmable ROM (“EPROM”), Electrically Erasable Programmable ROM (“EEPROM”), flash memory or other solid state memory technology, CD-ROM, digital versatile disks (“DVD”), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store data and which can be accessed by the compute resources1212.

The other resource(s)1215can include any other hardware resources that can be utilized by the compute resources(s)1212and/or the memory resource(s)1214to perform operations described herein. The other resource(s)1215can include one or more input and/or output processors (e.g., network interface controller or wireless radio), one or more modems, one or more codec chipset, one or more pipeline processors, one or more fast Fourier transform (“FFT”) processors, one or more digital signal processors (“DSPs”), one or more speech synthesizers, and/or the like.

The hardware resources operating within the hardware resource layer1202can be virtualized by one or more hypervisors1216A-1216N (also known as “virtual machine monitors”) operating within the hypervisor layer1204to create virtual resources that reside in the virtual resource layer1206. The hypervisors1216A-1216N can be or can include software, firmware, and/or hardware that alone or in combination with other software, firmware, and/or hardware, creates and manages virtual resources1218A-1218N operating within the virtual resource layer1206.

The virtual resources1218A-1218N operating within the virtual resource layer1206can include abstractions of at least a portion of the compute resources1212, the memory resources1214, and/or the other resources1215, or any combination thereof. In some embodiments, the abstractions can include one or more VMs, virtual volumes, virtual networks, and/or other virtualizes resources upon which one or more VNFs1219A-1219N can be executed. The VNFs1219A-1219N in the virtual function layer1208are constructed out of the virtual resources1218A-1218N in the virtual resource layer1206. In the illustrated example, the VNFs1219A-1219N can provide, at least in part, one or more services1220A-1220N in the service layer1210.

FIG. 13is a block diagram illustrating a computer system1300configured to provide the functionality in accordance with various embodiments of the concepts and technologies disclosed herein. It should be understood, however, that modification to the architecture may be made to facilitate certain interactions among elements described herein.

The computer system1300includes a processing unit1302, a memory1304, one or more user interface devices1306, one or more input/output (“I/O”) devices1308, and one or more network devices1310, each of which is operatively connected to a system bus1312. The bus1312enables bi-directional communication between the processing unit1302, the memory1304, the user interface devices1306, the I/O devices1308, and the network devices1310.

The processing unit1302may be a standard central processor that performs arithmetic and logical operations, a more specific purpose programmable logic controller (“PLC”), a programmable gate array, or other type of processor known to those skilled in the art and suitable for controlling the operation of the server computer. Processing units are generally known, and therefore are not described in further detail herein.

The memory1304communicates with the processing unit1302via the system bus1312. In some embodiments, the memory1304is operatively connected to a memory controller (not shown) that enables communication with the processing unit1302via the system bus1312. The illustrated memory1304includes an operating system1314and one or more program modules1316. The operating system1314can include, but is not limited to, members of the WINDOWS, WINDOWS CE, and/or WINDOWS MOBILE families of operating systems from MICROSOFT CORPORATION, the LINUX family of operating systems, the SYMBIAN family of operating systems from SYMBIAN LIMITED, the BREW family of operating systems from QUALCOMM CORPORATION, the MAC OS, OS X, and/or iOS families of operating systems from APPLE CORPORATION, the FREEBSD family of operating systems, the SOLARIS family of operating systems from ORACLE CORPORATION, other operating systems, and the like.

The program modules1316may include various software and/or program modules to perform the various operations described herein. The program modules1316and/or other programs can be embodied in computer-readable media containing instructions that, when executed by the processing unit1302, perform various operations such as those described herein. According to embodiments, the program modules1316may be embodied in hardware, software, firmware, or any combination thereof.

The user interface devices1306may include one or more devices with which a user accesses the computer system1300. The user interface devices1306may include, but are not limited to, computers, servers, PDAs, cellular phones, or any suitable computing devices. The I/O devices1308enable a user to interface with the program modules1316. In one embodiment, the I/O devices1308are operatively connected to an I/O controller (not shown) that enables communication with the processing unit1302via the system bus1312. The I/O devices1308may include one or more input devices, such as, but not limited to, a keyboard, a mouse, or an electronic stylus. Further, the I/O devices1308may include one or more output devices, such as, but not limited to, a display screen or a printer.

The network devices1310enable the computer system1300to communicate with other networks or remote systems via a network1318. Examples of the network devices1310include, but are not limited to, a modem, a radio frequency (“RF”) or infrared (“IR”) transceiver, a telephonic interface, a bridge, a router, or a network card. The network1318may include a wireless network such as, but not limited to, a Wireless Local Area Network (“WLAN”), a Wireless Wide Area Network (“WWAN”), a Wireless Personal Area Network (“WPAN”) such as provided via BLUETOOTH technology, a Wireless Metropolitan Area Network (“WMAN”) such as a WiMAX network or metropolitan cellular network. Alternatively, the network1318may be a wired network such as, but not limited to, a Wide Area Network (“WAN”), a wired Personal Area Network (“PAN”), or a wired Metropolitan Area Network (“MAN”). The network1318can be or can include the backbone network108and/or, one or more networks operating within the cloud computing environment100.

Turning now toFIG. 14, details of an overall network1400are illustrated, according to an illustrative embodiment. The overall network1400includes a cellular network1402, a packet data network1404, for example, the Internet, and a circuit switched network1406, for example, a public switched telephone network (“PSTN”). The backbone network108can be provided as part of the overall network1400or integrated within one or more of the sub-networks encompassed thereby.

The cellular network1402includes various components such as, but not limited to, base transceiver stations (“BTSs”), Node-B's or e-Node-B's, base station controllers (“BSCs”), radio network controllers (“RNCs”), mobile switching centers (“MSCs”), mobile management entities (“MMEs”), short message service centers (“SMSCs”), multimedia messaging service centers (“MMSCs”), home location registers (“HLRs”), home subscriber servers (“HSSs”), visitor location registers (“VLRs”), charging platforms, billing platforms, voicemail platforms, GPRS core network components, location service nodes, an IP Multimedia Subsystem (“IMS”), and the like. The cellular network1402also includes radios and nodes for receiving and transmitting voice, data, and combinations thereof to and from radio transceivers, networks, the packet data network1404, and the circuit switched network1406.

A mobile communications device1408, such as, for example, a cellular telephone, a user equipment, a mobile terminal, a PDA, a laptop computer, a handheld computer, and combinations thereof, can be operatively connected to the cellular network1402. The cellular network1402can be configured as a 2G Global System for Mobile communications (“GSM”) network and can provide data communications via General Packet Radio Service (“GPRS”) and/or Enhanced Data rates for Global Evolution (“EDGE”). Additionally, or alternatively, the cellular network1402can be configured as a 3G Universal Mobile Telecommunications System (“UMTS”) network and can provide data communications via the High-Speed Packet Access (“HSPA”) protocol family, for example, High-Speed Downlink Packet Access (“HSDPA”), Enhanced Uplink (“EUL”) (also referred to as High-Speed Uplink Packet Access “HSUPA”), and HSPA+. The cellular network1402also is compatible with 4G mobile communications standards such as Long-Term Evolution (“LTE”), or the like, as well as evolved and future mobile standards.

The packet data network1404includes various devices, for example, servers, computers, databases, and other devices in communication with one another, as is generally known. The packet data network1404devices are accessible via one or more network links. The servers often store various files that are provided to a requesting device such as, for example, a computer, a terminal, a smartphone, or the like. Typically, the requesting device includes software (a “browser”) for executing a web page in a format readable by the browser or other software. Other files and/or data may be accessible via “links” in the retrieved files, as is generally known. The circuit switched network1406includes various hardware and software for providing circuit switched communications. The circuit switched network1406may include, or may be, what is often referred to as a plain old telephone system (“POTS”). The functionality of a circuit switched network1406or other circuit-switched network are generally known and will not be described herein in detail.

The illustrated cellular network1402is shown in communication with the packet data network1404and a circuit switched network1406, though it should be appreciated that this is not necessarily the case. One or more Internet-capable devices1410, a PC, a laptop, a portable device, or another suitable device, can communicate with one or more cellular networks1402, and devices connected thereto, through the packet data network1404. It also should be appreciated that the Internet-capable device1410can communicate with the packet data network1404through the circuit switched network1406, the cellular network1402, and/or via other networks (not illustrated).

As illustrated, a communications device1412, for example, a telephone, facsimile machine, modem, computer, or the like, can be in communication with the circuit switched network1406, and therethrough to the packet data network1404and/or the cellular network1402. It should be appreciated that the communications device1412can be an Internet-capable device, and can be substantially similar to the Internet-capable device1410. In the specification, the network is used to refer broadly to any combination of the networks1402,1404,1406shown inFIG. 14.

Turning now toFIG. 15, a network topology1500for a data center cloud1502will be described, according to an illustrative embodiment. The illustrated network topology1500includes three layers: an application (“APP”) layer1504, a virtual network topology layer1506, and a physical network topology layer1508. The APP layer1504can include one or more application VNFs1510A-1510N, each of which can be divided to one or more sub-VNFs1512to be executed by one or more VMs1514.

The virtual network topology layer1506includes the VMs1514, one or more hypervisors1516, and one or more server modules (“blades”)1518. Each blade1518can support one hypervisor1516that, in turn, can manage one or more of the VMs1514. The blades1518provide computing capacity to support the VMs1514carrying the VNFs1512. The hypervisors1516provide resource management among the VMs1514supported thereby. A logical server cluster1520is created for resource allocation and reallocation purpose, which includes the blades1518in the same server host1522. Each server host1522includes one or more of the server clusters1520.

The physical network topology layer1508includes an Ethernet switch (“ESwitch”) group1524and a router group1526. The ESwitch group1524provides traffic switching function among the blades1518. The router group1526provides connectivity for traffic routing between the data center cloud1502and virtualized IP network(s)1528. The router group1526may or may not provide multiplexing functions, depending upon network design.

The virtual network topology1506is dynamic by nature, and as such, the VMs1514can be moved among the blades1518as needed. The physical network topology1508is more static, and as such, no dynamic resource allocation is involved in this layer. Through such a network topology configuration, the association among application VNFs1510, the VM1514supporting the application VNFs1510, and the blades1518that host the VM1514can be determined.

In the illustrated example, a first VNF is divided into two sub-VNFs, VNF 1-11512A and VNF 1-21512C, which is executed by VM 1-1-11514A and VM 1-N-11514C, respectively. The VM 1-1-11514A is hosted by the blade 1-11518A and managed by the hypervisor 1-11516A in the server cluster 11520of the server host1522. Traffic switching between the blade 1-11518A and the blade 1-N1518N is performed via ESwitch-11524A. Traffic communications between the ESwitch group1524and the virtualized IP network(s)1528are performed via the router group1526. In this example, the VM 1-1-11514A can be moved from the blade 1-11518A to the blade 1-N1518N for VM live migration if the blade 1-11518A is detected to have difficulty to support the VNF 1-11512A performance requirements and the blade 1-N1518N has sufficient capacity and is available to support the VNF 1-11512A performance requirements. The virtual network topology1506is dynamic by nature due to real-time resource allocation/reallocation capability of cloud SDN. The association of application, VM, and blade host in this example is the VNF 1-11512A is executed on the VM 1-1-11514A hosted by the blade 1-11518A in the server cluster 11520A.