Patent Description:
Telecommunication networks are currently being transformed from a system of network appliances operating on dedicated hardware to a system of virtualized network functions (VNFs) implemented on top of a cloud platform. The VNFs may be implemented in both dedicated clouds controlled by a communication service provider (CSP) and public clouds or multi-tenant clouds. Additionally, a single telecommunications network can span multiple cloud platforms controlled by different cloud providers.

The adoption of cloud technologies for telecommunication services provides a number of benefits, such as increased flexibility and agility, ability to rapidly scale services, and lower costs, but these benefits come at the cost of increased complexity. With the move towards VNFs, the underlying execution platform for network functionality may span multiple cloud platforms and providers, each with different characteristics, e.g., control-plane latency, scaling mechanisms, etc. This increase in complexity presents a challenge from an operations management standpoint. Performance may vary widely between cloud platforms. Additionally, each cloud platform will be subject to varying workloads and changing environments. The varying performance and conditions in each cloud platform makes management of the telecom network more cumbersome and less predictable.

As an example of the increased complexity, a cloud-based telecom network should be able to scale with demand so that the service requests can be handled. Additionally, it is important for CSPs to be able to accept new services within a given time frame, e.g., a deadline configured by the CSP. Currently, scaling a network deployed in a multi-cloud environment in order to meet an admission deadline or other Quality of Service (QoS) criteria is difficult because the characteristics of the underlying cloud platforms are not known. As demand increases, the CSP will request allocation of additional resources from the cloud providers to meet the increasing demand. Because the different cloud platforms will differ in the latency to allocate new resource, the CSP which will be faced with an unpredictable delay in accepting new service requests. In order to ensure that the admission deadline is met, the CSP may over allocate resources, which increases the costs of the CSP.

Accordingly, there is a need for improvements in management of telecommunications network implemented on cloud platforms.

<CIT>, provides a system and method for accelerated network service and/or network slice provisioning in response to customer requests or requirements.

The present invention provides an automation solution for telecommunications networks deployed in a multi-cloud environment. The automation solution autonomously controls the amount of pre-allocated cloud resources (e.g., CPU, storage, VNFs, etc.) within each underlying cloud platform with the goal of ensuring that the admission delay for accepting new flows into the system is below a desired admission deadline. Objects of the invention are a method, a system and a computer program as claimed in the appended independent claims. Preferred embodiments are covered by the appended dependent claims.

The present disclosure provides an automation solution for telecommunications networks deployed in a multi-cloud environment. From time to time, new flows will need to be implemented, for example, to meet increased demand or to provide a new service. Generally, a flow is a distributed connected service that spans over one or more cloud systems. The automation solution autonomously controls the minimum amount of pre-allocated cloud resources (e.g., CPU, storage, VNFs, etc.) within each underlying cloud platform with the goal of ensuring that the admission delay for accepting new flows into the system is below a desired admission deadline. It fulfills this goal by controlling the amount of pre-allocated resources that are standing by to handle new flows.

<FIG> schematically illustrates a telecommunications network <NUM> implemented in a multi-cloud environment. The telecommunications network <NUM> may, for example, comprise a mobile communication network operating according to the Fifth Generation (<NUM>) standard. The <NUM> network may, for example, comprise one or more network slices, where each network slice contains the core network functions of a <NUM> core (5GC) (e.g., Access and Mobility Management Function (AMF), Session Management Function (SMK, User Plane Function (UPF) etc.) tailored to a specific service provided by the network operator. In this example, the core network functions are implemented as virtual network functions (VNFs) in a cloud infrastructure comprising three cloud systems <NUM>, which can be operated by the same cloud provider or different cloud providers.

<FIG> shows three flows traversing the network, denoted as Flows A - C. A flow generally corresponds to a network slice that chains together a set of cloud services. In this example, Flow A uses services provided by Clouds I and II, Flow B uses services provided by Clouds I, II and III, and Flow C uses services provided by Clouds II and III.

The arrival of a new flow will potentially require the individual clouds to allocate more resources to implement a new network slice. The slowest cloud with the longest allocation time will determine how fast a new flow may be accepted. To serve the request, resources must be allocated across several clouds or several services within a single cloud. For example, the arrival of a new flow, i.e., Flow D in <FIG>, will require the allocation of new resources in all three clouds (I, II, III). The new flow can be accepted first when all requested resources are ready. However, the different cloud systems will differ in the latency to allocate new resource, which will give rise to unpredictable delay in accepting new flows. In addition, user requests must typically be granted within a specified deadline not to be deemed as failure.

One aspect of the claimed invention is to provide an "over-the-top" automation system <NUM> (<FIG>) which interacts automatically with the underlying cloud systems <NUM> to ensure a guaranteed admission delay or other Quality of Service (QoS) criteria. The automation system <NUM> is based on two data-driven system identification models: i) a model for control-plane latency, ii) a model for new flow requests. The model for control-plane latency includes the time it takes to scale the capacity within the cloud platform (e.g., add storage, CPU, boot new VNFs, etc.). The model for the new flow requests reflects how many and when new flows will request to join the system. Together, these two models allow an automation system <NUM> to determine on the amount resources to pre-allocate in order to ensure the desired admission delay. The model for control plane latency and new flow requests can be data driven. In other embodiments, the CSP may provide one or more configuration parameters that characterize the new flow requests. In such case, the configuration parameters provided by the CSP can be used as the model of new flow requests.

Generally, the amount of pre-allocated resources and/or the configuration of the pre-allocated resources needed to ensure the admission delay will depend on the control plane latency, also referred to herein as the scaling latency, and the arrival rate of new flows. The pre-allocated resources are on standby (i.e., activated but not currently in use) and are available to cover sudden bursts of new flows. High latency and high arrival rates for new flows mean that more resources need to be pre-allocated to guarantee the admission delay for new flows. Low latency and low arrival rates mean that fewer resources need to be pre-allocated. If a cloud system <NUM> is slow to add resources, then more resources will need to be pre-allocated in that cloud system <NUM>. At the other end of the spectrum, if the telecommunications network is hosted on really responsive execution platform (e.g., microservices funning on a virtual infrastructure), there will be a smaller delay and fewer resources will need to be pre-allocated. Similarly, a burst that contains a large number of new flows over a short period will require more resources on standby to ensure the admission delay. In contrast, a burst with a smaller number of flows over a longer time period will require fewer pre-allocated resources.

<FIG> schematically illustrates an automation system <NUM> according to one embodiment for the same telecommunication network shown in <FIG>. As previously described, the telecommunication network comprises one or more network slices implemented by VNFs spread across multiple cloud systems <NUM>. The automation system <NUM> communicates with the underlying cloud systems <NUM> and controls the amount of pre-allocated cloud resources (e.g., CPU, storage, VNFs, etc.) within each underlying cloud system <NUM> to ensure that the admission delay for accepting new flows by the telecommunications network is below a desired admission deadline.

As one example, the telecommunications network is assumed to comprise a total of M different network slice-types, each with its own unique characteristics (e.g., "low-latency, low bandwidth", "high-latency, high bandwidth", etc.). The total number of network slices hosted in the cloud systems <NUM> may be larger than M, since there may be multiple instances of each network slice type. There is also assumed to be a total number of N different cloud services needed to host these network slices. These services comprise VNFs, such as admission controllers, packet core nodes, IMU lookup, firewalls, etc. Additionally, Radio Access Network (RAN) functions can be virtualized and implemented using Open Radio Access Network (ORAN) and cloud RAN technologies.

To accept a new network slice or a change to an existing network slice, the CSP issues a request to the cloud system <NUM> to add or remove services needed by the network slice. This request states the characteristic of the network slice (i.e., the services it needs). The CSP must then wait for these services to be allocated and, once everything is allocated, the network slice will be accepted. If the pre-allocated resources available at the time of the request are not sufficient, the CSP will have to wait for the underlying cloud systems <NUM> to allocate resources for the network slice before it is available to support the new flow. In this context, the resources comprise specific service instances needed to host the network slice.

To ensure a high level of QoS, the automation system <NUM> is configured with an admission deadline D. The admission deadline D is the maximum time allowed for new network slices to be accepted following a request to join. Based on the admission deadline, the automation system <NUM> determines the "headroom" needed for each of the services provided by the cloud infrastructure. As used herein, the term "headroom" refers to the extra capacity on standby that is available for new requests. Stated another way, the headroom comprises the amount of standby resources need to ensure the admission deadline. The headroom, i.e. amount of standby resources, for service i is computed as follows: <MAT> where Li (t) is the scaling latency (e.g., the time it takes to add/remove resources) of service i, D is the admission deadline for accepting new network slices, M is the number of available services used to construct the flow, <MAT> is the maximum arrival rate of requests for network slice type j, and Ri,j(t) is an amount of each service i for each network slice type j , also referred to herein as flow type. The value of Ri,j(t) is zero if the network slice type or flow type does not use the service. Note that the term <MAT> Rj,i(t) is the aggregate amount of service i needed to implement the different network slice types (or flow types) using a particular service given the arrival rates <MAT> for all the network slice types.

To ensure that the scaling latency of the services is known and accurate, the automation system <NUM> uses a combination of data-modeling and measurement to continuously learn and update a model of the scaling latency for different services. In a similar way, the system can use data-modeling in order to continuously measure and model the amount of resource needed for different network slice types as well as the arrival rates of the different network slice types. By continuously updating these models, it is possible to ensure that the delay for accepting new network slices remains below the admission deadline.

<FIG> illustrates the main functional components of the automation system <NUM> according to one exemplary embodiment. The automation system <NUM> comprises a control plane latency estimator <NUM>, a resource allocator <NUM> and a sending unit <NUM>. The various components of the automation system <NUM> can be implemented by one or more microprocessors, application specific integrated circuits, hardware circuits, or a combination thereof. The control plane latency estimator <NUM> interacts with the cloud systems <NUM> to measure the control plane latency for each service provided by the cloud system <NUM>. The control plane latency estimator <NUM> provides the estimated control plane latency for each service i to the resource allocator <NUM>. In addition to the control plane latency for each service, the resource allocator <NUM> receives the admission deadline D, and the characteristics for each network slice type or flow type. The characteristics of a network slice or flow include the arrival rates <MAT> of new flows and the amount Ri,j(t) of each service required for each network slice type or flow. The arrival rate can be expressed as a number of new flows of the same type for some unit of time. These parameters can be provided by the CSP. Additionally, a statistical approach can be used to derive the arrival rate using a data driven approach. For example, instead of using a maximum arrival rate, a model of new flows can be used to compute the arrival rate corresponding to a predetermined probability, e.g. (<NUM>%). Based on the inputs, the resource allocator <NUM> generates models of the control plane latency and flow requests, which can be used to determine values to be entered into Equation <NUM>.

The control plane latency estimator <NUM> can measure the control plane latency during normal operation when new requests are sent to the cloud system <NUM>. When a new request is sent, the control plane latency estimator <NUM> can log the time of the request and measure the time delay until the new services are available. The time delay equals the difference between the time at which the requesting services are available and the time of the request. Additionally, the control plane latency estimator <NUM> can also spoof a request if measurement data is needed and measure the time delay until the new services are available. In this case, the new services can be removed after the measurements are made. <FIG> illustrates exemplary pseudocode for measuring the time delay.

The control plane latency estimator <NUM> can use a variety of well-known statistical methods to model the control plane latency. As one example, <FIG> is a representative histogram showing the distribution of control plane latency measurements. The dashed line is an estimated latency derived by applying a smoothing function to the control plane latency measurements. In this example, the control plane latency estimator can report the measurements corresponding to the <NUM> percentile in the control plane latency distribution (i.e., the right-most bin) to the resource allocator <NUM> as the control plane latency estimate. Similar techniques can be applied to modeling the arrival rate of requests for different network slice types and services.

<FIG> illustrates an exemplary method <NUM> implemented in an automation system <NUM> of managing network resources to guarantee flow admission in a network operating on a multi-vendor cloud system. For each of one or more services, the automation system <NUM> determines a control plane latency for the service (block <NUM>). The automation system <NUM> further computes an amount of standby resources needed to ensure a desired admission deadline based on the control plane latency and arrival rates for new flows using the service (block <NUM>). After determining the amount of standby resources needed for the service, the automation systems sends one of more requests to the cloud system to configure standby resources for the service based on the computed amount of standby resources (block <NUM>).

In some embodiments of the method <NUM>, determining a control plane latency for the service comprises sending a provisioning request to add the service, and measuring a time required to provision the resources for the service.

In some embodiments of the method <NUM>, measuring a time required to provision resources for the service comprises computing a delay between a time at which the provisioning request is sent and a time when the service is available for use.

Some embodiments of the method further comprise removing the service after measuring the delay if it is determined that the service is not required.

In some embodiments of the method <NUM>, computing an amount of standby resources needed to ensure a desired admission deadline based on the control plane latency and the one or more arrival rate comprises computing a scaling coefficient for the service based on the control plane latency and the admission deadline, computing an aggregate amount of the service needed for one or more flow types based on the one or more arrival rates, and computing the amount of standby resources needed based on the scaling coefficient and the aggregate amount of the service.

In some embodiments of the method <NUM>, computing a scaling coefficient for the service comprises computing the scaling coefficient based on a difference between the control plane latency and the admission deadline.

In some embodiments of the method <NUM>, sending one or more requests to configure standby resources comprises sending a request to add or remove standby resources.

<FIG> illustrates an embodiment of an automation system <NUM> according to an exemplary embodiment. The automation system <NUM> may serve as the automation system <NUM> shown in <FIG>. The automation system <NUM> comprises communication circuitry <NUM>, processing circuitry <NUM> and memory <NUM>.

The communication circuitry <NUM> comprises user interface circuitry <NUM> for communicating with the CSP or other user of the system and cloud interface circuitry for communicating with the cloud systems <NUM>. The user interface circuitry <NUM> enables the CSP or other user to interact with the automation system <NUM> to configure the automation system and provide data needed for operation, such as the arrival rates and resource amounts used to calculate the headroom for services. The cloud interface circuitry <NUM> implements an Application Programming Interface (API) or other protocols for communicating with the cloud systems <NUM> to manage resources and services provided by the cloud systems <NUM>.

The processing circuitry <NUM> controls the overall operation of the automation system <NUM> and implements the methods as herein described. The processing circuitry <NUM> may comprise one or more microprocessors, hardware, firmware, or a combination thereof. In one embodiment, the processing circuitry <NUM> is configured to perform the method shown <FIG>. In other embodiments, the processing circuitry <NUM> is configured to perform the method <NUM> shown in <FIG>.

Memory <NUM> comprises both volatile and non-volatile memory for storing computer program code and data needed by the processing circuit <NUM> for operation. Memory <NUM> may comprise any tangible, non-transitory computer-readable storage medium for storing data including electronic, magnetic, optical, electromagnetic, or semiconductor data storage. Memory <NUM> stores computer program <NUM> comprising executable instructions that configure the processing circuitry <NUM> to implement the method <NUM> according to <FIG> as described herein. A computer program <NUM> in this regard may comprise one or more code modules corresponding to the means or units described above. In general, computer program instructions and configuration information are stored in a non-volatile memory, such as a ROM, erasable programmable read only memory (EPROM) or flash memory. Temporary data generated during operation may be stored in a volatile memory, such as a random access memory (RAM). In some embodiments, computer program <NUM> for configuring the processing circuitry <NUM> as herein described may be stored in a removable memory, such as a portable compact disc, portable digital video disc, or other removable media. The computer program <NUM> may also be embodied in a carrier such as an electronic signal, optical signal, radio signal, or computer readable storage medium.

A computer program comprises instructions which, when executed on at least one processor of an apparatus, cause the apparatus to carry out any of the respective processing described above. A computer program in this regard may comprise one or more code modules corresponding to the means or units described above.

Claim 1:
A method (<NUM>) implemented in an automation system of managing network resources to guarantee flow admission in a network operating on a cloud system (<NUM>), the method comprising, for each of one or more services:
determining (<NUM>) a control plane latency for the service;
computing (<NUM>) an amount of standby resources needed to ensure a desired admission deadline based on the control plane latency and one or more arrival rates for new flows using the service, wherein computing (<NUM>) comprises:
computing a scaling coefficient for the service based on the control plane latency and the desired admission deadline,
computing an aggregate amount of the service needed for one or more the flow types based on the one or more arrival rates, and
computing the amount of standby resources needed based on the scaling coefficient and the aggregate amount of the service; and
sending (<NUM>) one or more requests to the cloud system to configure standby resources for the service based on the computed amount of standby resources.