Performance modeling for cloud applications

A method (1000) for performance modeling of a plurality of microservices (215) includes deploying the plurality of microservices (215) within a network (1260). The plurality of microservices (215) are communicatively coupled to generate at least one service chain (310) for providing at least one service. Based on a resource allocation configuration, an initial set of training data for the plurality of microservices within the network (1260) is determined. At least a portion of data is excluded from the initial set of training data to generate a subset of training data. A Quality of Service (QoS) behaviour model is generated based on the subset of the training data.

PRIORITY

This nonprovisional application is a U.S. National Stage Filing under 35 U.S.C. § 371 of International Patent Application Serial No. PCT/EP2019/076760 filed Oct. 2, 2019 and entitled “Performance Modeling for Cloud Applications”, which is hereby incorporated by reference in its entirety.

TECHNICAL FIELD

The present disclosure relates, in general, to wireless communications and, more particularly, systems and methods for performance modeling of a plurality of microservices.

BACKGROUND

One of the main promises of the next-generation 5G networks is to preserve consistent Quality of Experience (QoE) while decreasing end-to-end latency. Simultaneously, Mobile Network Operators (MNOs) need to lower their Capital and Operational Expenses (CapEx and OpEx). To achieve these goals, the telecommunication industry has adopted virtualization technology to modularize and decouple network functions from other network functions and from specialized proprietary hardware.

Concurrently, container-based cloud native technologies, as a more lightweight solution to virtualization, has developed with the goal to maintain at least the same level of performance and isolation as virtualization provides, if not better. To mitigate several performance and stability challenges exposed by such a transformation, several optimization solutions have been developed to both reactively and predicatively respond to different traffic demand situations leading big data-driven intelligent agents that can manage resource allocation, routing, and mobility of network functions.

However, there exist certain challenges. For example, the extra-granularity of cloud-native applications increases the uncertainty about initial resource allocation configuration of each microservice, which can also be named as resource footprint configuration. If kept uncharted, it imposes multitudinous scaling operations and dramatically decreases Meantime Between Failure (MTBF) of microservices.FIG.1illustrates example Service Level Agreement (SLA) violations and scaling operations when resource requirements are too small or too big. More specifically,FIG.1demonstrates SLA violations over time as a result of uncharted resource requirements of a set of microservices in a 5G Cloud-native Network Function (CNF).FIG.1also shows the corresponding scaling operations (both scale-ups and scale-downs) to heal the failures in the Network Function Cloudification (NFC) system.FIG.1shows a relatively high failure rate or a relatively low MTBF as a result of wrong or no minimum resource requirements.

As another example, another challenge with CNFs, like Virtual Network Functions (VNFs), is the extra flexibility of placing them in the network: where to deploy Network Functions (NFs) in the network such that the total cost of system deployment is minimized and also users agreed QoS level assured.

Moreover, a telecommunications network is a complex system to configure. Each network node has many configuration parameters whose optimal values can change depending on many things including its geographical location, traffic load, and traffic types. It is even more difficult to optimize configuration for a CNF where each CNF contains many separately configurable microservices. The sheer number of combinations of configuration parameters of the microservices in the system makes it difficult for a human to configure the system in an optimal way.

Perhaps with smaller systems, obtaining an optimal initial configuration may be unnecessary. With their granularity and faster boot-up times, cloud-native microservices are very fluid and scalable. Simple threshold-based auto-scalers like Kubernetes' Horizontal Pod Autoscaler (HPA) or Elasticdocker can guarantee fair elasticity for most applications. However, auto-scaling leads to other problems like maintaining agreed QoS constraints while preventing multitudinous scale-up and scale-down actions as traffic demands change. A thorough understanding each microservice's resource usage characteristics and the system's dynamics from the beginning has utmost importance for maintaining a consistent QoE.

The Kubernetes Vertical Pod Autoscaler (VPA) Recommender is an attempt to solve this. It can be used to gather metrics from Prometheus and make pod-level generic resource limit recommendations for watched pods based on each pod's historical resource usage data. However, this solution is too generic and does not consider the interaction between microservices within a pod, nor the traffic model supported by the network function.

There have also been a proposal to produce a polynomial model to relate resource utilization of a virtual machine (VM) to its throughput. This technique uses a measure of how sensitive the VM is to different resources and how it behaves under resource contention to decide where to place the VM with respect to other VMs in an optimal way. However, this solution does not consider minimum resource requirements or application-level metrics.

Static tools and testing are available to minimize the guessing involved in how configuration parameters interact within a network node and how their values are affected by the presence or absence and performance of different microservices or other network nodes. However, it takes a significant amount of time and resources. There is a need to avoid these costs by obtaining the optimal initial configuration for a network node quickly without extensive testing. The need is even greater with the additional complexity of deploying the multiple microservices of a CNF in the network.

In conclusion, some shortcomings of existing solutions include the fact that there is no automated solution/framework in the industry or academia that extracts the QoS behaviour model of microservices in the cloud to recommend an initial resource footprint configuration on a given network and keep correcting its errors afterwards. Similar solutions only work on a specific problem by providing a tailor-made approach. These solutions use a static set of metrics and do not consider user-level SLAs. The solutions also have limitations on the number of resource types such as, for example, only considering a static set of flavors in their evaluations (e.g. Amazon flavors). Additionally, existing solutions lack a solution to place and allocate resources to each microservice in a CNF. Finally, none of the previously proposed solutions provide a way to minimize the benchmarking need to create a machine learning model needed to recommend the initial resource allocation configuration.

SUMMARY

To address the foregoing problems with existing solutions, disclosed is systems and methods for model-based resource recommendation for cloud applications. For example, certain embodiments include comprehensive, automated and model-based solutions for modeling cloud-native applications such as NFC systems QoS metrics behaviour corresponding to its microservices resource allocation configuration. Such models may be used to recommend a configurable, predictive resource capacity and replication size for all microservices on a given substrate production network for each user.

According to certain embodiments, a method for performance modeling of microservices includes deploying the microservices within a network such that the microservices are communicatively coupled to generate at least one service chain for providing at least one service. Based on a resource allocation configuration, an initial set of training data is determined for the microservices within the network. At least a portion of data is excluded from the initial set of training data to generate a subset of training data. A QoS behaviour model is generated based on the subset of the training data.

According to certain embodiments, a network node for performance modeling of a plurality of microservices includes processing circuitry configured to deploy the plurality of microservices within a network. The plurality of microservices are communicatively coupled to generate at least one service chain for providing at least one service. Based on a resource allocation configuration, an initial set of training data is determined for the plurality of microservices within the network. At least a portion of data is excluded from the initial set of training data to generate a subset of training data. A QoS behaviour model is generated based on the subset of the training data.

According to certain embodiments, a computer program comprises instructions which when executed on a computer perform a method for performance modeling of microservices. The instructions may be executed to deploy the microservices within a network such that the microservices are communicatively coupled to generate at least one service chain for providing at least one service, determine an initial set of training data based on a resource allocation configuration, and exclude at least a portion of data from the initial set of training data to generate a subset of training data. A QoS behaviour model is generated based on the subset of the training data.

According to certain embodiments, a computer program product comprises a computer program comprising instructions which when executed on a computer perform a method for performance modeling of microservices. The instructions may be executed to deploy the microservices within a network such that the microservices are communicatively coupled to generate at least one service chain for providing at least one service, determine an initial set of training data based on a resource allocation configuration, and exclude at least a portion of data from the initial set of training data to generate a subset of training data. A QoS behaviour model is generated based on the subset of the training data.

According to certain embodiments, a non-transitory computer readable medium stores instructions which when executed by a computer performs a method for performance modeling of microservices. The instructions may be executed to deploy the microservices within a network such that the microservices are communicatively coupled to generate at least one service chain for providing at least one service, determine an initial set of training data based on a resource allocation configuration, and exclude at least a portion of data from the initial set of training data to generate a subset of training data. A QoS behaviour model is generated based on the subset of the training data.

Certain embodiments of the present disclosure may provide one or more technical advantages. For example, certain embodiments may provide an automated and configurable approach for modeling any given cloud-native system's dynamics by modeling the correlation between any QoS metrics with each and all microservice's resource types configuration (CPU size, memory size, etc.) such that the extracted regression models can predict the system's behaviour in a scaled environment with given contention ratios (even environments that had not been tested beforehand, but with the same or very similar hardware/OS/network configuration). This may dramatically reduce both OPEX and CAPEX such as by reducing the cost of dimensioning procedure, reducing SLA violations, reducing energy consumption and infrastructure cost, and increasing customer satisfaction as this gives a better estimate on infrastructure cost during initial SLA negotiation.

Another advantage may be that certain embodiments extract a behavioural model for microservices with any kind of QoS/performance behaviour, either linearly or non-linearly, or any other kind of distribution. Moreover, certain embodiments may provide a parametrized solution, so that one can change, add, delete or adjust modeling parameters to get the best results if needed. Moreover, certain embodiments work with any type of resource types, QoS metrics and SLAs.

Still another advantage may be that certain embodiments reduce the SLA violations and increase the MTBF by providing an optimally overprovisioned initial resource footprint configuration and replication size for each microservice based on the extracted QoS behaviour models and with regards to the given user's SLA constraints.

Yet another advantage may be that certain embodiments may work on either a lab/test environment (and extrapolate the results to predict QoS in production environment/firm and recommend resource with high accuracy), or the production environment/firm).

Still another advantage may be that, even though the disclosed techniques and solutions were applied to cloud-native applications, one can apply the techniques and solutions to any kind of cloud applications.

Yet another advantage may be that certain embodiments effectively reduce the cost of dimensioning through reduced testing and more accurate predictions of initial resource footprint and replication size configuration. Typically, it is too expensive to measure all traffic profiles and dimensional parameters. Therefore, traditionally it's been calculated based on the input parameters from the customer on-demand. However, certain embodiments disclosed herein make it possible to accurately estimate the QoS metrics beforehand and with lower cost. This may be essential when running microservices as the they may have many more dimensional parameters.

Another technical advantage may be that certain embodiments reduce the energy consumption by assigning an efficient capacities and replication sizes to each microservice.

Still another technical advantage may be that certain embodiments are compatible with user's custom configurations (such as custom affinity/anti-affinity rules).

Other advantages may be readily apparent to one having skill in the art. Certain embodiments may have none, some, or all of the recited advantages.

DETAILED DESCRIPTION

Particular embodiments are described inFIGS.1-15of the drawings, like numerals being used for like and corresponding parts of the various drawings.

Certain embodiments of the present disclosure may provide solutions enabling model-based resource recommendation for cloud applications by the network node. Certain embodiments may include functionality providing a model based approach for extracting resource capacity and replication size versus application's QoS behaviours. In particular, certain embodiments may perform only a minimal set of benchmarks (instead of trying out all possible combinations) to recommend an efficient initial resource configuration for each microservice. According to certain embodiments, the recommended resource configuration may satisfy all the required QoS constraints agreed on a given user's SLA. In a particular embodiment, the techniques may be implemented on a cloud-native 5G HSS-frontend prototype together with an MME simulator, which both are deployed on top of Kubernetes. Such a system may reach 87% accuracy in the initial modeling phase.

FIG.2illustrates a high-level sequence diagram100of the proposed solution, according to certain embodiments. Note that actors, steps and actions in this and other illustrations may be different in various contexts.

At step105, the sequence begins with the instantiation of the procedure by a system architect, a customer, a vendor, or another user of the system. As depicted, in particular embodiments, the user may instantiate the process by providing resource types, quality metrics, capacities to test, service chains, microservice(s) details, lab environment configurations, affinity/anti-affinity rules, and other parameters.

At step110, the modeling component of the NFV-Inspector prepares the lab environment and deploys the microservices in the lab environment. As part of the preparation process, the saturation points and minimal test points may be discovered.

At step115, the benchmarking procedures are performed. As described in more detail below, the objective is to select and benchmark only a small subset of configurations and predict the value of the rest using ML models.

At step120, the automated testing tool provides QoS measurements. The QoS measurements are used to generate a QoS behaviour model at step125. In a particular embodiment the QoS behaviour model includes a ML QoS behaviour model.

FIG.3illustrates an example organization of an NFC system200, according to certain embodiments. As shown inFIG.3, the NFC system is composed of a set of CNFs {CNFi}i=i|CNF|210in which consists of a number of microservices

{MSi,j}j=1❘"\[LeftBracketingBar]"MSi❘"\[RightBracketingBar]"
215. According to certain embodiments, each microservice MSi,j215may be load-balanced among a set of replication instances {msi,jl}l=1|msi,j|and placed on a substrate network, which is the underlying physical network. Worker nodes220that compose network denotes as M={mo}o=1|M|. Each worker node220can host one or several microservice215instances. Each mo∈M may be specified with its resource types capacities Crd(mo), where rd∈R is a resource types (CPU, memory, huge-page table, etc.). In the same way, each microservice instance msi,jl215capacity is denoted as Crd(msi,jl) with respect to the minimum and maximum allocatable capacity to each resource type denoted as Cminrd(MSi,j) and Cmaxrd(MSi,j) respectively. The total amount of assigned resource capacities of a microservice MSi,j215is denoted on each resource type rdas follows:

FIG.4illustrates a service chain flow300in an NFC system. The goal of these microservices215is to serve a series of service chains {SCk}k=1|SC|310a-z, as depicted inFIG.4. These service chains310a-zrepresent the connection and traffic flow between microservices215, which may be specified as a directed graph G(SCk)=(MSk, Ek), where MSkis a subset of all microservices215that have a role in providing service SCkand Ekis the subset of all edges belongs to the service chain SCk310a-z. Each service chain SCk310a-zhas a request entry point that receives requests from potential requesters via the Network Function Virtualization Infrastructure (NFVI). To provide a service Sk, each SCk310a-ztraverses an ordered set of microservices (MSi,j, . . . , MSi′,j′)215belonging to (CNFi, . . . , CNFi′). These microservices215are connected through one or more directed virtual links {ek,1(MSa,b, MSa′,b′), . . . , ek,n(MSc,d, MSc′,d′)}. Moreover, according to certain embodiments, the NFC system comes with a set of affinity/anti-affinity rules for the placement of the microservices215in an environment. For example, a pair of microservices215may need to be placed on the same worker node220if they have a high affinity relation. A pair of microservices215would have a high affinity, for example, if the pair communicate frequently with each other and exchange a large amount of data. On the other hand, some microservices215may need to be placed on separate worker nodes220if they have a high anti-affinity relation. This can happen, for example, when there exist two critical microservices215and there is a need to ensure the service availability in the case of failure of a physical node. These criteria are assumed and will be provided by the system admin.

Matrix U and U′ are created to hold all the affinity and anti-affinity rules respectively. Each matrix has dimension |MS|×|MS| where each row and column represents one of the microservices. An entry Uabtakes on a value of 1 if microservices a and b have a high affinity and must be placed at the same worker node. Additionally, an entry Uab′ takes on a value of 1 if microservice a and b have a high anti-affinity and must not be placed at the same worker node220.

To evaluate the performance of an NFC system, one should measure the key QoS metrics (serving throughput, latency, etc.). This set of QoS metrics may be defined as Q={qc}c=1|Q|. Assuming that an SLA agreement for a customer a on service chain SCk310a-zand QoS metric qc∈Q is in place, SLAαk(qc) may be defined such that it denotes the guaranteed value of QoS metric qcin user αSLA agreement on service chain SCk310a-z.

According to certain embodiments, CNFs QoS metrics behaviour modeling may be based on an algorithm that performs a minimal set of benchmarks and recommends a resource allocation and replica size configuration for each microservice215in a service chain315. In other words, for each SCk,215given a set of SLA requirements {SLAαk(qc)}qc∈Q, the final goal is to recommend an optimal amount of resource capacities for each microservice MSi,j215in a production environment based on a minimum number of experiments in a smaller-scale deployment and guaranty the agreed QoS constraints on all QoS metrics qc∈Q for a user α. It is assumed that the NFC system comes with a sophisticated benchmark tool that allows to define a test scenario based on a given service chain(s), arrival rate(s), duration(s) and burst setting(s).

FIG.5illustrates an example modeling process400, according to certain embodiments. Specifically, at a step410, an initial placement of microservices215is determined. At step420, the saturation points for a set of data are found. At step430, training data is generated. At step440, a model is fit based on a hypothesis. Each of these steps are described in more detail below.

For example, with regard to the initial placement of the microservices215at step410, it is recognized that before starting to perform benchmarks, the CNF's microservices are deployed on a lab environment. According to certain embodiments, in order to get accurate results, this deployment may follow one or more of the following criteria:Each microservice215may be placed on a completely isolated worker node220, such that other microservices215may not have negative performance effect over a microservice215(no noisy neighbour). If worker nodes220are VMs, they need to be explicitly pinned to CPU cores (using NUMA filters in case of OpenStack).The placement of microservices215may need to hold all affinity rules, according to certain embodiments.There should be minimum systemic performance bottlenecks in the deployment (no misconfigurations, scalability issues, etc.) such that the resource allocation configuration of microservices would be the only bottleneck for getting higher QoS.Except for resource capacities and scale, the lab environment infrastructure (equipment/hardware, operating system, bandwidth, etc.) must be as similar as possible to the production environment, to minimize prediction errors.

FIG.6illustrates an example initial placement of a number of microservices215a-don worker nodes220a-d, respectively, according to certain embodiments. Specifically, as depicted each of microservices215a-dare placed on a respective one of worker nodes220a-d. It is recognized, however, thatFIG.6is merely an example of an initial placement. Multiple microservices215may be placed on a single worker node220, in other embodiments.

As mentioned in previous sections, one of the main challenges with CNFs is the huge number of separately configurable microservices215as well as sheer number of combinations of configuration parameters which makes it very time consuming to benchmark all possible setting. If there is no systemic bottleneck in a deployment (based on the criteria discussed above) and QoS measures in a service chain315behave harmonically (as is typical) with the amount of resource capacities assigned to its microservices215, an algorithm is proposed to effectively select and benchmark only a small subset of configurations and predict the value of the rest using machine learning models. Because it is not necessary to try out all possible resource configurations with this methodology, both the dimensioning time as well as the number of benchmarks required is dramatically reduced for modeling the effect of resource configuration of microservices215on a service chain's QoS measurements.

The goal of certain embodiments is to model the effect of resource allocation configuration (e.g. CPU and memory capacities) of each microservice215in a service chain315on overall QoS metric values (e.g. throughput). But, before starting the modeling procedure, a set of data points is required to train the machine learning models, as discussed above with regard to step430ofFIG.5.

According to certain embodiments, such training data is gathered by selecting a single microservice215at a time and assigning a resource allocation configuration Tp=(tp(rd1), . . . , tp(rd|R|)) (e.g. a tp(cpu) can be measured as the milicpu, or for memory can be measured as bytes) to each of its resource types rd∈R (e.g., CPU, memory, etc.), while assigning maximum possible resources to other microservices in the service chain315. In this way and considering the placement of all microservices215are following all the criteria discussed above, it can be safely assumed that the only microservice215that can possibly become a bottleneck (before the QoS metrics start saturating) is the one under the test. With this method, QoS metric qc∈Q (e.g. serving throughput, latency, CPU utilizations, etc.) values Vk(qc) (e.g., throughput or latency) start saturating after a specific CPU and memory capacity threshold. Even in some cases, assigning more resources (e.g., CPU cores) may cause lower QoS (e.g., less serving throughput) due to possible contentions. If a microservice CPU or memory capacity has a small contribution on the service chain's final QoS, utilizing such benchmarking procedure may result in quick saturation of the QoS metric values. In contrast, if the microservice215plays a key role in providing the intended service, system's QoS metric values may start saturating on the edge of worker node's CPU capacity limits (or even never saturate). This means that if bigger worker nodes are used and more CPU or memory capacities are assigned to these key microservices215in the service chain315or if more replicas/instances of the microservice215are created, then other less-contributing microservices215may start saturating slower. Therefore, to form an accurate prediction model, all saturation points may be excluded from the test data. For the rest of the test space, only a random number of points are kept so as to both reduce the number of experiments (and as a result reduce the dimensioning time) as well as to avoid over-fitting of the models.

FIG.7illustrates a minimal benchmarking procedure, according to certain embodiments. At step610, the saturation point is determined. For example, a binary search may be performed to find the saturation point on one resource type (e.g., memory capacity). The same search may then be repeated for other resource types (e.g., CPU capacity). The process will result in finding the key saturation point. The saturation point corresponds to the point at which increasing resource capacities after that point will not have dramatic effect on the system's performance. As a result, QoS metrics starts saturating after that point.

FIG.8illustrates an example plot700for determining saturation points, according to certain embodiments. Specifically,FIG.8represents all the attempts to find the saturation point in a Lightweight Directory Access Protocol (LDAP) Server prototype, according to one particular example embodiment.

Returning toFIG.7, after finding this key saturation point, the saturated area associated with a saturation point is removed from the test space, at step620. For example, the test space may be divided into divisions and all the points in the saturated area may be excluded.

At step630, random tests are then performed on the non-saturated area. The random tests are performed with random resource capacities. For the rest of the points, a random set of points is selected in the test space and the number of test points is kept proportional to the size of each division to avoid over-fitting of the machine learning models in the next step.FIG.9illustrates an example plot800depicting the performance of random benchmarks in a non-saturated area, according to certain embodiments.

At step640, measured QoS values and corresponding capacities are used as input for ML models. More specifically, according to certain embodiments, regression models are used to form QoS behaviour models of an NFV system based on the training data gathered in step630, so that for a given a resource allocation configuration, the value of a given QoS metric may be predicted when a particular service chain is running. Once this data is collected, for a given microservice MSi,j, QoS metric qcand a service chain SCka separate regression model(qc) is trained to predict the value of qc, when resource allocation size of MSi,jis Tp.

FIG.10represents the entire data collection and modeling workflow, including hypotheses selection and training procedure. Specifically, at step910, the training data is obtained. According to a particular embodiment, to train each regression model(qc) we use the training input set

{Tp}
that was constructed using the process described above with regard toFIG.7along with corresponding targets

{Vi,jk(qc,Tp)}
the former of which represents a multi-dimensional matrix as each Tpitself is a set of resource allocation sizes for all resource types.

At step920, the hypotheses are trained. For example, to learn an approximation regression model with the objective of making accurate prediction of QoS metric value Vk(qc) for previously unseen template Tp, the disclosed method tries out different hypothesis functions hi(.) and, at step930, chooses the one that fits best with the training data such that the selected hypothesis function has the highest Goodness of Fit (GoF) as well as better cross-validation results. Naturally, each microservice215may behave differently, and therefore different hypotheses may need to be tested. Studies have shown that the throughput of around 85% of all type of applications can be modeled using a polynomial hypotheses. For instance, assuming there are two resource types r1and r2(e.g., CPU, memory, etc.) and a quality metric qc(e.g., serving throughput), and assuming all microservices follows a monotonically increasing performance model, such that by increasing the capacity of each resource type (up to a limit) QoS measurements will either increase or remain the same, the hypotheses {hi(Ctotalr1, Ctotalr2)}i+1|h|may be defined as (but are not limited to) as follows:
h1=λ1Ctotalr1+λ2Ctotalr2+λ3
h2=λ1Ctotalr1+λ2Ctotalr2+λ3Ctotalr1Ctotalr2+λ4
h3=λ1Ctotalr1+λ2Ctotalr2+λ3Ctotalr1Ctotalr2+λ4(Ctotalr1)2+λ5
h4=λ1Ctotalr1+λ2Ctotalr2+λ3Ctotalr1Ctotalr2+λ4(Ctotalr2)2+λ5
h5=λ1Ctotalr1+λ2Ctotalr2+λ3Ctotalr1Ctotalr2+λ4(Ctotalr1)2+λ5(Ctotalr2)2+λ6

After training all hypotheses {hi(Ctotalr1, Ctotalr2)}i=1|h|and based on their GoF values and cross validation results, a final hypothesiswill be selected at step930and a regression model(qc) is trained to estimate the value of Vk(qc) when the resource allocation configuration of MSi,jis equal to Tp.

FIG.11depicts another example method for performance modeling of a plurality of microservices. At step1002, the method begins when the plurality of microservices are deployed within a network. The plurality of microservices are communicatively coupled to generate at least one service chain for providing at least one service. At step1004, an initial set of training data is determined for the plurality of microservice based on a resource allocation configuration. At step1006, at least a portion of data is excluded from the initial set of training data to generate a subset of training data. At step1008, a QoS behaviour model is generated based on the subset of the training data.

In a particular embodiment, excluding the portion of data from the initial set of training data to generate the subset of training data includes: isolating resource allocation to each of the plurality of microservices; selecting one of the plurality of microservices; assigning a maximum respective resource allocation configuration to each of a plurality of resources associated with the microservices; determining a saturation point for the one of the plurality of microservices; and excluding a saturation area associated with the saturation point from the initial set of training data when generating the subset of training data.

In a particular embodiment, the saturation point includes a point when a quality of service associated with the one of the plurality of microservices starts saturating while increasing resources.

In a particular embodiment, the method further includes repeating the steps of isolating, selecting, assigning, determining, and excluding for a randomly selected subset of the plurality of microservices.

In a particular embodiment, the environment comprises a lab environment, which may include, as one example, a performance testing environment. The lab environment includes at least one infrastructure element. The at least one infrastructure element of the lab environment is selected to emulate a production environment to minimize prediction errors.

In a particular embodiment, the microservices may be isolated during placement, at a host level. If that is not possible, the microservices may be pinned to a CPU. For example, Kubernetes allows a docker process to be pinned to a particular set of cores. This may be done to avoid the nosy neighbor problem.

In a particular embodiment, a network topology of the lab environment emulates a network topology of the production environment.

In a particular embodiment, the at least one infrastructure element includes at least one of equipment, hardware, operating system, and bandwidth selected to emulate the production environment.

In a particular embodiment, deploying the plurality of microservices within the network includes using a network emulator.

In a particular embodiment, deploying the plurality of microservices within the network comprises determining at least one network node to host each of the plurality of microservices.

In a particular embodiment, a communication capacity of each of the plurality of microservices is considered when determining the at least one network node to host each of the plurality of microservices. In particular embodiments, the communication capacity may be the network bandwidth.

In a particular embodiment, the plurality of microservices are hosted on a plurality of network nodes. Each of the plurality of network nodes is elected to host at least one of the plurality of microservices based on at least one affinity rule or randomly.

In a particular embodiment, deploying the plurality of microservices within the network includes minimizing factors affecting QoS such that the resource allocation configuration is the only deterrent to getting a higher QoS value.

In a particular embodiment, selecting the QoS behaviour model based on the set of training data includes: testing a plurality of hypothesis functions for the plurality of microservices in the service chain; selecting one of a plurality of hypothesis functions that has a highest goodness of fit and cross-validation results to the set of training data; and training the QoS behaviour model to estimate at least one QoS metric. Though the steps described herein include training the QoS behaviour model to estimate at least one QoS metric, it is recognized that the techniques may be used to estimate any other kind of QoS metric, resource type, and/or hypothesis function.

In a particular embodiment, the method further includes using the QoS behaviour models to determine a value of the at least one QoS metric for the plurality of microservices in each service chain.

In a particular embodiment, the method further includes using the QoS behaviour models to determine an optimal amount of resource capacities of each of the plurality of microservices belonging to the service chain.

In a particular embodiment, the QoS behaviour model is a ML model.

FIG.12illustrates a schematic block diagram of a virtual apparatus1100in a network. The apparatus may be implemented in a wireless device or network node. Apparatus1100is operable to carry out the example method described with reference toFIG.11and possibly any other processes or methods disclosed herein. It is also to be understood that the method ofFIG.11is not necessarily carried out solely by apparatus1100. At least some operations of the method can be performed by one or more other entities.

According to certain embodiments, deploying module1110may perform certain of the deploying functions of the apparatus1100. For example, deploying module1110may deploy a plurality of microservices within a network. The plurality of microservices are communicatively coupled to generate at least one service chain for providing at least one service.

According to certain embodiments, determining module1120may perform certain of the determining functions of the apparatus1100. For example, determining module1120may determine an initial set of training data for the plurality of microservice based on a resource allocation configuration.

According to certain embodiments, excluding module1130may perform certain of the excluding functions of the apparatus1100. For example, excluding module1130may exclude at least a portion of data from the initial set of training data to generate a subset of training data.

According to certain embodiments, generating module1140may perform certain of the generating functions of the apparatus1100. For example, generating module1140may generate a QoS behaviour model based on the subset of the training data.

InFIG.14, network node1260includes processing circuitry1270, device readable medium1280, interface1290, auxiliary equipment1284, power source1286, power circuitry1287, and antenna1262. Although network node1260illustrated in the example wireless network ofFIG.14may represent a device that includes the illustrated combination of hardware components, other embodiments may comprise network nodes with different combinations of components. It is to be understood that a network node comprises any suitable combination of hardware and/or software needed to perform the tasks, features, functions and methods disclosed herein. Moreover, while the components of network node1260are depicted as single boxes located within a larger box, or nested within multiple boxes, in practice, a network node may comprise multiple different physical components that make up a single illustrated component (e.g., device readable medium1280may comprise multiple separate hard drives as well as multiple RAM modules).

Similarly, network node1260may be composed of multiple physically separate components (e.g., a NodeB component and a RNC component, or a BTS component and a BSC component, etc.), which may each have their own respective components. In certain scenarios in which network node1260comprises multiple separate components (e.g., BTS and BSC components), one or more of the separate components may be shared among several network nodes. For example, a single RNC may control multiple NodeB's. In such a scenario, each unique NodeB and RNC pair, may in some instances be considered a single separate network node. In some embodiments, network node1260may be configured to support multiple radio access technologies (RATs). In such embodiments, some components may be duplicated (e.g., separate device readable medium1280for the different RATs) and some components may be reused (e.g., the same antenna1262may be shared by the RATs). Network node1260may also include multiple sets of the various illustrated components for different wireless technologies integrated into network node1260, such as, for example, GSM, WCDMA, LTE, NR, WiFi, or Bluetooth wireless technologies. These wireless technologies may be integrated into the same or different chip or set of chips and other components within network node1260.

Processing circuitry1270may comprise a combination of one or more of a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application-specific integrated circuit, field programmable gate array, or any other suitable computing device, resource, or combination of hardware, software and/or encoded logic operable to provide, either alone or in conjunction with other network node1260components, such as device readable medium1280, network node1260functionality. For example, processing circuitry1270may execute instructions stored in device readable medium1280or in memory within processing circuitry1270. Such functionality may include providing any of the various wireless features, functions, or benefits discussed herein. In some embodiments, processing circuitry1270may include a system on a chip (SOC).

In some embodiments, processing circuitry1270may include one or more of radio frequency (RF) transceiver circuitry1272and baseband processing circuitry1274. In some embodiments, radio frequency (RF) transceiver circuitry1272and baseband processing circuitry1274may be on separate chips (or sets of chips), boards, or units, such as radio units and digital units. In alternative embodiments, part or all of RF transceiver circuitry1272and baseband processing circuitry1274may be on the same chip or set of chips, boards, or units.

In certain embodiments, some or all of the functionality described herein as being provided by a network node, base station, eNB or other such network device may be performed by processing circuitry1270executing instructions stored on device readable medium1280or memory within processing circuitry1270. In alternative embodiments, some or all of the functionality may be provided by processing circuitry1270without executing instructions stored on a separate or discrete device readable medium, such as in a hard-wired manner. In any of those embodiments, whether executing instructions stored on a device readable storage medium or not, processing circuitry1270can be configured to perform the described functionality. The benefits provided by such functionality are not limited to processing circuitry1270alone or to other components of network node1260but are enjoyed by network node1260as a whole, and/or by end users and the wireless network generally.

Interface1290is used in the wired or wireless communication of signalling and/or data between network node1260, network1206, and/or WIRELESS DEVICEs1210. As illustrated, interface1290comprises port(s)/terminal(s)1294to send and receive data, for example to and from network1206over a wired connection. Interface1290also includes radio front end circuitry1292that may be coupled to, or in certain embodiments a part of, antenna1262. Radio front end circuitry1292comprises filters1298and amplifiers1296. Radio front end circuitry1292may be connected to antenna1262and processing circuitry1270. Radio front end circuitry may be configured to condition signals communicated between antenna1262and processing circuitry1270. Radio front end circuitry1292may receive digital data that is to be sent out to other network nodes or WIRELESS DEVICEs via a wireless connection. Radio front end circuitry1292may convert the digital data into a radio signal having the appropriate channel and bandwidth parameters using a combination of filters1298and/or amplifiers1296. The radio signal may then be transmitted via antenna1262. Similarly, when receiving data, antenna1262may collect radio signals which are then converted into digital data by radio front end circuitry1292. The digital data may be passed to processing circuitry1270. In other embodiments, the interface may comprise different components and/or different combinations of components.

In certain alternative embodiments, network node1260may not include separate radio front end circuitry1292, instead, processing circuitry1270may comprise radio front end circuitry and may be connected to antenna1262without separate radio front end circuitry1292. Similarly, in some embodiments, all or some of RF transceiver circuitry1272may be considered a part of interface1290. In still other embodiments, interface1290may include one or more ports or terminals1294, radio front end circuitry1292, and RF transceiver circuitry1272, as part of a radio unit (not shown), and interface1290may communicate with baseband processing circuitry1274, which is part of a digital unit (not shown).

Antenna1262may include one or more antennas, or antenna arrays, configured to send and/or receive wireless signals. Antenna1262may be coupled to radio front end circuitry1290and may be any type of antenna capable of transmitting and receiving data and/or signals wirelessly. In some embodiments, antenna1262may comprise one or more omni-directional, sector or panel antennas operable to transmit/receive radio signals between, for example, 2 GHz and 66 GHz. An omni-directional antenna may be used to transmit/receive radio signals in any direction, a sector antenna may be used to transmit/receive radio signals from devices within a particular area, and a panel antenna may be a line of sight antenna used to transmit/receive radio signals in a relatively straight line. In some instances, the use of more than one antenna may be referred to as MIMO. In certain embodiments, antenna1262may be separate from network node1260and may be connectable to network node1260through an interface or port.

Antenna1262, interface1290, and/or processing circuitry1270may be configured to perform any receiving operations and/or certain obtaining operations described herein as being performed by a network node. Any information, data and/or signals may be received from a wireless device, another network node and/or any other network equipment. Similarly, antenna1262, interface1290, and/or processing circuitry1270may be configured to perform any transmitting operations described herein as being performed by a network node. Any information, data and/or signals may be transmitted to a wireless device, another network node and/or any other network equipment.

Power circuitry1287may comprise, or be coupled to, power management circuitry and is configured to supply the components of network node1260with power for performing the functionality described herein. Power circuitry1287may receive power from power source1286. Power source1286and/or power circuitry1287may be configured to provide power to the various components of network node1260in a form suitable for the respective components (e.g., at a voltage and current level needed for each respective component). Power source1286may either be included in, or external to, power circuitry1287and/or network node1260. For example, network node1260may be connectable to an external power source (e.g., an electricity outlet) via an input circuitry or interface such as an electrical cable, whereby the external power source supplies power to power circuitry1287. As a further example, power source1286may comprise a source of power in the form of a battery or battery pack which is connected to, or integrated in, power circuitry1287. The battery may provide backup power should the external power source fail. Other types of power sources, such as photovoltaic devices, may also be used.

Alternative embodiments of network node1260may include additional components beyond those shown inFIG.14that may be responsible for providing certain aspects of the network node's functionality, including any of the functionality described herein and/or any functionality necessary to support the subject matter described herein. For example, network node1260may include user interface equipment to allow input of information into network node1260and to allow output of information from network node1260. This may allow a user to perform diagnostic, maintenance, repair, and other administrative functions for network node1260.

The functions may be implemented by one or more applications1420(which may alternatively be called software instances, virtual appliances, network functions, virtual nodes, virtual network functions, etc.) operative to implement some of the features, functions, and/or benefits of some of the embodiments disclosed herein. Applications1420are run in virtualization environment1400which provides hardware1430comprising processing circuitry1460and memory1490. Memory1490contains instructions1495executable by processing circuitry1460whereby application1420is operative to provide one or more of the features, benefits, and/or functions disclosed herein.

Virtualization environment1400, comprises general-purpose or special-purpose network hardware devices1430comprising a set of one or more processors or processing circuitry1460, which may be commercial off-the-shelf (COTS) processors, dedicated Application Specific Integrated Circuits (ASICs), or any other type of processing circuitry including digital or analog hardware components or special purpose processors. Each hardware device may comprise memory1490-1which may be non-persistent memory for temporarily storing instructions1495or software executed by processing circuitry1460. Each hardware device may comprise one or more network interface controllers (NICs)1470, also known as network interface cards, which include physical network interface1480. Each hardware device may also include non-transitory, persistent, machine-readable storage media1490-2having stored therein software1495and/or instructions executable by processing circuitry1460. Software1495may include any type of software including software for instantiating one or more virtualization layers1450(also referred to as hypervisors), software to execute virtual machines1440as well as software allowing it to execute functions, features and/or benefits described in relation with some embodiments described herein.

Virtual machines1440, comprise virtual processing, virtual memory, virtual networking or interface and virtual storage, and may be run by a corresponding virtualization layer1450or hypervisor. Different embodiments of the instance of virtual appliance1420may be implemented on one or more of virtual machines1440, and the implementations may be made in different ways.

During operation, processing circuitry1460executes software1495to instantiate the hypervisor or virtualization layer1450, which may sometimes be referred to as a virtual machine monitor (VMM). Virtualization layer1450may present a virtual operating platform that appears like networking hardware to virtual machine1440.

As shown inFIG.15, hardware1430may be a standalone network node with generic or specific components. Hardware1430may comprise antenna14225and may implement some functions via virtualization. Alternatively, hardware1430may be part of a larger cluster of hardware (e.g. such as in a data center or customer premise equipment (CPE)) where many hardware nodes work together and are managed via management and orchestration (MANO)14100, which, among others, oversees lifecycle management of applications1420.

In the context of NFV, virtual machine1440may be a software implementation of a physical machine that runs programs as if they were executing on a physical, non-virtualized machine. Each of virtual machines1440, and that part of hardware1430that executes that virtual machine, be it hardware dedicated to that virtual machine and/or hardware shared by that virtual machine with others of the virtual machines1440, forms a separate virtual network elements (VNE).

Still in the context of NFV, Virtual Network Function (VNF) is responsible for handling specific network functions that run in one or more virtual machines1440on top of hardware networking infrastructure1430and corresponds to application1420inFIG.15.

In some embodiments, one or more radio units14200that each include one or more transmitters14220and one or more receivers14210may be coupled to one or more antennas14225. Radio units14200may communicate directly with hardware nodes1430via one or more appropriate network interfaces and may be used in combination with the virtual components to provide a virtual node with radio capabilities, such as a radio access node or a base station.

In some embodiments, some signaling can be affected with the use of control system14230which may alternatively be used for communication between the hardware nodes1430and radio units14200.

Abbreviations used in the preceding description include:NFC Network Function CloudificationCNF Cloud-native Network FunctionNFCI Network Function Cloudification InfrastructureNFV Network Function VirtualizationSLA Service Level Agreement: e.g. max arrival rate and max latencyCPU Central Processing UnitQoS Quality of Service: e.g. latency, CPU usage level, memory usage levelVNF Virtual Network FunctionVPA Vertical Pod AutoscalerQoE Quality of ExperienceSON Self-Organizing NetworksCapEx Capital ExpendituresOpEx Operational ExpendituresMNO Mobile Network OperatorHPA Horizontal Pod AutoscalerFFD First Fit DecreasingLDAP Lightweight Directory Access Protocol