Predictive scaling of application based on traffic at another application

Some embodiments provide a method that identifies a first number of requests received at a first application. Based on the first number of requests received at the first application, the method determines that a second application that processes requests after processing by the first application requires additional resources to handle a second number of requests that will be received at the second application. The method increases the amount of resources available to the second application prior to the second application receiving the second number of requests.

BACKGROUND

The increase in traffic diversity and accelerated capacity demand in mobile networks have pushed design of innovative architectural solutions and cost-effective paradigms for 5G evolution. Network Functions Virtualization (NFV) is an emerging trend in networking that involves migration of network functions (NFs) into virtualized environments, which leads to reduced capital investment. Traditionally, NFs are embedded on dedicated hardware devices (middleboxes or network appliances), but service providers and operators decouple NFs from their underlying hardware and run them on commodity servers. This has given birth to NFV technology that converts NFs into virtualized network functions (VNFs) hosted in virtual machines or containers. Network policies often require these VNFs to be stitched together as service chains to deliver various services or network functionality. These service chains define a sequence of services (network functions) through which traffic is steered.

Microservices allow parts of applications (or different services in a service chain) to react independently to user input. Kubernetes is considered as the most popular microservices orchestration. Within Kubernetes, auto-scaling is a mechanism by which applications can be scaled in or scaled out based on triggers. These triggers are typically based on observing an individual service and scaling that service as necessary. However, these reactive measures can cause issues for latency-sensitive workloads such as those implemented in a 5G service chain. As such, better techniques for auto-scaling such workloads would be useful.

BRIEF SUMMARY

Some embodiments provide a method for pre-emptively scaling resources allocated to one application based on identifying an amount of traffic received at another, related application. The method identifies a first number of requests received at a first application and, based on this first number of requests, determines that a second application that processes at least a subset of the requests after processing by the first application requires additional resources to handle a second number of requests that will be received at the second application. The method increases the number of resources available to the second application prior to the second application receiving this second number of requests, in order to avoid processing delays and/or dropped requests at the second application.

In some embodiments, the first and second applications are services in a service chain (e.g., for a 5G or other telecommunications network) that includes at least two services applied to the requests (e.g., audio and/or video calls). For each respective service in the service chain, the method uses a respective scaling factor that estimates a percentage of the requests received at the first service that will subsequently be received at the respective service in order to deploy additional resources to the respective service.

The services of the service chain, in some embodiments, are implemented as virtualized network functions. For instance, some embodiments deploy each service as one or more Pods in a Kubernetes cluster for the service chain. Each Pod is allocated a particular amount of resources (e.g., memory and processing capability) that enable the Pod to perform its respective service on a particular number of requests in a given time period (e.g., requests/second). In this environment, a front-end load balancer is configured to receive the requests and measure the number of requests received in real-time or near-real-time. Specifically, a data plane component of the load balancer receives and load balances the traffic at least among instances of the first service in the service chain in addition to providing information about the processed traffic to a control plane component of the load balancer. The control plane component uses this traffic information to measure the number of requests and provide that information to a scaler module that also operates (e.g., as a Pod or set of Pods) in the Kubernetes cluster.

The scaler module, in some embodiments, (i) computes the scaling factors for each service in the service chain and (ii) handles auto-scaling the services based on these scaling factors, traffic measurements from the load balancer (e.g., a number of requests forwarded by the load balancer to the first service), and data indicating the processing capabilities of each Pod for each service (enabling a judgment as to when the number of Pods should be increased or decreased for a given service). The scaler module computes the scaling factors either once at initial setup of the cluster or on a regular basis (e.g., depending on whether the inputs to the scaling factors use predefined or real-world data).

To compute the scaling factors, the scaler module of some embodiments generates a graph (e.g., a directed acyclic graph) of the service chain. In this graph, each service is represented as a node and each direct path from one service to another is represented as an edge. Each edge from a first node to a second node has an associated coefficient that specifies an estimate of the percentage of requests received at the service represented by the first node that are forwarded to the service represented by the second node (as opposed to being dropped, blocked, or forwarded to a different service). These coefficients may be specified by a user (e.g., a network administrator) or based on real-world measurement of the number of requests received at each of the services (e.g., over a given time period).

For each service, the scaler module uses the graph to identify each path through the service chain from the first service in the service chain to that service. For each such path, the scaler module multiplies each coefficient along the path in order to compute a factor for the path. The scaling factor for a given service is then the sum of the computed factors for each of the paths from the first service to that service, representing the estimated percentage of the requests received by the first service that will need to be processed by that service (the scaling factor for the first service will be 1). Other embodiments use a different equivalent computation that performs the component calculations in a different order in order to reduce the number of multiplications.

In real-time, as the load balancer provides measurements to the scaler module, the scaler module determines whether each of the services needs to be scaled (e.g., whether additional Pods should be instantiated for each service). Specifically, for one or more metrics (e.g., total requests, requests per second, latency (which is correlated with the rate of incoming traffic), etc.), the capacity of each Pod is specified for each service. A current value for each metric (based on the metrics from the load balancer and the scaling factor for the service) is divided by the Pod capacity for a given service to determine the number of Pods that will be required for the service. If the actual number of Pods is less than the required number of Pods, then the scaler module manages the deployment of additional Pods for the service. In this manner, if a large increase in traffic is detected at the load balancer, all of the services can be scaled up to meet this demand prior to the receipt of all of those requests at the services.

DETAILED DESCRIPTION

Some embodiments provide a method for pre-emptively scaling resources allocated to one application based on identifying an amount of traffic received at another, related application. The method identifies a first number of requests received at a first application and, based on this first number of requests, determines that a second application (which processes at least a subset of the requests after processing by the first application) requires additional resources to handle a second number of requests that will be received at the second application. The method increases the number of resources available to the second application prior to the second application receiving this second number of requests, in order to avoid processing delays and/or dropped requests at the second application.

In some embodiments, the first and second applications are services in a service chain (e.g., for a 5G or other telecommunications network) that includes at least two services applied to the requests (e.g., audio and/or video calls). For each respective service in the service chain, the method uses a respective scaling factor that estimates a percentage of the requests received at the first service that will subsequently be received at the respective service in order to deploy additional resources to the respective service.

The services of the service chain, in some embodiments, are implemented as virtualized network functions. For instance, some embodiments deploy each service as one or more Pods in a Kubernetes cluster for the service chain.FIG.1conceptually illustrates the architecture of such a service chain deployment100of some embodiments. As shown, the deployment100includes a Kubernetes cluster105as well as a front-end load balancer. The Kubernetes cluster105includes an ingress controller120for the load balancer, a Kubernetes ingress object125, a scaler module130, and the service chain135(which, as described below, includes multiple services).

The front-end load balancer, in some embodiments, includes both a data plane110and a controller115. In some embodiments, front-end load balancers such as Avi Vantage can be configured to define virtual services as a front-end for Kubernetes-based applications, generally via ingress controllers. Each virtual service maps to back-end server pools, serviced by application pods in the Kubernetes cluster. In some embodiments, these virtual services are the ingress point for incoming traffic when used at the edge of the cluster, as in the deployment100. That is, all traffic sent to the cluster passes initially through the front-end load balancer. In the example of a 5G or other telecommunication network, this traffic may include audio and/or video calls, potentially in addition to other types of traffic.

In some embodiments, an initial data message or set of data messages for a call (which can be referred to as a request) is sent to the service chain via the front-end load balancer (e.g., for the service chain to perform authentication and other tasks for the call), while subsequent traffic (carrying audio and/or video data) does not need to be processed through the service chain. In other embodiments, all traffic for the network passes through the front-end load balancer and service chain.

In some embodiments, the load balancer data plane110receives the incoming requests and load balances this ingressing traffic (possibly in addition to providing additional services, such as web application firewall). The load balancer data plane110may be implemented by a single appliance, a centralized cluster of appliances or virtualized data compute nodes (e.g., bare metal computers, virtual machines, containers, etc.), or a distributed set of appliances or virtualized data compute nodes. In some embodiments, the load balancer data plane110may load balance traffic between different Pods implementing the first service in the service chain and then forward the traffic to the selected Pods. In other embodiments, the load balancer data plane110performs additional service chaining features, such as defining a path through the service chain for an incoming data message (e.g., by selecting Pods for each of the services and embedding this selection in a header of that data message). In addition, while forwarding the incoming requests to the service chain135in the cluster105, the load balancer data plane110gathers information about the incoming requests and provides this traffic information to the load balancer controller115.

The load balancer controller115is a centralized control plane that manages one or more instances of the load balancer data plane110. The load balancer controller115defines traffic rules (e.g., based on administrator input) and configures the data plane110to enforce these traffic rules. In addition, the controller115gathers various traffic metrics from the data plane110(and aggregates these metrics in the case of a distributed data plane). The controller115also makes these aggregated metrics accessible (e.g., to an administrator, the scaler module130, etc.). In some embodiments, the metrics are accessible (to administrators, other modules, etc.) via application programming interfaces (APIs) such as representational state transfer (REST) APIs.

The ingress controller120for the load balancer, in some embodiments, handles conversion of data between the Kubernetes cluster and the front-end load balancer. In some embodiments, the ingress controller120is implemented on one or more Pods in the Kubernetes cluster105. This ingress controller120listens to a Kubernetes API server and translates Kubernetes data (e.g., the ingress object125, service objects, etc.) into the data model used by the front-end load balancer. The ingress controller120communicates the translated information to the load balancer controller115via API calls in some embodiments to automate the implementation of this configuration by the load balancer data plane110.

The ingress object125is a Kubernetes object that defines external access to the Kubernetes cluster105. The ingress object125exposes routes to services within the cluster (e.g., the first service in the service chain135) and can define how load balancing should be performed. As noted, the ingress controller120is responsible for translating this ingress object into configuration data to provide to the front-end load balancer controller115.

The scaler module130, in some embodiments, also operates on one or more Pods within the Kubernetes cluster105. The scaler module130is responsible for (i) computing the scaling factors for each service in the service chain135and (ii) initiating pre-emptive auto-scaling of the services based on these computed scaling factors, traffic measurements from the front-end load balancer controller115, and data indicating the processing capabilities of each service. The scaling factors for each respective service, as mentioned, estimates the percentage of the traffic received at the front-end load balancer (and thus received at the first service in the service chain) that will be subsequently received at the respective service. Depending on whether input coefficients to the scaling factor computation use user-defined or real-world observation data, the scaling factors can either be computed once (at initial setup of the cluster) or on a regular basis. The pre-emptive auto-scaling decisions output by the scaler module130specify, in some embodiments, when the number of Pods should be increased or decreased for a given service in the service chain. The operation of the scaler module130is described in additional detail below by reference toFIG.2.

The service chain135is a set of services deployed in a specific topology. In the Kubernetes context of some embodiments, each of these services is implemented as a “micro-service” and is deployed as a set of one or more Pods. However, it should be understood that in other embodiments the service chain may be implemented as a set of virtual machines (VMs) or other data compute nodes (DCNs) rather than as Pods in a Kubernetes cluster. In this case, a front-end load balancer can still be configured to measure incoming traffic and provide this data to a scaler module (executing, e.g., on a different VM) that performs similar auto-scaling operations outside of the Kubernetes context.

In this example, the service chain135includes three services140-150. The first service140receives traffic directly from the load balancer data plane110(potentially via Kubernetes ingress modules) and sends portions of this traffic (after performing processing) to both the second service145and the third service150. The second service145also sends a portion of its traffic (after it has performed its processing) to the third service150. The first service140is implemented using two Pods, the second service145is implemented using three Pods, and the third service150is implemented using a single Pod. Each Pod is allocated a particular amount of resources (e.g., memory and processing capability) that enable the Pod to perform its respective service on a particular number of requests in a given time period (e.g., requests/second). This per-Pod capacity can vary from one service to the next based on the physical resources allocated to each Pod for the service as well as the resources required to process an individual request by the service. In various embodiments, the services can include firewalls, forwarding elements (e.g., routers and/or switches), load balancers, VPN edges, intrusion detection and prevention services, logging functions, network address translation (NAT) functions, telecommunications network specific gateway functions, and other services, depending on the needs of the network.

FIG.2conceptually illustrates a more detailed view of the architecture of the scaler200of some embodiments (e.g., the scaler module130). As shown, the scaler200includes a modeler205, a metrics receiver210, and an auto-scaling and deployment module215. The modeler205receives the service chain topology as well as direct path coefficients and uses this information to (i) define a graph representing the service chain and (ii) compute the scaling factors for each of the services in the service chain (i.e., the scaling factors for estimating the percentage of traffic received at the first service that will be subsequently received at the other services). The modeler stores (e.g., in memory) the graph and scaling factors220for use by the auto-scaling and deployment module215.

In some embodiments, the modeler205receives the service chain topology information from the services themselves or from other Kubernetes constructs. This topology indicates the direct paths between services in the service chain. The direct path coefficients specify, for each direct path in the service chain topology from a first service to a second service, the portion of traffic received at the first service that is forwarded on to the second service. For the example service chain135shown inFIG.1, the paths from the first service140to the second service145, from the first service140to the third service150, and from the second service145to the third service150each have their own associated direct path coefficients. In different embodiments, the direct path coefficients may be administrator-specified or be based on recent observation of the service chain traffic. In the latter case, traffic information from the services in the service chain is provided to the modeler205, which regularly determines the ratios of traffic forwarded from one service to the next. The operations of the modeler205to define the service chain graph and compute the scaling factors are described further below by reference toFIG.3.

The metrics receiver210receives traffic metrics, including those indicating the amount of requests received at the first service in the service chain. In some embodiments, the metrics receiver210receives API schema information for the front-end load balancer from the ingress controller and uses this API information to retrieve the traffic metrics from the load balancer controller via API calls. The specific metrics received can include a total number of requests, requests per unit time (e.g., requests per second or millisecond), etc. As these metrics are retrieved, the metrics receiver210provides the metrics to the auto-scaling and deployment module215.

The auto-scaling and deployment module215uses the scaling factors computed by the modeler205to determine, in real-time, whether any of the services in the service chain need to be scaled (e.g., either instantiation of additional Pods or removal of Pods) based on the traffic metrics. The capacity of each Pod is specified for each service (the capacity can vary between services) for one or more metrics (e.g., requests per unit time) and provided to the auto-scaling and deployment module215(e.g., as an administrator-provided variable or based on observation). The current value for this metric (as received from the load balancer controller and multiplied by the scaling factor for a given service) is divided by the Pod capacity for the service to determine the number of Pods that will be required for the service. If the actual number of Pods is less than the required number of Pods, then the auto-scaling and deployment module215manages the deployment of additional Pods for the service. In this manner, if a large increase in traffic is detected at the load balancer, all of the services can be scaled up to meet this demand prior to the receipt of all of those requests at the services. On the other hand, if the actual number of Pods deployed is greater than the required amount for the service, the auto-scaling and deployment module215manages the deletion of one or more Pods for the service. In different embodiments, the auto-scaling and deployment module215either handles the deployment/deletion operations directly or provides the necessary instructions to a Kubernetes control plane module that handles these operations for the cluster. The operations of the auto-scaling and deployment module215to predictively auto-scale the services of a service chain will be described in detail below by reference toFIG.7.

FIG.3conceptually illustrates a process300of some embodiments for computing scaling factors for a set of services in a service chain. In some embodiments, the process300is performed by a scaler module (e.g., the modeler205of the scaler module200shown inFIG.2). In different embodiments, this process300(or a similar process) may be performed once at initial configuration of the service chain or at regular intervals (if the direct path coefficients change over time). The process300will be described by reference toFIGS.4and5, which illustrate an example of the calculation of scaling factors for a service chain.

As shown, the process300begins by receiving (at305) a service chain topology and a set of direct path coefficients. The service chain topology indicates which services forward traffic directly to other services (i.e., indicates direct paths between services in the service chain). As noted above, the service chain topology information can be received from the services themselves or from other Kubernetes constructs. The direct path coefficients specify, for each direct path in the service chain topology from a first service to a second service, the portion of traffic received at the first service that is forwarded on to the second service. In different embodiments, the direct path coefficients may be administrator-specified or be based on recent observation of the service chain traffic. In the latter case, traffic information from the services in the service chain is provided to the scaler, which regularly determines the ratios of traffic forwarded from one service to the next (e.g., on an hourly basis based on the past hour of traffic).

The service chain topology, in some embodiments, is user-specified information that defines connections between services. Specifically, in the Kubernetes context, the service chain topology (referred to as NetworkServiceTopology) is a cluster-scoped construct capable of chaining services from different namespaces together in the cluster, thereby allowing service administrators to operate in their own namespaces while handing the job of service chaining to the infrastructure administrator. The following provides an example declaration for a connection between a first service (serviceA) in red namespace and a second service (serviceB) in blue namespace, with a direct path coefficient of 0.7:

Next, the process300defines (at310) a graph for the service chain. Specifically, some embodiments define a directed acyclic graph (DAG) based on the service chain topology (e.g., based on each user-specified connection). In this graph, each service is represented as a node and each direct path from one service to another is represented as an edge. Each edge from a first node to a second node has an associated coefficient (i.e., the direct path coefficient for the connection represented by that edge) that specifies an estimate of the percentage of requests received at the service represented by the first node that are forwarded to the service represented by the second node (as opposed to being dropped, blocked, or forwarded to a different service).

FIG.4conceptually illustrates an example of a DAG400for a service chain that includes five services. Each service (A-E) is represented by a node in the DAG400, with each of the edges having an associated direct path coefficient. As shown, service A is expected to forward 70% of its traffic to service B, which is expected to forward 50% of its traffic to service C and 80% of its traffic to service E (meaning that service B is expected to forward at least some of its traffic to both services). Service E is expected, in turn, to forward 50% of its traffic to service C, which is expected to forward only 30% of its traffic to service D.

With the graph defined, the process generates scaling factors for each of the services in the service chain. Different embodiments use different specific calculations to compute these scaling factors, though they reduce to the same computation. For instance, some embodiments traverse through the graph starting from the beginning of the service chain and compute scaling factors for nodes that build on the computations for previous nodes in the graph. Other embodiments, as in the process300, compute the scaling factor for each service separately.

As shown, the process300selects (at315) a service in the service chain. In some embodiments, the process300begins with the first node in the directed graph and then proceeds to select nodes using a breadth-first traversal of the graph. Other embodiments select the nodes randomly. However, embodiments that compute the scaling factors for later services in the chain by building on previous computations cannot use a random selection.

The process300uses (at320) the graph to identify paths from the first service in the service chain to the selected service in the service chain. Assuming a single ingress point for the service chain, when the first service is selected, there is no path discovery required (and no scaling factor computation needed, as the scaling factor is always equal to 1). In the example graph400shown inFIG.4, Service B has a single path (from service A). Service C, on the other hand has two paths (one from Service A to Service B to Service C and another from Service A to Service B to Service E to Service C). Service E only has a single path (Service A to Service B to Service E), while Service D also has two paths (one from Service A to Service B to Service C to Service D and another from Service A to Service B to Service E to Service C to Service D).

It should be noted that the example graph shown inFIG.4includes a single ingress node (Service A) and a single egress node (Service D, which does not forward traffic to any other service). However, other service chains may have other structures. In the case of multiple ingress services, the process300(or other processes) can be expanded to accommodate these multiple entry points to the graph. The scaling factors for the ingress services will depend on whether the front-end load balancer provides traffic metrics indicating the number of requests forwarded to each different ingress service (in which case the scaling factors are all equal to 1) or the total number of services provided to the service chain as a whole (in which case each ingress service has its own scaling factor). In addition, in the former case, scaling factors relative to each ingress service are calculated for each service in the service chain and in real-time the number of requests predicted for each service is a weighted sum over the requests being provided to each ingress service. Multiple egress services do not require a change to the computations of the scaling factors in some embodiments.

Next, for the selected service, the process300computes (at325) an estimated percentage of the traffic received at the first service that arrives at the selected service via each path by using the direct path coefficients. The process300then sums (at330) these percentages from the various different paths to the selected service in order to compute the scaling factor for the selected service. For a single path, the estimated percentage of traffic is computed by multiplying all of the direct path coefficients along that path.

FIG.5conceptually illustrates a table500showing these computations for each of the services shown in the graph400(relative to x, the ingress traffic). For Service A (the ingress service), the scaling factor is simply equal to 1. For Service B, there is only a single path represented by a single graph edge having a coefficient of 0.7, so the scaling factor of 0.7 is arrived at by simply using this single coefficient. The computation for Service C is more complicated. For Service C, two paths are identified; each of these includes the path from Service A to Service B, so the 0.7 coefficient can be factored out of the computation. The path from Service B to Service C has a coefficient of 0.5 while the path from Service B to Service E to Service C has a coefficient of 0.5 multiplied by 0.8. This results in a scaling factor of 0.63 for Service C, as shown in the graph. Service D is simply the scaling factor for Service C multiplied by the coefficient 0.3 for the path from Service C to Service D, equal to 0.189. Finally, there is a single path to reach Service E and the multiplication of the coefficients 0.7 and 0.8 results in a scaling factor of 0.56.

In the above example, all of the scaling factors are less than 1, as is common. However, in certain cases, a service may actually receive more traffic than the front-end load balancer if multiple different paths exist to the service. For instance, a logging application might receive log data from most or all applications in a cluster such that it receives multiple times the traffic that enters the cluster.

Returning toFIG.3, after computing the scaling factor for the selected service, the process300determines (at335) whether more services remain in the service chain. If this is the case, the process returns to315to select the next service. On the other hand, once all of the scaling factors have been computed, the process300ends. It should be understood that the process300is a conceptual process. Not only might the scaling factor computations be computed slightly differently but some embodiments compute all or some of the scaling factors in parallel rather than using the serial process shown in the figure.

As mentioned, some embodiments use slightly different processes to compute the scaling factors based on the directed acyclic graph for the service chain. Specifically, for the following algorithm, the inputs are a DAG represented with an adjacency list (DG), a coefficient for each directed edge in the graph, and a starting point in the graph (S). The following pseudocode describes the algorithm:

In the above, the first step of the ModifiedBFS algorithm is to define the incoming edge graph (IE) for a node. The incoming edge graph for a particular node represents all the nodes that have an edge to that particular node. For example, in the graph500shown inFIG.5, IE(B)==[A]. To compute the IE for a node, the algorithm traverses through the keys of the adjacency list DG and finds out the next set of vertices for a key. For each node N, it appends the key in IE(N) list.

Once the IE map is computed for all of the nodes, the start node S is added to a queue and a modified version of breadth first search is performed. For a node in the queue, first the node is dequeued in V. After this, the neighbors of V are fetched. For each neighbor, its scaling factor is calculated and its incoming edge from V to N is deleted. Once no other incoming edges to N are found, N is enqueue to the queue. This ensures that nodes are not enqueued unless all of the incoming edges to that node are exhausted, because the scaling factor is only complete if all of the incoming edges are visited.

As described above, the scaling factors may be computed once or on a regular basis, depending on whether the direct path coefficients are fixed or the scaler receives statistics with which to determine those coefficients. In the latter case, the direct path coefficients are calculated dynamically in some embodiments, either because the user does not have the information to set these values or because the values are not constant and change over time. Such a heuristics-based approach allows for more accurate auto-scaling calculations, especially when these values change over time.

For input into the dynamic calculations, some embodiments use traffic metrics from the services. This information may be retrieved from the services themselves or from load balancers interposed between the services in a service chain. Some embodiments send traffic to the front-end load balancer for inter-service load balancing while other embodiments use other load balancers to handle traffic between one service and the next. The following pseudocode for an algorithm CalculateEdgeWeight uses the following variables: FS(e, t) represents flow data for a period of time on a given direct path, time range (TR) is the period of time over which a sliding window average is calculated, total flow (TF) is the sum of all incoming flow data for a node (e.g., a service or other application) in the given time range TR, and average flow (AvgF) is the average value of the incoming flow data for a node, which is obtained by dividing the total flow TF by the time range TR.

The above algorithm determines the value of the average flow over a time period using a sliding window method to determine the direct path coefficient for an edge in the graph. An array Flow at time t stores the flow data for all of the edges and is stored in FlowStore (FS). To calculate the average weight, the algorithm finds the total flow (TF) for an edge by adding all of the flows for the edge from the current time t to (t-TR). At later times, the total flow for an edge is calculated by adding the latest flow for the edge and subtracting the value of the flow at the time t-TR. The average flow for an edge is calculated by dividing the total flow for the edge by the time range.

FIG.6conceptually illustrates a table600providing an example of flow data for a particular path between two applications. In this example, data points are retrieved for every 12 minutes, and a time range value of 5 provides a one-hour time period. This allows, at 9:48, the total flow to be calculated as 540 (100+110+120+90+120) for an average flow value of 108. At 10:00, the new flow value to add is 60 and so the total flow is equal to 500 (540−100+60), for an average flow value of 100. These flow values can then be used to calculate the direct path coefficient (i.e., by identifying what percentage of traffic sent to a particular service is forwarded to the next service in the service chain), and subsequently to calculate the scaling factors.

With the scaling factors determined, the scaler module determines in real-time whether each of the services needs to be scaled. Based on the traffic being received at the first service in the service chain (from the load balancer), some embodiments calculate the traffic expected at each service and determine whether the current deployment for that service has the capacity required to handle that traffic without dropping traffic or imposing longer latency. If the expected traffic is larger than current capacity for a given service, the scaler initiates deployment of additional instances (e.g., additional Pods) for that service.

FIG.7conceptually illustrates a process700of some embodiments for determining whether scaling of services in a service chain is required based on traffic expected to arrive at those services and initiating that scaling if needed. In some embodiments, the process700is performed by a scaler module (e.g., the auto-scaling and deployment module215of the scaler shown inFIG.2). In some embodiments, the process700is performed at regular intervals or as metrics are retrieved from the front-end load balancer. The process700will be described by reference toFIGS.8-12, which illustrate examples of scaling the services in a service chain.

Specifically,FIG.8conceptually illustrates an example of a service chain800as deployed. The service chain800includes a first service (A) for which two Pods are instantiated, a second service (B) for which three Pods are instantiated, a third service (C) for which two Pods are deployed, and a fourth service (D) for which a single Pod is deployed. This figure also shows the direct path coefficients for each of the connections in the service chain800. Service A receives data messages directly from the front-end load balancer805then forwards 40% of its traffic to Service B and 50% of its traffic to Service C. Service B sends 80% of its traffic to Service D and Service C sends 90% of its traffic to Service D.

As shown, the process700begins by receiving (at705) traffic measurements at the ingress of a service chain, corresponding to the traffic at the first service in the service chain. As described, the front-end load balancer of some embodiments generates these metrics, which are retrievable by the scaler module (e.g., using API calls in the load balancer schema). The received metrics provide a measure of incoming traffic to the first service. This may be measured in an absolute number of requests, a rate of requests (e.g., requests per second), a latency measure (which can be assumed to scale linearly with the request rate, or other metrics.

The scaler is then able to use these received metrics to scale each of the services in the service chain. The scaler determines, for each service, the expected traffic to reach that service (based on the scaling factor) and whether the current deployment for the service will have adequate capacity to handle that expected traffic. If the current deployment is inadequate, the scaler initiates deployment of one or more additional instances; if the current deployment should be reduced, the scaler initiates deletion of one or more existing instances.

FIG.9conceptually illustrates scaling the deployment of the service chain800over two stages905-910in response to receiving a first traffic measurement. The first stage905ofFIG.9shows the service chain800as deployed inFIG.8, while the second stage910shows this service chain800after being scaled up in response to the traffic measurement.

FIG.10illustrates a table1000showing the computations to arrive at the scaling decisions for the example shown inFIG.9. As shown, each of the services in the service chain800has an associated scaling factor (1 for Service A, 0.4 for Service B, 0.5 for Service C, and 0.77 for Service D) computed as described above based on the direct path coefficients shown inFIG.8.

Returning toFIG.7, the process700selects (at710) a service in the service chain. Some embodiments select the services by traversing the graph of the service chain topology (as with the scaling factor calculation), while other embodiments select services using different techniques (e.g., randomly). It should be noted that the process700is a conceptual process and other processes might perform slightly different operations or perform the operations in a different order. For instance, although the process700shows each service being evaluated serially, other embodiments might evaluate whether to scale each of the services in parallel.

The process700then determines (at715) the current capacity of the selected service. In some embodiments, each service has a different capacity per instance (e.g., per Pod). This capacity may vary based on (i) the physical resources allocated to each instance for the service and (ii) the type of processing performed by the service. In some embodiments, different administrators for the different services set their own configurations for the physical resources allocated to each Pod, which may vary from service to service as a result. In addition, different types of data message processing can require different amounts of memory and/or processing power. For instance, a service that performs layer 7 processing (e.g., a deep packet inspection service) might require more resources per data message than a service that only performs layer 2 or layer 3 processing (e.g., an L2/L3 firewall service). Typically, the current capacity for a given service is the per-instance capacity multiplied by the currently-deployed number of instances.

In the example ofFIG.10, each of the services has a different per-Pod capacity. Each Pod for Service A can handle 1000 requests/second, each Pod for Service B can handle 300 requests/second, each Pod for Service C can handle 600 requests/second, and each Pod for Service D can handle 1250 requests/second. Given the current deployment, Service A can handle 2000 requests/second, Service B can handle 900 requests/second, Service C can handle 1200 requests/second, and Service D can handle 1250 requests/second.

Next, the process700computes (at720) the traffic expected to be received at the selected service based on the scaling factor for the service and the traffic at the first service. Some embodiments compute this expected traffic for the service by simply multiplying the traffic seen at the first service (i.e., from the front-end load balancer) by the scaling factor for the service. As shown inFIG.10, the first service (Service A) is expected to receive 3200 requests/second, the traffic flow retrieved from the load balancer. Service B is expected to receive 1280 requests/second (multiplying 3200 by the scaling factor of 0.4), Service C is expected to receive 1600 requests/second (multiplying 3200 by the scaling factor of 0.5), and Service D is expected to receive 2464 requests/second (multiplying 3200 by the scaling factor of 0.77).

Based on the per-instance capacity and the expected traffic for the selected service, the process700computes (at725) the required number of instances for that service. This value can be computed by dividing the expected traffic by the per-instance capacity and applying the ceiling function that rounds a decimal value up to the next integer (i.e., 3.1 and 3.9 are both rounded to 4). Some embodiments also add a safety factor to the value before applying the ceiling function (e.g., adding 0.1 so that 2.95 becomes 3.05, which rounds to 4) in case the traffic to a service increases more than expected (e.g., based on more traffic than expected being forwarded by one or more of the services in the chain).

In the example shown inFIG.10, the expected traffic for Service A is 3200 requests/second, which requires 4 Pods when divided by the per-Pod capacity of 1000 requests/second (with 3.2 rounding up to 4). The expected traffic for Service B is 1280 requests/second, which requires 5 Pods when divided by the per-Pod capacity of 300 requests/second (with 4.27 rounding up to 5). The expected traffic for Service C is 1600 requests/second, which requires 3 Pods when divided by the per-Pod capacity of 600 requests/second (with 2.67 rounding up to 3). Lastly, the expected traffic for Service D is 2464 requests/second, which requires 2 Pods when divided by the per-Pod capacity of 1250 requests/second (with 1.97 rounding up to 2). However, if a small safety factor were applied, 3 Pods would be required for Service D.

The process700then determines (at730) whether to scale the selected service. If the service should be scaled, the process determines (at735) the scaling action for the selected service. As discussed, some embodiments scale the services predictively so that the additional instances are deployed prior to the existing instances for the service being overloaded by the incoming traffic. In the example, shown in the table1000ofFIG.10, Services A and B should be scaled by adding 2 Pods for each service while Services C and D should be scaled by adding 1 Pod for each service. Owing to (1) the different per-instance capacity of different applications (in this case, the services) and (2) the different expected traffic flow reaching each different applications based on the computed scaling factors, in many cases different numbers of instances will be required due to the same change in ingress traffic.

Next, the process700determines (at740) whether any more services in the service chain remain for evaluation. If additional services remain, the process700returns to710to select the next service and determine whether to scale that service. As mentioned, in some embodiments the various services are evaluated in parallel to determine whether to scale each of the services, rather than in a serial loop as shown in the figure.

Once all of the services have been evaluated, the process700initiates (at745) deployment of additional instances or removal of instances based on the determined scaling actions (i.e., the actions determined at735for each service), then ends. In some embodiments, the scaler module itself modifies the deployment to change the number of instances for the different services. In the Kubernetes context, the scaler module edits one or more configuration objects that define the deployment to change the number of Pods implementing each service in some embodiments. In other embodiments, the scaler module provides the deployment edits to another component (e.g., an auto-scaling application in the Kubernetes control plane) that handles the modification to the deployment.

The second stage910ofFIG.9shows that, as a result of the calculations made by the scaler module915(as shown in the table1000), the scaler has modified the deployment of the service chain800. Service A now has 4 Pods deployed, Service B now has 5 Pods deployed, Service C now has 3 Pods deployed, and Service D now has 2 Pods deployed. While the figure conceptually shows the scaler915directly modifying the services, as noted above in some embodiments the scaler either modifies configuration objects for the services directly or hands off the deployment to a separate component (e.g., an auto-scaling application) that modifies these configuration objects.

As noted, in addition to scaling up a set of applications (e.g., the services in a service chain, as shown inFIG.9), in some embodiments the scaler module can also determine when to scale down one or more applications. That is, if the number of incoming requests drops, the scaler can perform similar calculations to determine that fewer instances of at least some of the applications are required. For scaling up due to increased traffic, the predictiveness helps avoid increases in latency and dropped packets. For scaling down, the predictiveness is not as crucial but can help free up resources more quickly when those resources are not needed for the current traffic levels.

FIG.11conceptually illustrates scaling down the deployment of the service chain800over two stages1105-1110in response to receiving a second traffic measurement. The first stage1105ofFIG.11shows the service chain800as deployed in the second stage910ofFIG.9, while the second stage1110shows this service chain800after some of the services have been scaled down in response to the second traffic measurement.

FIG.12illustrates a table1200showing the computations to arrive at the scaling decisions for the example shown inFIG.11. The scaling factors and per-Pod capacity for these services are the same as shown in the table1000ofFIG.10. In this case, Service A is expected to receive 2500 requests/second (the traffic flow retrieved from the load balancer), a drop in traffic as compared to the measurement shown inFIG.10that caused all of the services to be scaled up. Service B is expected to receive 1000 requests/second (multiplying 2500 by the scaling factor of 0.4), Service C is expected to receive 1250 requests/second (multiplying 2500 by the scaling factor of 0.5), and Service D is expected to receive 1925 requests/second (multiplying 2500 by the scaling factor of 0.77).

The expected traffic for Service A of 2500 requests/second requires 3 Pods when divided by the per-Pod capacity of 1000 requests/second (with 2.5 rounding up to 3). The expected traffic for Service B of 1000 requests/second requires 4 Pods when divided by the per-Pod capacity of 300 requests/second (with 3.33 rounding up to 4). The expected traffic for Service C of 1250 requests/second requires 3 Pods when divided by the per-Pod capacity of 600 requests/second (with 2.08 rounding up to 3). Lastly, the expected traffic for Service D of 1925 requests/second requires 2 Pods when divided by the per-Pod capacity of 1250 requests/second (with 1.54 rounding up to 2).

As a result of these calculations, the scaler915determines that one Pod should be removed from the deployments of Services A and B, but the deployments of Services C and D do not require updating. As such, the second stage1110ofFIG.11shows that, as a result of the calculations made by the scaler module915(as shown in the table1200), the scaler915has modified the deployment of the service chain800. Service A now has only 3 Pods deployed, Service B now has 4 Pods deployed, while Service C and Service D are unchanged.

FIG.13conceptually illustrates an electronic system1300with which some embodiments of the invention are implemented. The electronic system1300may be a computer (e.g., a desktop computer, personal computer, tablet computer, server computer, mainframe, a blade computer etc.), phone, PDA, or any other sort of electronic device. Such an electronic system includes various types of computer readable media and interfaces for various other types of computer readable media. Electronic system1300includes a bus1305, processing unit(s)1310, a system memory1325, a read-only memory1330, a permanent storage device1335, input devices1340, and output devices1345.

The bus1305collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of the electronic system1300. For instance, the bus1305communicatively connects the processing unit(s)1310with the read-only memory1330, the system memory1325, and the permanent storage device1335.

The read-only-memory (ROM)1330stores static data and instructions that are needed by the processing unit(s)1310and other modules of the electronic system. The permanent storage device1335, on the other hand, is a read-and-write memory device. This device is a non-volatile memory unit that stores instructions and data even when the electronic system1300is off. Some embodiments of the invention use a mass-storage device (such as a magnetic or optical disk and its corresponding disk drive) as the permanent storage device1335.

Other embodiments use a removable storage device (such as a floppy disk, flash drive, etc.) as the permanent storage device. Like the permanent storage device1335, the system memory1325is a read-and-write memory device. However, unlike storage device1335, the system memory is a volatile read-and-write memory, such a random-access memory. The system memory stores some of the instructions and data that the processor needs at runtime. In some embodiments, the invention's processes are stored in the system memory1325, the permanent storage device1335, and/or the read-only memory1330. From these various memory units, the processing unit(s)1310retrieve instructions to execute and data to process in order to execute the processes of some embodiments.

The bus1305also connects to the input and output devices1340and1345. The input devices enable the user to communicate information and select commands to the electronic system. The input devices1340include alphanumeric keyboards and pointing devices (also called “cursor control devices”). The output devices1345display images generated by the electronic system. The output devices include printers and display devices, such as cathode ray tubes (CRT) or liquid crystal displays (LCD). Some embodiments include devices such as a touchscreen that function as both input and output devices.

Finally, as shown inFIG.13, bus1305also couples electronic system1300to a network1365through a network adapter (not shown). In this manner, the computer can be a part of a network of computers (such as a local area network (“LAN”), a wide area network (“WAN”), or an Intranet, or a network of networks, such as the Internet. Any or all components of electronic system1300may be used in conjunction with the invention.

Hypervisor kernel network interface modules, in some embodiments, is a non-VM DCN that includes a network stack with a hypervisor kernel network interface and receive/transmit threads. One example of a hypervisor kernel network interface module is the vmknic module that is part of the ESXi™ hypervisor of VMware, Inc.

It should be understood that while the specification refers to VMs, the examples given could be any type of DCNs, including physical hosts, VMs, non-VM containers, and hypervisor kernel network interface modules. In fact, the example networks could include combinations of different types of DCNs in some embodiments.