Kepler — A ML based capacity prediction system
Tags: AI, Capacity Planning, Prediction, Forecasting, Machine Learning

At MobiTV, our business relies on meeting and exceeding strong demands for availability and reliability for applications used by our customers to watch live streaming content. MobiTV’s multi-tenant platform serves live video and VOD traffic to our customers on a variety of devices and platforms over hls and dash streams.

MobiTV’s data centers contain hundreds of servers that deliver content to our customers. In the past, we predicted server capacity using spreadsheet models using data obtained by running performance and stress tests on staging environments. While such models are valuable, the method for obtaining data is time-consuming, error-prone and not scalable beyond a few teams. We needed a deterministic way to forecast server capacity and associated costs based on the expected user growth, new user features and software efficiencies over time.

Consequently, we built a tool (Kepler) that predicts the computing resources needed to support our explosive growth. Kepler does this by running ML algorithms on real-time performance data obtained using our production monitoring systems such as Dynatrace, Prometheus, Elk, Icinga, Datadog, etc.

Kepler Architecture

Kepler takes user input parameters such as desired CPU utilization for scaling, cache hit ratios, and number of subscribers.

Here is a screenshot for some of the inputs to Kepler.

We store real-time metrics on a NoSql db (Cassandra) and run various ML models on the collected data on a nightly schedule. The results of the models are cached so that the Kepler is able to serve the prediction results quickly.

The graphs below summarizes the results of running a regression model generated using the tool for our Media Muxer nodes. Media Muxers are responsible for serving live and vod streams in a variety output formats such as dash, hlsv3, hlsv4, fmp4 and SmoothStreaming. Note that Kepler shows a significant correlation (correlation coefficient: 0.98) between CPU and Transaction per second for Media Muxer nodes.

Kepler can dynamically choose the best fit ML model based on the node type, distribution of the data and results of rmse value seen during a run. We used the following chart to try out various ML algorithms for use in Kepler.

1. Linear Regression: It finds the optimal fit equation using the least squares method. Due to significant correlation between metrics — CPU usage and transaction per second, we’re using this model to predict future transactions and nodes based on current subscribers, and CPU usage.

2. Polynomial Regression: Polynomial regression is useful only if there were non-linear relationships between the variables. However, we observed that none of our services manifest this relationship so we did not use polynomial regression. Note that using polynomial regression with degree > 1, will result in overfitting and is not recommended.

3. Stochastic Gradient Descent: Keeping Linear Regression as the base model, we implemented minimization through stochastic gradient descent. We observed that method provides a decent alternative to regression however we do observe higher rmse at times. In future, we might try out alternatives such as Batch Gradient Descent with optimizers like Momentum, RMSProp and Adam, which are used to overcome convergence problems around local and global minima as well as pathological curvature.

4. Random Forest: This is a robust non-parametric predictor because it constructs large number of uncorrelated tree models which train on random slices of train cases. This injects randomness that makes them very accurate regressors. For capacity prediction scenarios, we often want to predict CPU required for TPS values we have not yet seen in the production environment. We observed that the Random Forest algorithm performs poorly for values it has not yet seen. We chose not to use evaluate algorithms like Random Forest, and LightGBM for our current use case.

5. Support Vector Regression: A support vector regression algorithm attempts to create a function that keeps the target result within a threshold. This is slightly different from linear regression which tries to create a function that minimizes the error by using sum of least squares. SVR tends to be computationally expensive, however since we value recency of our data more than its volume, SVR works well for us in predicting capacity.

The following table showing the root mean squared error (rmse) values for the various model’s test cases. Note that the rmse values are almost similar for all of them. To calculate rmse, we split 30 days worth of data ingested with hourly granularity into training and test set. Models were fitted on the training set and rmse values were calculated using the test set.

rmse values for various models/services

The chart below shows results from predicting CPU for given TPS. Tree based model like Random Forest perform poorly as expected while the regression functions provide similar results.

The chart below depicts predicted TPS for a given CPU utilization % for Service A.

So far, we have only predicted TPS (y-axis) given desired CPU (x-axis), however in practice we want to predict nodes/cores/pods given the rise in the number of subscribers to our video platform. For simplicity, let’s assume that for Service A, a percentage increase in subscribers results in similar percentage increase in TPS. Given this assumption, we utilize the following heuristic:

Total Desired TPS = 90th percentile of TPS for Service A per node * current nodes * (1 + % rise in subscribers)

Predicted TPS Per Node for a desired CPU utilization is derived using the ML models above

Now, we can easily extrapolate:

Number of Desired Nodes = Total Desired TPS / Predicted TPS per Node

Finally, here is a chart that shows Kepler forecasting a 370, 100 and 266 percent rise for nodes running Service A, B and C respectively with SVR model - given a 1000% percent rise in number of subscribers: