Automating Forecasting in Scale
Tags: AWS, Machine Learning, Big Data, Forecasting

“YOU CAN BUILD AI”

How to use Amazon Forecast (AF) and other supporting AWS data services to improve, simplify, and scale your business forecasting.

Amazon Forecast Pipeline Flow

Building Forecasting Pipeline

Forecasting of demand or sales is often done using 20th-century tools, such as Microsoft Excel. The benefits of Excel in terms of interactivity, familiarity, and popularity are obvious. However, the limitations of Excel as a manual tool, running on a single machine are blocking the practice to scale for the amount of data, the number of forecasts needed, and the ability to experiment with newer methods. The set of tools in the cloud can bring forecasting methods to the 21st century and allow more accurate and more frequent forecasts.

In this post, we will build a forecasting pipeline that is starting with a database query dump put into S3, transforms using Amazon Athena (managed Presto), develop with Jupyter notebook in SageMaker, build models using Amazon Forecast, and visualize the data and the forecast results using QuickSight. The benefits of using the native AWS services in the pipeline are the cost, scale, and integration of most of AWS services. However, you can replace any of the tools with others such as PowerBI for visualization, or RDBMS for the queries and transformation.

Securing the Data in the Cloud

The first question I get when a large “not-born-in-the-cloud” enterprise is evaluating the migration or usage of the cloud, is where to start and how to do it with minimal risk. My answer is often to start with building new machine learning systems that you can’t grow in the current infrastructure. If we choose this path, we need to find secure and straightforward ways to get the data that is required for the machine learning algorithms. Therefore, most of these projects are starting with a massive query to the current data warehouse or database, and dumping the result set as files and upload them to S3.

In our case, the data that we need for the forecast pipeline is sensitive business information and must be protected well. The best practices to allow the creation of the process is to create a new bucket in S3, encrypt it by default, create it as a private endpoint that allows access only from the internal organization VPC, and enable audit using CloudTrail.

Using Amazon Athena to transform data

Once the data is securely stored in an S3 bucket, data engineering, data science, and data Ops teams can start processing it. I can’t stress enough the need for the teams to collaborate on the problem. Forecasting is not a science problem or data optimization or process automation only. It is a combination of all the above.

Nevertheless, the process starts with transforming the flat files data into a format that allows efficient queries on it, for data exploration, aggregation, and similar manipulation. Smaller datasets were able to be loaded into Excel or similar software tools. However, the data here is more detailed and includes multiple years of daily sales data for each of the thousands of SKU products. Here is a partial view of the dozens of files in S3:

A simple way to turn the flat CSV files to allow SQL queries on is to use AWS Glue Crawler that analyzes the format of the data files and creates a schema. The wizard in Glue can lead you through the process, using Security Configuration (1), Data Location (2), Access Permission (3) and Target Database (4):

AWS Glue Crawler Configuration

A couple of minutes later, the following table will appear in the Glue data catalog and will be available to query using SQL syntax with Amazon Athena:

Raw Data Schema in AWS Glue Data Catalog

Although the data is now ready for filtering, aggregations, and other transformation, it is recommended to compress the data and partition it to reduce the cost of the next steps dramatically. You can perform this step using Create Table As Select (CTAS) query in Athena:

/*Original Data Compressed*/ CREATE TABLE forecast.compressed_data WITH (

format='PARQUET',

external_location='s3://forecast-xxxxxx/sagemaker/data/',

partitioned_by = ARRAY['year']

) AS

SELECT *, year(calendar_date) as year FROM forecast.data;

Let’s check what did we improve with this transformation. First, let’s count how many distinct products we have in the raw data in the CSV format:

SELECT count (distinct item_id) FROM forecast.data where year(calendar_date) = 2018;

(Run time: 3.3 seconds, Data scanned: 1.57 GB)

Now, let’s run the same query on the compressed data using the partition:

SELECT count (distinct item_id) FROM forecast.compressed_data where year = 2018;

(Run time: 2.95 seconds, Data scanned: 114.16 KB)

The run time is similar. However, the cost is 10,000 times less expensive. I used an extreme example that is benefiting both from the columnar format of Parquet with the single-column count and from the yearly partition that I used. Nevertheless, any other query that we will use on the compressed and partitioned data will have a significant (at least *100) better cost performance.

The modular and flexible architecture we can build in the cloud is a complete game-changer in the field of big data and machine learning. The time it takes to build a working pipeline, the ability to focus on any part of it and optimized it to an extreme level, and the ability to put it to production on any scale are intoxicated, especially for people with a background of similar systems in the past using on-premise infrastructure.

Building the first Forecasting Model

Creating Training Dataset

Now that we have our data ready for analysis using different tools such as python libraries as Pandas and Scikit-learn, or SQL tools for visualization and reporting, we can turn to train our first baseline forecasting model using Amazon Forecast. We will not jump directly to the fanciest models on the highest level of details resolution since most machine learning projects require many (often failed) experiments that are leading us to the best result that the business needs. Every business has different requirements, different products, different customers in different markets, and there is no one-size-fits-all solution that can work on the first trial. The pipeline that we are building here is designed to allow quick experiments and iterations to facilitate the exploration and development process of our Forecasting models.

For the first model, we will take the highest aggregation on the data and try to forecast the sales of the category level. The original data has more details that include the brand and item_id level, as well as the demand for each location:

Sample of the RAW data

We will use Athena again to create the aggregated data for the model:

/*train_category training data aggregated by category*/

CREATE TABLE forecast.train_category

WITH (

format='TEXTFILE',

external_location='s3://XXXX/sagemaker/train_category/',

field_delimiter = ','

) AS

SELECT category as item_id, date_format(calendar_date, '%Y-%m-%d') as timestamp, sum(sales) as demand FROM forecast.compressed_data

WHERE calendar_date < CAST('2018–01–01' AS DATE)

GROUP BY 1,2

ORDER BY 2,1

The points to note in the code are:

Line 4: format=’TEXTFILE’ , Amazon Forecast (AF) is expecting to get CSV format files and not Parquet. The data is compressed with gzip, which is the default for Create-Table-As-Select (CTAS) commands.

, Amazon Forecast (AF) is expecting to get CSV format files and not Parquet. The data is compressed with gzip, which is the default for Create-Table-As-Select (CTAS) commands. Line 8: category as item_id , for simplicity we are converting the category level to be named item_id, which is the default column name in AF for the ID of the product.

, for simplicity we are converting the category level to be named item_id, which is the default column name in AF for the ID of the product. Line 8: date_format(calendar_date, ‘%Y-%m-%d’) as timestamp , another convention of AF regarding the name of the column and its format.

, another convention of AF regarding the name of the column and its format. Line 8: sum(sales) as demand , the last mandatory column for AF and its default name

, the last mandatory column for AF and its default name Line 9: FROM forecast.compressed_data , using the compressed version of the data we created above for efficiency.

, using the compressed version of the data we created above for efficiency. Line 10: where calendar_date < CAST(‘2018–01–01’ AS DATE), creating the split on the data for training (before 2018) and test (2018).

Importing the Dataset to Amazon Forecast

For the more advanced recipes of Amazon Forecast, we will use additional datasets such as item metadata and related time series. However, for the first simple model, we will use only the target time-series, which is expecting the following schema:

item_id (string) – A unique identifier for the item or product that you want to predict the demand for. timestamp (timestamp) demand (float) – The number of sales for that item at the timestamp. This is also the target field for which Amazon Forecast generates a forecast.

We can use the AWS management console to create the dataset and import the data from S3; however, since we want to create an automated pipeline, we will use the Python SDK for it. I’m also using Jupyter notebooks on top of Amazon SageMaker to make the development more interactive and sharing simpler.

After a few minutes, the import job is successful and turns to ‘ACTIVE.’ I’ll use the Jupyter notebook to check that the data statistics of the data set make sense:

Screenshot of Jupyter Notebook with Import Job Statistics

Training a Forecast Predictor on the Dataset

Once the dataset is ready, it is time to choose the recipe of the predictor. Amazon Forecast is offering out-of-the-box a broad set of recipes:

Amazon Forecast Built-in Recipes

In the next parts, we will explore the different recipes and use them to improve on the baseline. For now, we will choose the most straightforward ARIMA recipe for our first predictor.

predictorName= project+'_ARIMA' # We will create forecast prediction for two months (60 days)

forecastHorizon = 60 # Starting the predictor creation job

createPredictorResponse=forecast.create_predictor(

RecipeName=recipe,

DatasetGroupName= datasetGroupName ,

PredictorName=predictorName,

ForecastHorizon = forecastHorizon)



predictorVerionId=createPredictorResponse['VersionId'] # Waiting for the predictor to be ready

while True:

predictorStatus = forecast.describe_predictor(

PredictorName=predictorName,

VersionId=predictorVerionId)['Status']

print(predictorStatus)

if predictorStatus != 'ACTIVE' and predictorStatus != 'FAILED':

sleep(30)

else:

break

A few minutes later, the predictor is ready, and we can check the accuracy metrics it calculated on its internal validation datasets:

Screenshot of the Predictor Accuracy Metrics

Creating Forecasts with the Predictor

To generate a forecast using the predictor that we just trained, we need to deploy it first.

forecast.deploy_predictor(PredictorName=predictorName) while True:

deployedPredictorStatus = forecast.describe_deployed_predictor(

PredictorName=predictorName

)['Status']

print(deployedPredictorStatus)

if deployedPredictorStatus != 'ACTIVE' and deployedPredictorStatus != 'FAILED':

sleep(30)

else:

break

When the predictor is deployed and ACTIVE, we can go to the AWS management console and get a simple visualization of the forecast to the first two months of the year for one of the categories (item_id = “012”):

Screenshot of the Weekly Forecast for one of the products

However, we need a more scalable way to see the accuracy of the forecasts, compare it across the different categories, and later on, compare it to other recipes. We will want to use a visualization tool such as QuickSight, and for it, we need to get the forecasts to S3 and query them using Athena.

Exporting Forecasts to S3 for Further Analysis

The forecast export job is triggered in a similar way to the previous job that we started:

# Request for a forecast based on time and product filter

forecastResponse = forecastquery.get_forecast(

PredictorName=predictorName,

StartDate='2018-01-01T00:00:00Z',

EndDate='2018-01-10T00:00:00Z',

Interval="day",

Filters={"item_id":"012"} ) # Creating a dataframe for the prediction output

predictions = forecastResponse['Forecast']['Predictions']

pd.concat({k: pd.DataFrame(v).T for k, v in predictions.items()}, axis=0).T forecastId = forecastResponse['Forecast']['ForecastId'] forecast.create_forecast_export_job(

ForecastId= forecastId,

OutputPath = {

"S3Uri":"s3://forecast-xxxxxx/output/"+project+"/",

"RoleArn":roleArn

}

)

And a few minutes later, we see a new folder in our S3 bucket under the “output” folder with a few files with forecasts results on all the products and all the days of the forecast horizon we defined.

Next, we will create another AWS Glue crawler to analyze the forecast output files and generate a schema to allow us to query the data with Athena and QuickSight. We will use the wizard and get a similar crawler definition, except for pointing to the output folder and create a table with “output_” prefix:

AWS Glue Crawler for Forecast Output Files

And after a couple of minutes, the table is ready to query in Athena (and QuickSight):

Forecast Export Output Schema

Joining Forecast Output with Actual Values for Evaluation

For the last step of evaluating the performance of the simple predictor we created, we will open QuickSight, create a new data set, connect to Athena, and use the following custom SQL:

select * from (select cast(category as integer) as item_num_history, date_trunc('day', calendar_date) as sales_day, sum(sales) as demand



from forecast.compressed_data group by 1,2) as actual right join (select *, date_trunc('day', from_iso8601_timestamp(date)) as forecast_day, cast(item_id AS integer) as item_num_forecast

from forecast.output_train_1217_category) as forecast on actual.sales_day = forecast.forecast_day AND actual.item_num_history = forecast.item_num_forecast

The loading of the data through the Athena connector takes a few seconds, and then a few clicks to generate the first visualization:

Screenshot of QuickSight Visualization to compare

Recap and Next Steps

In a couple of hours, we built a pipeline to automate forecasting experiments and later for production usage. We used a combination of AWS Glue and Athena, Amazon Forecast (finally out of beta) and QuickSight. In the next steps, we will explore ways to improve the accuracy of the forecast, by adding more data sets (store level, brand and product level, product metadata, and related time-series), other forecast recipes (DeepAR, Prophet, etc.), and additional transformation (using local calendar, for example). We will also explore ways to automate the process further and allow refresh of the models daily with no manual work, using Step Function, Lambda, and other serverless services.