5 Machine Learning Techniques for Sales Forecasting
By: Molly Liebeskind
Tags: Machine Learning, Sales Forecasting, Arima, Lstm, Time Series Analysis

5 Machine Learning Techniques for Sales Forecasting

Forecasting sales is a common and essential use of machine learning (ML). Sales forecasts can be used to identify benchmarks and determine incremental impacts of new initiatives, plan resources in response to expected demand, and project future budgets. In this article, I will show how to implement 5 different ML models to predict sales.

The data for this demonstration can be found on Kaggle and the full code is on GitHub.

Getting Started

The first step is to load the data and transform it into a structure that we will then use for each of our models. In its raw form, each row of data represents a single day of sales at one of ten stores. Our goal is to predict monthly sales, so we will first consolidate all stores and days into total monthly sales.

def load_data():

url = """https://www.kaggle.com/c/demand-forecasting-kernels

only/download/ryQFx3IEtFjqjv3s0dXL%2Fversions%2FzjbSfpE39fdJl

MotCpen%2Ffiles%2Ftrain.csv"""



return pd.read_csv(url) def monthly_sales(data):

data = data.copy() # Drop the day indicator from the date column

data.date = data.date.apply(lambda x: str(x)[:-3]) # Sum sales per month

data = data.groupby('date')['sales'].sum().reset_index()

data.date = pd.to_datetime(data.date)

data.to_csv('../data/monthly_data.csv')

return data data = load_data()

monthly_data = monthly_sales(data)

In our new data frame, each row now represents total sales in a given month across all stores.

If we plot the total monthly sales over time, we see that average monthly sales increase over time, which means that our data is not stationary. To make it stationary, we will calculate the difference between sales in each month and add this into our data frame as a new column. Additional details on stationery and differencing can be found here.

# Calculate difference in sales month over month

def get_diff(data):

data['sales_diff'] = data.sales.diff()

data = data.dropna()

return data stationary_df = get_diff(monthly_data)

Below is a visual representation of how our data looked before and after the differencing transformation. Code for all plots can be found here.

Comparing stationarity before and after differencing. Image by Molly Liebeskind.

Now that our data represent monthly sales and we have transformed it to be stationary, we will set up the data for our different model types. To do this, we will define two different structures: one will be used for ARIMA modeling and the other will be used for the rest of the models.

For our Arima model, we will need only a datetime index and the dependent variable (diff in sales) columns.

def generate_arima_data(data):

dt_data = data.set_index('date').drop('sales', axis=1)

dt_data.dropna(axis=0)

dt_data.to_csv('../data/arima_df.csv')

return dt_data arima_data = generate_arima_data(stationary_df)

For our other models, we will create a new data frame where each feature represents a previous month’s sales. To determine how many months to include in our feature set, we will observe the autocorrelation and partial autocorrelation plots and use the rules for selecting lags in ARIMA modeling. This way, we can keep consistent a look-back period for our ARIMA and regressive models.

Autocorrelation and partial autocorrelation plots. Image by Molly Liebeskind.

Based on the above, we will choose our look-back period to be 12 months. We will, therefore, generate a data frame that has 13 columns, 1 column for each of the 12 months and the column for our dependent variable, difference in sales. Code for the below was grabbed from Baris Karaman’s data driven growth series.

def generate_supervised(data):

supervised_df = data.copy()



#create column for each lag

for i in range(1,13):

col = 'lag_' + str(i)

supervised_df[col] = supervised_df['sales_diff'].shift(i)



#drop null values

supervised_df = supervised_df.dropna().reset_index(drop=True)

supervised_df.to_csv('../data/model_df.csv', index=False)



return supervised_df model_df = generate_supervised(stationary_df)

Now we have two separate data structures, our Arima structure that includes a datetime index and our supervised structure that includes lags as features.

ARIMA and Supervised data frames for sales forecasting. Image by Molly Liebeskind.

Modeling

To create and assess all of our models, we use a series of helper functions that perform the following functions. The code for all of the helper functions can be found here.

Train test split: we separate our data so that the last 12 months are part of the test set and the rest of the data is used to train our model Scale the data: using a min-max scaler, we will scale the data so that all of our variables fall within the range of -1 to 1 Reverse scaling: After running our models, we will use this helper function to reverse the scaling of step 2 Create a predictions data frame: generate a data frame that includes the actual sales captured in our test set and the predicted results from our model so that we can quantify our success Score the models: this helper function will save the root mean squared error (RMSE) and mean absolute error (MAE) of our predictions to compare performance of our five models

Regressive Models: Linear Regression, Random Forest Regression, XGBoost

For our regressive models, we can use the fit-predict structure of the scikit-learn library. We therefore can set up a base modeling structure that we will call for each model. The function below calls many of the helper functions outlined above to split the data, run the model, and output RMSE and MAE scores.

def regressive_model(train_data, test_data, model, model_name):



# Call helper functions to create X & y and scale data

X_train, y_train, X_test, y_test, scaler_object =

scale_data(train_data, test_data)



# Run regression model

mod = model

mod.fit(X_train, y_train)

predictions = mod.predict(X_test) # Call helper functions to undo scaling & create prediction df

original_df = load_data('../data/monthly_data.csv')

unscaled = undo_scaling(predictions, X_test, scaler_object)

unscaled_df = predict_df(unscaled, original_df) # Call helper functions to print scores and plot results

get_scores(unscaled_df, original_df, model_name)

plot_results(unscaled_df, original_df, model_name)



# Separate data into train and test sets

train, test = tts(model_df) # Call model frame work for linear regression

regressive_model(train, test, LinearRegression(),'LinearRegression') # Call model frame work for random forest regressor

regressive_model(train, test,

RandomForestRegressor(n_estimators=100,

max_depth=20),

'RandomForest') # Call model frame work for XGBoost

regressive_model(train, test, XGBRegressor(n_estimators=100,

learning_rate=0.2),

'XGBoost')

The output below shows the predictions (red) overlaid on top of the actual sales (blue) for each of the regressive models. While the results look similar, the nuanced differences equate to a few thousand dollars in sales, which we will see in the comparison section below.

Regressive model predictions vs actual sales. Image by Molly Liebeskind.

Long Short-Term Memory (LSTM)

LSTM is a type of recurrent neural network that is particularly useful for making predictions with sequential data. For this purpose, we will use a very simple LSTM. For additional accuracy, seasonal features and additional model complexity can be added.

def lstm_model(train_data, test_data):

# Call helper functions to create X & y and scale data

X_train, y_train, X_test, y_test, scaler_object =

scale_data(train_data, test_data) X_train = X_train.reshape(X_train.shape[0], 1, X_train.shape[1])

X_test = X_test.reshape(X_test.shape[0], 1, X_test.shape[1]) # Build LSTM

model = Sequential()

model.add(LSTM(4, batch_input_shape=(1, X_train.shape[1],

X_train.shape[2]),

stateful=True))

model.add(Dense(1))

model.add(Dense(1))

model.compile(loss='mean_squared_error', optimizer='adam')

model.fit(X_train, y_train, epochs=200, batch_size=1, verbose=1,

shuffle=False)

predictions = model.predict(X_test, batch_size=1) # Call helper functions to undo scaling & create prediction df

original_df = load_data('../data/monthly_data.csv')

unscaled = undo_scaling(predictions, X_test, scaler_object,

lstm=True)

unscaled_df = predict_df(unscaled, original_df) # Call helper functions to print scores and plot results

get_scores(unscaled_df, original_df, 'LSTM')

plot_results(unscaled_df, original_df, 'LSTM')

The resulting plot looks similar to the three regressive plots above, so we will hold off on comparing the results until we look at the errors below.

LSTM model predictions vs actual sales. Image by Molly Liebeskind.

ARIMA:

The ARIMA model looks slightly different than the models above. We use the statsmodels SARIMAX package to train the model and generate dynamic predictions. The SARIMA model breaks down into a few parts.

AR: represented as p, is the autoregressive model

I : represented as d, is the differencing term

MA: represented as q, is the moving average model

S: enables us to add a seasonal component

In the code below, we define our model and then make dynamic predictions for the last 12 months of the data. For standard, non-dynamic predictions, the following month’s prediction is made using the actual sales from the prior months. In contrast, for dynamic predictions, the following month’s prediction is made using the predicted sales from the prior months.

def sarimax_model(data):

# Model

sar = sm.tsa.statespace.SARIMAX(data.sales_diff, order=(12, 0,

0), seasonal_order=(0, 1, 0,

12), trend='c').fit() # Generate predictions

start, end, dynamic = 40, 100, 7

data['pred_value'] = sar.predict(start=start, end=end,

dynamic=dynamic) # Call helper functions to undo scaling & create prediction df

original_df = load_data('../data/monthly_data.csv')

unscaled_df = predict_df(data, original_df) # Call helper functions to print scores and plot results

get_scores(unscaled_df, original_df, 'ARIMA')

plot_results(unscaled_df, original_df, 'ARIMA')

Again, the results look pretty good. We will dig in further below.

ARIMA model predictions vs actual sales. Image by Molly Liebeskind.

Comparing Models

To compare model performance, we will look at root mean squared error (RMSE) and mean absolute error (MAE). These measurements are both commonly used for comparing model performance, but they have slightly different intuition and mathematical meaning.

MAE: the mean absolute error tells us on average how far our predictions are from the true value. In this case, all errors receive the same weight.

RMSE: we calculate RMSE by taking the square root of the sum of all of the squared errors. When we square, the larger errors have a greater impact on the overall error while smaller errors do not have as much weight on the overall error.

From the helper functions above, we use get_scores to calculate the RMSE and MAE scores for each model. These scores were saved in a dictionary and pickled. For comparison, we will transform that dictionary into a Pandas data frame and plot the results.

def create_results_df():

# Load pickled scores for each model

results_dict = pickle.load(open("model_scores.p", "rb")) # Create pandas df

results_df = pd.DataFrame.from_dict(results_dict,

orient='index', columns=['RMSE', 'MAE', 'R2']) results_df = results_df.sort_values(by='RMSE',

ascending=False).reset_index() return results_df results = create_results_df()

This gives us the following data frame.

We can see that although our model outputs looked similar in the plots above, they do vary in their degree of accuracy. Below is a visual to help us see the difference. Plot code can be found here.

Comparing model performance. Image by Molly Liebeskind.

What we see is that overall the XGBoost model had the best performance followed closely by the ARIMA and LSTM models. A caveat here is that all of the models above were derived in their most basic form to demonstrate how they can be used for sales forecasting. The models were only lightly tuned to minimize complexity. The LSTM for example could have many additional nodes and layers to increase performance.

To identify which model is right for your use case, you should consider the following.

The degree of model complexity vs interpretability you are comfortable with.

Models can be tuned and features can be engineered to include seasonal information, holidays, weekends, etc.

Understand how you will be using the results and how data will be coming in to update your model.

Tune models using cross-validation or similar techniques to avoid overfitting data.

The full code for this introduction into sales forecasting can be found here. Please add any questions or comments below.