Walmart-Sales Forecasting. Sales are the lifeblood of a business…
Tags: Time Series Forecasting, Imputation, Data Science, Machine Learning, Prophet

Sales are the lifeblood of business. With an accurate sales forecast in hand, one can plan for the future wisely. If the sales forecast predicts that during March, one could make 30 percent of one’s entire yearly sales, then production needs to be ramped up in September itself! Perhaps a smarter company would also invest in more seasonal salespeople.

In this specific project, Walmart is seeking help from the Kaggle community to better forecast sales for 45 stores located across the USA. Each store is home to several departments, and this challenge requires us to project the sales for each department in each store, for all 45 stores. To add to the challenge, selected holiday markdown events are included in the dataset which is known to affect sales.

1) Dataset Understanding

We are provided with three tables namely: Sales, Features, and Stores.

Sales:

Store — the store number

Dept — the department number

Date — the week

Weekly_Sales — sales for the given department in the given store

IsHoliday — whether the week is a special holiday week

Sales.csv

Features:

Store — the store number

Date — the week

Temperature — the average temperature in the region

Fuel_Price — the cost of fuel in the region

MarkDown1–5 — anonymized data related to promotional markdowns that Walmart is running. MarkDown data is only available after Nov 2011 and is not available for all stores all the time. Any missing value is marked with an NA.

CPI — the consumer price index

Unemployment — the unemployment rate

IsHoliday — whether the week is a special holiday week

Features.csv

Stores :

Store — the store number

Type — three types of stores ‘A’, ‘B’ or ‘C’

Size — the size of a store

Stores.csv

2) Data Cleaning

In time-independent data (non-time-series), a common practice is to fill the gaps with the mean or median value of the field. However, this is not applicable to time series. For example, we expect the temperature value of January to be very far from its value in July. Imputing the missing temperature values in July using the mean or median value in January doesn’t seem to be a wise option. Following Time-Series specific methods do work well but have their caveats:

Last observation carried forward (LOCF)/Next observation carried backward (NOCB) — These methods perform poorly when data has a visible trend

— These methods perform poorly when data has a visible trend Linear interpolation/Spline interpolation — These methods work well for data with a trend, but do not work well when there is a presence of strong seasonality.

— These methods work well for data with a trend, but do not work well when there is a presence of strong seasonality. Seasonal Adjustment + Linear Interpolation — This method works well for data with both trend and seasonality

CPI & Unemployment values are available for dates between 2/5/2010 and 4/26/2013 but are missing between dates 5/3/2013 and 7/26/2013. We can also observe an upward trend in CPI, a downward trend in Unemployment. Missing values for both these features can be imputed using methods such as rolling average, interpolation techniques (using interpolate method in python) & ARIMA. For our analysis, we use rolling average to impute missing values (ARIMA code for imputation attached for reference).

3) Feature Engineering

Feature engineering, also known as feature creation, is the process of constructing new features from existing data to train a machine learning model. This step can be more important than the actual model fitting step because a machine learning algorithm only learns from the data we give it and creating features that are relevant to a task is absolutely crucial.

Do we need to scale features? In general, algorithms that exploit distances between data samples, such as k-NN and SVM, are sensitive to feature scaling. Graphical-model based classifiers, such as Fisher LDA or Naive Bayes, as well as Decision trees and Tree-based ensemble methods (RF, XGB) are invariant to feature scaling. Random Forest/Boosted trees use the Information gain/Gini coefficient inherently which will not be affected by scaling. In the model fitting phase, which we will explore in the further sections, we fit tree-based models and hence normalizing/scaling features is not required.

In general, algorithms that exploit distances between data samples, such as k-NN and SVM, are sensitive to feature scaling. Graphical-model based classifiers, such as Fisher LDA or Naive Bayes, as well as Decision trees and Tree-based ensemble methods (RF, XGB) are invariant to feature scaling. Random Forest/Boosted trees use the Information gain/Gini coefficient inherently which will not be affected by scaling. In the model fitting phase, which we will explore in the further sections, we fit tree-based models and hence normalizing/scaling features is not required. Do we need to address the issue of multicollinearity? Multicollinearity occurs when independent variables in a regression model are correlated. Multicollinearity between quantitative (continuous) variables can be detected by reviewing scatterplot, correlation matrix (heatmap), Variance Inflation Factor (VIF). Categorical features like sex (male or female), season (summer, winter, spring, fall), are represented by dummy variables — variables containing values such as 1 or 0 representing the presence or absence of the categorical value. By including the dummy variables in a regression model, one should be careful of the Dummy Variable Trap. The Dummy Variable trap is a scenario in which independent variables are multicollinear. In order to avoid the dummy variable trap, we always declare one less dummy variable (n-1) than the categorical values (n). In the above case, we have four categorical values (summer, winter, spring, fall). So we can have only 3 Dummy variables instead of 4.

3.1) Categorical Encodings

One hot encode “Type” categorical feature since it contains more than 2 levels (A/B/C)

Label encode the “IsHoliday” categorical feature since it contains just 2 levels (True/False)

Create new features namely — “IsSuperbowl”, “IsLaborday”, “IsThanksgiving”, “IsChristmas” to identify the type of holiday

3.2) Datetime Features

Date fields are rich sources of information that can be used with machine learning models. However, these date-time variables do require some feature engineering to convert them into numerical data. We extract new features from “Date” feature such as year, month, week of the year, week of the month, day, etc

3.3) Featuretools

Typically data scientists spend the majority of their time finding features to improve the accuracy of their models. Undoubtedly this is a tedious manual process and requires a lot of intuition, data manipulation and extensive knowledge of the domain itself. Automated feature engineering aims to help the data scientist by automatically creating features out of a dataset from which the best can be selected and used for training. Featuretools is a framework that automatically creates features from temporal and relational datasets.

3.4) Features from sales value

Lag features can be extremely beneficial for time series forecasting. Creating lag features is a classical way of transforming time series forecasting problems into supervised learning problems. A key function to help create lagged features from time-series data is Pandas shift(). Here we create a few lagged features:

· Previous year sales for each store+dept combo for the current week i.e. lag of 52 weeks

· The exponentially weighted mean value of previous year sales for each store+dept combo for the current week

Aggregated monthly sales values for each store+dept combo

4) Supervised Learning model

If we optimize the model for training data, then our model will score very well on the training set but will not be able to generalize to new data, such as in a test set. When a model performs highly on the training set but poorly on the test set, this is known as overfitting, or essentially creating a model that knows the training set very well but cannot be applied to new problems, essentially reducing its flexibility. An overfit model may look impressive on the training set but will be useless in a real application. Therefore, we tune hyperparameters while performing cross-validation.

Using Scikit-Learn’s RandomizedSearchCV method, we can define a grid of hyperparameter ranges and randomly sample from the grid, performing K-Fold CV with each combination of values. The most important arguments in RandomizedSearchCV are n_iter, which controls the number of different combinations to try and cv, which is the number of folds to use for cross-validation (we use 10 and 5 respectively). Increasing n_iter will cover a wider search space, whereas increasing cv folds reduce the chances of overfitting, but raising either one of them will increase run time.

Machine learning is a field of trade-offs, and model performance vs computational fidelity is the most fundamental one.

4.1) By Store & Department combination

My approach involved fitting one model per store-department combination (so about 3300 simple models). The following store-department combos are present in the test set but not in the train set: 5–99, 9–99, 10–99, 18–43, 24–43, 25–99, 34–39, 36–30, 37–29, 42–30, 45–39

For any newcomers to programming, object-oriented programming allows us to write code in a slightly different way than we would write if we didn’t have object-oriented programming. The syntax for object-oriented programming or to define a class in Python is to type out a keyword class, and then the name of your class. Inside the class, you can define a special function called the __init__ (constructor). So this function name is very important, and it has two underscores at the start and two at the end. There’s one more thing you can do which is going to simplify your life greatly, and that is the main point of object-oriented programming, which is to define methods (functions defined inside of a class) inside the class. Methods will take self as a parameter. When you call a class, an object is created and python will automatically call the init method for you. Then whenever you want to call other methods, all you have to do is type out the “object_name.method_name”.

In the code below, we create a class TrainByStoreDeptCombo, that allows one to define methods (such as __init__, train_predict). This cuts out redundancy as we won’t need to write the same method ten times if we wanted to invoke ten different regressors. For hyperparameters associated with each algorithm, refer to the documentation on the official Sklearn website, where one can find an entire host of other useful parameters that one can play around with.

5) Time Series Modeling

In total there are ~3300 store department combinations.

The training data has gaps in dates. For any store department combinations that have missing dates, I simply used 0 for weekly sales and make sure that any store department combinations has 143 weekly dates.

5.1) Prophet

Create a holidays dataframe with two columns (holiday and ds) and a row for each occurrence of the holiday. It must include all occurrences of the holiday, both in the past (back as far as the historical data go) and in the future (out as far as the forecast is being made).

5.2) ARIMA

6) Stacking models

After the feature engineering and modeling stages, we finally arrive at a crucial point of this article: how to champion any Kaggle competition.

(Hint: It involves the word “averaging”)

As you can see, we have averaged predictions from two models: RF and Prophet. Of course, we could easily add more models into the mix.

7) Conclusion

Several of these tidbits that I have learned so far have helped me gain a wealth of knowledge, especially related to improving the quality of forecasts and demonstrating how feature engineering can potentially lead to improved estimates.

Uploading the average sales predictions from Random Forest and Prophet in Kaggle, I achieved a WMAE score of 2543 with a rank of 12!! Using a hybrid approach of statistical time-series analysis and machine learning methods resulted in a more robust model.