Walmart Sales Forecasting
Tags: Machine Learning, Forecasting, Data Science, Time Series Analysis

Building train-test set

splitting final data into train and test. We kept 80%of train data and 20% test data.

#train-test split

X_train,X_test,y_train,y_test=train_test_split( X, y, test_size=0.20, random_state=0)

Out of 421570, training data consists of 337256 and test data consists of 84314 with a total of 15 features.

Machine Learning Models

We are going to use different models to test the accuracy and will finally train the whole data to check the score against kaggle competition.

Standardizing train and test data :

from sklearn.preprocessing import StandardScaler

sc_X = StandardScaler()

X_train = sc_X.fit_transform(X_train)

X_test = sc_X.transform(X_test)

1) KNN Regressor

Out of all the machine learning algorithms I have come across, KNN has easily been the simplest to pick up. KNN can be used for both classification and regression problems. The algorithm uses ‘feature similarity’ to predict the values of any new data points. This means that the new point is assigned a value based on how closely it resembles the points in the training set.

from sklearn.metrics import mean_absolute_error

from sklearn.neighbors import KNeighborsRegressor

knn = KNeighborsRegressor(n_neighbors=10,n_jobs=4)

knn.fit(X_train,y_train)

y_pred = knn.predict(X_test)

scatter plot for predicted values

Accuracy KNNRegressor: 56.78497373157646 %

2 ) Decision tree Regessor

Decision tree builds regression or classification models in the form of a tree structure. It breaks down a dataset into smaller and smaller subsets while at the same time an associated decision tree is incrementally developed. The final result is a tree with decision nodes and leaf nodes. A decision node (e.g., Outlook) has two or more branches (e.g., Sunny, Overcast and Rainy), each representing values for the attribute tested. Leaf node (e.g., Hours Played) represents a decision on the numerical target. The topmost decision node in a tree which corresponds to the best predictor called root node. Decision trees can handle both categorical and numerical data.

from sklearn.tree import DecisionTreeRegressor

dt = DecisionTreeRegressor(random_state=0)

dt.fit(X_train,y_train)

y_pred = dt.predict(X_test)

accuracy DTR: 96.20101070234142 %

3) Random Forest Regressor

Random forest is a bagging technique and not a boosting technique. The trees in random forests are run in parallel. It operates by constructing a multitude of decision trees at training time and outputting the class that is the mode of the classes (classification) or mean prediction (regression) of the individual trees. The number of features that can be split on at each node is limited to some percentage of the total (which is known as the hyperparameter)

# After Hyper-parameter tunning

rfr = RandomForestRegressor(n_estimators = 400,max_depth=15,n_jobs=5)

rfr.fit(X_train,y_train)

y_pred=rfr.predict(X_test)

accuracy RandomForestRegressor: 96.56933672047487 %

4) XGBRegressor

XGBoost (eXtreme Gradient Boosting) is an advanced implementation of gradient boosting algorithm. XGBRegressor Handling sparse data.XGBoost has a distributed weighted quantile sketch algorithm to effectively handle weighted data. For faster computing, XGBoost can make use of multiple cores on the CPU. This is possible because of a block structure in its system design. Data is sorted and stored in in-memory units called blocks. Hyperparameters are objective, n_estimators, max_depth, learning_rate.

xgb_clf = XGBRegressor(objective='reg:linear', nthread= 4, n_estimators= 500, max_depth= 6, learning_rate= 0.5)

xb = xgb_clf.fit(X_train,y_train)

y_pred=xgb_clf.predict(X_test)

accuracy XGBRegressor: 97.21754267971075 %

4) ExtraTreesRegressor

The Extra-Tree method (standing for extremely randomized trees) was proposed with the main objective of further randomizing tree building in the context of numerical input features, where the choice of the optimal cut-point is responsible for a large proportion of the variance of the induced tree. With respect to random forests, the method drops the idea of using bootstrap copies of the learning sample, and instead of trying to find an optimal cut-point for each one of the K randomly chosen features at each node, it selects a cut-point at random.

from sklearn.ensemble import ExtraTreesRegressor

etr = ExtraTreesRegressor(n_estimators=30,n_jobs=4)

etr.fit(X_train,y_train)

y_pred=etr.predict(X_test)

Accuracy ExtraTreesRegressor: 96.40934076228986 %

All model Comparison :

from prettytable import PrettyTable



x = PrettyTable() x.field_names = ["Model", "MAE", "RMSE", "Accuracy"] x.add_row(["Linear Regression (Baseline)", 14566, 21767, 8.89])

x.add_row(["KNNRegressor", 8769, 14991, 56.87])

x.add_row(["DecisionTreeRegressor", 2375, 7490, 96.02])

x.add_row(["RandomForestRegressor", 1854, 5785, 96.56])

x.add_row(["ExtraTreeRegressor", 1887, 5684, 96.42])

x.add_row(["XGBRegressor", 2291, 5205,97.23 ]) print(x)

Getting averages of best models

The trick is to get the average of the top n best models. The n top models are decided by their accuracy and rmse. Here we have taken 4 models as their accuracies are more than 95%. The models are DecisionTreeRegressor, RandomForestRegressor, XGBRegressor and ExtraTreesRegressor.

Note that just taking top models doesn’t mean they are not overfitting. This can be verified by checking RMSE or MAE. In the case of a classification problem, we can use the confusion matrix. Also, there should not be much difference in test accuracy and train accuracy.

# training top n models

dt = DecisionTreeRegressor(random_state=0)

etr = ExtraTreesRegressor(n_estimators=30,n_jobs=4)

xgb_clf = XGBRegressor(objective='reg:linear', nthread= 4, n_estimators= 500, max_depth= 6, learning_rate= 0.5)

rfr = RandomForestRegressor(n_estimators = 400,max_depth=15,n_jobs=4) dt.fit(X_train,y_train)

etr.fit(X_train,y_train)

xgb_clf.fit(X_train,y_train)

rfr.fit(X_train,y_train) # predicting on test data

etr_pred=etr.predict(X_test)

xgb_clf_pred=xgb_clf.predict(X_test)

rfr_pred=rfr.predict(X_test)

dt_pred = dt.predict(X_test)

Getting averages of models :

final = (etr_pred + xgb_clf_pred + rfr_pred + dt_pred)/4.0

prediction of the final model

Conclusion

Here we can see that our RMSE reduced in comparison to our best performing single model i.e. XGBRegressor with RMSE of 3804. Hence we can conclude that taking averages of top n models helps in reducing loss.

As here available data is less, so loss difference is not extraordinary . But in large datasets of sizes in Gigabytes and Terabytes, this trick of simple averaging may reduce the loss to a great extent.

Kaggle Score

Now without splitting the whole data into a train-test, training it on the same and testing it on future data provided by kaggle gives a score in the range of 3000 without much deep feature engineering and rigorous hypertuning.

Future Work

Modifying date feature into days, month, weeks.

The dataset includes special occasions i.e Christmas, pre-Christmas, black Friday, Labour day, etc. On these days people tend to shop more than usual days. So adding these as a feature to data will also improve accuracy to a great extent.

Also there are a missing value gap between training data and test data with 2 features i.e. CPI and Unemployment. If that gap is reduced then also performance can be improved.

References :

Thank you for your attention and reading my work

If you liked this story, share it with your friends and colleagues !

Also, follow me on