The Most Feature-Rich ML Forecasting Methods Available: Compliments of RemixAutoML
By: Adrian Antico
Tags: Catboost, Feature Engineering, Forecasting, Machine Learning, Optimization

Breakdown of AutoCatBoostCARMA()

This is my go-to method. The main difference between the CatBoost, XGBoost, and H2O versions relate to the ML parameters available for tuning. All functions listed in this blog have working examples in the GitHub README, the R help files (which can be opened in your R session) or the package reference manual.

Five feature aspects of AutoCatBoostCARMA():

Feature engineering Data series enhancements Machine learning parameters Grid tuning parameters Model insights

F eatures Engineering

The function has arguments for specifying the types of feature engineering you’d like generated. Most of your accuracy lift will come from exploring these.

Most ML forecasting methods do not have an option for utilizing moving averages (let alone other rolling statistics) unless they are for single time series models (no grouping variables). I’ve spend a lot of time optimizing the run times for generating those for both the training data creation and more importantly the scoring data which is generated after each update step during forecasting (the features have to be generated after each forecast step, just like an SARIMAX). These features always dominate the variable importance so it’s important to make use of them.

Image by author

Data Series Enhancements

These parameters can be set to alter the underlying data used in the ML training and forecasting output. Time weighting, for example, is especially useful if you are forecasting KPIs from a business unit that modifies their processes from time to time. It will allow the model to pick up on the changes more quickly.

Image by author

Feature Comparison to Neural Prophet

Image by author

ML Parameters

These arguments are the same ones you would see if you were building an ML regression model for regression types of projects using the AutoCatBoostRegression() function from RemixAutoML. CatBoost has plenty more parameters to tweak but I haven’t had the need to add them at this point.

Image by author

Grid Tuning Parameters

I typically don’t run grid tuning on the underlying regression model itself but it can be done and a multi-armed bandit system using randomized probability matching is used on discrete subspaces of the parameters to achieve that. I, however, prefer to grid tune on the out of sample forecast evaluation which is not done internally, and you can see how I set that up in the code at the bottom of this article.

Model Insights

The model insights output is useful for data scientists and ML engineers when it comes to understanding the underlying regression model. Business folks can also benefit from the insights output to help them understand what is going on inside the black box, when you’re trying to get buy-in.

TrainOnFull: When you set this argument to TRUE the regression model is trained and then a forecast is generated. When it’s set to FALSE only the model is trained but get back all the model insight information. I typically run in FALSE mode first to get the model insight info for sanity checking.

The following model insights information is returned. You can also have everything except the validation data saved to file in PDF form by supplying a file path to the PDFOutputPath argument.

Validation data for further analysis (includes model predictions) Regression evaluation plots (box-plot version below) Regression performance metrics on holdout data (R2, MAE, MAPE, RMSE) Variable importance data.table and a formatted bar plot (below) Interaction importance data.table (below) Partial Dependence Calibration Plots and Box Plots (below)

Model calibration box-plot by every 5th percentile of the predicted values range (Image by author)

Variable importance plot (Image by author)

Below are two examples of the partial dependence calibration plots that are generated. You can tell the system how many plots you want generated by setting the NumParDepPlots argument to a specific value.

Example of one of the partial dependence box-plots. GroupVar is the interaction of the categorical variables. (Image by author)

Partial dependence calibration plot for categorical variables (Image by author)

Pairwise interaction importance is a new offering from the CatBoost team.