Weighted Ensemble Forecasting (FFORMA Method)
By: Justin Joyce
Tags: Ensemble Learning, Time Series Forecasting, Time Series Analysis, Ensemble

In the 2018 M4 forecasting competition , the two top models combined elements of traditional statistical modeling with modern machine Learning. The first place model used a combination of ETS and RNNs and the second place model used a meta-learning forecast model.

The second place model was pioneered by Thiyanga S Talagala, Rob J Hyndman and George Athanasopoulos at the University of Monash. Their Feature-based FORecast Model Averaging (FFORMA) model extracts features from multiple time series , trains a forest model on the “best model” from a selection of models then predicts the probability of each model for a weighted ensemble prediction.

This article goes through a step by step code example of the second place team’s method which can be reviewed in detail on the link below http://htmlpreview.github.io/?https://github.com/robjhyndman/M4metalearning/blob/master/docs/M4_methodology.html

Some of the features included in this model are:

Linearity

Seasonality

Trend

Hurst Coefficient (long term patterns or not)

Spectral Entropy (noise or not)

ACF and PACF values (used in ARIMA modelling)

Code Demo

In this demo, I am using 36 monthly time series from https://fred.stlouisfed.org/ based on global prices of products (Wheat, Wood, Coffee etc). The github repo https://github.com/GinoWoz1/Learnings contains the data under fforma_method_data. In the end, the FFORMA model performs best on 3 out of 10 time series on the test dataset. Personally I have seen it do well on product forecasts when there are hundreds of thousands of time series (better representation of features) but the goal of this article is to just provide a resource for those who want to try it out.

Load Data

#load pacman

library(pacman)



#load related packages

p_load(stringr,caret,purrr,fabletools,tidyr,dplyr,magrittr,tsibble,feasts,future,tidymodels,cowplot,gratis,DataExplorer) path = "~//GitHub/Learnings/fforma_method_data/" # read list of global prices

files = list.files(path = path, pattern= "*Global.*.csv", all.files=FALSE,

full.names=FALSE) read_files = function(accessor){



file = read.csv(str_glue(path,'/',files[accessor])) %>%

set_colnames(c('DATE','value')) %>%

mutate(dataset = gsub(".csv",'',files[accessor]))



} # create tsibbles for each data set

df_monthly = map_dfr(1:length(files),read_files) %>%

mutate(month = tsibble::yearmonth(as.character(DATE))) %>%

as_tsibble(key = dataset,index =month) %>%

fill_gaps(value =0)

Feature Extraction and Exploration

Next we extract features from the time series using Feasts.I plot a sample of the features below; the ACF values of these time series are high suggesting ARIMA may work best for the time series. Definitions of the features can be found at the bottom of this link here.

Extract features and plot

# extract time series features from the time series using feasts

features_df = df_monthly %>% features(value,feature_set("feasts")) # load libraries

p_load(caret,rsample) # plot the densities of the time series

DataExplorer::plot_density(features_df)

Feature Distribution — 30 Global Price Datasets

Generate Time Series

The “best model” for the above time series were mostly ARIMA and features reflect this. Since we will be training a classifier against the “best model” and its features, it may be best if we generate some artificial time series. For this I use GRATIS .

Generate artificial time series

# generate 3 sets of 100 time series based on nComp from GRATIS

generate_time_series = function(accessor){ x = generate_ts(n.ts = 100,freq=12,nComp =accessor,n=97,output_format = "tsibble")

}



# generate time series via future_map

# create 100 time series of 3,4 and 5 component mixture models plan(multisession, gc = TRUE,workers = 3)

ts = furrr::future_map(3:5,generate_time_series) future:::ClusterRegistry("stop")

Next I unpack the list of lists of time series and grab the features and tsibbles.

#function to create a mappable df to unpack the time series and feaqtures

map_df = function(accessor){

empty = tibble(rows = 1:100) %>%

mutate(list_element = accessor)

} #mapping df

df_mapped = map_dfr(1:3,map_df) %>%

mutate(row = row_number()) # function to unpack the time series from list of lists

get_timeseries = function(list_number,length,row){

time_series = ts[[list_number]][[length]] %>% as_tibble() %>% mutate(dataset = str_c("time_series_",row)) %>%

set_colnames(c('month','value','dataset'))

time_series

} # function for getting features from the list of list of time series

get_features = function(list_number,length,row){

feats = ts[[list_number]][[length]] %>% as_tsibble() %>% features(value,feature_set(pkgs = "feasts")) %>%

mutate(dataset = str_c("time_series_",row))

feats

} # generated time series

df_generated = pmap_dfr(list(df_mapped$list_element,df_mapped$rows,df_mapped$row),get_timeseries) # generated time series features

features_df_generated = pmap_dfr(list(df_mapped$list_element,df_mapped$rows,df_mapped$row),get_features)

Then we plot the time series and then explore the generated time series features. On the second image below, the ACF features now has a long tailed distribution showing more proper representation of the time series features.

Plot Generated Time Series

# initiate list to collect time series plots

plot_list = list()

for(i in 1:length(ts[[1]])){

g = ts[[1]][[i]] %>% autoplot()

plot_list[[i]] = g

} # use cowplot to plot time series

plot_grid(plotlist = plot_list,ncol = 10)

Sample of Randomly Generated Time Series (from GRATIS)

Plot Features of Generated Time Series

Generated Time Series Features — note ACF1 feature distribution changed

Modelling

Next we train the Global Prices Data and Generated time series in fable using a sliding window of four years then predicting each subsequent month.

Create Train and Test set from the Global Prices Data

# create test and train

split <- initial_split( features_df, prop = .7,seed =3)

train_df <- training(split)

test_df <- testing(split)

Create Sliding Window Cross Validation Data

# create cv dataset for Global Prices data

cv_ts = df_monthly %>% filter(dataset %in% (train_df %>% pull(dataset))) %>%

filter(lubridate::year(DATE) >= 2013) %>% # only choose last 8 years of data

stretch_tsibble(.init = 48, .step = 1) %>%

fill_gaps(value = 0) %>%

filter(.id > 36) %>% select(dataset,month,.id,value) # create cv dataset for generated time series

cv_ts_generated = df_generated %>% as_tsibble(key = dataset,index = month) %>%

stretch_tsibble(.init=48,.step = 1) %>%

fill_gaps(value = 0)%>%

filter(.id > 36)

Train CV model

# set number of cores

num_cores = 8 # function to take the time series and parallelize training

train_data = function(ts){



# mappings to then be used in future_map

data_mappings = ts %>% distinct(dataset,.id)



# model and predict

fable_generated =

future_map2_dfr(data_mappings$dataset,data_mappings$.id,

~ ts %>% filter(ts$dataset == .x & .id == .y) %>%

model(snaive = fable::SNAIVE(value ~ lag(lag=12)),

naive = fable::NAIVE(value),

arima = fable::ARIMA(value),

theta = fable::THETA(value),

ets = fable::ETS(value)) %>%

forecast(h = "1 month"))

} # collect forecasts for generated time series and global prices time series

plan(multisession, gc = TRUE,workers = num_cores) # perform forecasts

global_forecasts = train_data(cv_ts )

generated_forecasts = train_data(cv_ts_generated) future:::ClusterRegistry("stop")

After training we collect the “best model” based on MASE score, which is a measure of whether a model is better than a naive one-step-ahead prediction. We then pump this into a xgboost model training against the “best model” and predict out the weights for the test data set.

Calculate “best model” and prepare test and training data

# best model from seasonal time series

best_model = global_forecasts %>%

fabletools::accuracy(df_monthly) %>%

group_by(dataset) %>%

filter(MASE == min(MASE,na.rm = TRUE)) # recipe for splitting data for training the forest model

item_recipe <- training(split) %>%

left_join(best_model %>% select(dataset,.model), by = c("dataset"

= "dataset")) %>%

recipe(.model ~.) %>%

step_rm(dataset) %>%

step_naomit(all_predictors()) %>%

prep() # prepare the test data set

item_testing <- item_recipe %>%

bake(testing(split)) # juice the training data set

item_training <- juice(item_recipe)

Train a XGBoost model against the “best model”

# set parallel plan to train model

all_cores <- parallel::detectCores(logical = FALSE) library(doParallel)

cl <- makePSOCKcluster(all_cores)

registerDoParallel(cl) #set model for classification

boost_model = boost_tree(mode = "classification",trees = 5000,learn_rate = 0.001) %>%

set_engine("xgboost") %>%

fit(.model ~., data = item_training %>% head(100))

Predict the Test Weights

# weights

weights_df <- boost_model %>%

predict(item_testing, type = "prob") #append the weights to the test data

model_weights = test_df %>%

select(dataset) %>%

bind_cols(weights_df)

Weights from Global Prices Data from Arima, ETS, naive,theta

Predict on test (to be used with weightings)

# get the predictions for 2020 for test df

cv_ts_test = df_monthly %>% filter(dataset %in% (test_df %>% pull(dataset))) %>%

filter(lubridate::year(DATE) >= 2013) %>% # only choose last 8 years of data

stretch_tsibble(.init = 48, .step = 1) %>%

fill_gaps(value = 0) %>%

filter(.id > 36) # run models for cross validation period

plan(multisession, gc = TRUE,workers = num_cores) df_fable_test =

cv_ts_test %>%

model(snaive = fable::SNAIVE(value ~ lag(lag=12)),

naive = fable::NAIVE(value),

arima = fable::ARIMA(value),

theta = fable::THETA(value),

ets = fable::ETS(value)) %>%

forecast(h = "1 month") future:::ClusterRegistry("stop")

Finally I weight the predictions , score the best model over that time period and plot an example of the weighted ensemble versus base models versus actuals.

Weight the Predictions

# apply the weights to the last 12 months

weighted_predictions = df_fable_test %>% as_tibble()%>%

filter(lubridate::year(month) == 2020 ) %>%

select(-c(.id,value)) %>%

pivot_wider(id_cols = c(dataset,month),names_from = .model,values_from = .mean) %>%

inner_join(model_weights, by =c("dataset" = "dataset")) %>%

mutate(naive_pred = naive * .pred_naive,

arima_pred = arima * .pred_arima,

theta_pred = theta * .pred_theta,

ets_pred = ets * .pred_ets) %>%

mutate(weighted_ensemble = naive_pred + theta_pred + ets_pred + arima_pred ) %>%

select(-c(contains('.pred'),contains('_pred')))

Score Weighted Ensemble (with Generated Time Series)

# score the models

weighted_predictions %>%

pivot_longer(cols = !c(dataset,month),values_to = 'predictions') %>%

inner_join(df_monthly, by =c("dataset" = "dataset","month" = "month"))%>%

group_by(dataset,name)%>%

yardstick::mase(value,predictions) %>%

group_by(dataset) %>%

filter(.estimate == min(.estimate))%>%

rename(best_model = name) %>%

select(-c(.estimator))

Best Model over last 12 months

Plot Global Cocoa Predictions

Black Actuals , Goldenrod Weighted Ensemble

As you can see above, the Weighted Ensemble Model above is similar to ARIMA (90% weighted ARIMA).

Thats it! Just a quick example in R to highlight how to run the FFORMA model. Other things that could be investigated further that I didn’t dive into.