Time Series Machine Learning Regression Framework | by Pourya | Towards Data Science

Time! One of the most challenging concepts in the Universe. I have studied Physics for many years and seen how most brilliant scientists have been struggling to deal with the concept of the time. In machine learning, even we are far from those complicated Physics theories, in which the common understanding of the concept of the time modifies, the existence of time and the sequence of observations in an ML problem can make the problem much more complicated.
A time series is a sequence of observations taken sequentially in time. Time series forecasting involves taking models then fit them on historical data then using them to predict future observations. Therefore, for example, min(s), day(s), month(s), ago of the measurement is used as an input to predict the
next min(s), day(s), month(s). The steps that are considered to shift the data backward in the time(sequence), called lag times or lags. Therefore, a time series problem can be transformed into a supervised ML by adding lags of measurements as inputs of the supervised ML. see Fig.3 right. Generally, explore the number of lags as a hyperparameter.
Cross-validation for time series is different from machine-learning problems that time or sequence is not involved. In the case of the absence of time, we select a random subset of data as a validation set to estimate the accuracy of the measurement. In time series, we often predict a value in the future. Therefore, the validation data always has to occur after the training data. There are two schemas sliding-window and Forward Chaining validation methods, that can be used for the time series CV.
Fig. 5 top shows the Sliding Window method. For this method, we train on n-data points and validate the prediction on the next n-data points, sliding the 2n training/validation window in time for the next step. Fig. 5 bottom shows Forward Chaining method. For this method, we train on the last n-data points and validate the prediction on the next m-data points, sliding the n+m training/validation window in time. In this way, we can estimate the parameters of our model. To test the validity of the model, we might use a block of data at the end of our time series which is reserved for testing the model with the learned parameters.
Fig 6. shows how the Forward Chaining CV works. Here, there is one lag. Thus, we train the model from the first to third second/min/hour/day, etc. then validate on forth and so on. Since, now, we get familiar with TS problem, let’s chose a time series problem and build a forecasting model.
Imagine a manager of a shop asked us to build an ML model to forecast the number of sales for the next week. The model must be run every Sunday and the result of the prediction must be reported on every Monday morning. Then, the manager can make a decision about the number of orders for the week. The manager provides us the data of sales 811 products for 52 weeks. The sales data can be found in the UCI Repository or kaggle.
Let’s look at the data.
Many data scientists might create a single model for each product to forecast the number of sales. And while this can work well, we may have problems due to having only 52 data points for each model, which is really low! Although this approach is possible, it might not the best solution. Besides, if there are interactions among the number of sales of two or more products, we might miss their interactions by building a single model for each product. Therefore, at this post, we will investigate how we can build a multiple time series forecasting models.
The raw data has a column for the product code and 52 weeks columns for sales. First, we are going to create a new data frame by melting the data on the weeks. Thus, the new data frame has three columns, product code, week and sales. Besides, the “W” & “P” are dropped from the week and the product respectively. So, let’s take look at the head and the tail of the new data frame
In order to get familiar with the dataset, the distribution of sales is plotted at Fig.7. It is seen that there are big numbers of products with a very small amount of sales, and the data skew toward the left as well. The effect of the issue on modeling will be discussed later.
Since the goal of this post is not Feature Engineering for TS, we will try to keep this part as simple as possible. Let’s create two features which usually are used for time series. One step back in time, 1-lag(shift =1) and the difference between the number of purchase a week ago(W 1) and its previous week, means, two weeks ago (W2). After that, since the lag and diff result in having null in the dataset, see Fig. 4, we drop them. Therefore, when we look at the head of the data frame, it starts from week = 2.
“ToSupervised” and “ToSupervisedDiff” classes, code 1 and code 2, shown at the coding section, are used to obtain the new data frame through a simple pipeline:
steps = [('1_step',          ToSupervised('Sales','Product_Code',1)),         ('1_step_diff',          ToSupervisedDiff('1_Week_Ago_Sales',          'Product_Code',1,dropna=True))]super_1 = Pipeline(steps).fit_transform(df)
Now, the data has a proper shape for using it at supervised ML.
The other problem when we are working on time series, we have to deal with its CV for time series. We chose forward-chaining for the model validation. To avoid having a very good model in a small number of weeks, we will use every week from 40 to 52, repeating the process for one at a time, and compute the score. Therefore, the k-fold code at this schema can be found at C. 3.
kf = Kfold_time(target='Sales',date_col = 'Week',                    date_init=40, date_final=52)
Since this post is just a demonstration, I am not separating a test dataset. In a real project, always keep some periods out, as a test dataset, to evaluate the model on unseen data.
Since the problem is Regression, there are several well-known metrics to evaluate the model, such as Mean Square Error (MSE), Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), Root Mean Squared Log Error (RMSLE), R-squared, so forth. Each of these metrics has its own use case and they punish the error differently while there are some similarities among them as well. In this post, RMSLE is chosen to evaluate the model.
Usually, when we build a model, we might come up with a very simple assumption that we expect using ML can improve it. Here, let’s assume that the number of each product is sold at the current week, it will be the same for the next week. It means that if product-1 is sold 10 times at week-1, its sales number also will be the same for week-2. It generally is not a bad assumption. So, let’s consider this assumption as our baseline model.
The baseline model is coded at C. 5, let’s see how the baseline model works
base_model = BaseEstimator('1_Week_Ago_Sales')errors = []for indx,fold in enumerate(kf.split(super_1)):    X_train, X_test, y_train, y_test = fold    error = base_model.score(X_test,y_test,rmsle)    errors.append(error)    print("Fold: {}, Error: {:.3f}".format(indx,error))    print('Total Error {:.3f}'.format(np.mean(errors)))
Fold: 0, Error: 0.520Fold: 1, Error: 0.517Fold: 2, Error: 0.510Fold: 3, Error: 0.508Fold: 4, Error: 0.534Fold: 5, Error: 0.523Fold: 6, Error: 0.500Fold: 7, Error: 0.491Fold: 8, Error: 0.506Fold: 9, Error: 0.505Fold: 10, Error: 0.522Fold: 11, Error: 0.552Total Error 0.516
Here, fold 0 to 11 indicate week = 40 to week = 52. The mean of RMSLE for the baseline model over these 12 weeks is 0.51. This can be considered as a big error which might originate from the huge number of items were sold very little amounts as shown in Fig.7.
Now, we will apply the ML to improve the baseline prediction. Let’s define a Time Series Regressor Class, C. 5, which work with our time series cross-validation. This class gets the cv and model and it returns, the model prediction and its score. There is a wide range of ML algorithms that can be used as an estimator. Here, we select a Random Forest. Simply, RF can be considered as a combination of bagging and randomly selecting columns of features on top of decision trees. Therefore, it reduces the variance of the prediction of a decision tree model. So, it usually has better performance than a single tree and has weaker performance than ensemble methods which are designed to reduce the bias error of the decision tree model.
model = RandomForestRegressor(n_estimators=1000,                               n_jobs=-1,                                random_state=0)steps_1 = [('1_step',              ToSupervised('Sales','Product_Code',1)),           ('1_step_diff',              ToSupervisedDiff('1_Week_Ago_Sales',                       'Product_Code',1,dropna=True)),           ('predic_1',              TimeSeriesRegressor(model=model,cv=kf))]super_1_p = Pipeline(steps_1).fit(df)Model_1_Error = super_1_p.score(df)
we get
Fold: 0, Error: 0.4624Fold: 1, Error: 0.4596Fold: 2, Error: 0.4617Fold: 3, Error: 0.4666Fold: 4, Error: 0.4712Fold: 5, Error: 0.4310Fold: 6, Error: 0.4718Fold: 7, Error: 0.4494Fold: 8, Error: 0.4608Fold: 9, Error: 0.4470Fold: 10, Error: 0.4746Fold: 11, Error: 0.4865Total Error 0.4619
It seems that the model works and the error decrease. Let’s add more lags and evaluate the model again. Since we built the pipeline, add more lags would be very simple.
steps_3 = [('1_step',            ToSupervised('Sales','Product_Code',3)),           ('1_step_diff',            ToSupervisedDiff('1_Week_Ago_Sales','Product_Code',1)),           ('2_step_diff',            ToSupervisedDiff('2_Week_Ago_Sales','Product_Code',1)),           ('3_step_diff',            ToSupervisedDiff('3_Week_Ago_Sales',                  'Product_Code',1,dropna=True)),           ('predic_3',            TimeSeriesRegressor(model=model,cv=kf,scoring=rmsle))]super_3_p = Pipeline(steps_3).fit(df)
Fold: 0, Error: 0.4312Fold: 1, Error: 0.4385Fold: 2, Error: 0.4274Fold: 3, Error: 0.4194Fold: 4, Error: 0.4479Fold: 5, Error: 0.4070Fold: 6, Error: 0.4395Fold: 7, Error: 0.4333Fold: 8, Error: 0.4387Fold: 9, Error: 0.4305Fold: 10, Error: 0.4591Fold: 11, Error: 0.4534Total Error 0.4355
It seems that the error of the prediction reduces again and the model learns more. We can continue to add lags and see how the performance of the model changes; however we will postpone this process until we use LGBM as an estimator.
The distribution of sales, Fig. 7 shows that the data skews toward the low sales number or to the left. Usually, Log transforms are useful when applied to skewed distributions as they tend to expand the values which fall in the range of lower magnitudes and tend to compress or reduce the values which fall in the range of higher magnitudes. The interpretability of the models changes when we do statistical transformations since the coefficients no more tell us about the original features but the transformed features. Therefore, while we apply np.log1p on the sales number to transform its distribution to become closer to the normal distribution, we also apply np.expm1 on the result of the prediction, see C. 6, TimeSeriesRegressorLog. Now, we repeat the calculation with the mentioned transformation
steps_3_log = [('1_step',                 ToSupervised('Sales','Product_Code',3)),               ('1_step_diff',                 ToSupervisedDiff('1_Week_Ago_Sales',                                    'Product_Code',1)),               ('2_step_diff',                 ToSupervisedDiff('2_Week_Ago_Sales',                                    'Product_Code',1)),               ('3_step_diff',                 ToSupervisedDiff('3_Week_Ago_Sales',                                    'Product_Code',1,dropna=True)),               ('predic_3',                 TimeSeriesRegressorLog(model=model,                                     cv=kf,scoring=rmsle))]super_3_p_log = Pipeline(steps_3_log).fit(df)
so we have
Fold: 0, Error: 0.4168Fold: 1, Error: 0.4221Fold: 2, Error: 0.4125Fold: 3, Error: 0.4035Fold: 4, Error: 0.4332Fold: 5, Error: 0.3977Fold: 6, Error: 0.4263Fold: 7, Error: 0.4122Fold: 8, Error: 0.4301Fold: 9, Error: 0.4375Fold: 10, Error: 0.4462Fold: 11, Error: 0.4727Total Error 0.4259
This shows that the performance of the model improves and the error reduces again.
Now, it is time to use a stronger ML estimator to improve forecasting. We chose LightGBM as a new estimator. So let’s repeat the calculation
model_lgb = LGBMRegressor(n_estimators=1000, learning_rate=0.01)steps_3_log_lgbm = [('1_step',                       ToSupervised('Sales','Product_Code',3)),                    ('1_step_diff',                       ToSupervisedDiff('1_Week_Ago_Sales',                                          'Product_Code',1)),                    ('2_step_diff',                       ToSupervisedDiff('2_Week_Ago_Sales',                                          'Product_Code',1)),                    ('3_step_diff',                       ToSupervisedDiff('3_Week_Ago_Sales',                                          'Product_Code',1,                                                dropna=True)),                   ('predic_3',                       TimeSeriesRegressorLog(model=model_lgb,                                               cv=kf,scoring=rmsle))]super_3_p_log_lgbm = Pipeline(steps_3_log_lgbm).fit(df)
then,
Fold: 0, Error: 0.4081Fold: 1, Error: 0.3980Fold: 2, Error: 0.3953Fold: 3, Error: 0.3949Fold: 4, Error: 0.4202Fold: 5, Error: 0.3768Fold: 6, Error: 0.4039Fold: 7, Error: 0.3868Fold: 8, Error: 0.3984Fold: 9, Error: 0.4075Fold: 10, Error: 0.4209Fold: 11, Error: 0.4520Total Error 0.4052
Again, we successfully improve the prediction.
At this section, we are going to tune the number of steps (lags/diffs). I intentionally postpone tuning the steps this section after using LGBM as the regressor because it is faster than RF. Fig. 8 clearly shows that by adding more steps to the model the error reduces; however, as we expect, it is seen that after passing a threshold around steps = 14, adding further steps will not reduce the error significantly. You might be interested in defining an error threshold for stopping this process. Steps= 20 is chosen for the rest of the calculation. Please check code C 7. A and B for tuning.
model_lgbm = LGBMRegressor(n_estimators=1000, learning_rate=0.01)list_scores2 = stepsTune(df,TimeSeriesRegressorLog(model=model_lgbm,                           scoring=rmsle,cv=kf,verbosity=False),20)
At this part, we are going to implement the grid search method in the way that we can apply it through the pipeline, see Code 8 A and B. C.8.A code is borrowed from Sklearn library. The aim of this part is not building a fully tuned model. We trying to show how the workflow is. After slightly tuning, the error becomes
RMSLE= 0.3868
for these two hyperparameters {‘learning_rate’: 0.005, ‘n_estimators’: 1500}.
params = {'n_estimators':[100,500,1000,1500,2000],         'learning_rate':[0.005,.01,.1]}steps_20 = getDataFramePipeline(20)super_20 = Pipeline(steps_20).fit_transform(df)model_lgbm2 = LGBMRegressor(random_state=0)tune_lgbm =TimeSeriesGridSearch(model = model_lgbm2, cv = kf,                  param_grid=params,verbosity=False,scoring=rmsle)
When the best tuning hyperparameters are at the edge of tuning parameters means that we must reconsider the range of hyperparameters and recalculate the model although we will not do it at this post.
Fig 9. shows the prediction value versus the sales value for week 52. It is seen that the model works well for the sales numbers up to 15; however, it forecasts poorly for the sales around 30. As we discuss in Fig. 7, we might build different models for the different range of sales to overcome this problem and to have a more robust forecasting model although further modeling is beyond this post and this post already is so long.
Finally, Fig. 10 shows all of our attempts to forecast the sales. We started with a very simple assumption as the baseline and we tried to improve it by using different lags/diff, statistical transformation, and applying different machine learning algorithms to improve the forecasting. The baseline error was 0.516 and the tuned model error was 0.3868 which means 25% reduction in the error.
There are still many ways to improve the presented model, for example, dealing with products as categorical variables properly, more extensive feature engineer, tuning hyperparameters, using various machine learning algorithms and blending and stacking.
We built a time series forecasting pipeline to predict weekly sales transaction. We started with a simple logical assumption as a baseline model; then, we could reduce the baseline error 25% by building a pipeline including a basic feature engineering, a statistical transformation and applying Random forest and LGBM and finally tuning them. Besides, we discussed different time series cross-validation methods. Moreover, we show how we can use Sklearn base classes to build a pipeline.
The complete code of this post can be found on my GitHub.
Code 1.
class ToSupervised(base.BaseEstimator,base.TransformerMixin):        def __init__(self,col,groupCol,numLags,dropna=False):                self.col = col        self.groupCol = groupCol        self.numLags = numLags        self.dropna = dropna            def fit(self,X,y=None):        self.X = X        return self        def transform(self,X):        tmp = self.X.copy()        for i in range(1,self.numLags+1):            tmp[str(i)+'_Week_Ago'+"_"+self.col] =              tmp.groupby([self.groupCol])[self.col].shift(i)                     if self.dropna:            tmp = tmp.dropna()            tmp = tmp.reset_index(drop=True)                                        return tmp
Code 2.
class ToSupervisedDiff(base.BaseEstimator,base.TransformerMixin):        def __init__(self,col,groupCol,numLags,dropna=False):                self.col = col        self.groupCol = groupCol        self.numLags = numLags        self.dropna = dropna            def fit(self,X,y=None):        self.X = X        return self        def transform(self,X):        tmp = self.X.copy()        for i in range(1,self.numLags+1):            tmp[str(i)+'_Week_Ago_Diff_'+"_"+self.col] =                tmp.groupby([self.groupCol])[self.col].diff(i)                     if self.dropna:            tmp = tmp.dropna()            tmp = tmp.reset_index(drop=True)                    return tmp
Code 3.
from itertools import chainclass Kfold_time(object):        def __init__(self,**options):                        self.target     = options.pop('target', None)        self.date_col   = options.pop('date_col', None)        self.date_init  = options.pop('date_init', None)        self.date_final = options.pop('date_final', None)        if options:            raise TypeError("Invalid parameters passed: %s" %                               str(options))                    if ((self.target==None )|(self.date_col==None )|            (self.date_init==None )|(self.date_final==None )):                          raise TypeError("Incomplete inputs")        def _train_test_split_time(self,X):        n_arrays = len(X)        if n_arrays == 0:            raise ValueError("At least one array required as input")         for i in range(self.date_init,self.date_final):            train = X[X[self.date_col] < i]            val   = X[X[self.date_col] == i]             X_train, X_test = train.drop([self.target], axis=1),                                val.drop([self.target], axis=1)            y_train, y_test = train[self.target].values,                               val[self.target].values            yield X_train, X_test, y_train, y_test     def split(self,X):        cv_t = self._train_test_split_time(X)        return chain(cv_t)
Code 4.
class BaseEstimator(base.BaseEstimator, base.RegressorMixin):    def __init__(self, predCol):        """            As a base model we assume the number of sales             last week and this week are the same            Input:                     predCol: l-week ago sales        """        self.predCol = predCol    def fit(self, X, y):        return self    def predict(self, X):        prediction = X[self.predCol].values        return prediction    def score(self, X, y,scoring):                prediction = self.predict(X)            error =scoring(y, prediction)        return error
Code 5.
class TimeSeriesRegressor(base.BaseEstimator, base.RegressorMixin):        def __init__(self,model,cv,scoring,verbosity=True):        self.model = model        self.cv = cv        self.verbosity = verbosity        self.scoring = scoring                         def fit(self,X,y=None):        return self                def predict(self,X=None):                pred = {}        for indx,fold in enumerate(self.cv.split(X)):            X_train, X_test, y_train, y_test = fold                self.model.fit(X_train, y_train)            pred[str(indx)+'_fold'] = self.model.predict(X_test)                    prediction = pd.DataFrame(pred)            return prediction    def score(self,X,y=None):        errors = []        for indx,fold in enumerate(self.cv.split(X)):            X_train, X_test, y_train, y_test = fold                self.model.fit(X_train, y_train)            prediction = self.model.predict(X_test)            error = self.scoring(y_test, prediction)            errors.append(error)            if self.verbosity:                print("Fold: {}, Error: {:.4f}".format(indx,error))         if self.verbosity:            print('Total Error {:.4f}'.format(np.mean(errors)))        return errors
Code 6.
class TimeSeriesRegressorLog(base.BaseEstimator,                                 base.RegressorMixin):        def __init__(self,model,cv,scoring,verbosity=True):        self.model = model        self.cv = cv        self.verbosity = verbosity        self.scoring = scoring                        def fit(self,X,y=None):        return self                def predict(self,X=None):                pred = {}        for indx,fold in enumerate(self.cv.split(X)):            X_train, X_test, y_train, y_test = fold                self.model.fit(X_train, y_train)            pred[str(indx)+'_fold'] = self.model.predict(X_test)                    prediction = pd.DataFrame(pred)            return prediction    def score(self,X,y=None):#**options):        errors = []        for indx,fold in enumerate(self.cv.split(X)):            X_train, X_test, y_train, y_test = fold                self.model.fit(X_train, np.log1p(y_train))            prediction = np.expm1(self.model.predict(X_test))            error = self.scoring(y_test, prediction)            errors.append(error)            if self.verbosity:                print("Fold: {}, Error: {:.4f}".format(indx,error))        if self.verbosity:            print('Total Error {:.4f}'.format(np.mean(errors)))        return errors
Code 7.
A:
def getDataFramePipeline(i):    steps = [(str(i)+'_step',              ToSupervised('Sales','Product_Code',i))]    for j in range(1,i+1):        if i==j:            pp = (str(j)+'_step_diff',                  ToSupervisedDiff(str(i)+'_Week_Ago_Sales',                                   'Product_Code',1,dropna=True))            steps.append(pp)        else:            pp = (str(j)+'_step_diff',                    ToSupervisedDiff(str(i)+'_Week_Ago_Sales',                                   'Product_Code',1))            steps.append(pp)                return steps
B:
from tqdm import tqdmdef stepsTune(X,model,num_steps,init=1):    scores = []    for i in tqdm(range(init,num_steps+1)):        steps = []        steps.extend(getDataFramePipeline(i))        steps.append(('predic_1',model))        super_ = Pipeline(steps).fit(X)        score_ = np.mean(super_.score(X))        scores.append((i,score_))            return scores
Code 8.
A:
from collections.abc import Mapping, Sequence, Iterablefrom itertools import productfrom functools import partial, reduceimport operatorclass TimeGridBasic(base.BaseEstimator, base.RegressorMixin):        def __init__(self,param_grid):                    if not isinstance(param_grid, (Mapping, Iterable)):                raise TypeError('Parameter grid is not a dict or '                                'a list ({!r})'.format(param_grid))        if isinstance(param_grid, Mapping):                # wrap dictionary in a singleton list to support                  either dict                # or list of dicts                param_grid = [param_grid]        if isinstance(param_grid, Mapping):                # wrap dictionary in a singleton list to support                  either dict                # or list of dicts                param_grid = [param_grid]        # check if all entries are dictionaries of lists        for grid in param_grid:            if not isinstance(grid, dict):                raise TypeError('Parameter grid is not a '                                'dict ({!r})'.format(grid))            for key in grid:                if not isinstance(grid[key], Iterable):                    raise TypeError('Parameter grid value is not                                     iterable '                                    '(key={!r}, value={!r})'                                    .format(key, grid[key]))        self.param_grid = param_grid                    def __iter__(self):        """Iterate over the points in the grid.        Returns        -------        params : iterator over dict of string to any            Yields dictionaries mapping each estimator parameter to            one of its            allowed values.        """        for p in self.param_grid:            # Always sort the keys of a dictionary, for             reproducibility            items = sorted(p.items())            if not items:                yield {}            else:                keys, values = zip(*items)                for v in product(*values):                    params = dict(zip(keys, v))                    yield params
B:
class TimeSeriesGridSearch(TimeGridBasic,base.BaseEstimator,                              base.RegressorMixin):            def __init__(self,**options):                self.model      = options.pop('model', None)        self.cv         = options.pop('cv', None)        self.verbosity  = options.pop('verbosity', False)        self.scoring    = options.pop('scoring', None)        param_grid      = options.pop('param_grid', None)        self.param_grid = TimeGridBasic(param_grid)                if options:            raise TypeError("Invalid parameters passed: %s" %                              str(options))        if ((self.model==None )| (self.cv==None)):            raise TypeError("Incomplete inputs")                            def fit(self,X,y=None):        self.X = X        return self    def _get_score(self,param):        errors = []        for indx,fold in enumerate(self.cv.split(self.X)):            X_train, X_test, y_train, y_test = fold                self.model.set_params(**param).fit(X_train, y_train)            prediction = self.model.predict(X_test)            error = self.scoring(y_test, prediction)            errors.append(error)            if self.verbosity:                print("Fold: {}, Error: {:.4f}".format(indx,error))        if self.verbosity:            print('Total Error {:.4f}'.format(np.mean(errors)))                                return errors     def score(self):        errors=[]        get_param = []        for param in self.param_grid:                        if self.verbosity:                print(param)                            errors.append(np.mean(self._get_score(param)))            get_param.append(param)        self.sorted_errors,self.sorted_params =           (list(t) for t in zip(*sorted(zip(errors,get_param))))                return self.sorted_errors,self.sorted_params            def best_estimator(self,verbosity=False):        if verbosity:            print('error: {:.4f} \n'.format(self.sorted_errors[0]))            print('Best params:')            print(self.sorted_params[0])        return self.sorted_params[0]