AutoML for Time Series Forecasting | by Denis Vorotyntsev | Towards Data Science

Although I’m not a fan of nowadays AutoML, I enjoy participating in AutoML competitions. I believe that such competitions are a great learning opportunity and could give strong pipelines for future projects. AutoML competitions are something in between Kaggle-like competitions and LeetCode tasks: your code should give high scores, and it should be steady and fast.
Recently I took part in AutoSeries — AutoML competition on time-series data, in which I managed to get the first place among 40 competitors (15 in the finals). This post is an overview of my solution.
AutoSeries is one of the competitions of the Web Search and Data Mining (WSDM) conference. The competition is the 10th AutoML competition organized by 4Paradigm and ChaLearn. The previous ones were aimed to provide automated machine learning solutions for tabular data, computer vision, natural language processing, and speech recognition tasks. The full list of past challenges could be found on the official website of AutoML challenges.
This challenge aims at proposing automated solutions for the time series regression task. AutoSeries is restricted to multivariate regression problems, which come from different time series domains, including air quality, sales, work presence, city traffic, and other. Each dataset in the challenge is a tabular data whose features are of three main types: Id (could be multiple features or none), timestamp (each dataset has only one timestamp), other features (either numerical or categorical), and target to predict. The combination of Id features identifies a variable (a time series).
Participants have to submit code, which will be run in Docker container (CPU: 4 cores, 16 Gb RAM, no GPU). At first, the model is trained on full data, but during inference, it could be updated or retrained. The public leaderboard is calculated on five datasets, private — 5 new datasets without human intervention. The results of the private datasets determine the final ranking.
This section is devoted to the main parts of the final solution. The code of it is published here:
github.com
The solution consists of several steps. At first, common features are generated for time-series tasks. The number and types of these features are hyperparameters of the pipeline, which should be optimized for each task separately, but due to lack of computational time, I decided to generate the same features for all tasks and remove useless ones during the features selection stage.
The first batch of features are features based on the most critical numerical features. Numerical operations (addition, subtraction, multiplication, and division) of pairs of numerical features always improve the score of tree-based models because new features might reveal some hidden relations in the data.
Let’s take, for example, predicting the price of the apartment. The known information on the apartment’s floor (ap_floor) and the total number of floors (total_floors) in the building might be enriched by adding a new feature — relative_floor — which shows the relative position of the apartment’s floor in the building:
rel_floor = ap_floor / total_floors
This feature might help the model to understand that the third floor in row house (rel_floor is closer to 1) is not the same as the third in a skyscraper (rel_floor is closer to 0).
However, there are two significant problems with such features engineering strategy, if we perform numerical operations of all possible pairs: overfitting, which is especially vital in time-series tasks, and memory issues (16 RAM docker was used). To reduce the negative effect, a small subset of features is selected and used for pairs. It is done by fitting the whole dataset to a shallow LigtGBM model (10 trees). All used features are sorted by “gain” importance, i.e. a total sum of gains of splits that use the feature. Then top-n most important numerical features are selected for pairs.
The next batch of features is based on the time-series nature of the data: previous values and differences. I calculate the lagged values of target, most important numerical and categorical features, the difference between last value of the target (lag = 1), and lagged value of the target (lag > 1). These new features were among the most important ones.
The last batch is time-series features: year, month, day of the week, day of the year, and hour. I could add many more time-based features, such as a minute of the day, quartal of the year, and so on, but decided not to do that, so my solution would be universal. Treating these new features as categories sometimes improved scores, but in other cases, it significantly reduced it. I didn’t have enough computational time during inference for optimizing this hyperparameter (i.e., treating as numerical or categorical), so all of them are treated as numerical.
After generating new features, the baseline model is trained. The baseline model uses all initial and created features. It encodes categories with a CatBoost encoder and uses the target as-is. The model is trained in two steps:
At first, the data is split into the train and validation parts. Commonly, you want your train/validation/test split to emulate the usage of model in “production settings”. In the case of time series, that means that the model won’t be updated frequently, and you take 20–30% of data in the validation part (or use rolling window with the same proportions). In this competition, frequent updating of the model was possible, and thus, the validation part should be smaller: the validation part is 10% of the full training data. It is used for early stopping, i.e., to optimize the number of trees in boosting ensemble. After this step, the model could start making predictions, and all following steps are optional (bt crucial for a high score).The model is refit on full data using the optimal number of trees.
At first, the data is split into the train and validation parts. Commonly, you want your train/validation/test split to emulate the usage of model in “production settings”. In the case of time series, that means that the model won’t be updated frequently, and you take 20–30% of data in the validation part (or use rolling window with the same proportions). In this competition, frequent updating of the model was possible, and thus, the validation part should be smaller: the validation part is 10% of the full training data. It is used for early stopping, i.e., to optimize the number of trees in boosting ensemble. After this step, the model could start making predictions, and all following steps are optional (bt crucial for a high score).
The model is refit on full data using the optimal number of trees.
I used the solo LigthGBM model for making predictions. I tested CatBoost (too slow without GPU) and LinearModels (not accurate enough). I also tested bagging and training with different seeds to reduce the variance of the predictions, but these approaches took a lot of time and didn’t improve the scores enough to be included in the final solution.
I had too little time during inference for hyperparameters optimization, so I decided to narrow all possible hyperparameters combinations to the most promising ones, namely:
Treating categorical variables: treat categorical features as pandas category type and leave the headache to LightGBM or encode each category with CatBoost encoding (check my previous article on the encoding categories, if you haven’t’ read it yet).Target preprocessing: use target as is or calculate a new target for regression by differentiating: new_target(t)=target(t)-target(t-1). Differencing could help to overcome non-stationarity time series data. I also tested power transformation (taking root of target and Box-Cox) to reduce stationarity, but it didn’t improve the score enough to be included in the final solution.
Treating categorical variables: treat categorical features as pandas category type and leave the headache to LightGBM or encode each category with CatBoost encoding (check my previous article on the encoding categories, if you haven’t’ read it yet).
Target preprocessing: use target as is or calculate a new target for regression by differentiating: new_target(t)=target(t)-target(t-1). Differencing could help to overcome non-stationarity time series data. I also tested power transformation (taking root of target and Box-Cox) to reduce stationarity, but it didn’t improve the score enough to be included in the final solution.
towardsdatascience.com
Each set of parameters is first validated, and, if the new validation score is better — refit model. After selecting the best set of pipeline hyperparameters, the model starts feature selection: the model is refit using the top 5, 10, 20, etc. percents of the most important features (“gain” importance). If the scores improve — a new set of features is used for the final optional step — optimizing hyperparameters (RandomGrid).
Updating is simple: refit optimal model with full data (training data plus new train data). Yet frequent updating was crucial for a high score.
I put a lot of effort into this project, and the struggle paid off. I got third place in public leaderboard and a first place in the private leaderboard.
During the competition, I faced a lot of bugs, which cost me time and submissions. Bugs are inevitable, but there are notes, which helped me a lot if I used them from the start:
Log as much useful information as possible: columns in the data frame (columns order in train and test data might be different), data types(data types in train and test data frames might be different), time per training, time left for training, etc. It will help a lot to understand why submission failed or got a low score. For example, models in some of my submissions were not updated due to a silly bug, and I haven’t noticed that. Thus, my scores were much lower than they should be. A simple message print(“Model is updated!”) saved the day and helped me found the bug.The testing on unseen data is essential in AutoML. You may easily overfit your solution for a public part, and it could crash on the unseen data. That was my case — my submission failed on the first task. Try to collect more data at the beginning of the competition. New datasets should be diverse, for example, have missing values or strings in categories and other factors, which might crash your code. Test them with different time budgets: try to output even poor model when you’re tight on time.Organize your code in the “plug and play” manner: each part of the pipeline should not rely on the other parts. For example, I want to fit Linear Regression instead of LightGBM, would it be easy to do? If your code is well-organized, it will be. During this competition, I’ve made a great step towards a more clean, organized code. Yet sometimes I think it is still garbage, I see the progress.Do not use inplace operations in pandas because it is buggy. Simply don’t use it when possible.
Log as much useful information as possible: columns in the data frame (columns order in train and test data might be different), data types(data types in train and test data frames might be different), time per training, time left for training, etc. It will help a lot to understand why submission failed or got a low score. For example, models in some of my submissions were not updated due to a silly bug, and I haven’t noticed that. Thus, my scores were much lower than they should be. A simple message print(“Model is updated!”) saved the day and helped me found the bug.
The testing on unseen data is essential in AutoML. You may easily overfit your solution for a public part, and it could crash on the unseen data. That was my case — my submission failed on the first task. Try to collect more data at the beginning of the competition. New datasets should be diverse, for example, have missing values or strings in categories and other factors, which might crash your code. Test them with different time budgets: try to output even poor model when you’re tight on time.
Organize your code in the “plug and play” manner: each part of the pipeline should not rely on the other parts. For example, I want to fit Linear Regression instead of LightGBM, would it be easy to do? If your code is well-organized, it will be. During this competition, I’ve made a great step towards a more clean, organized code. Yet sometimes I think it is still garbage, I see the progress.
Do not use inplace operations in pandas because it is buggy. Simply don’t use it when possible.
I would like to thank organizers of the AutoSeries competition — 4Paradigm and ChaLearn teams. They did a great job: collected the data, prepared submissions scoring engine, wrote a robust baseline solution and answered the arising questions on the forum and in emails. Thank you for your work!