Predicting Volcanic🌋 Eruption With tsfresh & lightGBM | by Ekhtiar Syed | Towards Data Science

A volcanic eruption can take tens of thousands of lives, destroy millions of dollars of infrastructure, and cause a great deal of inconvenience. What if scientists could anticipate volcanic eruptions as they predict the weather? That was the aim of the INGV — Volcanic Eruption Prediction Kaggle competition organized by the Istituto Nazionale di Geofisica e Vulcanologia (INGV).
I created a simple solution for this competition with tsfresh and lightGBM, and it ranked 18th place on the competition’s public leaderboard. In this article, I want to describe my solution in detail. I also cleaned up and documented my competition code as a public kernel on Kaggle. You can view the full code for this article with the link below.
www.kaggle.com
If you are starting off with multivariate time-series data, I hope reading this article will help you with the following two topics:
Automatically generating features for multivariate time-series data.Designing a robust machine learning model for time-series prediction.
Automatically generating features for multivariate time-series data.
Designing a robust machine learning model for time-series prediction.
Feature generation for time-series data can be time-consuming. However, many of the techniques/features we want to generate for time-series data are well known and standardized. With tsfresh you can automatically calculate a large number of these known time series features effortlessly. It even has built-in methods to evaluate the feature importance of the generated features. The entire project is opensource, and you can view the repo on GitHub.
github.com
Using tsfresh is fairly simple. The API is very clean, you just describe the features you want from their exhaustive list of available features, and ask tsfresh to extract them. However, at the start of exploration, it is very common to not know the kind of features you want. So tsfresh also ships feature extraction settings pre-built. I will describe a few of them below:
ComprehensiveFCParameters: For processing with all the time-series features available. Useful when you don’t know which features you need.
MinimalFCParameters: Contains a very small and basic set of time-series features (min, max, standard deviation, and so on). Useful for setting up a quick test before kicking your pipeline off.
EfficientFCParameters: Contains all the features that are not of high computation time. Useful if you have a really large dataset and have no way to process the comprehensive set of features.
Below is a code snippet of how you would use one of these settings to extract time-series features on your dataframe.
import pandas as pdfrom tsfresh.feature_extraction import extract_featuresfrom tsfresh.feature_extraction import ComprehensiveFCParametersdf = pd.read_csv('/path/to/file.csv')settings = ComprehensiveFCParameters()features_df = extract_features(df, default_fc_parameters=settings)
Once you have processed enough data, you can apply different techniques to understand the importance of the features. Once you know the features you want, you can “hand-write” the settings you want to extract from your dataset. There is also a handy trick, to go from column names to settings using the from_columns function. Below is a code snippet of how this would work.
from tsfresh.feature_extraction.settings import from_columns# fictions function to get top features from your datasettop_features_column_names = features_df.get_top_features()# get the config for the selected columnsconfig = from_columns(top_features_column_names)# get selected features for another dataframeselected_features_df = extract_features(another_df, column_id =   'id', kind_to_fc_parameters = config)
In the case of this competition, I first processed the training dataset with ComprehensiveFCParameters. Then I removed columns that are highly correlated and quasi constant and generated custom settings to process the test set. More details on this are given in the next section. Note that lightGBM will not accept the column names generated by tsfresh because they often have the minus sign (-). You have to write a small function to match the names between the two.
This competition provides ten minutes of seismic sensor readings and a time to erupt for each segment. In the training dataset, we have 4231 segments, and in the test dataset, we have 4520 segments. We need to predict the time to erupt for each segment of test data. For each segment, we have 60,000 data-points (for 10 minutes); so we have good quality and high-frequency data.
As some features are very computation heavy, for each segment I divided the data points into 6 batches (10,000 data-points per batch). I could have also down-sampled the total number of data points, and that would be more conventional, but I wanted to go with the non-conventional.
At present, we have 2854 features for our dataset. At first, I processed all the possible features of the training dataset (ComprehensiveFCParameters). This generated almost 8000 features. Then I removed highly correlated columns and quasi-constant features. This brought our features down to 2854. I also applied a recursive feature elimination to take the top 501 features (500 seemed too goodie-to-shoe of a number). These columns are hard-coded in this notebook. The full dataset is also uploaded as a public dataset on Kaggle, and you can view it from the following link:
www.kaggle.com
For predicting the time of an eruption, I used a tree-based learning algorithm with the lightGBM (LGBM) framework. It has been a go-to framework for many people on Kaggle, and this was also my starting point. At first, I used a vanilla lightGBM model using the top features; and it yielded decent results. From here the most common method to improve my result could be creating more models with algorithms and averaging the results. However, in a real-life production model, a blend of multiple models is difficult to manage. So, instead of mixing multiple models, I decided to do something a little different.
For improving my results, I created two layers of lightGBM models. I call this a generalized-specialized layering. At first, a single model is used for making an initial prediction on the full dataset. Then on the second layer, I trained multiple models. These models are more specialized as they are trained with data from a specific range of time to eruption. To deal with a greater degree of error from the first model, I created overlaps between the ranges of the second models. One last bit of trivial detail, I am also using a min-max scaling to scale our data before fitting it, to get the best performance.
For predicting the test data, the model from the first layer is used first to make an initial prediction. Based on this prediction, we use a specific model of the second layer to make a more accurate prediction. As we have overlaps in the time to eruption ranges, it can be that a single test segment is evaluated by two models. Finally, we have six outputs or predictions for each segment in our test dataset. We then take the median of these outputs to get our final predicted time to erupt. I tried to visualize this process in the diagram below (to the best of my abilities).
LightGBM has a seriously high number of hyperparameters to tune from. Like all tree-based algorithms, lightGBM is also well-known for overfitting. One of the main reasons for overfitting is model complexity. I prevented lightGBM from building over-complex models by controlling the number of leaves (num_leaves) and maximum depth (max_depth) parameters. I also used a large number of estimators (n_estimators) and regularization (lambda_l1 and lambda_l2) to reduce overfitting. The total number of iterations (num_iterations) was also controlled, and early stopping (early_stopping_round) was also introduced to prevent overfitting. Finally, I also tried to introduce a lot of randomness by using extra trees, feature fraction, and bagging_fraction. Below is the hyperparameter configuration for the second layer of models.
params = {         'application':'regression',         'boosting ': 'dart',         'num_iterations':8000,          'learning_rate':0.05,          'num_leaves': 95,         'extra_trees': True,         'feature_fraction':0.8,          'bagging_fraction':0.9,         'lambda_l1':0.1,          'lambda_l2':0.1,          'min_split_gain':0.01,          'early_stopping_round':100,          'max_depth': 7,         'min_child_weight':40,          'n_estimators': 400,         'metric':'mae',         }
You will also notice that the first layer is much less complex (higher number of leaves and depth). Experiments showed that keeping the first model less complex than the second produces good results. This also enables the first model to generalize and the second model to specialize.
This competition uses a mean average score (MAE) and this solution had a private score of 4200311. This ranked 18th on the leaderboard, and I am pretty happy that my simple solution ranked that high in the end. The bar chart below shows the mean average score for the different ranges of time to eruption values. The bar chart on the left is from the first model, and the one on the right is from the second model.
As the second model can focus on a smaller range of time, they tend to do much better. Also, the overall score of the model improved. For both extreme ends (very quick or late time to erupt) I created one extra model. As the range between lowest and highest is really big, even the specialized models tend to lean more towards the middle of the range. These extra models for both extremes helped better predict the extreme cases.
This was an interesting Kaggle competition and a big thanks to the organizers for organizing it. Like with any other competition, I got to learn a lot from other Kaggler’s work while participating in this competition.
I wish I had the time to do a more systematic study and visualization of feature importance. I also think using multiple classifiers in the first layer instead of a single model might be a better approach. Oh well, I will take this experience with me to do better in the future.