Time series analysis for predictive maintenance of turbofan engines | by Koen Peters | Towards Data Science

<disclaimer: I aim to showcase the effect of different methods and choices made during model development. These effects are often shown using the test set, something which is considered (very) bad practice but helps for educational purposes.>
Welcome to another installment of the ‘Exploring NASA’s turbofan dataset’ series. This will be the third analysis on FD001, where all engines run on the same operating condition and develop the same fault.
Initially we assumed the Remaining Useful Life (RUL) of the engines to decline linearly. In my last post we re-examined this assumption by clipping any values above 125. Clipping the RUL improved the baseline linear regression by 31% (from an RMSE of 31.95 to an RMSE of 21.90). We then switched to a Support Vector Regression and squeezed out another 6% improvement for a total RMSE of 20.54.
Today, we’ll focus on time series analysis to forecast when the engines are due for maintenance. But, before getting into the time series part, we first have to recap a few processing steps. Let’s get started!
First, we’ll import some libraries and read the data.
Looks good. Next, the linear declining RUL is computed (clipping is added later) and uninformative columns are dropped.
Nice, with that out of the way let’s discuss our approach.
Time series analysis and forecasting for this data set can be done with one of three general approaches:
Predicting sensor values and setting ‘alarm’ thresholds. The component is deteriorated to a state in which breakdown is imminent when a sensor is predicted to cross this threshold. There are a few caveats using this approach for the current dataset.(a) Every sensor has to be modelled individually, therefore we cannot make use of interactions between sensors.(b) it requires a lot of domain knowledge for setting the appropriate thresholds and (c) it doesn’t directly inform us about our target feature, the Remaining Useful Life (RUL).Vector Autoregression (VAR) type models. These types of models can handle multivariate timeseries, however the model initially creates lags of both our X and Y variables. Normally in timeseries the past values of Y play a large role in determining future values of Y. But since we assume our values for Y to be constant or linearly declining in the training set, incorporating these self-defined targets in the model and using them for prediction would highly influence model results and overemphasize the effect of Yt-1 on Yt. You could write some custom code to set the coefficients for the weights of the lagged Y variables to 0, but in my opinion, there is an easier method.Distributed lag models. Basically, a regression model where you have full control over how many lags you add for each variable.
Predicting sensor values and setting ‘alarm’ thresholds. The component is deteriorated to a state in which breakdown is imminent when a sensor is predicted to cross this threshold. There are a few caveats using this approach for the current dataset.(a) Every sensor has to be modelled individually, therefore we cannot make use of interactions between sensors.(b) it requires a lot of domain knowledge for setting the appropriate thresholds and (c) it doesn’t directly inform us about our target feature, the Remaining Useful Life (RUL).
Vector Autoregression (VAR) type models. These types of models can handle multivariate timeseries, however the model initially creates lags of both our X and Y variables. Normally in timeseries the past values of Y play a large role in determining future values of Y. But since we assume our values for Y to be constant or linearly declining in the training set, incorporating these self-defined targets in the model and using them for prediction would highly influence model results and overemphasize the effect of Yt-1 on Yt. You could write some custom code to set the coefficients for the weights of the lagged Y variables to 0, but in my opinion, there is an easier method.
Distributed lag models. Basically, a regression model where you have full control over how many lags you add for each variable.
In creating a distributed lag model, we can use many of the tools and tests from both time series and regression analyses. However, we’ll start small and add complexity as needed.
In its basic form the model we’re going to implement looks as follows:
Yt = a + B1*Xt + B2*Xt-1 + ... + Bn*Xt-n
Where a is the intercept and B1 to Bn are the coefficients. Xt-1 and Xt-n are the lagged variables of Xt, with a lag of one till n respectively and Yt is the target variable at the current time t.Note: to be in line with previous posts we’re predicting RUL at Yt, if you want to predict the RUL of the next time_cycle (Yt+1) you can simply shift the target by one, as explained below.
We can use Pandas shift function to add lagged variables.
You can see the lagged columns contain data from the previous time stamps. Row 2 for example, has the data of time 0 and 1 appended. This is how we can get data from multiple time instances represented in one row, which is a convenient representation for use by our model.
Now let’s apply this principle to our dataset. We drop the rows where NaN values were introduced as a result of adding lagged variables and fit a new regression model.
# returns# train set RMSE:39.367692351135005, Variance:0.6709929595362338# test set RMSE:31.423109839706022, Variance:0.42820706786532714
Great! Adding a lag of one already gives a small improvement upon our baseline model, which had a RMSE of 31.95. The above example showcases the basics of what we’re trying to achieve.
Correction: In the above example I forgot to drop the index_names. The results do not improve compared to baseline when these are dropped. Thanks Xuanang Liu for pointing this out! Fortunately this has little impact on the overall flow, adding more lags will start to improve results (as you can read further down below)
Let’s wrap this functionality in two functions for later use, one to add lags from 1 till n, and one to add specific lags.
There are a few assumptions to consider when applying time series analysis and regression, let’s look at those now.
Stationarity implies the statistical properties of the time series, such as mean and variance do not change over time, in addition the covariance (spread of the time series) should not be dependent on time (see Figure 1 below).
It is important that these statistical properties stay constant because a lot of statistical models rely on these properties for their predictions. Hence, if the mean, variance or covariance changes with time the model would not be able to accurately predict the next value [1].
Fortunately, there are tests to check for stationarity in your data, and there are also methods to make time series stationary if the tests are not passed.
To test for stationarity, we’re going to use the Augmented Dickey-Fuller test, you can check [1–5] for more information. For now, the adf-test-statistic and p-value are the most important. The more negative the test statistic, the more stationary the signal. When the p-value is below 0.05, it can be assumed the time-series is stationary. Let’s perform this test on a column of a single engine.
# returns# single unit, single column test results# adf: -2.276666927948019 # pvalue: 0.17960380425358546# Significant: False
The test statistic is negative but not significant. We can difference the data once in an attempt to remove the trend and test again [6].
# returns# single unit, single column test results after differencing# adf: -14.824168539718967 # pvalue: 1.9431018596392618e-27# Significant: True
Much better, the p-value is now < 0.05 allowing us to assume stationarity.
Note: It’s important to take into account how you implement your test, for example testing stationarity of one sensor column has the time series of all 100 engines. All engines together form a stationary signal, but signals of individual engines are not stationary! (see Figure 2)
Since we don’t know beforehand how many times the timeseries of each sensor of each engine needs to be differenced before becoming stationary, it’s best to create some functions which handle these steps for us. We’ll use the p-value as the decisive factor to assume stationarity.
We apply these functions to make our data stationary.
Let’s fit a new linear regression to get an idea of the effect of stationary data on model performance.
# returns# train set RMSE:51.63375613791181, Variance:0.4340864011385228
Training RMSE has become quite a bit worse as a result of making the data stationary. This can be explained by the data no longer having a trend, making it difficult to fit a regression line. Performance will improve again when adding lagged variables, as this will inform the model of the relation between the changes in sensor values over time.
Next, we can start our search to determine how many lags to add for optimal model results.
The Akaike Information Criterion (AIC) is a metric which reflects model quality and is often used in time series analysis. It takes both over- and underfitting into account. When training multiple similar models, the AIC-scores can be compared for model selection [7]. A lower score is generally better, but there is usually a trade-off between complexity and model quality improvement. Let’s train some models with 0 to n lags and calculate the AIC to determine the highest quality model.
We’re looking for a tipping point in the AIC’s indicating diminishing returns of model improvement by adding more lags. However, based on the table of metrics it does not become clear how many lags to add as the metrics seem to keep improving steadily. Let’s try plotting the AIC for any visual ques.
Based on the graph, quality improvement seems to stagnate after 14 lags. To me this indicates the trade-off between model complexity and quality is reached at this point, as adding more lags (complexity) no longer substantially improves model quality.
One problem with adding multiple lags of the same variable is the possibility of introducing multicollinearity, let’s check that next.
Multicollinearity occurs when independent variables in a multiple regression model are highly correlated [8]. Due to the nature of time series, where the value of a previous timepoint is inherently related to the next value in time, the chances of introducing multicollinearity when you add lagged variables is substantial. Multicollinearity influences the models’ coefficients and stability, increasing the chance of incorrect results on new data. Let’s test the multicollinearity of our model by calculating the Variance Inflation Factor (VIF), as a rule of thumb vif-values below 5 are considered safe [9].
Our model seems to suffer greatly from multicollinearity with vif-values far bigger than 5. When searching for ways to reduce multicollinearity I found the method used by statsmodels for calculating multicollinearity may be heavily influenced by scaling [10]. This would be quite an easy fix, so let’s implement a scaling transformation and hope we don’t have to look further to solve our problems.
Much better! We now have all components to train, predict and evaluate to find the best model.
We first repeat our procedure for identifying the number of lags to add.
The point shifted a little bit after adding scaling, model quality improvement now stagnates at 9 lags instead of 14.
Let’s put our pipeline together and train our model with 9 lags to see how it performs. We’ll also incorporate the clipped RUL from here on.
# returns# train set RMSE:20.80233928213837, Variance:0.7523208224684113# test set RMSE:21.14895857012398, Variance:0.7409888687595052
A RMSE of 21.14 isn’t bad, but I think we can squeeze out a little bit more. Currently our model uses consecutive lagged variables. However, adjacent lags do not necessarily have to hold distinctive information. For example, the change in sensor values at lag 6, 7 and 8 could be minimal and therefore not really informative to the model. Changes in sensor values become more apparent when increasing the timesteps in between lags. For example, sensor values at lag 5, 10 and 20 might differ enough to be informative. Note, by adding higher lagged variables you also throw away more rows of data, so there’s a trade-off to take into account.
When the engines deteriorate and get closer to breakdown there are clear trends in the sensors which are distinguishable between single timesteps. So personally, I choose to keep the first few lags consecutive before increasing the timesteps between lags. Let’s try a few variations and see which scores best.
# Results of different lags and test RMSE# RMSE = 21.76, lags = [1,2,3,4,5]# RMSE = 21.14, lags = [1,2,3,4,5,6,7,8,9]# RMSE = 21.06, lags = [1,2,3,4,5,10,20,30]# RMSE = 20.85, lags = [1,2,3,4,5,10,20]
There are of course many more possibilities, but with minimal testing and taking into account the stagnating improvement from adding more continuous lags, lags [1,2,3,4,5,10,20] performed best with a RMSE of 20.85. These results come pretty close to our SVR from last time (RMSE = 20.54).
In the end, I feel the distributed lag model is a valuable addition to the possible methods for solving this use-case. It gives great flexibility and control during model development while also being easy to explain and interpret by the business. For the complete notebook you can check out my github page here.
I would like to thank Wisse Smit and Maikel Grobbe for their inputs and reviewing my article. Next time we’ll delve into survival analysis. Are you able to find a combination of specific lags (not fully continuous) which performs better? Let me know in the comments below!
References:[1] https://medium.com/open-machine-learning-course/open-machine-learning-course-topic-9-time-series-analysis-in-python-a270cb05e0b3[2] https://en.wikipedia.org/wiki/Augmented_Dickey%E2%80%93Fuller_test[3] https://www.statsmodels.org/stable/generated/statsmodels.tsa.stattools.adfuller.html[4] https://www.quora.com/What-is-unit-root-in-a-time-series[5] https://www.machinelearningplus.com/time-series/time-series-analysis-python/[6] https://www.quora.com/What-is-the-purpose-of-differencing-in-time-series-models[7] https://en.wikipedia.org/wiki/Akaike_information_criterion[8] https://en.wikipedia.org/wiki/Multicollinearity[9] https://www.statisticshowto.com/variance-inflation-factor/[10] https://stats.stackexchange.com/questions/332428/regression-model-constant-causes-multicollinearity-warning-but-not-in-standardi