Predictive Maintenance Of Equipment
By: Sarang Sonar
Tags: Kaggle, Machine Learning, Deep Learning

Introduction

Conoco Phillips is a multinational oil and transportation company, which operates stripper wells and massive oil rigs in more than 17 countries. Stripper wells have a maximum daily average production of 15 barrels per day, despite such low production capacity, these are popular because of low production and running costs. This promises a steady cash flow. Oil rigs on the other hand have an average production capacity of 1000 barrels per day. Many onshore and offshore oil rigs depend upon initial fixed investment, because of the scale of the infrastructure and expensive equipment that are employed while production.

Business Problem

These oil rigs are highly scrutinized as a measure of prevention of any equipment damage and failure, as well as the safety of the foremen working onsite. Any halt in workflow and production can cause massive losses financially. The expensive equipment installed might take days to be replaced, and cause a dent in the investments. This is a highly unfavorable scenario. This situation can be avoided by routine maintenance and additional measure like predictive maintenance can be taken to mitigate the occurrence of failure.

Problem statement

We have been provided with recorded data of the sensors when faults on surface and downhole have occurred. A total of 107 sensors are employed to collect data when faults on the surface and downhole have occurred. This case study focuses on predicting the downhole and surface faults, by using these documented sensor data. This information is crucial for the engineers operating on the rig, to mitigate the fault before critical damage causes the halt of production.

Data Overview

The data has been provided by ConocoPhillips in association with Kaggle.

There are a total of 172 features, which represent real-time readings from 109 sensors. There are a total of 100 non-time based sensors and 7 time-based sensors. The time-based sensors provide 10 bins of data at one instance, therefore 7 time-based sensors provide 70 bins/features. The remaining two features are ids and targets. The targets are binary classified as 0’s representing surface faults and 1’s representing downhole fault.

Existing Solution

The best solution ranked on the Kaggle is an XGboost model and has an F1- Score of 0.993.

My Approach

The major issue with this dataset is that it is a highly imbalanced dataset. We try to find different methods to mitigate this issue, by using different methods like using basic random oversampling, synthetic minority oversampling technique (SMOTE), borderline SMOTE, Adaptive synthetic oversampling technique. We compare their performance by plotting those points using PCA and TSNE and also compare them by using a base model like logistic regression. Once we find the best oversampling technique we employ that dataset on our model. We experiment with machine learning models like Random Forest and Xgboost and try to find their best hyperparameter by cross-validation. We compare their results with our deep learning model.

The deep learning model consists of three parts -

The first model is designed for taking in time-based sensor data. We have a total of 7 time-based sensors, which are recording 10 bins of data at an instance. This data is given to LSTM layers followed by dense layers. The second model consists of dense layers which are used to feed in the non-time based sensor data recorded by a total of 100 sensors as an input. We also add drop out layers to mitigate overfitting. The third model is the final model which also consists of dense layers. This model takes in the predicted data of the first two models as an input to give a final prediction of the classes/faults.

Exploratory Data Analysis

First, we try to understand the distribution of both the classes in our dataset.

Distribution of the classes.

We can observe that this is a highly imbalanced dataset with less than 2 % of data points are classified to be downhole faults. This gives an obvious advantage to the other class, as any model will have a higher bias towards the majority class which happens to be the surface faults in our case. We have to overcome this problem by oversampling techniques, which are discussed later in the essay.

EDA on Time-based sensors

First, we try to fetch all the time-based sensors from our dataset.

[‘sensor7’, ‘sensor24’, ‘sensor25’, ‘sensor26’, ‘sensor64’, ‘sensor69’, ‘sensor105’]

These are the sensors that are time-based. Each sensor is having 10 reading, for every instance, the fault occurs.

To understand the very moment when the fault has occurred we plot the sensor data along the time-axis concerning the nature of the fault i.e surface fault and downhole fault. By doing this we can exactly know which sensors are responding positively to a certain nature of the fault.

The time-based sensors that are more sensitive to the downhole fault.

Fig 1.00

Fig 1.01

Fig 1.02

Fig 1.03

In the above diagram, we can see that the blue spikes represent the downhole faults and the blue line parallel to the x-axis are the surface faults. The red line/spike shows the response of the particular sensor to the nature of the fault. We can observe that for downhole faults, “sensor 7 bin4”, “sensor-25 bin2", “sensor 26 bin3 ”, “sensor 69 bin2” are responding more sensitively with the least error.

The time-based sensors that are more sensitive to the surface faults.

Fig 1.04

Fig 1.05

Fig 1.06

Fig 1.07

Fig 1.08

Fig 1.09

It can be seen that “sensor 7 bin8”, “sensor 24 bin6”, “sensor 25 bin3”, “sensor 26 bin5 ”, “sensor 64 bin6 ”, “sensor 69 bin8” are very sensitive towards surface faults.

Therefore these sensors/features can be very essential while distinguishing the nature of the fault.

Now we try to plot the PDF of these time-based sensors.

The sensors useful to classify the downhole plots :

Fig 2.01

Fig 2.01

It can be observed that “sensor 25 bin0”, “sensor 64 bin2” are having the minimum overlap for downhole faults. Therefore they become important to classify downhole fault.

The sensors useful to classify the surface plots:

Fig 2.03

Fig 2.04

It can be observed that in the plot “sensor 7 bin0”, “sensor 25 bin0” is having the minimum overlap for surface faults. Therefore they become important to classify surface faults.

EDA for Non-Time based sensors

We now plot the PDF for the non-time based sensors:

Fig 3.01

Fig 3.02

Fig 3.03

Fig 3.04

It can be observed that “sensor 27 measure”, “sensor 32 measure”, “sensor 45 measure”, “sensor 46 measure” are having minimum overlaps for surface and downhole faults, therefore can be very useful for classifying the nature of the fault effectively.

Feature Engineering

In addition to the existing features, we try to find more cases where we can effectively distinguish the surface faults from the downhole faults. As we are aware that time-based sensors are used, we can therefore employ methods like simple moving averages(SMA), Auto-Regressive Integrated Moving Averages(ARIMA) on the time-based sensors.

Simple Moving Averages(SMA)

Fig 4.00

Fig 4.01

Fig 4.02

Fig 4.03

In the above figure, we have plotted the moving averages of the time-based sensors, they are represented by the red line. It can be observed that the “sensor 7 bin3”, “sensor 25 bin3”, “sensor 26 bin ”, “sensor 69 bin4” are responding very well to the downhole faults compared to the rest of the sensors. Therefore we can use simple moving averages of these sensors as a feature to get better accuracy.

Auto-Regressive Integrated Moving Averages

Sensors useful to classify Downhole faults:

Fig 4.04

Fig 4.05

The red response peaks are occurring exactly at the same time when the downhole fault is occurring. This can be seen in time plots of “sensor 25 bin2”, “sensor 26 bin1”

Sensors useful to classify Surface Fault:

Fig 4.06

Fig 4.07

Fig 4.08

The red response peaks are occurring exactly at the same time when the surface fault is occurring. This can be seen in time plots of “sensor 25 bin3”, “sensor 26 bin6”, “sensor 69 bin9”.There is a considerable error happening in the case of surface faults, but we can still consider them as they aren't that prevalent in downhole faults either.

Therefore we can use the ARIMA of these sensors as features to get more accurate results.

Plotting the count of Nan values in non-time based sensor/Feature.

Fig 5.00

We notice that sensor number 36–43 and ‘sensor 68 measure’ are having the most number of Nan values and hence these features can be eradicated from the main dataset as they have the majority of the Nan values.

Fig 5.01

As compared to non-time-based features, time-based features are all having relatively fewer nan values and also the same number of nan values, implying that in certain instances the readings were not taken, hence we can conclude that nan values aren't a major problem while handling the time-based features.

Handling of Imbalance Dataset

The methods employed for handling the imbalance of dataset are :

Random sampling:

In this technique, the minority class is randomly oversampled, irrespective of the existing clusters. These oversampled points are plotted randomly in the dataset plane. This can be observed with the help of TSNE and converting the dataset n-dimensions to 2 dimensions.

Fig 6.00 Random Oversampling

It can be observed here, the minority points(yellow) are randomly plotted forming an outer circle, whereas the majority points(purple) are clustered in the circle. This kind of oversampling techniques can lead to inaccurate results and aren’t very reliable.

2.Synethetic Minority OverSampling Technique (SMOTE):

This is the most ubiquitous oversampling technique. This technique involves the formation of clusters around the existing minority points. This makes them more accurate than random sampling and hence more reliable in the real world. When we perform TSNE using SMOTE we get

Fig 6.01 Oversampling using SMOTE

The yellow minority points have formed a cluster and are all located near each other, which makes more sense in real-world datasets. SMOTE cannot be always relied upon when the dimensionality of the data is high.

3.Adaptive Synthetic Sampling(ADASYN):

This technique involves a density distribution of the minority data points and hence the number of clusters formed are less compared to SMOTE. The clusters also tend to be closer to one another in the case of ADASYN. In SMOTE there is a uniform distribution of the minority data points which cannot be relied upon when the dimensionality is high. When we perform TSNE we get

Fig 6.02 Oversampling using ADASYN

The clusters formed here are much closer to each other when compared to SMOTE. This technique takes into account the overall density of the minority datapoints i.e where most of the minority points are located.

Comparision of the oversampling Techniques:

Now we apply Logistic Regression (base model) on the oversampled dataset and compare their performance.

Fig 6.03 Comparison of OverSampling Techniques

The F1-score of SMOTE is found to be consistently higher compared to randomly oversampled data and ADASYN. Hence, we select the SMOTE dataset for building our final model.

Building and Comparing Models

As we have the SMOTE dataset, we start building models like Random Forest and Xgboost with hyperparameter tuning and compare their results.

Random Forest Classifier

Fig 7.00 Random Forest Classifier

Fig 7.01 Confusion Matrix (Random Forest)

The F1-score on test data is 0.798 using random forest on the SMOTE oversampled dataset. It can be observed that the False-negative and False Positives are almost the same but highly inaccurate for the minority points when we are using random forest.

We can also know the most important features which are being used by random forest.

Fig 7.02 Top 10 important features using Random Forest

It can be seen that “sensor_35_measure”, “sensor_46_measure”, “sensor_17_measure”, “sensor_16_measure” are the most important features according to Random Forest. They are the same features that we obtained during the plotting of PDFs of the non-time based sensors. The time-based features are not used by Random Forest hence we can conclude that random forest doesn't make use of all the features present in the dataset.

Xgboost Classifier

Fig 7.03 Xgboost Classifier

Fig 7.04 Confusion Matrix (Xgboost)

The test data score is 0.9938 which is exactly similar to the competition leaderboard score. The number of misclassified points is 73 i.e 7 less misclassified points than Random Forest model. Over here we can see that the number of False-positives has increased from 41 to 44. The score has increased due to an increase in True -Negatives. Even though the F1 score has increased, minority points are still misclassified. We can observe that some of the time-based features are also providing good information for classification, unlike the case of random forest.

Fig 7.05 Best Features using Xgboost

The best features according to Xgboost are “sensor_44_measure”, “sensor_52_measure”, “sensor_105_bin_7”, “sensor_24_bin_8”, “sensor_17_measure”.It can be seen that “sensor_17_measure” is used as an important feature by both Random Forest and Xgboost model.

Deep Learning Model (Final Model)

Our final Deep Learning model consists of three separate models

LSTM model: This model takes the time-based sensor data as input. The shape of the input data is (number_of_datapoints,7,10 ).

Fig 8.00 LSTM Input Data Shape

Fig 8.01 Structure of the LSTM Model

Fig 8.02 Outputs of the LSTM Model

It can be seen that by only using the time-based features, we can achieve an f1 score of 0.98, which is very good.

No case of overfitting is observed here, and therefore the outputs of this model can be used as an ensemble with other models.

The dropout layers are added to reduce the overfitting.

2. Dense Model: This model takes the non-time based sensor data as input.

Fig 8.03 Shape of Non-time-based-Features

The shape of the input to the model is (number_of_datapoints,89). We have selected the best 89 features out of 100 non-time-based features using EDA.

The structure of the Dense Model is shown below

Fig 8.04 Structure of the Dense Model

This is a typical and simple dense neural network with dropout layers added to mitigate the overfit of the train data, the parameter are kept low to again reduce the overfit.

Fig 8.05 Result of the dense model (non_time_based sensors)

It can be seen that we get a train F1 score of 0.9737 and the test score is 0.9818, using a very simple dense network and only using the non-time-based sensors.

We can combine the results of the LSTM and this dense network by using their outputs in another dense network.

3. Final Dense Model: The structure of this dense model is exactly similar to the previous dense model, as the number of trainable parameters is kept low and dropout layers have been added to overcome overfitting in the prior model.

The input to this model is the predicted values of the LSTM model and dense model 2 which are concatenated and fed as train datapoints, with corresponding classes. The code snippet for the process is shown below

Code for Concatenation of Model Outputs

Fig 8.06 Result of the Final model

We can observe that we, achieve an F1 score of 0.9881 for the data set.

We can therefore conclude that using our deep learning model with a combination of LSTM and dense networks we can achieve a better F1 score. In this case, the MLP is extracting the best features automatically by itself therefore we obtain these results.

We can see that the Xgboost model gives us a test data F1 score of 0.80, and random forest gives us a score of 0.72, therefore compared to the Machine Learning models, the deep learning models are performing more accurately.

Final Comparision of Models

Fig 8.06 Comparison of all models

Future Work

More robust deep learning models can be built and their predictions can be used along with Random Forest and XgBoost models.

Also, we have used all features after our preprocessing step (total 176 features) for all our models. In the future, we will try using feature selection using Random forest or XgBoost and discard the less important features.

New feature engineering can also be implemented. Example: Rate of change of dynamic sensor data, Ratio of moving average, moving standard deviation, etc.

References