Predictive Equipment Failures
By: Shivam Srivastava
Tags: Kaggle, Data Science, Machine Learning, Data Visualization, Imbalanced Dataset

Predict down hole or surface failure in pump jacks

Table of content:-

1: Introduction

2: Detailed Business Problem

3: Existing Solution

4: My approach

5: Details of data

6: ML Formulation of business problem

7: Data per-processing

8: Exploratory Data Analysis

9: Performance Metric

10: Challenges

11: Feature Engineering

12: Feature Selection

13: Modelling

14: Result

15: What worked and what didn’t worked

16: Best Model In Production

17: Conclusion and Future work

18: Learning

19: Reference

20: GitHub repository

21: LinkedIn Profile

1: Introduction:-

We would be predicting equipment failure of pump jack(Mechanical system used for crude oil extraction), using data provided by sensors.

In USA 80% of wells are stripper well(Oil producing well, that produce 10 to 15 barrel oil per day),

Companies use these stripper wells for earning high profit , because they need less investment and give high returns.

These wells account for 19% oil production of USA(Source:- https://nswa.us/stripper-wells/)

Image Source:- Wikipedia

Please refer this video for more information(https://www.youtube.com/watch?v=X0Dpd52pfp0)

2: Detailed Business Problem:-

Company named ConocoPhillips has posted a kaggel problem (https://www.kaggle.com/c/equipfails/overview) . In this they have given us data of 107 sensors .

When ever there is failure of any equipment in down hole or on surface, company has documented sensors data along with region of equipment failure(done hole or surface)

Using these sensors data we have to predict, the region of failure. If failure has happend on surface then workers could solve the problem on surface, if failure has happend in down hole then, workers can pull the down hole equipment to surface and solve the problem.

This prediction would help workers in quickly identifying , where has the failure occurred and without wasting time in diagnosing both down hole and surface, they could directly reach to the problematic area.

This in turn safe time and would increase oil production.

3: Existing Solution:-

In existing solution for this problem, XgBoost performs the best with

0.9937 f1 score.

4: My approach:-

This competition has been closed, we can’t get result from test data set.This is why we can not compare the result in this blog to result in kaggle competition.

I will do detailed EDA, Feature Engineering ,Feature selection and will use all the metrics that is helpful in this type of problem

I will be using many ML model like logistic regression, KNN , SVM , Naive bayes , Decision Tree, Random Forest, GBDT and Stacking Classifier.

5: Details of data:-

They have given data of 107 sensors

100 of them is static sensor

7 of them is dynamic sensor, these sensors contains 10 histogram bins

total features = 100(static sensor) + 7(dynamic sensor)*10(bins) = 170 features

Using these features we need to predict that where the equipment failure has happened, under ground(down hole) or on surface

Down hole failure is signified by target value = 1

Surface failure is signified by target value = 0

6: ML Formulation of business problem:-

This is a binary classification problem.

Problem statement:- We need to predict , where the equipment failure has happened , down hole or on surface

Real word constraints:-

We do not have strict time constraint, but our model should not take more than few minutes to give result.

If probabilistic value is given then good, else it is not very necessary

Cost of miss-classification is high, because if miss-classified time would be wasted in wrong place, and this will decrease the profit of company

Interoperability is not important at all.

7: Data per-processing:-

Instead of np.nan values, data have “na”, so we will replace it

It have use less columns like “id”, we will remove it.

Mostly columns are of object type, we will make them float

Raw data

Cleaned data

8: Exploratory Data Analysis:-

Data is highly imbalanced , 1.67% down hole failure, rest is surface failure.

Data Distribution

Using box plot we could see that , data is some what separable , but there is lot of outliers

Univariate analysis of highly correlated feature to target

Using Pair plot we could observe that many features are highly inter-correlated

Pair-Plot of highly correlated feature to target

Using 3d plot of all the feature, we found that if we use all features ,then model will not be able to Separate both the target.

We might need to create new features to make data set more seperable

3D plot using all the features

By looking at the nan value distribution in both the classes we, could find that there is some information in nan values also. So in feature engineering we would be using these nan values also

%age nan distribution {“Green” : Down-hole , “Orange” : Surface , “Blue” : Total}

Here we could see that , features that are highly correlated with class, they are correlated with themselves also.

We need to remove highly correlated features, because these features do not help in prediction of class

Heat map of highly correlated feature to target

Here few features are highly correlated, few features have very less correlation, and few are negatively correlated with each other. Which is good for model.

Heat map of randomly selected data

9: Performance Metric:-

Since data set is highly imbalanced, so accuracy would not be good metric to use.

We would be using F1 score because it take account of both precision and recall.

10: Challenges:-

There is problem of outliers in the data set.

Too many nan values.

Features are inter-correlated

Highly imbalanced data-set

11: Feature Engineering:-

For each feature create new feature, that tells presence of nan, because nan values also contains some information.

If nan value is there then insert 1.0 , else insert 0.0, in corresponding nan feature of original feature.

nan feature ,corresponding to original feature

Replace nan value with median , because, features have either very low value or very high value.This is why replacing with mean is not sensible.

We have 100 simple sensor, and 7 time based sensor. Here we will extract min, max and mean from those time based sensors.

Featured, time based sensors

We are not removing outliers, because most of the values of class 1 (Down-hole) lies in outliers only.

Box plot of features

12: Feature Selection:-

Remove all the features which are least correlated to our feature(This is simple)

Remove all the inter-correlated feature(This is little complicated)

Function for removing inter-correlated feature

Algorithm of this function:-

Let we have 10 features and 1 target,

2. It will first select highly correlated feature to target, let corr(feature10,target) = 0.95

3. Not it will remove all the features that are highly correlated to feature10, let corr(feature10,feature9) ≥0.9 #Here 0.9 is threshold, but you can change it . corr(feature10,feature8) ≥0.9, similarly corr(feature10,feature7) ≥0.9. then we will remove feature9, feature8, and feature7 from data frame

4. Now repeat step 2nd to 3rd, but this time we, will consider 2nd most correlated feature, and will not touch 1st most correlated feature.

5. Repeat this step until all the features are not considered.

Now in this 3d plot we could see that data is little separable, i.e. more separable than before. Still it is not completely separable, but in higher dimensions, it will be separable.

3d Plot after feature engineering and feature selection

13: Modelling:-

Since we know that our data is highly imbalanced, so we will be using both Random Up-sampling and SMOTE

We Used grid search CV on Gaussian NB, Logistic Regression, Decision Tree, Gradient Boosting DT on both data-set i.e. Random Up-sampled and SMOTE.

Gaussian NB and Logistic regression had given worst result on validation set, even after using grid search CV. So we dropped those models from stacking classifier.

In order to get the best result, we used 1 Decision Tree and 2 GBDT in classifier, and used GBDT as meta classifier of Stacking Classifier.

Use best model in Stacking classifier, and get result.

14: Result:-

Train F1 Score = 1.0

Validation F1 Score = 0.8125

Test F1 Score = 0.8222811671087533

ROC curve with AUC scores

Train precision Score = 1.0

Test precision Score = 0.8659217877094972

Train recall Score = 1.0

Test recall Score = 0.7828282828282829

Confusion Matrix

Precision Matrix

Recall Matrix

Just using 70% of total data, we managed to get F1 score of 0.82. Which is pretty good.

15: What worked and what didn’t worked:-

Creating new features from time based sensors, improved the model performance.

Using nan distribution , improved the performance.

No model worked on test data, but GBDT worked on.

Using all model in stacking classifier, degraded the model performance.

Using Probabilistic value of classifiers in stacking classifier didn’t worked at all.

Using features in stacking classifier , improved performance of model.

GBDT meta classifier worked best.

16: Best Model In Production:-

Download the template file, fill the values, then upload that file.

Model in production, upload csv file

Click on “Upload & predict” to get result.

Model in production, with result

17: Conclusion and Future work:-

Thank you soo much for reading till end, I hope you liked my work.

For source code please go to my github code.

For more information or any suggestion please contact me on linked in.

We could improve its performance by following below mentioned techniques.

Use 90% data for training , instead of 70%

Replace nan values using inter-correlated feature, instead of replacing nan with median.

Use LSTM on time based sensors.

try using dense layers network on other features.

Try using create more features from time based sensors, i.e. rate of change of data, number of times mean crossed , checking frequency etc.

18.Learning:-

We have learned how to utilize nan distribution in the data-set.

We learned how to improve model performance from small data, using Stacking Classifier.

We learned how to handle imbalanced data-set

We learned how to Select best feature and how to create new feature from existing feature.

We learned how to analyze 3D plot to see if any model would work on that data or not.

19: Reference:-

20: GitHub repository:-

21: LinkedIn Profile:-