Extreme Rare Event Classification: A Straight Forward Solution For a Real World Dataset | by Roberto Mansur | Towards Data Science

Initially, thank you Chitta Ranjan for the real-world dataset of a web break on a paper mill. It is a challenging time series dataset and a common problem on predictive maintenance domain. This article proposes a feasible and straight forward solution for the problem. I strongly recommend the reading of Chitta’s previous articles (Link).
On the predictive maintenance domain, a challenge that many companies are facing is predicting failures before its occurrence based on the equipment behavior (Condition Based Maintenance). Curently, a common tool used by maintenance crew is APM (Asset Performance Management) which is based on risk and reliability techniques (such as Weibull curves for example).
Summarizing: the idea of this type of curves is to separate the failures on a probability curve with 3 distinct areas: premature failures, random failures and end of life failures.
This is a simple idea that can give us insights about features that are important on this domain, such as the running time (counter).
First you need to create the frame of the problem with the following code:
df['y_1'] = df['y'].shift(-1)df['y_2'] = df['y'].shift(-2)df = df.loc[df.y==0] #deleting the downtime eventdf['y'] = df.apply(lambda x: 1 if ((x['y_1'] == 1) | (x['y_2'] == 1)) else 0, axis=1)features = df.columns.tolist()# adding delayed infofeatures.remove('time')features.remove('y_1')features.remove('y_2')features.remove('x61')target = 'y'features.remove(target)
Some highlights: the time features were eliminated since the period of data (less than a month) does not justify any new feature creation (hour, day, shift, quarter, day of week, etc.). Another variable eliminated were x61 that is completely freezed.
The minority class is ~ 1.3% of all data. A very simple way to improve it a bit is to determine that 6 minutes before the event is as good as 4 minutes. So, it will improve 50% of minority class and it is fair to assert that it is not a terrible sin:
df['y'] = df.apply(lambda x: 1 if ((x['y_1'] == 1) | (x['y_2'] == 1) | (x['y_3'] == 1) ) else 0, axis=1)
The counter(run time) called count_y for now on, is the feature created to contextualize the model’s lapse of time.
The simplest strategy for anomaly detection is to use one class algorithms such as SVM or Mahalanobis distance to understand what is normal in data and what is not normal.
First the data will be separated into train and test.
#Splittingfrom sklearn.model_selection import train_test_splitx_train, x_test, y_train, y_test = train_test_split(df[features],df[target],shuffle=True,random_state=10)#Scallingfrom sklearn.preprocessing import StandardScalerscale = StandardScaler()x_train = pd.DataFrame(data=scale.fit_transform(x_train),columns=features,index=y_train.index)x_test = pd.DataFrame(data=scale.transform(x_test),columns=features,index=y_test.index)
After data separation comes the SVM as a candidate model:
from sklearn.svm import OneClassSVMfrom sklearn.metrics import classification_reportfrom sklearn.metrics import confusion_matrixsvm1 = OneClassSVM(kernel='rbf', gamma='auto',nu=0.05)svm1.fit(x_train_pos[features])y_pred = pd.DataFrame(data=svm1.predict(x_test[features]))# need to convert to the same standard, 0 == normal and 1 for breaky_pred.loc[y_pred[0]==1] = 0 # normal y_pred.loc[y_pred[0]==-1] = 1 #breakprint(classification_report(y_test,y_pred,digits=3))confusion_matrix(y_test,y_pred)
Result bellow the benchmark (F1<0.1)
precision    recall  f1-score   support           0      0.984     0.947     0.965      4486           1      0.040     0.123     0.061        81   micro avg      0.933     0.933     0.933      4567   macro avg      0.512     0.535     0.513      4567weighted avg      0.967     0.933     0.949      4567
The Mahalabonis distance provided a similar result:
clf_total = MahalanobisOneclassClassifier(x_train_pos[features], significance_level=0.001)mahalanobis_dist_total = clf_total.predict_proba(x_test_pos[features].values)print(classification_report(y_test,clf_total.predict(x_test[features].values),digits=3))precision    recall  f1-score   support           0      0.984     0.886     0.932      4486           1      0.032     0.210     0.056        81   micro avg      0.874     0.874     0.874      4567   macro avg      0.508     0.548     0.494      4567weighted avg      0.967     0.874     0.917      4567
Both algorithms were bellow benchmark using all the features as input. The dimension of the problem is surely a complication for this type of algorithm. One possible strategy is to divide the features into affinity groups.
Usually a good working session with the domain expert can help with that. An alternative is to use the GradientBoost Tree (features importance) to support this definition.
By using the top 20 tags, separating them into 2-tag groups and calculating the Mahalanobis distance in a one class approach, we artificially created 10 new features that represent the distance from the standard operation of training data.
df['maha_dist'] = clf.predict_proba(df[feat_maha].values)df['maha_dist2'] = clf2.predict_proba(df[feat_maha2].values)df['maha_dist3'] = clf3.predict_proba(df[feat_maha3].values)df['maha_dist4'] = clf4.predict_proba(df[feat_maha4].values)df['maha_dist5'] = clf5.predict_proba(df[feat_maha5].values)df['maha_dist6'] = clf6.predict_proba(df[feat_maha6].values)df['maha_dist7'] = clf7.predict_proba(df[feat_maha7].values)df['maha_dist8'] = clf8.predict_proba(df[feat_maha8].values)df['maha_dist9'] = clf9.predict_proba(df[feat_maha9].values)df['maha_dist10'] = clf10.predict_proba(df[feat_maha10].values)
New features for the problem frame:
['x1', 'x2', 'x3', 'x4', 'x5', 'x6', 'x7', 'x8', 'x9', 'x10', 'x11', 'x12', 'x13', 'x14', 'x15', 'x16', 'x17', 'x18', 'x19', 'x20', 'x21', 'x22', 'x23', 'x24', 'x25', 'x26', 'x27', 'x28', 'x29', 'x30', 'x31', 'x32', 'x33', 'x34', 'x35', 'x36', 'x37', 'x38', 'x39', 'x40', 'x41', 'x42', 'x43', 'x44', 'x45', 'x46', 'x47', 'x48', 'x49', 'x50', 'x51', 'x52', 'x53', 'x54', 'x55', 'x56', 'x57', 'x58', 'x59', 'x60', 'count_y', 'maha_dist', 'maha_dist2', 'maha_dist3', 'maha_dist4', 'maha_dist5', 'maha_dist6', 'maha_dist7', 'maha_dist8', 'maha_dist9', 'maha_dist10']
It will be used the GradientBoostClassifier as the main classifier receiving this new features (Figure 4).
Some cautions should be taken to avoid mixing the train and test samples using the same random state:
x_train, x_test, y_train, y_test = train_test_split(df[features],df[target],shuffle=True,random_state=10)gbc = XGBClassifier(n_estimators=1000,subsample=0.9,max_depth=6,random_state=10, max_features=0.9, n_jobs=2)%time gbc.fit(x_train,y_train)Wall time: 1min 4s
The F1 result for minority class ~0.5 is above the benchmark for the problem.
precision    recall  f1-score   support           0      0.989     1.000     0.994      4486           1      0.968     0.370     0.536        81    accuracy                          0.989      4567   macro avg      0.978     0.685     0.765      4567weighted avg      0.988     0.989     0.986      4567
A simpler strategy is creating AD (anomaly detectors) for each feature on the model input using the same top 20 tags pointed by the first XGBoost feature importance.
#top 20 tagsoriginal_features = ['count_y','x51', 'x28','x26', 'x43','x60', 'x8','x54', 'x50','x14', 'x18','x24', 'x53','x23', 'x15','x22', 'x52','x42', 'x21', 'x36', 'x3']features_individual_ad = original_features.copy()ad_maha = []for feat in original_features:    if not feat == 'count_y':        _model = MahalanobisOneclassClassifier(x_train_pos[[feat]], significance_level=0.01)        ad_maha.append(_model)        _ad_name = feat + '_ad'        df[_ad_name] = _model.predict_proba(df[[feat]].values)        features_individual_ad.append(_ad_name)
And the result was even better than the combination of paired features.
precision    recall  f1-score   support           0      0.992     0.998     0.995      4486           1      0.865     0.556     0.677        81    accuracy                          0.991      4567   macro avg      0.929     0.777     0.836      4567weighted avg      0.990     0.991     0.990      4567
A very straight forward solution using artificial features (run time feature and Mahalanobis distance) and a good ML classifier algorithm (Gradient Boost) provided an above the initial benchmark result for the problem (F1>0.1). The best topology for the algorithm was able to achieve 0.67 F1 for the minority class. A more robust separation between train and test (cross validation) is recommended to validate this result and also a fine tune on model parameters.
The complete code is available on GIT .