Unsupervised Machine Learning Example in Keras | by Andrej Baranovskij | Towards Data Science

This post is about unsupervised learning and about my research related to the topic of fraudulent claims detection in health insurance.
There are several challenges related to fraudulent claims detection in health insurance. First, of them — there is no public data related to health insurance claims fraud, this is related to the data privacy issues. Second, it is very hard to identify a set of rules, which would help to identify fraudulent claims. This means a supervised machine learning approach with labeled data would hardly work for our case. Unsupervised machine learning seems like it will be a better match. In unsupervised machine learning, network trains without labels, it finds patterns and splits data into the groups. This can be specifically useful for anomaly detection in the data, such cases when data we are looking for is rare. This is the case with health insurance fraud — this is anomaly comparing with the whole amount of claims.
I would recommend going through Hands-On Unsupervised Learning Using Python book for a better hands-on understanding of the topic. There is a GitHub repo for this book. I’m using an anomaly score calculation function from this book in my example.
The way how we can identify fraud is by detecting an anomaly. In unsupervised learning, an anomaly can be detected with autoencoders. Autoencoder translates original data into a learned representation, based on this we can run a function and calculate how far is learned representation from the original data. Fraudulent data is reconstructed with a higher error rate, this helps to identify anomalies. Autoencoders are suitable for unsupervised learning — labeled data is not required for training.
My example is based on the idea of locating fraudulent health insurance claims. There could be various types of fraud, one example — hospitals may overcharge insurance companies. Our task is to detect fraudulent claims, the model is trained in Keras using unsupervised manner, without labels.
It is hard to get public data for health insurance (privacy issues). For this reason, I’m using generated data (based on this post — Data Science With No Data):
Training: 100 000 (valid claims), 1 000 (fraudulent claims)
Testing: 30 000 (valid claims), 300 (fraudulent claims)
Fraudulent claim rule: claimed insurance cost is doubled
Anomaly score function to detect data reconstruction error rate:
def anomalyScores(originalDF, reducedDF):    loss = np.sum((np.array(originalDF) - \                   np.array(reducedDF))**2, axis=1)    loss = pd.Series(data=loss,index=originalDF.index)    loss = (loss-np.min(loss))/(np.max(loss)-np.min(loss))        print('Mean for anomaly scores: ', np.mean(loss))        return loss
Autoencoder is implemented with Keras/TensorFlow. Neural Network is defined with 3 layers (number of nodes = data dimension). Using linear activation, optimizer Adam:
# Call neural network APImodel = Sequential()# Apply linear activation function to input layer# Generate hidden layer with 14 nodes, the same as the input layermodel.add(Dense(units=14, activation='linear',input_dim=14))model.add(Dense(units=14, activation='linear'))# Apply linear activation function to hidden layer# Generate output layer with 14 nodesmodel.add(Dense(units=14, activation='linear'))# Compile the modelmodel.compile(optimizer='adam',              loss='mean_squared_error',              metrics=['accuracy'])
This is unsupervised learning — fraud label is not included into training:
# Train the modelnum_epochs = 10batch_size = 256history = model.fit(x=dataX, y=dataX,                    epochs=num_epochs,                    batch_size=batch_size,                    shuffle=True,                    validation_data=(dataX, dataX),                    verbose=1)
The model is trained in 10 runs. After each run, the anomaly score is calculated to measure reconstruction accuracy. The best model is saved. Accuracy is measured using Precision-Recall and Receiver Operating Characteristic. Sample training result:
Fraud detection logic is implemented in fraud_detection notebook. Predict function is executed on test data without labels. Anomaly scores are calculated using the anomaly score function. I have calculated the mean anomaly score, this will help to define the threshold to distinguish between valid and fraudulent claims. Calculated mean based on test data:
In the end, I’m comparing predicted labels with actual test set labels — this allows us to measure fraud detection accuracy by the trained model.
Fraud detection threshold = 0.01.
Results:
Valid claims: 30 000
Fraud: 286 (accuracy 95%)
Not identified as fraud: 14
Fraud detection threshold = 0.005 (closer to calculated mean 0.003).
Results:
Valid claims: 29 636
Fraud: 293 (accuracy 98%)
Valid identified as fraud: 364
Not identified as fraud: 7
Fraud detection threshold = 0.003 (equal to calculated mean 0.003).
Results:
Valid claims: 26 707
Fraud: 300 (accuracy 100%)
Valid identified as fraud: 3293
Conclusion
With anomaly scores getting smaller, we apply a more strict check for the anomaly. This allows us to catch all fraudulent claims, but in return, we flag more valid claims as a fraud.
You can see from the charts above, it is harder to identify fraudulent claims with a lower value. This makes sense, when amounts are smaller — higher chance it can be missed.
Source code: GitHub repo