Supervised Machine Learning Technique for Anomaly Detection: Logistic Regression | by Mahbubul Alam | Towards Data Science

This is the 11th (and final) piece in a series of articles I am writing about anomaly detection algorithms. The following are the previous 10 articles if you want to check out, each focusing on a different anomaly detection algorithm:
1. Isolation Forest
2. K-Nearest Neighbors (kNN)
3. Support Vector Machines (SVM)
4. DBSCAN, an unsupervised algorithm
5. Elliptic Envelope
6. Local Outlier Factor (LOF)
7. Z-score
8. Boxplot
9. Statistical techniques
10. Time series anomaly detection
With a closer look, you’ll discover that all of those algorithms are either statistical or unsupervised ML techniques. Unsupervised, but many of those algorithms are actually built around supervised classification theories (such as kNN, SVM or DBSCAN). Today I am going to take on a “purely” machine learning approach for anomaly detection — meaning, the dataset will have 0 and 1 labels representing anomaly and non-anomaly respectively.
So why supervised classification is so obscure in this domain? A few reasons are behind it but a key one is the severe class imbalance, meaning only a tiny fraction of the data represents anomaly. Anomalies are rare events and finding them is like finding a needle in the haystack. That’s a big barrier for a supervised algorithm because there’re not enough examples to learn from!
In any case, below I am writing an implementation of supervised classification with publicly available data, using a popular algorithm — Logistic Regression.
Typically in previous articles, I create a small synthetic dataset on the fly and implement the algorithms with bare minimum codes to give an intuition on how they work. It’s going to be different today since it is a supervised classification problem and I have to follow all the essential steps. As you will see, data preparation alone is going to take quite a bit of space.
We need quite a few libraries for this exercise for data wrangling, preparation of model inputs, model building and validation — all libraries coming from three big packages: pandas, nunpy and sklearn.
# data wranglingimport pandas as pdimport numpy as np# inputs data preparationfrom sklearn.preprocessing import RobustScalerfrom sklearn.model_selection import train_test_split# modelingfrom sklearn.linear_model import LogisticRegression# model validationfrom sklearn.metrics import classification_reportfrom sklearn.metrics import confusion_matrix
I am using a popular dataset from Kaggle on credit card fraud detection. The dataset was released in the public domain by European cardholders after removing any user identifier. Let’s import and take a look at the datasets and see the features.
# import datadf = pd.read_csv("..\creditcard.csv")# view the column namesdf.columns
# view the column namesdf.columns>> Index(['Time', 'V1', 'V2', 'V3', 'V4', 'V5', 'V6', 'V7', 'V8', 'V9', 'V10', 'V11', 'V12', 'V13', 'V14', 'V15', 'V16', 'V17', 'V18', 'V19', 'V20', 'V21', 'V22', 'V23', 'V24', 'V25', 'V26', 'V27', 'V28', 'Amount', 'Class']
The dataset has 31 columns. The first column “Time” is the transaction timestamp, the second last column “Amount” is the transaction amount and the last column “Class” designates whether the transaction is fraudulent or not (fraud = 1, non-fraud = 0). The rest of the columns, “V1” to “V28” are unknown features and the values were scaled.
Now check out the number of fraud and no-fraud cases.
# number of fraud and non-fraud observations frauds = len(df[df.Class == 1])nonfrauds = len(df[df.Class == 0])print("Frauds", frauds); print("Non-frauds", nonfrauds)>> Frauds 492>> Non-frauds 284315
As I said, that’s quite an unbalanced dataset, only 492 fraud cases out of a quarter of a million observations. This represents only a 0.17% fraud cases in all of the transactions.
The dataset is pretty clean, so in the next step, we are just going to scale the “Amount” and “Time” columns to make sure they are of a similar format to the other columns.
## scaling the "Amount" and "Time" columns rob_scaler = RobustScaler()df['scaled_amount'] = rob_scaler.fit_transform(df['Amount'].values.reshape(-1,1))df['scaled_time'] = rob_scaler.fit_transform(df['Time'].values.reshape(-1,1))# now drop the original columnsdf.drop(['Time','Amount'], axis=1, inplace=True)
Now let’s define the X and y input variables.
# define X and y variablesX = df.loc[:, df.columns != 'Class']y = df.loc[:, df.columns == 'Class']
Since this is an unbalanced dataset, we need to take an extra step to take a subsample — known as undersampling — so the machine doesn’t have to find a needle in the haystack to learn from examples.
# number of fraud casesfrauds = len(df[df.Class == 1])# selecting the indices of the non-fraud classesfraud_indices = df[df.Class == 1].indexnonfraud_indices = df[df.Class == 0].index# from all non-fraud observations, randomly select observations equal to number of fraud observationsrandom_nonfraud_indices = np.random.choice(nonfraud_indices, frauds, replace = False)random_nonfraud_indices = np.array(random_nonfraud_indices)# appending the 2 indicesunder_sample_indices = np.concatenate([fraud_indices,random_nonfraud_indices])# undersample datasetunder_sample_data = df.iloc[under_sample_indices,:]# now split X, y variables from the under sample dataX_undersample = under_sample_data.loc[:, under_sample_data.columns != 'Class']y_undersample = under_sample_data.loc[:, under_sample_data.columns == 'Class']
Splitting data into training and testing sets before feeding into the model.
# split datasetX_train_undersample, X_test_undersample, y_train_undersample, y_test_undersample = train_test_split(X_undersample, y_undersample, test_size = 0.3, random_state = 0)
As usual, the actual model building takes only 3 lines of code, to instantiate, fit and predict on the given dataset. Since this is for demonstration purposes only, we are going to use default parameters without tuning anything.
# instantiate modelmodel = LogisticRegression()# fit model.fit(X_train_undersample, y_train_undersample)# predicty_pred = model.predict(X_test_undersample)
As a standard model evaluation metric, we are producing a classification report and confusion metrix.
classification_report = classification_report(y_test_undersample, y_pred)confusion_matrix = confusion_matrix(y_test_undersample, y_pred)print("CLASSIFICATION REPORT")print(classification_report)print("CONFUSION MATRIX") print(confusion_matrix)
The objective of this article was to demonstrate a purely supervised machine learning approach for anomaly detection. The use of supervised techniques is rare in this domain because of the severe class imbalance. Since there are only a few anomalous observations in a dataset, algorithms don’t have enough examples to learn from.
None of the 11 algorithms I wrote about so far is good or better in an absolute sense, it all comes down to the nature of the dataset and the domain it is coming from. Often a simple statistical algorithm (e.g. boxplot) is good enough to filter possible candidates and in other cases, a sophisticated algorithm can be useless.
Hope you’ve found this series useful, feel free to leave a comment below and follow me on Medium, Twitter or LinkedIn.