How to find outliers and anomalies using Isolation Forest
Tags: Unsupervised Learning, Kaggle, Artificial Intelligence, Python, Isolation Forests

How to find outliers and anomalies using Isolation Forest Tracyrenee Follow Dec 11 · 4 min read

I made a previous video about finding outliers in the candidate votes of the US presidential elections from 1976 to 2020, the link being found here:- A Data-Driven Analysis Of Trump’s ‘Voter Fraud’ Claims In The 2020 Presidential Election | by Tracyrenee | AI In Plain English | Dec, 2020 | Medium

The presidential elections only come around once every four years, so I felt it would be a good idea to search for outliers in more practical, day to day activities, such as the housing market. I chose the train dataset from Kaggle’s Ames House Price competition to carry out this test, with the web address for this dataset being found here:- House Prices: Advanced Regression Techniques | Kaggle

Once I downloaded the dataset and saved it to my personal GitHub account, i created a program using Google Colab because it is free and has many libraries already installed on it. The link for Google Colab can be found here:- Welcome to Colaboratory — Colaboratory (google.com)

Once my program was initiated, I imported the libraries I would need:-

I then loaded the dataset onto the program and read it:-

I carried out a graphical analysis and found that most of the house prices were between $100,000 and $300,000, but there were a few outliers of over $700,000:-

I also created a box plot and found that the better the overall quality of the home, the higher in price it was likely to be:-

I then checked for any null values and found there were quite a few columns that had missing data:-

I imputed the missing values by replacing the null values in categorical columns with the most used values and the null values in numerical columns with the median values:-

I then ordinal encoded all of the categorical columns because most models want this type of data to be converted to numeric data before fitting it:-

I defined the X and y variables that were used to train, fit and make predictions. The y variable, being the target, was composed of train.SalePrice. I employed feature selection to define the X variable, so dropped a number of columns in an attempt to improve accuracy.

I then used sklearn’s train_test_split function to break the X dataset into four smaller datasets for training, fitting and predicting:-

I then created a pipeline where the data would be scaled and then trained and fit into sklearn’s GradientBoostingRegressor. I achieved 99.96% accuracy when I employed this technique.

I predicted on the validation set and attained 95.21% accuracy:-

I created a dataframe where the actual values in the validation set can be compared to the predicted values:-

After making predictions on the Ames House Prices train dataset, I wanted to obtain information of outliers. I created another model, which was sklearn’s IsolationForest in this instance. I set contamination to 1%, which means that 1% of the examplies would be considered outliers.

I created two new columns called “scores” and “anomaly” in the train dataset. Each row is given a score based on the “SalePrice” and values of either 1 or -1 are placed in the “anomaly” column. If the value of -1 is present in the “anomaly” column then this indicates the price is an outlier:-

I then printed out a list of anomalies. Fifteen anomalies came up, which is 1% of the number of rows in the dataset:-

I then decided I wanted to see all of the sale prices greater than $500,000 and there were 9 properties that met this criteria:-

I checked the accuracy and it was 166.67%, the reason being I was dividing 15 anomalies by 9 outliers.

I also made a visual representation of the scores of each property as compared to its sale price:-

The code for this program can be found in its entirety in my personal GitHub account, found here:- House-Prices/Kaggle_Ames_Housing_Prices_Isolation_Forest.ipynb at master · TracyRenee61/House-Prices (github.com)