Predicting Claims Severity: A Machine Learning Approach
By: Fardil Bhugaloo
Tags: Ensemble, Insurance, Claims, Severity, Machine Learning

3. Prepare Data

Detect and handle outliers in the dataset

Many machine learning models are susceptible to outliers, if there are outliers the machine learning (ML) model will take them into considerations, thus degrading the performance of the model and increasing the error of the model. That’s why they need to be handled.

From the whiskers of the boxplot above it can be seen that cont7, cont9 and cont10 have outliers.

We use the IQR (interquartile range) method of outlier detection. Any data point that fell outside of either 1.5 times the IQR below the first quartile or 1.5 times the IQR above the third quartile is “outside” or “far out”.

Compared to the previous boxplot, the whiskers have shifted inside, there are still some but most of them are inline, we have been able to reduce the variability of the dataset.

Feature elimination techniques for continuous and categorical features

Why feature elimination/selection?

Model interpretability becomes easier with less features Easier to implement machine learning with fewer features as it takes much lesser time to train such model Less features means less overfitting Help us remove redundant data Reduced features makes model less prone to error

Multivariate method

Detecting correlation between variables in the dataset: correlation — indicates the extent to which two or more variables fluctuate in relation to each other. If 2 variables are correlated then adding such variable in the dataset do not tend to add much information as both are feeding similar information and model will not be able to make judgement by taking both into consideration so it is best to eliminate one of them and retain only one, it also help to stabilize the model and help it make better prediction

Eliminated variables:

cont6 - high correlation with cont11 , so cont6 has been removed

high correlation with , so has been removed cont10 - high correlation with cont1 and cont6 , so cont10 has been removed

high correlation with and , so has been removed cont12 - high correlation with cont7 and cont13, so cont12 has been removed

Now to decide which variables to remove depend on business understanding as well and the business need. If you want to keep certain variables in the dataset which will be easier to explain to the business, then you should retain that and eliminate other variables which are correlated to it.

Handling correlation between categorical variables

Machine learning models require all input and output variables to be numeric. This means that if your data contains categorical data, you must encode it to numbers before you can fit and evaluate a model.

There are 2 types of features transformation: Label encoding and One-hot Encoding. We used Label encoding

Identify the correlated categorical variables and drop them

Chi-Square Test

A Chi-Square test is a test of statistical significance for categorical variables. We start by defining the null hypothesis (H0) which states that there is no relation between the variables. An alternate hypothesis would state that there is a significant relation between the two. We verify the hypothesis by using p-value. We use a 5% significance level to determine whether the relation between the variables is of considerable significance. If p-value is less than 0.05, then reject H0, there exist a relationship between the variable and one variable is dropped.

We maintain a file columns_to_drop_df so that whenever we load a validation dataset or a new dataset, these columns can be directly dropped before generating prediction, as these columns we will no longer be used for building the ML models.

Visualizing the Output Variable

The density plot of loss below provides the information about the distribution of the output variables. It indicates that the observation is skewed to the right. To prevent high leverage from the outliers, we did a shifting and log transformation on ‘loss’, so it can be more normally distributed without changing the order. Second density plot is the result after transforming the output variable.

Loss without transformation