Life Insurance Risk Prediction using Machine Learning Algorithms- Part I: Data Pre-Processing and Dimensionality Reduction
By: Bharat Sethuraman Sharman
Tags: Insurance, Classification, Predictive Analytics, Machine Learning

Life Insurance Risk Prediction using Machine Learning Algorithms- Part I: Data Pre-Processing and Dimensionality Reduction

In this two-part series, we will describe our experience of working on the Prudential Life Insurance Dataset to predict the risk of life insurance applications using supervised learning algorithms. We worked on this dataset as a part of our final group project in a graduate course on Statistical Learning that we took at the University of Waterloo in which we reproduced the results of a paper¹ and improved upon the work of the authors.

The link to the project GitHub repository is here and you can find the link to our YouTube video here

Business Context

Companies that underwrite life insurance policies have to evaluate applications carefully. The payouts from life insurance claims are very high relative to the insurance premiums that companies collect from an individual customer. For example, a person who purchases a $ 500 a year, $1 million sum-assured 40-year term plan would be paying $ 20,000 over 40 years but in case if a claim arises, the insurance company would have to pay $ 1 million to the dependents of this individual. So the company has to be selective about people whom it chooses to insure to keep its business financially viable.

Traditionally, companies have used actuarial tables that involve applying detailed rule-based procedures to qualify a life insurance application. Some of these rules are formulae based and others are rules of thumb. These rules do incorporate the wealth of experience that actuaries have gained over time in evaluating applications and are therefore generally accurate. The flip side however is that they involve a lot of manual tasks like information gathering and subjective evaluations about various aspects of an application. It is not uncommon for life insurance applications to take over a month from the date of application to be issued.

Researchers have applied machine learning techniques to perform predictive analytics and automate the life insurance application evaluation process. The fundamental idea here is that rating a life insurance application is a supervised learning problem. Here, an application is treated as a data point, the data columns provided by an applicant are the features of this data point and the risk rating of the applicant is the output that we are trying to predict.

So the business context of this data science problem is that we seek to significantly reduce the processing time involved in the issuance of a life insurance application and save costs for the company through greater automation while making sure that the accuracy of risk assessment is not compromised.

Description of the Dataset

The Prudential Life Insurance Dataset consists of information from 59,381 life insurance applications and the risk rating that these applications have been assigned by the company. Each application consists of 126 features that are either continuous, discrete, or categorical. The table below shows the names of the features and gives a brief description of what they represent:

Brief Description of the Features of the Dataset¹

The features that are categorical, continuous, and discrete have been mentioned in the dataset link here. We use the word features and variables interchangeably in this context.

The features can be broadly classified into the following categories:

Medical Information- This includes medical history, height, age, weight, BMI, and presence of any medical conditions Family History- While it is not mentioned what this term means but it can include variables that indicate marital status, number of children, etc Insurance History- Again, this is not stated explicitly what this set of features implies but it could mean whether the person was insured previously, whether one has missed paying premiums in the past, the insurance coverage level, etc Personal Information: This category could include employment status, type of occupation, salary scale, seniority, and also information such as safety level of the PIN code of residence, ownership of a car, driver rating among other such information that can be relevant for making life insurance-related decisions. Product Information: This is product-specific information. So this can mean things like the sum assured amount, length of the plan, whether the payment is lumpsum or staggered amongst others. It might not be immediately obvious how product features could influence the risk rating of an applicant. Perhaps applicants who opt for a short-term plan of say 10 years (which is considered short in the life insurance context) could be considered high risk by the company and classified as such.

There are 60 categorical, 48 dummy, 13 continuous, and 5 discrete features, making up a total of 126 variables. The response or the output variable is the risk rating and it has 8 levels — 1 to 8 where 1 is the lowest and 8 is the highest risk rating.

Dealing with Missing Values

There were 9 features that had more than 30% of data missing. These features are shown in the table below:

Features that have a high percentage of missing values (Image by authors)

We deleted these features from the dataset.

There were 4 features that had anywhere between 0.03% to 18.28% missing values. These are shown in the table below.

Features that have a relativeyl low percentage of missing values (Image by authors)

Not all missing values are created alike. There can be many reasons why data might be missing and based on why the data might be missing, missing data can be classified into three categories:

Missing Completely at Random (MCAR): When data is missing completely at random, it is equally likely for each data point to be missing from the dataset. Let’s say that we are taking the measurements of ambient air temperature every 15 minutes between 8 AM and 12 Noon. So, if everything went on well, we would have 17 measurements taken at 8, 8:15,8:30….. and so on up to 11:45 and 12. Now, let’s suppose there is a power shut down for an hour between 8:55 and 9:35, and as a result, we don’t have the measurements for 9,9:15, and 9:30. The power could have gone at any time and therefore, it is equally likely for any of the 17 data points to be missing. There is no specific reason to assume that the measurements of these time points are more likely to be missing than other time points. Therefore, such missing values are said to be missing completely at random.

Missing at Random (MAR): Let’s say that there is electrical maintenance work going on in the area where the measurements are being taken. The technicians perform the repair work generally between 8 AM and 10 AM and the repair work is going to take a week’s time. It is expected that there will be a power outage for about 30 minutes during this two-hour period every day. In this situation, we will find that there are two categories of measurements — those taken in the 8–10 AM time interval and those taken in the 10 AM –12 Noon time interval. There will be more data that will be missing in the former class than in the latter class. However, within that class, it is equally likely for any data point to be missing. In other words, while we are going to see fewer measurements between 8–10 AM than 10 AM-12 Noon, within 8–10 AM time interval, 8:15, 8:45, 9:15, and 9:45 are going to be equally likely to be missing as the half an hour time interval of the power outage is random within the 8–10 AM time interval. Such missing values are said to be missing completely at random.

Missing Not at Random (MNAR): Let’s say that there is no power outage in the area and the measurements are being taken over a period of 5 years. The working life of the equipment is 5 years. As the equipment ages, it starts to miss recording values more often, especially after 3 years. Let’s say that the researcher is not aware of this issue with the equipment. He will observe that more values are missing after 3 years than before. At the same time, let’s say that the environment is getting warmer with time. So, there will be hotter weather for a period of 3 years and beyond. Now, the researcher will not know whether the equipment has missed recording these values because the temperature was higher or whether the values went missing because of the aging of the equipment. Such missing values are hard to handle and are known as missing not at random.

To check whether the data is MCAR, there is a test known as the Little’s Test and can be implemented in R. Here’s the code for performing the test:

Code for performing Little’s MCAR Test in R (Image by authors)

We got a p-value less than 0.05 implying that the data is not MCAR. The alternative mechanisms could then be either MAR or MNAR.

The plot of the top three variables with missing values is shown below: