Exploratory Data Analysis (EDA) solution to Kaggle caravan insurance challenge on R
By: Kieran Tan Kah Wang
Tags: Kaggle, Data Analytics, Caravan, R Programming, Exploratory Data Analysis

Photo by Kevin Schmid on Unsplash

The Caravan Insurance Challenge was posted on Kaggle with the aim in helping the marketing team of the insurance company to develop a more effective marketing strategy. The dataset consists of 5822 records of customer data collected by the insurance company on 85 different socio-demographic and product-ownership data features.

Machine learning (ML) algorithms is the solution to help identify those customers who are most likely to purchase caravan policies especially since we are given 5822 training data and 238 testing data, but the purpose of writing this article, is to tackle the problem using just Exploratory Data Analysis (EDA) without the complicated ML models.

Overall data statistics

After obtaining the datasets from the Kaggle link, we can read the training data ticdata2000.txt into R. I did mine using Jupyter Notebook with the R kernel installed as I find Jupyter Notebook more user-friendly and effective for me to debug any errors. You can find the link to install R kernel onto Jupyter Notebook with this link.

After reading in the data, we can get a sense of how each data attribute looks like and as mentioned, each record has 86 features represented by V1 — V86. It was further mentioned all attributes have been discretised and attribute 1–43 are assigned based on the zip code the customers stay in.

Using the summary command, we can get the statistics of each attribute and get a general sense of each attribute better.

Using the describe command, more detailed statistics can be displayed.

Missing values is one of the most problematic in every data analysis or data science project especially if we need to decide to discard these entries or try to fill in the empty values using methods like mean, max or even regression. Thankfully in this training data, there were no missing values for any of the 86 attributes.

Summarising from the overall statistics, there are 86 attributes (columns) in which attributes 1–43 contains socio-demographic data and attributes 44–86 contains product ownership, with attribute 86: Number of mobile home policies being the target variable. Attributes consist of mainly categorical with subtype nominal or ordinal, as well as numerical with discrete subtype.

The biggest challenge

The reason why this was a posted as a Kaggle challenge was the fact that though there were 5822 customer records, there were only a small percentage of customers who bought caravan insurance, making any data analysis or machine learning harder as this dataset is greatly imbalanced. In fact, only 348 (6%) of the customers bought them.

Correlation between variables

We proceed to explore possible correlation between variables in our EDA using the cor command. A correlation coefficient near to 1 or -1 means both attributes are highly correlated (i.e. a rise in variable A will give a rise in variable B if correlation coefficient is near to 1).

But yes, it’s hard to read especially since we have 86 attributes and the cor command trying to provide us with correlation coefficients of each attribute against the remaining 85. Let’s try to visualise it using a graphical representation, level plot. Note that you will need the library lattice to run the levelplot command together with cor.

Hmm, it doesn’t seem very helpful but it certainly shows there’s some attributes being correlated to others. We make use of a loop to identify those variables that have high correlation between them, by filtering the correlation coefficient to be <-0.5 or >0.5.

Note that the clear light blue diagonal line in the middle is expected as each variable is positively correlated to itself.

We see that it’s easier to know which attributes each attribute is correlated to and we can easily observe the following:

Customer Sub Type (V1) and Customer Main Type (V5) are highly correlated.

and are highly correlated. Average Household Size (V3) increases as Household with Children (V15) increases.

increases as increases. Average Household Size (V3) decreases as Singles (V13) increases.

decreases as increases. Singles (V13) are more probable to own no car (V34) .

are more probable to own . High level education (V16) , High Status (V19) , Social Class A (V25) are more likely to purchase Private health insurance (V36) .

, , are more likely to purchase . Lower level education (V18) , Social Class C (V28) are more likely to purchase National Health Service (V35) .

, are more likely to purchase . Contribution of a certain policy is highly correlated to the number of that policy. For instance, V44 - Contribution private third party insurance is highly correlated to Number of private third party insurance . This is consistent from V44 - V64 against V65 - V85 . The higher the number of a particular policy you have, the higher your contribution towards that policy.

is highly correlated to . This is consistent from against . The higher the number of a particular policy you have, the higher your contribution towards that policy. There is greater fire policies (V80) insured under third party insurance (V65) .

insured under . There is greater tractor policies (V73) insured under third party agriculture insurance (V67).

We explore the relationship between Customer Main Type (V1) and Customer Sub Type (V5) since the correlation coefficient between them is really high (0.993). We will use the ggplot command which is a common visualisation command in R under ggplot2 library, and plot a coloured bar chart.

Looking at the plot, we can conclude that:

Customer Main Type 1 is represented by Customer Sub-Type 1–5

is represented by Customer Main Type 2 is represented by Customer Sub-Type 6–8

is represented by Customer Main Type 3 is represented by Customer Sub-Type 9–13

is represented by Customer Main Type 4 is represented by Customer Sub-Type 15–19

is represented by Customer Main Type 5 is represented by Customer Sub-Type 20–24

is represented by Customer Main Type 6 is represented by Customer Sub-Type 25–28

is represented by Customer Main Type 7 is represented by Customer Sub-Type 29–32

is represented by Customer Main Type 8 is represented by Customer Sub-Type 33–37

is represented by Customer Main Type 9 is represented by Customer Sub-Type 38–39

is represented by Customer Main Type 10 is represented by Customer Sub-Type 40–41

is represented by There is no Customer Sub Type 14 (Junior cosmopolitan) in this training dataset

(Junior cosmopolitan) in this training dataset The most common belongs to Customer Main Type 8 (Family with Grown Ups) with Customer Subtype 33 (Lower class large families)

Distribution of attributes against the target variable (V86)

What we would like to know is whether there are any variables that exhibit certain patterns/trends towards our target variable V86, as the observations can be particularly useful when we are trying to predict and identify potential mobile home customers. Hence, we explore the distribution of attributes against V86.

We first explore V5 — Customer Main Type using ggplot and plotting the bar chart similar to before. Note that we use grid.arrange to plot two charts in one console output in Jupyter Notebook.

We see that the number of caravan policies is most common in customer main type 8 (Family with grown ups) followed by customer main type 2 (Driven Growers). Note that customer main type 4 (Career Loners) has 0 mobile home policies for all of them.

We do the same to customer subtypes.

We see that the number of mobile home policies is most common in customer sub type 8 (Middle class families) and type 33 (Lower class large families).

Also note that the following customer sub type has 0 mobile home policies:

Customer Sub Type 14 (Junior cosmopolitan)

Customer Sub Type 15 (Senior cosmopolitans)

Customer Sub Type 16 (Students in apartments)

Customer Sub Type 17 (Fresh masters in the city)

Customer Sub Type 18 (Single youth)

Customer Sub Type 19 (Suburban youth)

Customer Sub Type 21 (Young urban have-nots)

Customer Sub Type 28 (Residential elderly)

Customer Sub Type 40 (Large family farms)

We then explore if there are any noticeable trend or pattern in attribute V2 (Number of houses), V3(Avg size household) and V4(Avg age) with the target variable V86.

Looking at the plots, we can observe the following:

For V2 — Number of houses,

Mobile home policy buyers owns 1 or 2 houses

There are 0 buyers from those who owns 3 to 10 houses

For V3 — Avg size household,

The most common is of size 3 that purchased mobile home policy

There are 0 buyers from those whose average household size is 6

For V4 — Avg age,

The highest group of mobile home policy buyers belong to group 3 (40–50 years old)

The least common groups are group 1 (20–30 years old) and group 6 (70–80 years old)

Next, we would like to know if religion plays a part in affecting the purchase of mobile home policies. We can plot histogram using hist command for each of V6 — V9 which represents different religion groups, against our target variable V86. Note that par(mfrow) can be used to plot multiple plots in a single console output.

From the 4 histograms plot for the attributes relating to religion (V6 — V9), it seems to suggest V6 — Roman Catholic are least likely to buy mobile home policies, while frequency of V7 — Protestant and V9 — No Religion of buying is higher.

Next, we explore the marital status (V10 — V13) as well as presence of children (V14 — V15). Since from V1, we have identified the main buyers of mobile home policies belong to Middle class families as well as lower class families, we will expect a higher percentage of V10 — Married to own mobile home policies.

As expected, those who owns mobile home policies are most common in marital status of V10 — Married. There seems to be a fair distribution between household without children and household with children.

In the correlation coefficient matrix, we discovered that singles are more likely to have no car. Since we know that the majority of the mobile home policies buyers are not single, we will also expect those without any car to not have purchased the mobile home policies logically.

We plot the number of cars (V32 — V34) to explore this.

From the plots, we see majority of mobile home policies owners to have 1 car. What’s interesting is there are some who have no car and purchased mobile home policies. Hence, we can conclude the term cars in this dataset does not include mobile home caravan.

V16 — V18 carries information about the education level and from the correlation coefficient/matrix we have observed earlier on, education level seems to be highly correlated to social class status (V25 — V29). Let’s see if we can observe any noticeable pattern in any particular group with respect to buying mobile home policy. Note that plot() is another inbuilt R command to build bar chart.

Mobile Home policies seem to be less common in high level education and more common with medium and low education. This also corresponds to mobile home policies being less common in social class A and more common in social class B1, B2 and C. It is also notable that it is not common for social class D.

From the observations from education level and social class, it should be expected that mobile home policies will be more popular with those drawing average income. Let’s plot the income attributes V37 — V42 as well as the purchasing power V43.

As seen in the plots, mobile home policies are more popular with customers drawing average income of $30000 -$75000, and those with medium to high purchasing power.

Now, let’s explore V19 — V24 which are attributes pertaining to the occupation of the customers. From earlier plots, we know mobile home policies are more common in customers from social class B1, B2 and C, and those drawing average income of $30000-$75000. Furthermore in the correlation coefficient matrix, we know different occupations are correlated to different social class.

Looking at the plots, it seems to suggest that the occupations of mobile home policies buyers are diverse, in which there is not really a particular occupation who is most likely to purchase the policy. The only observation that can be observed is if there is a high percentage of farmers in the household, that particular household is least likely to purchase mobile home policies.

Let’s see if V30 — rented house and V31 — home owners has any observable pattern towards purchasing of mobile home policies.

Since there is a high correlation between this 2 attributes (i.e. if the customer is not owning the house, they will definitely be renting the house). What’s observable is home owners are more likely to purchase mobile home policies compared to customers who are renting their houses.

In our correlation matrix above, we noted that customers with high level education, high status and belonging to social class A are more likely to purchase private health insurance. Whereas for customers with lower level education and belonging to social class C, they are more likely to purchase National Health Service. Furthermore, we have also observed that customers of low education level and lower social class are more likely to purchase the mobile home policies. Let’s explore this by plotting V35 — National Health Service and V36 — Private health insurance against our target variable V86.

Even though lower level education and lower social class are more likely to purchase mobile home policies, and they are also more likely to purchase national health service insurance, the relationship of customers who bought national health service insurance and purchase of mobile home policies is not distinctly distinguishable as suggested by the plots. Both group of customers in the plot above are equally likely to purchase mobile home policies.

Attributes V44 to V85 are related to insurance products ownership and in the correlation matrix earlier on, we noted that contribution to each policy is highly correlated to number of that policy. What we would like to find out is are there any particular policies which is highly correlated to mobile home policies as well, meaning what other policies will the customer buy when they purchased the mobile home policy or vice versa.

We first plot all the contributions towards each insurance type from V44 to V85 as a bar chart and the their respective numbers as a pie chart using pie command for comparison.

Plotting all insurance type from V44 to V85 for those customers who bought caravan policies, we can get the following summary from the product ownership attributes:

From these observations, we can conclude that mobile home policies will not buy lorry and agricultural machine policies. On the other hand, they are also likely to purchase car policy and contribute $1000 to $4999, fire policy and contribute $200 — $499, and maybe private third party insurance with contribution at $100 — $199.

To get a better representation on the exact numbers of other policies bought by caravan policy customers, we can do a simple aggregation.

It is observed that for the 348 customers who bought mobile home policies:

276 (79%) of them also bought car policies (V68)

239 (69%) of them also bought fire policies (V80)

201 (58%) of them also bought private 3rd party insurance (V65)

0 of them bought lorry policies (V71), agricultural machine policies (V74)

Overall summary

Summarising our findings from the distributions of other attributes against the target variable, a high percentages of them out of the 348 customers who bought caravan policies came from the following:

comes from Middle class families & lower class families

are family with grown ups or drivern growers

possesses 1 or 2 houses

average household size is 3

belongs to 40–50 years old

Protestant & No religion dominates

Married

received medium and low education level

social class B1, B2, C

possess 1 car

average income is around $30000 to $75000

owns car policy and contribute $1000 to $4999

owns fire policy and contribute $200 — $499

owns private third party insurance and contribute $100 — $199

Intuition from these observations tell us customers of mobile home policies are middle-sized families with average income, in which they own property and car, resulting in them owning car and fire policy as well. Possible explanation is if a family is drawing average income, they are at higher chance to purchase a caravan as it is relatively cheaper to go on a holiday with a caravan in their home country compared to taking the plane overseas for holiday. Also, caravan are normally towed by car which explains why there are high number of mobile home policies customers owning car and car policy as well. These customers owns property and will have the tendency to purchase fire policy for their property/car unassumingly.

Looking at the other side of the spectrum, these 348 customers did not come from:

career loners

Junior cosmopolitan, Senior cosmopolitans, Students in apartments, Fresh masters in the city, Single youth, Suburban youth, Young urban have-nots, Residential elderly, Large family farms

who owns 3 to 10 houses

whose average household size 6

who owns lorry policy

who owns agricultural machine policy

Looking at these observations, it is highly unlikely for customer who is alone and young, or customer with large families who either are elderly or stay in farms to purchase caravan. Logically speaking, this makes sense as if you are young, your priority would not be to buy a caravan as you will not have the income to do so, and if you are alone, buying a car is more enticing than buying a caravan. Similarly, if you are elderly, you will not have the energy to go on a holiday with a caravan as it’s considered more adventurous. Buying agricultural machine/lorry will also be more meaningful if you are staying in the farm.

As such, the possible factors that we have identified through EDA that can help the marketing team of the insurance company to develop a more effective marketing strategy are as follows:

Before we dive into using techniques subset selection or shrinkage method, based on the exploratory data analysis, we expect the following attributes to be significant in helping us predict if a customer will purchase a mobile home policy:

V1 — Customer Subtype

2) V2 — Number of houses

3) V3 — Avg size household

4) V4 — Avg age

5) V5 — Customer main type

6) V10 — Married

7) V17 — Medium level education

8) V18 — Lower level education

9) V26 — Social class B1

10) V27 — Social class B2

11) V28 — Social class C

12) V32–1 Car

13) V38 — Income 30–45.000

14) V44 — Contribution private third party insurance

15) V47 — Contribution car policies

16) V50 — Contribution lorry policies

17) V53 — Contribution agricultural machines policies

18) V59 — Contribution fire policies

19) V65 — Number of private third party insurance

20) V68 — Number of car policies

21) V71 — Number of lorry policies

22) V74 — Number of agricultural machines policies

23) V80 — Number of fire policies

Thank you for reading, and in my next post, I will continue the machine learning portion to this particular challenge which utilises features selection methods and ML algorithms like Logistic Regression, Linear Discriminant Analysis and Quadratic Discriminant Analysis to identify possible attributes for the marketing team to consider looking into.