Data Analysis Project — Telco Customer Churn
By: John Chen, Yueh-Han
Tags: Analytics, Data, Data Science, Data Visualization, Towards Data Science

Step 1: Gather the data

For the statsmodels.api module, since I will use multiple linear regression analysis for question 8, I included here.

Step 2: Assess and clean the data

Here, since the purpose of this article is to show what I did and my process of conducting a data analysis project broadly, so I won’t go over every detail. Instead, I will pick the most important procedures that I did. In case you really want to check out every detail I did, see here.

Assessing:

Usually, I would love to view all the column all at once, so I use pandas.set_option to do this:

df.describe()

df.Contract.value_counts()

Month-to-month: one-year: two-year is approximately 2.5: 1: 1

# show the distribution of tenure.

plt.hist(data = df, x = 'tenure');

This is apparently not a normal distribution. And with two peaks, there are two extreme kinds of people among all customers, and I will investigate what services have kept those who stay more than 70 months the most.

Assessment report:

There is no tidiness issue and only 2 issues here to consider it a very clean dataset. Here are the 2 Quality issues:

The data type of “TotalCharges” should be the float64 type instead of the object type.

Many rows of “TotalCharges” do not equal each tenue times monthly charges.

First thing first, we should copy our original dataset:

df_copy = df.copy()

Cleaning 1: The data type of “TotalCharges” should be the float64 type instead of the object type

Action: Give None value to all row, then convert it to the data type of float64. (will recalculate it later)

df_copy.TotalCharges = None

df_copy.TotalCharges=df_copy.TotalCharges.astype(float)

Test the results:

df_copy.TotalCharges.dtype

Output: dtype(‘float64’)

Cleaning 2: Many rows of “TotalCharges” do not equal each tenue times monthly charges.

Action: recalculate it, let each tenures times monthly charges

df_copy.TotalCharges = df_copy.tenure * df_copy.MonthlyCharges

Test the results:

df_copy[df_copy.tenure * df_copy.MonthlyCharges != df_copy.TotalCharges].shape

Output: (0, 21)

The final step of the cleaning process: save the data.

# store the clean data

df_copy.reset_index(drop=True)

df_copy.to_csv('Telco-Customer-Churn_clean.csv')

Step 3: Conduct exploratory data analysis to answer the questions & create visualizations (Final visualization code)

Before writing any visualization, I like to create a reusable function, so I can save lots of time without writing the same codes:

Research Question 1: For two groups of unsubscribed and who are paying for the service, how long did they usually stay in the service? And what was their average LTV(Life Time Value)?

Firstly, I extracted those who unsubscribed from the service and examined the distribution of tenure.

Churn_df = clean_df.query('Churn=="Yes"')

Churn_df.TotalCharges.describe()

Let’s see its distribution:

plt.hist(data = Churn_df, x = 'TotalCharges');

After viewing the distribution and the 5-number summary, I found that around 80% of the data are extremely high, so I decided to divide them to 80% and the rest 20% to see each data distribution. (Applying 80/20 rule in real life.)

#find the 80th percentile of the data in total charges

Churn_df.TotalCharges.quantile(0.8)

Output: 2827.5900000000006

# Divide the data by the 80th percentile of the data, and show the distribution of its TotalCharges under 80th percentile

TotalCharges_under80 = Churn_df.query('TotalCharges<=2827.59')

TotalCharges_above80 = Churn_df.query('TotalCharges>2827.59')

Let’s visualize our unsubscribed customers divided by 80/20 group:

Ok, now, let's see the graph that combines those who unsubscribed and those who are paying for the service.

Note: 80%(low LTV) of leaked customers only stayed under 10 months. And, the average LTV of 80% of those who unsubscribed is 750 dollars. On the other hand, the average LTV of top 20% of those who unsubscribed is 4750 dollars. And the ratio by the sum of total LTV by each groups is 750*4 : 4750 = 1 : 1.6, which suggests we should focus on serving those 20% customers with high LTV, which brought 60%(1.6/2.6) of our revenue from leaked customers

Research Question 2: For those who unsubscribed and still are paying the service, what are the proportion of people used and using phone service?

Note: 100% of those who brought high LTV used and are using internet service.

Research Question 3: For two groups of those who unsubscribed and who are paying for the service, what are the proportion of people used and using phone service with multiple lines?

Note: 81% of those who bring high LTVs tends to used lines. There is no big difference between those who are still paying for the service or who unsubscribed.

Research Question 4: For those who unsubscribed and still are paying the service, what are the proportion of people used and using internet service?

Note: Those who bring high LTVs, loved to use Fiber optic(75%-90%) and DSL(10–20%), and none of them weren’t using internet service. While those who bring low LTVs, in terms of current customers, around 30% of them don’t use internet service.

Now, let’s dive deeper to see the situation of each subset of internet service.

Research Question 5: For those who unsubscribed and still are paying the service, what are the proportion of people used and using each subset of internet service?

Firstly, Extract 80% with low LTV who used the internet service, and save each proportion of subsets of internet service in the variable “proportion_internet_sub_service”:

And then extract 20% with high LTV who used the internet service, and save each proportion of subsets of internet service in the variable “proportion_internet_sub_service_above80”

Secondly, Extract 80% with low LTV who used the internet service, and save each proportion of subsets of internet service in the variable “paying_proportion_internet_sub_service_under80”

And then extract 20% with high LTV who used the internet service, and save each proportion of subsets of internet service in the variable “paying_proportion_internet_sub_service_above80”

Let’s visualize it:

Note: Among 80%(low LTVs) of data of current customers, all the subsets of internet services are equally used by 40% of people. And among 80% (low LTVs) of current customers, streaming movies and streaming TV are two top subsets of internet service people use, and device protection and online backup are the second place, and tech support and online security are both third place, the gaps between these are near 10% of whole data.

Research Question 6: For those who are still in service with LTV higher than the average LTV of leaked customers, what are the differences compared to general current customers in each service?

First thing first, I extracted the groups of data I want to investigate:

1. Let’s look at the phone service with multiple lines:

2. For internet service:

Note: Among 80% of current customers, those whose LTV are more than the average LTV of 80% of leaked customers are 10% more likely to use multiple lines and internet service than the average 80% of current customers.

Research Question 7: For two groups of those who leaked and still are paying the service, what is the proportion of each kind of contract by each group?

Note: 89% of those who leaked used monthly contract, while only 42% of current customers are using monthly contract.

Research Question 8: Among ‘gender’, ‘Partner’, ‘Dependents’,’PhoneService’, ‘InternetService’, ‘contract’, and ‘PaymentMethod’, what variable affected the LTV the most?

I used multiple linear regression for this questions, so first, because most of them are categorical variables, so I needed to convert them into dummy variables:

Use multiple linear regression:

Note: Whether people use internet service is the most important factor to create high LTV, and yearly contract is second place.

Note: Among all the subsets of internet services, online backup is the most important factor to create high LTV

Step 4: Understand the limitations

Before drawing any conclusion, it is always better to inform the limitations. During the process of assessing data and conducting the exploratory data analysis, I have found some limitations:

Limitation 1: In this dataset, we can only see one type of each variables instead of real world situation of changing different options as time passes, i.e., in real world, people might wanna try streaming service, but they might change their mind to use tech supports next month.

Limitation 2 : We cannot consider all these variables as the exact reasons why customers left because they might leave for better price offered by competitors or the bad economy in certian time, etc. We also cannot see the time they leaked, so it’s hard to infer those external situation.

Step 5: Summaries

80%(low LTV) of leaked customers only stayed under 10 months. And, their average is 750 dollars. On the other hand, the average LTV of top 20% of those who leaked is 4750 dollars. And the ratio by the sum of total LTV by each groups is 750*4 : 4750 = 1 : 1.6, which suggests we should focus on serving those 20% customers with high LTV, which brought 60%(1.6/2.6) of our revenue from leaked customers 81% of those who bring high LTVs tends to used lines. There is no big difference between those who are still paying for the service or who unsubscribed. Those who bring high LTVs, loved to use Fiber optic(75%-90%) and DSL(10–20%), and none of them weren’t using internet service. While those who bring low LTVs, in terms of current customers, around 30% of them don’t use internet service. Among 80%(low LTVs) of current customers, all the subsets of internet services are equally used by 40% of people. And among 80% (low LTVs) of current customers, streaming movies and streaming TV are two top subsets of internet service people use, and device protection and online backup are the second place, and tech support and online security are both third place, the gaps between these are near 10% of whole data. Among 80% of current customers, those whose LTV are more than the average LTV of 80% of leaked customers are 10% more likely to use multiple lines and internet service than the average 80% of current customers. 89% of those who leaked used monthly contract, while only 42% of current customers are using monthly contract. Whether people use internet service is the most important factor to create high LTV, and yearly contract is second place. Among all the subsets of internet services, online backup is the most important factor to create high LTV

Step 6: Actionable insights

For retaining customers:

Focusing on marketing to those currently only subscribed to streaming movie and TV service to subscribe our tech support and online security and then device protection and online backup. Based on graph from Q5, the more subsets of the internet service they’re paying for, the longer they tend to stay.

For increasing customers’ lifetime values:

Increase marketing on our streaming movie and TV service, since these two sectors have biggest gaps between those who brought 80%(low LTV) and those who brought 20%(high LTV), no matter they leaked or are current customers. Increase marketing on online backup, except for paying for both of streaming movie and TV service, online backup alone is the most important factor to create high LTV. Increasing marketing/promotion on those who want to use multiple lines.

Photo by Franki Chamaki on Unsplash

Finally, data has a better idea, but you should always doubt any insight and the data to be some degree wrong or have some biases, no matter who provided them to you are experts or people with high credibility.