Data Analysis Project — Telco Customer Churn | by John Chen (Yueh-Han) | Towards Data Science

To extract actionable insights from the dataset. I listed all the questions that came to mind below after assessing the dataset, and I tried to investigate all of them to find the insights:
1. How long did unsubscribed people who are paying for the service usually stay in the service? And what was their average LTV(Life Time Value)?
2. How many of those who unsubscribed and still subscribe used the phone service?
3. How many people use phone services with multiple lines for those who unsubscribed and still subscribe?
4. How many people used the internet service for those who unsubscribed and subscribed?
5. How many people used and using each subset of internet service for those who unsubscribed and who are paying for the service?
6. What are the differences in each service internet service for those who are still in service with LTV higher than the average LTV of leaked customers,?
7. How many subscribed to each kind of contract for those who unsubscribed and still are paying for the service,?
8. Among ‘gender’, ‘Partner’, ‘Dependents’,’PhoneService’, ‘InternetService’, ‘contract’, and ‘PaymentMethod’, which variables affected the LTV the most?
This dataset is IBM Sample Data Sets that I founded at Kaggle.
Each row represents a customer; each column contains the customer’s attributes described in the column Metadata.
The data set includes information about:
Customers who left within the last month — the column is called Churn.
Services that each customer has signed up for — phone, multiple lines, internet, online security, online backup, device protection, tech support, and streaming TV and movies
Customer account information — how long they’ve been a customer, contract, payment method, paperless billing, monthly charges, and total charges
Demographic info about customers — gender, age range, and if they have partners and dependents
Let’s start by explaining my whole data analysis steps in this project:
Step 1: Gather the data
Step 2: Assess and clean the data
Step 3: Conduct exploratory data analysis to answer the questions & create visualizations (Final visualization code)
Step 4: Understand the limilations
Step 5: Summaries
Step 6: Actionable insights
Step 1: Gather the data
For the statsmodels.api module, since I will use multiple linear regression analysis for question 8, I included here.
Step 2: Assess and clean the data
Here, since the purpose of this article is to show what I did and my process of conducting a data analysis project broadly, so I won’t go over every detail. Instead, I will pick the most important procedures that I did. In case you really want to check out every detail I did, see here.
Usually, I would love to view all the columns all at once, so I use pandas.set_option to do this:
df.describe()
df.Contract.value_counts()
Month-to-month: one-year: two-year is approximately 2.5: 1: 1
# show the distribution of tenure.plt.hist(data = df, x = 'tenure');
This is apparently not a normal distribution. And with two peaks, there are two extreme kinds of people among all customers, and I will investigate what services have kept those who stay more than 70 months the most.
There is no tidiness issue and only 2 issues here to consider a very clean dataset. Here are the 2 Quality issues:
The data type of “TotalCharges” should be the float64 type instead of the object type.
Many rows of “TotalCharges” do not equal each tenue times monthly charges.
First thing first, we should copy our original dataset:
df_copy = df.copy()
Action: Give None value to all rows, then convert it to the data type of float64. (I will recalculate it later)
df_copy.TotalCharges = Nonedf_copy.TotalCharges=df_copy.TotalCharges.astype(float)
Test the results:
df_copy.TotalCharges.dtype
Output: dtype(‘float64’)
Action: recalculate it, let each tenures times monthly charges
df_copy.TotalCharges = df_copy.tenure * df_copy.MonthlyCharges
Test the results:
df_copy[df_copy.tenure * df_copy.MonthlyCharges != df_copy.TotalCharges].shape
Output: (0, 21)
The final step of the cleaning process: save the data.
# store the clean datadf_copy.reset_index(drop=True)df_copy.to_csv('Telco-Customer-Churn_clean.csv')
Step 3: Conduct exploratory data analysis to answer the questions & create visualizations (Final visualization code)
Before writing any visualization, I like to create a reusable function, so I can save lots of time without writing the same codes:
Firstly, I extracted those who unsubscribed from the service and examined the distribution of tenure.
Churn_df = clean_df.query('Churn=="Yes"')Churn_df.TotalCharges.describe()
Let’s see its distribution:
plt.hist(data = Churn_df, x = 'TotalCharges');
After viewing the distribution and the 5-number summary, I found that around 80% of the data are extremely high, so I decided to divide them to 80% and the rest 20% to see each data distribution. (Applying 80/20 rule in real life.)
#find the 80th percentile of the data in total chargesChurn_df.TotalCharges.quantile(0.8)
Output: 2827.5900000000006
# Divide the data by the 80th percentile of the data, and show the distribution of its TotalCharges under 80th percentile TotalCharges_under80 = Churn_df.query('TotalCharges<=2827.59')TotalCharges_above80 = Churn_df.query('TotalCharges>2827.59')
Let’s visualize our unsubscribed customers divided by 80/20 group:
Ok, now, let's see the graph that combines those who unsubscribed and those paying for the service.
Note: 80%(low LTV) of leaked customers only stayed under 10 months. And, the average LTV of 80% of those who unsubscribed is 750 dollars. On the other hand, the average LTV of the top 20% of unsubscribed is 4750 dollars. And the ratio by the sum of total LTV by each group is 750*4: 4750 = 1: 1.6, which suggests we should focus on serving those 20% customers with high LTV, which brought 60%(1.6/2.6) of our revenue from leaked customers.
Note: 100% of those who brought high LTV used and are using internet service.
Note: 81% of those who bring high LTVs tend to used lines. There is no big difference between those who are still paying for the service or those who unsubscribed.
Note: Those who bring high LTVs, loved to use Fiber optic(75%-90%) and DSL(10–20%), and none of them used internet service. While those who bring low LTVs, in terms of current customers, around 30% of them don’t use internet service.
Now, let’s dive deeper to see the situation of each subset of internet service.
Firstly, Extract 80% with low LTV who used the internet service, and save each proportion of subsets of internet service in the variable “proportion_internet_sub_service”:
And then extract 20% with high LTV who used the internet service, and save each proportion of subsets of internet service in the variable “proportion_internet_sub_service_above80”
Secondly, Extract 80% with low LTV who used the internet service, and save each proportion of subsets of internet service in the variable “paying_proportion_internet_sub_service_under80”
And then extract 20% with high LTV who used the internet service, and save each proportion of subsets of internet service in the variable “paying_proportion_internet_sub_service_above80”
Let’s visualize it:
Note: Among 80%(low LTVs) of data of current customers, all the subsets of internet services are equally used by 40% of people. And among 80% (low LTVs) of current customers, streaming movies and streaming TV are two top subsets of internet service people use, and device protection and online backup are the second places, and tech support and online security are both third places, the gaps between these are near 10% of whole data.
First thing first, I extracted the groups of data I want to investigate:
Note: Among 80% of current customers, those whose LTV is more than the average LTV of 80% of leaked customers are 10% more likely to use multiple lines and internet service than the average 80% of current customers.
Note: 89% of those who leaked used monthly contracts, while only 42% of current customers used a monthly contract.
I used multiple linear regression for these questions, so first, because most of them are categorical variables, so I needed to convert them into dummy variables:
Use multiple linear regression:
Note: Whether people use internet service is the most important factor in creating high LTV, and the yearly contract is second.
Note: Among all the subsets of internet services, online backup is the most important factor to create high LTV
Step 4: Understand the limitations
Before drawing any conclusion, it is always better to inform the limitations. During the process of assessing data and conducting the exploratory data analysis, I have found some limitations:
Limitation 1: In this dataset, we can only see one type of each variable instead of a real-world situation of changing different options as time passes, i.e., in the real world, people might want to try streaming services, but they might change their minds to use tech supports next month.
Limitation 2: We cannot consider all these variables as the exact reasons why customers left because they might leave for the better price offered by competitors or the bad economy at a certain time, etc. We cannot see the time they leaked, so it’s hard to infer those external situations.
Step 5: Summaries
80%(low LTV) of leaked customers only stayed under 10 months. And, their average is 750 dollars. On the other hand, the average LTV of the top 20% of those who leaked is 4750 dollars. And the ratio by the sum of total LTV by each group is 750*4: 4750 = 1: 1.6, which suggests we should focus on serving those 20% customers with high LTV, which brought 60%(1.6/2.6) of our revenue from leaked customers81% of those who bring high LTVs tend to used lines. There is no big difference between those who are still paying for the service or those who unsubscribed.Those who bring high LTVs loved to use Fiber optic(75%-90%) and DSL(10–20%), and none of them used internet service. While those who bring low LTVs, in terms of current customers, around 30% of them don’t use internet service.Among 80%(low LTVs) of current customers, all the subsets of internet services are equally used by 40% of people. And among 80% (low LTVs) of current customers, streaming movies and streaming TV are two top subsets of internet service people use, and device protection and online backup are the second places, and tech support and online security are both third places, the gaps between these are near 10% of whole data.Among 80% of current customers, those whose LTV is more than the average LTV of 80% of leaked customers are 10% more likely to use multiple lines and internet service than the average 80% of current customers.89% of those who leaked used a monthly contract, while only 42% of current customers use a monthly contract.Whether people use internet service is the most important factor in creating high LTV, and the yearly contract is second.Among all the subsets of internet services, online backup is the most important factor in creating high LTV.
80%(low LTV) of leaked customers only stayed under 10 months. And, their average is 750 dollars. On the other hand, the average LTV of the top 20% of those who leaked is 4750 dollars. And the ratio by the sum of total LTV by each group is 750*4: 4750 = 1: 1.6, which suggests we should focus on serving those 20% customers with high LTV, which brought 60%(1.6/2.6) of our revenue from leaked customers
81% of those who bring high LTVs tend to used lines. There is no big difference between those who are still paying for the service or those who unsubscribed.
Those who bring high LTVs loved to use Fiber optic(75%-90%) and DSL(10–20%), and none of them used internet service. While those who bring low LTVs, in terms of current customers, around 30% of them don’t use internet service.
Among 80%(low LTVs) of current customers, all the subsets of internet services are equally used by 40% of people. And among 80% (low LTVs) of current customers, streaming movies and streaming TV are two top subsets of internet service people use, and device protection and online backup are the second places, and tech support and online security are both third places, the gaps between these are near 10% of whole data.
Among 80% of current customers, those whose LTV is more than the average LTV of 80% of leaked customers are 10% more likely to use multiple lines and internet service than the average 80% of current customers.
89% of those who leaked used a monthly contract, while only 42% of current customers use a monthly contract.
Whether people use internet service is the most important factor in creating high LTV, and the yearly contract is second.
Among all the subsets of internet services, online backup is the most important factor in creating high LTV.
Step 6: Actionable insights
For retaining customers:
Create marketing campaigns to upsell those currently subscribed to streaming movies and TV services on our other internet services. (Based on the graph from Q5, the more subsets of the internet service they’re paying for, the longer they tend to stay.)
Create marketing campaigns to upsell those currently subscribed to streaming movies and TV services on our other internet services. (Based on the graph from Q5, the more subsets of the internet service they’re paying for, the longer they tend to stay.)
For increasing customers’ lifetime values:
Increase our marketing budget on our streaming movie and TV service since these two sectors have the biggest gaps between those who brought 80%(low LTV) and 20%(high LTV). (Based on the graph from Q5)Increase our marketing budget on online backup, which is the most important factor contributing to a high LTV. (Based on the graph from Q8)Increasing our marketing budget on those who want to use multiple lines. (Based on the graph from Q6)
Increase our marketing budget on our streaming movie and TV service since these two sectors have the biggest gaps between those who brought 80%(low LTV) and 20%(high LTV). (Based on the graph from Q5)
Increase our marketing budget on online backup, which is the most important factor contributing to a high LTV. (Based on the graph from Q8)
Increasing our marketing budget on those who want to use multiple lines. (Based on the graph from Q6)
Finally, data has a better idea, but you should always doubt any insight and the data to be some degree wrong or have some biases, no matter who provided them to you are experts or people with high credibility.
Thanks for spending your time reading this article.
If you’re an employer wanting to find a data analyst intern, here is my LinkedIn or contact me through my email(ychen596@insite.4cd.edu)
If you’re also working in the data science/business analytics field, feel free to give me any feedback through IG(@jchen_mmm), by email, or by comment below!