Extensive EDA and Prediction on Customers Subscription(on an Application) Analysis.
By: Luv Dhamija
Tags: Data Science, Data Analysis, Machine Learning, Exploratory Data Analysis, Data Visualization

Extensive EDA and Prediction on Customers Subscription(on an Application) Analysis.

This article explains various data science aspects that are applied to predict whether a customer subscribes the premium feature of an app. Luv Dhamija Jun 9, 2020·7 min read

We have a dataset which consists of the details about a user and shows that if the particular user has enrolled for the Premium version for that application. We would be using this dataset to determine various insights hidden in the dataset along with using this dataset for building some machine learning model as well.

Let us start the process by importing the dataset.

import pandas as pd

df=pd.read_csv("customer application subscription.csv")

df.head()

First five records of the dataset

After loading the dataset we would now write some lines to know about the dataset in general.

print("\t\t\tDataset Statistics")

print("Number of variables:-",len(df.columns))

print("Number of observations:-",df.shape[0])

print("Missing Values:-",df.isna().sum().sum())

print("Missing Values %:-",round(df.isna().sum().sum()/df.shape[0]*100,3))

print("Duplicate rows:-",df[df.duplicated(keep="first")].shape[0])

print("Duplicate rows%:-",round(df[df.duplicated(keep="first")].shape[0]/df.shape[0]*100,3))

print("Variable types:-")

a=pd.crosstab(df.dtypes,columns="Count")

for i in pd.crosstab(df.dtypes,columns="Count").index:

print("\t",str(i).split()[0],a.loc[i][0])

Information regarding the dataset.

If we look into the dataset we will observe that the missing values are present in the enrolled_date column only. Since there are only 0.03% of the duplicate rows in 50000 entries so we would not drop them as it is not required.

Now we will convert the first_open column into datetime format as it is of object type. first_open column specifies the Date and time of login on app for the first time.

df["first_open"]=pd.to_datetime(df["first_open"])

df['first_open_date'] = [d.date() for d in df['first_open']]

df['first_open_time'] = [d.time() for d in df['first_open']]

df=df.drop("first_open",axis=1)

Since we have obtained the date and time from the first_open column, we can see the trend of using the app for the first time within the date.

a=pd.crosstab(index=df["first_open_date"],columns="count")

from datetime import datetime, timedelta

import matplotlib.pyplot as plt

import seaborn as sns

sns.set_style('darkgrid')

import numpy as np

%matplotlib inline

x = np.arange(datetime(2012,11,23), datetime(2013,7,10), timedelta(days=1)).astype(datetime)

y = a["count"]

y_mean = [a["count"].mean()]*len(x)

fig,ax = plt.subplots(figsize=(17,12))

ax.set_xlabel("Month",fontsize=20)

ax.set_ylabel("Count",fontsize=20)

ax.tick_params(labelsize=15, labelrotation = 20, color ="k")

data_line = ax.plot(x,y, label='Data')

mean_line = ax.plot(x,y_mean, label='Mean', linestyle='--')

legend = ax.legend(loc='upper right')

plt.show()

Trend of using the application for first time

As we can see from the graph, the usage of the application is highly fluctuating. This can lead to conclusions that reasons could involve directly to the application such as the promotion of the app etc. instead of outside factors since it could not affect that much.

Now we would see some insights from dayofweek column. We would not focus on first_open_time as there are no insights to find from there.

import seaborn as sns

sns.set_style('darkgrid')

ct=pd.crosstab(df["dayofweek"],columns="Count")

stacked = ct.stack().reset_index().rename(columns={0:'value'})

g=sns.barplot(x=stacked.dayofweek, y=stacked.value)

plt.title("Count of On which day user logon")

Here, in dayofweek the unique values explain the following:

1: Monday

2: Tuesday

3: Wednesday

4: Thursday

5: Friday

6: Saturday

0: Sunday

This graph explains that the minimum usage of the application is on Tuesday and maximum usage is on Thursday.

Now let's analyze the hour column

ct=pd.crosstab(df["hour"],columns="Count")

plt.figure(figsize=(12,12))

plt.tick_params(labelsize=15, labelrotation = 90, color ="k")

stacked = ct.stack().reset_index().rename(columns={0:'value'})

g=sns.barplot(x=stacked.hour, y=stacked.value)

Usage of application over 24-hours

As we can see in this graph the user uses the app less around 9 AM while the maximum is around 3 PM.

Let's look at age column.

df["age"].describe()

Basic desception of age column

plt.hist(df["age"],edgecolor="black",bins=7)

plt.show()

Histogram of age column

This shows that most 75% of the users are below the age of 37. This implies that the makers and the company should cater to the need and mark those people whose age is under 37 as their target audience.

The other columns are:- screen_list: it describes the screens that is seen by customers, numscreens: number of screens watched by the user, minigame: If the customer played mini-game which was included within the app then 1 else 0,used_premium_feature: if the user took the premium feature of the application then it is 1 else 0, enrolled: If the user purchased a premium feature then 1 else 0, enrolled_date: the date and time at which the user purchased a premium features.

We will analyse the above columns now

groupedvalues=df.groupby('dayofweek')["numscreens","minigame","used_premium_feature","enrolled","liked"].sum().reset_index()

attributes=list(groupedvalues.columns)[1:]

att=attributes[1:]

for i in att:

print(i)

sns.countplot(df[i])

plt.show()

Let's see the other columns behaviour on the basis of dayofweek column:

for i in attributes:

print(i)

sns.barplot(x="dayofweek",y=i,data=groupedvalues)

plt.show()

dayofweek vs other columns

Let's see the other columns behaviour on the basis of hour column

for i in attributes:

print(i)

plt.figure(figsize=(12,12))

plt.tick_params(labelsize=15, labelrotation = 90, color ="k")

sns.barplot(x="hour",y=i,data=groupedvalues2)

plt.show()

hour vs other columns

With these graphs, we can surely analyze the inactiveness of users during day time as most of the users have to perform other tasks instead of using this app. This can further imply that all the promotional activities that can be performed on the application can be done during the afternoon, as most users are active during this time.

Let us analyze the usage of the application on the basis of age. Since there are many unique ages in 50000 records we would use the approach of visualizing the other columns on the basis of “AGE GROUP”.

groupedvalues3=df.groupby(pd.cut(df['age'], np.arange(15, 106, 10)))["numscreens","minigame","used_premium_feature","enrolled","liked"].sum().reset_index()

for i in attributes:

print(i)

plt.figure(figsize=(12,12))

plt.tick_params(labelrotation = 45)

sns.barplot(x="age",y=i,data=groupedvalues3)

plt.show()

agegroup vs other columns

In these graphs we can analyze that though 75% of our audience lies below the age of 37, the most active age group is between 15–35.

Since we performed the EDA we would be moving on to the feature selection.

Finding a correlation between various features:

for i in df.index:

x = df.at[i , 'hour']

y = x[0:3]

df.at[i , 'hour'] = y

df["hour"]=df["hour"].astype('int64')

import seaborn as sns

df2 = df.drop(['user', 'enrolled_date'], axis = 1)

sns.heatmap(df2.corr(), annot = True)

Correlation between various features

df3 = df2.drop(['enrolled'], axis = 1)

plt.figure(figsize=(8,8))

sns.barplot(x=df3.corrwith(df2.enrolled).index,y=df3.corrwith(df2.enrolled))

plt.tick_params(labelsize=15, labelrotation = 45)

Correlation between various features with respect to enrolled column

The above graph shows that hour, age, minigame(negatively correlated), numscreens, minigame(positively correlated) can be considered for model building.

After all this analysis we would revert back our dataframe so as to find the total time a user took to enroll for the app.

df=pd.read_csv('data.csv')

df["first_open"]=pd.to_datetime(df["first_open"])

df["enrolled_date"]=pd.to_datetime(df["enrolled_date"])

df['time_to_enrolled']=(df.enrolled_date - df.first_open).astype('timedelta64[h]')

import matplotlib.pyplot as plt

plt.figure(figsize=(6,6))

plt.hist(df['time_to_enrolled'].dropna(), range = (0,100))

plt.show()

Visual representation of time taken to enroll for the premium feature.

As we can see in this histogram the details before 50 hours are good as others are the outliers and can affect the model so we would consider the customers have enrolled as 0.

df.loc[df.time_to_enrolled > 50, 'enrolled'] = 0

The screen_list can be a really useful column as it can be used to modeling purpose. But this column is a string. So in order to use this, we need to encode this column which would take a lot of time since it has to process around 50000 strings.

Before continuing with this, we need another package which is “mlxtend” package. In order to install it use this line in cmd/Anaconda Prompt:-

pip install mlxtend

After this write the following lines:

for i in range(len(df["screen_list"])):

df["screen_list"][i]=df["screen_list"][i].split(',')

from mlxtend.preprocessing import TransactionEncoder

transactionEncoder = TransactionEncoder()

txnn=transactionEncoder.fit(df["screen_list"]).transform(df["screen_list"])

dfnn=pd.DataFrame(txnn,columns=transactionEncoder.columns_)

df=df.join(dfnn, how="outer")

from sklearn.preprocessing import LabelEncoder

le=LabelEncoder()

l=list(transactionEncoder.columns_)

for i in l:

try:

df[i]=le.fit_transform(df[i])

except:

pass

From this, we are going to convert our dataset and chose only those features which are required by the model as discussed above.

a1=df[["hour","age","numscreens","used_premium_feature","minigame"]]

for i in df.index:

x = a1.at[i , 'hour']

y = x[0:3]

a1.at[i , 'hour'] = y

a1["hour"]=a1["hour"].astype('int64')

a2=df[transactionEncoder.columns_]

a1=a1.join(a2, how="outer")

X=a1

Y=df["enrolled"]

from sklearn.model_selection import train_test_split

xtrain,xtest,ytrain,ytest=train_test_split(X,Y,test_size=0.3,random_state=42)

Now our data is made to pass through the models. Let us try to pass our model through various ML models.

Logistic Regression:

from sklearn.linear_model import LogisticRegression

le=LogisticRegression(random_state = 42, penalty = 'l1')

le.fit(xtrain,ytrain)

le.score(xtest,ytest)

After running the code we get the following score:

0.868

We got an impressive score from this dataset. Let us check from the training set that whether there is a case of overfitting or not.

le.score(xtrain,ytrain)

After running the code we get the following score:-

0.8619428571428571

It seems that there is no case of overfitting

2. Decision Tree Classifier

We would be using GridSearchCV for hyperparameter tuning\

from sklearn.tree import DecisionTreeClassifier

from sklearn import metrics

from sklearn.model_selection import GridSearchCV

tuned_parameters = [{'criterion': ['gini','entropy'],

'max_depth': range(2,10)}]

clf_tree = DecisionTreeClassifier()

clf = GridSearchCV(clf_tree,

tuned_parameters,

cv=10,

scoring='roc_auc')

clf.fit(xtrain, ytrain )

To find the parameters:-

clf.best_params_

This is what we got.

{'criterion': 'entropy', 'max_depth': 7}

So we would use it to find the score.

clf_tree2 = DecisionTreeClassifier(criterion = 'entropy',

max_depth = 7 )

clf_tree2.fit( xtrain, ytrain )

tree_predict2 = clf_tree2.predict( xtest )

metrics.roc_auc_score( ytest, tree_predict2 )



The following score is received:

0.8662688484194627

We can also use other ML models for the same. This is my first article so valuable feedback is highly appreciated. Thanks and happy coding!

Notebook link:-https://github.com/luv8860/Projects/blob/master/Customer%20application%20subscription.ipynb

Dataset link:-

https://github.com/luv8860/Projects/blob/master/customer%20application%20subscription.csv