Predicting App Subscription using Logistic Regression | by Luke Sun | Towards Data Science

The previous article is about EDA on raw data. Here, I will explain how to perform feature engineering, data processing, and eventually create a logistic regression model using mobile app behavior data. It is split into 7 parts.
Problem statementFeature engineeringData processingModel buildingModel testModel validationSummary
Problem statement
Feature engineering
Data processing
Model building
Model test
Model validation
Summary
Now let’s begin 🏃‍♀️🏃‍♂️.
1. Problem statement
We are tasked by a Fintech company to analyze mobile app behavior data to help direct customers to paid subscription services. Specifically, the task is to identify which users are most likely NOT to enroll. Figure 1 is a snippet of raw data with 12 columns and 50,000 rows.
2. Feature engineering
Feature engineering is the art of transforming raw data into features which best represents the problem. Only with correct feature engineering, the model makes the best prediction. We will perform feature engineering from two aspects.
2.1 Dependent variable engineering
The dependent variable is column ‘enrolled’. A column closely related to ‘enrolled’ is ‘enrolled_date’. Basically, users can enroll at any date with ‘enrolled’ as 1. Since most of the app features are not available for trials after the first 24 hours, we need to set a time limit for enrollment.
To set a valid enrollment time limit, we calculate the time difference between ‘first_open’ and ‘enrolled_date’, and investigate the enrollment distribution.
1st step is to calculate the time difference between ‘first_open’ and ‘enrolled_date’. Specifically, parse ‘first_open’ and ‘enrolled_date’ to datetime type, and calculate the time difference in hours.
dataset[“first_open”] = [parser.parse(row_date) for row_date in dataset[“first_open”]]dataset[“enrolled_date”] = [parser.parse(row_date) if isinstance(row_date, str) else row_date for row_date in dataset[“enrolled_date”]]dataset[“difference”] = (dataset.enrolled_date-dataset.first_open).astype(‘timedelta64[h]’)
2nd step is to review the enrollment distribution by plotting the histogram of the ‘difference’ column.
response_hist = plt.hist(dataset[“difference”].dropna(), color=’#3F5D7D’)plt.title(‘Distribution of Time-Since-Screen-Reached’)plt.show()
Figure 2 shows user enrollment is densely concentrated in the first 50 hours. Thus, we set the response limit as 48 hours.
Now, with a 48-hour enrollment limit, we reset the ‘enrolled’ column wherever ‘difference’ great than 48 to be 0, namely not enrolled.
dataset.loc[dataset.difference > 48, ‘enrolled’] = 0dataset = dataset.drop(columns=[‘enrolled_date’, ‘difference’, ‘first_open’])
Above we removed ‘enrolled_date’, ‘difference’, ‘first_open’ columns, as they are not needed for training.
2.2 Independent variable engineering
The only non-numerical column in the dataset is ‘screen_list’. It is a list containing all screen features that the user viewed. So we need to convert it to numerical variables. One method is to convert each unique screen in ‘screen_list’ as a categorical variable. However, there are too many unique screens. So, we will only focus on those most popular screens. Specifically,
top_screens = pd.read_csv(‘top_screens.csv’).top_screens.valuesdataset[“screen_list”] = dataset.screen_list.astype(str) + ‘,’ for sc in top_screens:    dataset[sc] = dataset.screen_list.str.contains(sc).astype(int)    dataset['screen_list'] = dataset.screen_list.str.replace(sc+",", "")
Above, we created a column for every screen in ‘top_screens’ which represents the most popular screens.
Next, we create an ‘Other’ column as a sum of all un-popular screens.
dataset[‘Other’] = dataset.screen_list.str.count(“,”)dataset = dataset.drop(columns=[‘screen_list’])
Finally, if we carefully examine the name of the column, we found many columns represent the same feature. For instance, ‘Saving1’, ‘Saving5’ are screens about saving, while ‘Credit1’ and ‘Credit3’ are screens about credit. We need to aggregate the same features by summing the count of the same feature screens. Specifically,
cm_screens = [“Credit1”, “Credit2”, “Credit3”, “Credit3Container”, “Credit3Dashboard”]dataset[“CMCount”] = dataset[cm_screens].sum(axis=1)dataset = dataset.drop(columns=cm_screens)
Repeating the same methods for other features (e.g., saving, loan, etc), we got the final dataset with all numerical variables. Figure 3 shows all the columns.
In summary, we used data mining techniques to refine and extract attributes that best represent mobile app user behaviors.
3. Data processing
Data processing consists of data split and feature scaling.
3.1 Data split
1st step is to split the independent and dependent variables. Specifically,
response = dataset[“enrolled”]dataset = dataset.drop(columns=”enrolled”)
2nd step is to split the data into training and test sets. Specifically,
X_train, X_test, y_train, y_test = train_test_split(dataset, response, test_size = 0.2, random_state = 0)
3.2 Feature scaling
Feature scaling is to avoid any variables dominating over other variables, namely taking higher weights and a strong influence on model learning. Here we standardize features by removing the mean and scaling to unit variance. Specifically,
from sklearn.preprocessing import StandardScalersc_X = StandardScaler()X_train2 = pd.DataFrame(sc_X.fit_transform(X_train))X_test2 = pd.DataFrame(sc_X.transform(X_test)) X_train2.columns = X_train.columns.valuesX_test2.columns = X_test.columns.valuesX_train2.index = X_train.index.valuesX_test2.index = X_test.index.valuesX_train = X_train2X_test = X_test2
Note, StandardScaler() returns a numpy array, which loses the column name and index. So we convert scaled data to dataframe again, to keep the row and column identifiers.
Great. Data is finally ready for the model. Figure 4 is a brief view of the final data with 50 columns and 50,000 rows.
4. Model building
Here we will create a logistic regression model to predict a binary dependent variable, namely enrolled or not. According to Wikipedia, the logarithm of the probability for the label as 1 is a linear combination of one or more independent variables. Essentially, we try to estimate the coefficients of a logistic model as shown in Figure 5.
Specifically,
from sklearn.linear_model import LogisticRegressionclassifier = LogisticRegression(random_state = 0, penalty = ‘l1’)classifier.fit(X_train, y_train)
Note we use Lasso (L1) regularization model, instead of the normal regression model. L1 regularization adds a penalty equal to the sum of the absolute value of the magnitude of coefficients to the loss function, as shown in Figure 6.
Note that the difference between L1 and L2 regularization is that the penalty L2 adds is the sum of squared value of the magnitude of coefficients, as Figure 7 shows.
5. Model test
With the model trained, let’s test the model on X_test.
y_pred = classifier.predict(X_test)cm = confusion_matrix(y_test, y_pred)
To better review the predicted results, let’s compare it with actual results. So, specifically,
final_results = pd.concat([y_test, test_identity],axis =1).dropna()final_results[‘predicted_results’] = y_predfinal_results[[‘user’, ‘enrolled’,‘predicted_results’]].reset_index(drop=True)
Figure 8 shows the actual enrolled result and predicted results.
Figure 9 illustrates the confusion matrix. This tells us a test accuracy of 0.768. Not bad results 😃. If you want to know how to calculate accuracy, please read this article.
6. Model validation
With the above test accuracy, one question as a data scientist you should ask: is that a true reflection of model performance 🤔? To answer this question, we will use the K-fold cross-validation.
Specifically, split the training data into 10 subsets, use 9 subsets to train the model, and the remaining for validation. Repeat this training and validation 10 times. In the end, average accuracy and loss.
accuracies = cross_val_score(estimator= classifier, X= X_train, y = y_train, cv = 10)
We got an average accuracy of 0.767 with a standard deviation of 0.10. Great, models show little variance, namely the model is consistently accurate.
7. Summary
To recap, we walked through feature engineering, data processing, model building, test, and validation. Feature engineering and data processing are time-consuming, but of most importance to get data ready for the model. If you want to learn about model optimization, please read this article.
Great! That’s all of the journeys! If you need the source code, feel free to visit my Github page 🤞🤞.