Customer lifetime value (CLV) prediction
Tags: Regression Analysis, Data Science, Xgboost, Customer Lifetime Value, Clv

Knowing that a customer is going to bring a certain amount of money for the company early enough can be very helpful to know how much the company can spend to retain it.

Indeed, if a customer only brought in € 10 during his life as a customer in the company, we will not spend more than € 10 to retain him.

There are many techniques out there for estimating Customer Lifetime Value and we are going to introduce you two of them that will allow you to estimate its value without waiting to have a significant purchase history.

The idea of ​​this article is to give you a method that will allow you to estimate the CLV of a new customer.

Usually the CLV is set for a defined period of time, for example how much will a customer bring after one year, two years.

In our study, we will use the behavior of clients from the first 6 months to estimate the CLV at 2 years. All scripts are available on my Github : https://github.com/Faouzizi/customerLifetimeValue/

I. Predicting CLV Using Purchase History

In our case, CLV = the value that the customer generated after 2 years.

The first step is to retrieve the data of all the people who became customers more than two years ago so we can calculate the CLV to build our data sample for our models.

For our modeling, we will use the purchase history of the first 6 months (which we will call explanatory variables) to estimate the value of the Customer Lifetime value generated after 2 years.

Table 1 : Sample data

####################################################################

############# Import some packages ######################

####################################################################

import pandas as pd

import numpy as np

from sklearn.model_selection import train_test_split

from sklearn.linear_model import LinearRegression

import sklearn.metrics





####################################################################

#################### Import DATA ##################

####################################################################

df = pd.read_csv('./data.csv')



####################################################################

## We check that all data have the right type ##

####################################################################

df.dtypes

df.head()





####################################################################

### we do a correlation analysis ###

####################################################################

df.drop('customer_id', axis=1).corr()

Table 2: Correlation Table

During our correlation analysis, we remove the ids column because it is not useful for our modeling. The last column fo table 2 represents the correlation between the explanatory variables and the target variable.

We note that our explanatory variables show a good correlation with the target variable (target variable = clv).

This explains why we have a good chance of being able to predict the value of the CLV using the first 6 columns.

We also calculated the correlation of the explanatory variables in pairs to see if we have any correlated variables that would introduce instability into our model.

In our case, the correlations between the explanatory variables are weak, so we can continue our study.

##################################################################

# oWe define explanatory variables and target variable #

##################################################################

variables_explicatives = df.drop(['customer_id','clv'], axis=1)

variable_cible = df.clv

###################################################################

# We split our data on train/test samples #

###################################################################x_train, x_test, y_train, y_test = train_test_split(variables_explicatives, cible, test_size=0.2)



# We fit our model

model = LinearRegression()

model.fit(x_train, y_train) # We print regression coefficients and out intercept

print("Coefficients :

", model.coef_)

print("Intercept :

", model.intercept_)

Coefficients :

[2.79722234, 0.99247278, 3.40547329, 2.80283652, 1.52911686, 2.80767086]

Intercept :

498.44473460759167

# We test our model on test sample

predictions = model.predict(x_test)

predictions

# Mean/Median of abs (diffrence between real values of CLV and CLV

# predicted using the model) abs(y_test-predictions).mean()

abs(y_test-predictions).median() # We compute R^2 to determine our model performance

sklearn.metrics.r2_score(y_test, predictions)

Mean : 51

Median : 48

R² = 0.83

The coefficient of determination R² varies between 0 and 1 and represents an indicator which makes it possible to judge the quality of a simple linear regression.

This coefficient measures the fit between the data predicted by the model and the observed data or how well the regression equation is suited to describe the distribution of points.

The closer the R² is to 0, the more difficult it is for the model to explain the distribution of points (we cannot correctly predict the values ​​of the CLV).

On the contrary, the closer the R² is to 1, the more the model explains the distribution of points.

In our case, our R² is 0.83 which is close to 1, our model allows us to predict the value of CLV thanks to the first 6 months of tenure.

II. Predicting CLV using other variables

In this section, we will be using variables other than purchase history within the company.

We are going to use explanatory variables such as geographic location, gender, salary….

Processing of the geolocation variable

This Location.Geo variable gives the amplitude and longitude of the location of our clients. Such a variable cannot be used in this state, so we will convert it to an address to keep only the country.

Thus, we obtain the following distribution:

####################################################################

# Replace geolocalisation by country name

####################################################################import reverse_geocode

coordonnees = tuple(df['Location.Geo'].map(lambda x: tuple(x.split(',')) if x.split(',')[0]!='NA' else (17,77))) df['country'] = [x['country'] for x in reverse_geocode

.search(coordonnees)]

####################################################################

# we delete geolocalisation variable

####################################################################df.drop('Location.Geo', axis=1, inplace=True) ####################################################################

# we compute frequency of each country ####################################################################

df.country.value_counts()

Note: We are not keeping the city because we have a too heterogeneous distribution and that would introduce a lot of instability in the model.

2. Imputation of missing data

The first thing we’ll do is deal with the missing data:

We have several solutions for imputing missing data and we will present two of them here:

- Static imputation (imputation by the mean or median for the quantitative

variables and imputation by the mode for the qualitative variables)

- Dynamic imputation (imputation using KNNs)

In which case to choose dynamic or static imputation:

- If the missing data is less than 5%, you can removing the lines with missing data will not affect your model.

- If the missing data is between 5% and 15%, you can opt for static imputation the consequences on your model will not be significant

- If the rate of missing data is between 15% and 30%, it is necessary to opt for dynamic imputation

- If the rate of missing data is greater than 30%, you must remove the variable from your model.

a. Static Imputation

###################################################################

# Here, we will replace missing value by the median for continuous

# variables and mode for qualitative variables

################################################################### def imputation_statique(df):

missing_data = df.apply(lambda x: np.round(x.isnull()

.value_counts()*100.0

/len(x),2)).iloc[0]

columns_MissingData = missing_data[missing_data<100].index

for col in columns_MissingData:

if df[col].dtype=='O':

df[col] = df[col].fillna(df[col].mode().iloc[0])

else:

df[col] = df[col].fillna(df[col].median())

return(df) imputation_statique(df)

b. Dynamic Imputation

Using a model to predict missing data is one of the most common solutions in data science.

This approach involves creating a model for each explanatory variable that has missing values.

You can use whatever algorithm you want, but in this example, we will use the k-nearest neighbor (KNN) algorithm (we choose this algorithm because it is quite efficient for this kind of problem).

###################################################################

# Here, we use KNN to impute missing values

###################################################################def imputation_dynamique(df):

############################

# we defines 3 neighbors

############################

imputer = KNNImputer(n_neighbors=3)

############################

# get dummies

############################

X = pd.concat([pd.get_dummies(df.drop('CustomerID', axis=1).select_dtypes('O')), df.drop('CustomerID', axis=1).select_dtypes(exclude='O')], axis=1)

############################

# Missing values imputations

############################

X_filled_knn = pd.DataFrame(imputer.fit_transform(X))

############################

# We renames columns

############################

X_filled_knn.columns = X.columns

############################

# We replace variables with missing values by variables

# imputed

############################

for col in columns_MissingData:

print(col)

if df[col].dtypes=='O':

df_temp =X_filled_knn.filter(regex='^'+col+'*')

df_temp.columns = [x.replace(col+'_', '') for x in

df_temp.columns]

df[col] = df_temp.idxmax(1)

else:

df[col] = np.round(X_filled_knn[col],2)

return(df)

3. Univariate study

The objective of this section is to analyze the explanatory variables one by one in order to identify possible inconsistencies in the data (extreme data for example).

For this kind of analysis, you have a large test set to detect these extreme values ​​such as the Chauvenet test, Grubbs ..

But in our case, we are going to use the quantiles: let Q1 and Q3 respectively be the first quartile and the third quartile, then we can define an extreme datum as being any value located outside the interval: [Q1 — k * (Q3-Q1); Q1 + k * (Q3-Q1)] where k is a positive integer.

For each continuous variable, all values ​​outside this interval will be considered as extreme values ​​and will be replaced by the mean (we can treat them as missing data and impute them with algorithms like the KNN).

####################################################################

############## Treat extreme values

####################################################################for col in df.select_dtypes(exclude='O'):

if col != 'Customer.Lifetime.Value':

q1 = df[col].quantile([0.25]).values[0]

q3 = df[col].quantile([0.75]).values[0]

IC_valeur_non_aberantes = [q1 - 2*(q3-q1), q3 + 2*(q3-q1)]

df.loc[df[col]<IC_valeur_non_aberantes[0], col] =

df[col].mean()

df.loc[df[col]>IC_valeur_non_aberantes[1], col] =

df[col].mean()

4. Bivariate study

The aim of this study is to identify the potential relationships between the target variable and the explanatory variables.

In our case, we will not delay too long on this step, but it is very important for understanding the business issues of the study.

5. Prediction

In this section we will test several models and we will select the best model.

Linear regression

To start with, we’ll start modeling with a simple linear regression to see the performance at this point.

####################################################################

########### Linear regression modelization ######

#################################################################### ##### Import packages

from sklearn.linear_model import LinearRegression

from sklearn.metrics import r2_score ##### RL

model = LinearRegression().fit(x_train, y_train)

model.score(x_train, y_train)

model.predict(x_train) #### We print coeffs

print(r2_score(y_train, model.predict(x_train)))

res = pd.DataFrame({'reel':y_train, 'pred':las.predict(x_train),

'dif':abs(y_train-las.predict(x_train))})

print("Coefficients :

", model.coef_)

print("Intercept :

", model.intercept_)

print(res) #### We print real CLV values vs predicted CLV

#### If our prediction are enought good, the distribution will

#### follow the curve y=x (first diagonale)

plt.scatter(res.reel, res.pred) #### We print mean and median of diffrence between real values and

#### our prediction

print('median :' + str((res.dif*100/res.reel).median()))

print('Mean :'+str((res.dif*100/res.reel).mean()))

The coefficient R² is 0.35, a fairly low value, moreover, if we calculate the average difference (or median) between the predicted values ​​and the real values, we see a difference, in absolute value, of around 30%.

The model is therefore not very reliable.

To visualize the performance of our model, we can display the scatter plot of the actual values ​​according to the predicted values.

If the cloud of points is distributed around the first diagonal our modeling is reliable which is not the case above.

Lasso regression

For the Lasso regression, we will test several penalization values ​​(alpha) in order to identify the best penalization.

####################################################################

########### Lasso regression modelization ######

####################################################################from sklearn.linear_model import Lasso

import sklearn # We define an alpha vector to test

my_alphas = np.array([0.001,0.01,0.02,0.025,0.05,0.1,0.25,0.5,0.8,1.0, 10,20, 20, 40,50, 100, 200, 300, 500, 700]) # We use cross validation to identify the best alpha

lcv = sklearn.linear_model.LassoCV(alphas=my_alphas,normalize=False,fit_intercept=False,random_state=0,cv=5) #fit the model on train data

lcv.fit(x_train,y_train) # Print alpha tested values

print(lcv.alphas_) #Print the MSE of cross validation

print(lcv.mse_path_) # Mean mse in cross validation for each alpha

avg_mse = np.mean(lcv.mse_path_,axis=1) #alphas vs. MSE in cross-validation

print(pd.DataFrame({'alpha':lcv.alphas_,'MSE':avg_mse})) #graphic

plt.plot(lcv.alphas_,avg_mse)

plt.xlabel('Alpha')

plt.ylabel('MSE')

plt.title('MSE vs. Alpha')

plt.show()

We find that the best alpha is alpha = 20 but the a MSE is too high. We can draw the same graph as the one in the previous section to visualize the distribution of the point cloud around the diagonal, but we will get the same results as above.

The R² of this model is 0.35.

This kind of simple model does not seem to be effective for this problem.

XGBoost

For this kind of model, the choice of parameters is very important, so we will start by choosing the optimal parameters:

####################################################################

########### XGBoost modelization ######

####################################################################

import xgboost as xgb

from sklearn.metrics import mean_squared_error ##### choice of parameters

xgb1 = xgb.XGBRegressor()

parameters = {'nthread':[4],

#when use hyperthread, xgboost may

#become slower

'objective':['reg:squarederror'],

'learning_rate': [.1, 0.2, .5], #so called `eta` value

'max_depth': [2, 3, 4],

'min_child_weight': [4],

'silent': [1],

'subsample': [0.1, 0.2, 0.5],

'colsample_bytree': [0.1, 0.2, 0.5, 0.7],

'n_estimators': [100, 200, 500],

'alpha':[1, 10, 50]}

xgb_grid = GridSearchCV(xgb1,

parameters,

cv = 5,

n_jobs = 5,

verbose=True)

xgb_grid.fit(X_train,y_train) ##### We print the best score and best parameters

print(xgb_grid.best_score_)

print(xgb_grid.best_params_) ##### results :

{'alpha': 1, 'colsample_bytree': 0.5, 'learning_rate': 0.1, 'max_depth': 4, 'min_child_weight': 4, 'n_estimators': 500, 'nthread': 4, 'objective': 'reg:squarederror', 'silent': 1, 'subsample': 0.5} ##### Final Model:

xg_reg = xgb.XGBRegressor(objective ='reg:squarederror', colsample_bytree = 0.5, learning_rate = 0.1,max_depth = 4, alpha = 1, n_estimators = 400)

xg_reg.fit(X,y) #### We compute R²

res = pd.DataFrame({'reel':y, 'pred':xg_reg.predict(X), 'dif':abs(y-xg_reg.predict(X))})

r2_score(res.reel, res.pred) #### We check that our model is still good for test sample

The cloud of points is distributed around the diagonal, moreover we obtain an R² of 0.87.

Our model allows us to determine 87% of the point distribution.

Conclusion:

The latest model is the best model. We will keep this model for the future.

If you liked my article press like button and do not hesitate to follow me to receive my next articles :)