Google analytics customer revenue prediction-Feature Engineering
Tags: Python, Kaggle, Features, Google Analytics, Time Series Analysis

This blog is a part of a series about Faeture Engineering techniques for Google Analytics using machine learning

You can check out the rest of the articles below

As we have seen how to load our data now and EDA. Now let’s see how to do Feature Engineering and many ways to predict the revenue.

Feature Engineering Time-Series Featurization Getting Train and Test data Features Extracted Machine Learning Models Hyper Parameter Tuning for Classification Models Hyper Parameter Tuning for Regression Models Getting Important Features Models and Results Challenges faced and its Solutions Future Work Profile References

1. Feature Engineering

Creating Features….

We have the date and visit start time through which we can get features like day, weekday, month, year, is_weekend, and visitHour.

numerical features

We convert all the numerical features into float.

Label Encoding

For all categorical features, we will apply label encoding and store all the labelencoder.classes_ so we use the same in production.

We fit on all categorical features and transform all the data

Hey Wait a minute One Question we are doing Feature engineering but what are we gonna predict and how ??

Let me explain….

We are gonna predict the revenue of the customers or users in the future.

This is a regression problem and most values are zero ., so we are going to solve these kinds of problems using hurdle models.

Lets deep dive into it now

The below is the data given from kaggle

* train data time period range is: Aug 1'st 2016 to Apr 30th, 2018 => total 638 days.

* test data time period range is: may 1'st 2018 to Oct 15th, 2018 => total 168 days.

* Prediction data time period range is: Dec 1'st 2018 to Jan 31'st 2019 => total 62 days.

* test data time period range is: may 1'st 2018 to Oct 15th, 2018 => total 168 days. * Prediction data time period range is: Dec 1'st 2018 to Jan 31'st 2019 => total 62 days. so here we need to predict the revenue of users from Dec 1st, 2018 to Jan 31'st 2019 by using the train and test data given to us.

so we had data until Oct 15th, 2018, and prediction data beginning date was Dec 1st, 2018., so the in-between period we call it as “ cooling period ” and it is from Oct 15th, 2018 to Dec 1'st 2018(46 days).

” and it is from Oct 15th, 2018 to Dec 1'st 2018(46 days). So Revenue can be generated only if the user has a visit in the future. So let's predict whether the user will come to the store or not after the “cooling period” of 46 days using the classification model.

suppose if he will come to the store then we will predict the revenue of that user by using the regression model.

So we need to create data for classification problems i.e if the user comes back to the store or not. Let's create data which is the most challenging and exciting part.

So as above part we analyze test_data (168 days) wait for (46 days) and predict if the user comes or nor for next (62 days)

In the above diagram, we can see many people are there by train. Less showed up in the test and 0 in the cooling period.

ex: I have a grocery store. Many customers come to the shop. I closed my shop for one month. Before opening shop next month I predict which known customers come to shop later when I open.

That means train data will consist of 168 days data and test data will consist of 62 days data and the gap between the train data and test data is 46 days.

We do not have data if the user comes back or not so let's create data. For this, we have to do Time-series Featurization.

2. Time Series Featurization

so by using this train data we need to predict whether the user will come to the store or not for test data that we prepared.

ex:

train data = Aug 1'st 2016 to Jan 15th 2017 (168 days)

test data = mar 2nd 2017 to may 3'rd 2017 (62 days)

The gap between train and test data is 46 days.

so by using the train data that we had we can make 4 sets of train and test frames. data set-1:

*train data = Aug 1'st 2016 to Jan 15th 2017 (168 days)

*test data = mar 2nd 2017 to may 3'rd 2017 (62 days)

data set-2:

*train data = Jan 16'st 2017 to Jul 2nd 2017 (168 days)

*test data = Aug 17th 2017 to Oct 18th 2017 (62 days)

data set-3:

*train data = Jul 3'rd 2017 to Dec 17th 2017 (168 days)

*test data = Feb 1'st 2018 to Apr 4th 2018 (62 days)

data set-4:

*train data = Dec 18th 2017 to Jun 4th 2018 (168 days)

*test data = Jul 20th 2018 to Sep 20th 2018 (62 days) From above data sets for the users which are common in train and test(that means they returned after cooling period) we will create a new feature ‘is_returned’ and we will set it to 1., for the users which not returned we will set ‘is_returned’ to 0. we will create some new features for the every user in ‘train data’ and finally we will merge all this data frames.

so now our target features are “ is_returned ” and “ revenue ”.

” and “ ”. where “ is_returned ” will indicate whether the user will come to the store or not in the future.

” will indicate whether the user will come to the store or not in the future. where “revenue” will indicate that the revenue generated by the user which is our target variable.

Note: I know for first time it is very difficult to understand. so here in fewer lines I will give the brief summary of time series featurization. we decided to build classification model and regression model. Here the task of classification model is to predict whether the user will come to store or not. If he is not coming to store store then the revenue from that user is zero. until here we are clear. But for building classification model we don’t have any labelled data with us. so we are trying to generate the data for classification model. so with the data that we have on our hand we are dividing it into train frame and test frame by replicating real world scenario( cooling period gap). If the user present in both train frame and test frame means he will come to store and label for that user is ‘1’., If he is not present in test frame then we will label that user with ‘0’. I hope now it’s clear for you.

So we create data for 4 sets as mentioned above to get is_returned.

above we created is_returned feature

There is a problem here. Let's take a user. A user can have transactions in London, New York, Ohio, and many more. What to pick and how to pick the data for a user.

Here comes the feature engineering part. For categorical variables, we take max() and min() as we cannot take min and max. For numerical we take min, max, mean and sum.

But Why?

ex:

if user has 15 transactions from 10 different places.We take min transacted place, max transacted place, amount of total min,max,mean and sum of transactions for user. This gives us broader view of customer from high to low end case.

3. Getting Train and Test data

train_test_data = pd.concat([train_df, test_df],axis=0).reset_index() %time train_frame_1 = get_time_series_features(train_test_data,1)

train_frame_1.to_pickle('train_frame_1') %time train_frame_2 = get_time_series_features(train_test_data,2)

train_frame_2.to_pickle('train_frame_2') %time train_frame_3 = get_time_series_features(train_test_data,3)

train_frame_3.to_pickle('train_frame_3') %time train_frame_4 = get_time_series_features(train_test_data,4)

train_frame_4.to_pickle('train_frame_4') #concatenating all our featurized frames: final_featurized_data = pd.concat([tr1, tr2, tr3, tr4], axis=0, sort=False).reset_index(drop=True) for all our test records we already fill the 'revenue' column with 'null' values.,

# so here we are separating our train and test records train_df=final_featurized_data[final_featurized_data['revenue']

.notnull()] test_df=final_featurized_data[final_featurized_data['revenue'].isnull()]

4. Feature Extracted

Summation of time on site

Min value of time on site

Max value of time on site

Mean value of time on site

Min value of weekday

Max value of weekday

Min value of day

Max value of day

Min value of month

Max value of month

Min value of year

Max value of year

Min value of visitHour

Max value of visitHour

Summation of pageviews

Min value of pageviews

Max value of pageviews

Mean value of pageviews

Summation of hits

Min value of hits

Max value of hits

Mean value of hits

Count of visitStartTime

Max value of visitNumber

Summation of transactionRevenue

first_ses_from_the_period_start — — Days from first shopping session for customer from the period start date

last_ses_from_the_period_end — — Days from last shopping session for customer before the period start date

interval_dates — — Difference between first and last shopping session for customer in the current frame

5. Machine Learning Models

Here we are using two models for building final that will predict revenue:-

Classification Model to predict whether customers would return during the test window.

Regression Model to predict transaction amount.

finally

predicted revenue = classification model output(probability)* regression model output(real value)

Note: If predicted revenue is negative then we will make it zero. since revenue is not negative.

6. Hyper Parameter Tuning for Classification Models

Here we are taking Random-Forest as our base model for the classification task. so by using random search we will find the hyper-parameter values.

After performing Hyperparameter tuning we got the best score for n_estimators = 1000

7. Hyper Parameter Tuning for Regression Models

Here we are taking Random-Forest as our base model for the classification task. so by using random search we will find the hyper-parameter values.

After performing Hyperparameter tuning we got the best score for n_estimators = 200

8. Getting Important Features

These are the important features for classification

These are the important features for Regression

9. Models and Results

We are building a classification model and regression model on the best hyperparameter values that we got and we run for multiple times(let say 10) and we are taking the average value of all the predictions generated in each iteration.



#Source :-

%%time

classifier = RandomForestClassifier(n_estimators = 200)

regressor = RandomForestRegressor(n_estimators=1000)classifier.fit(train_df.drop(target_columns, axis=1) , train_df['is_returned'])

regressor.fit(train_df.drop(target_columns, axis=1)[train_df['is_returned']==1], train_df['revenue'][train_df['is_returned']==1])final_pred = 0for i in range(10):

print("training model : ",i)



classifier_pred = classifier.predict_proba(test_df.drop(target_columns, axis=1))[:,1]

regressor_pred = regressor.predict(test_df.drop(target_columns, axis=1))



final_pred = final_pred + (classifier_pred*regressor_pred)final_pred /=10 #Running Random Forest for 10 iterations and took average of those.#Source :- https://www.kaggle.com/kostoglot/winning-solution %%timeclassifier = RandomForestClassifier(n_estimators = 200)regressor = RandomForestRegressor(n_estimators=1000)classifier.fit(train_df.drop(target_columns, axis=1) , train_df['is_returned'])regressor.fit(train_df.drop(target_columns, axis=1)[train_df['is_returned']==1], train_df['revenue'][train_df['is_returned']==1])final_pred = 0for i in range(10):print("training model : ",i)classifier_pred = classifier.predict_proba(test_df.drop(target_columns, axis=1))[:,1]regressor_pred = regressor.predict(test_df.drop(target_columns, axis=1))final_pred = final_pred + (classifier_pred*regressor_pred)final_pred /=10

Now we will create the final submission.csv file with ‘fullvisitorID’ and ‘PredictedLogRevenue’ as columns.

pred_df_new_date = pd.DataFrame({“fullVisitorId”:test_df[“fullVisitorId”].values})pred_df_new_date[“PredictedLogRevenue”] = final_predpred_df_new_date.columns = [“fullVisitorId”, “PredictedLogRevenue”]pred_df_new_date.to_csv(“pred_rf_new_date_pkl_rf_19.csv”, index=False)

Castboost

Catboost is a new tree-based splitting algorithm. It is very easy to use and can be run on GPU’s which helps us to run even the biggest computations in way lesser time and gives amazing results when compared with xgboost for this problem.

10. Challenges faced and its Solutions

Data Loading

We cannot load large size if our ram size is small. There is a hack if you are loading your data using pandas.

There is a parameter called chunk size in pandas. It means the data loaded will be read in several chunks given at a time but not all the data together.

So in this way you can load the data even if you have less computing power.

Other Alternative anyway is load your data using Dask

Data Downloading

The other issue I faced is when downloading the data from kaggle. I have downloaded the train and test files separately where the total data is not downloaded for the train.

The solution is to try to download the total zip file of Competition at a time. So you would not miss any data when downloaded.

Hyperparameter Tuning

While doing hyperparameter tuning for RF try to do tune only for n_estimatore as other parameter tuning did not give better results than the default.

The above challenges and solutions are subjective in my opinion. Wrote as it helps others if you are facing the same issue.

11. Future Work

We can use LSTM’s, Neural Networks, and perform Time-series featurization.

We can even use Matrix factorization methods for getting clear similarity and combine with Deep Neural Networks which gives better results.

By using the above feature engineering and models I have got good results than top kaggle score.

12. Profile

13. References