AIR BNB NEW USER’S TRAVEL DESTINATION PREDICTION
By: J Jayasri
Tags: Python, Kaggle, Machine Learning

In this article, we will look into how Airbnb uses machine learning to predict the user's travel destination

Pre-requisites:

Python

Machine Learning

Here’s my approach:

Business problem Use of ML Source of Data Evaluation Metric Solving approaches to the problem Exploratory Data Analysis-EDA Pre-processing data Feature Engineering ML Modelling Comparison table Kaggle score Future work References

1.Business Problem:

Airbnb is a travel community wherein people can list out, discover and book rooms around the world. By providing accurate predictions where a new user would book their first travel experience, so that Airbnb can personalize their content according to user’s preference and better be able to identify the demand of country destination bookings and to increase their revenue.

2.Use of ML:

Here Machine learning is used to predict Top 5 countries a new user would make his or her first travel booking.

3.Source of Data:

Here the data is collected from Kaggle AirBnB New User Bookings Competition

Overview of Data:

Here the data given is age_gender_buckets.csv, sessions.csv,countries.csv,train_users.csv,test_users.csv

countries.csv: summary statistics of destination countries in this dataset and their locations

age_gender_buckets.csv: summary statistics of users’ age group, gender, country of destination

train_users.csv: user’s information

sessions.csv: web sessions log for user

test_users.csv: features contains the same as train_users set except for the class label

4.Evaluation Metric:

Here the metric used for this Kaggle competition is NDCG Score

It is a measure for ranking quality to evaluate a recommendation engine.

To know more about it, check this

5.Solving approaches to the problem:

a.Existing approaches to the problem:

Considering the train and sessions data

Impute Null values,Mean,Mode,Median

Extracted meaningful information from date features of train data and considered the 3-dimensional vector by combining the first device type, first browser, signup app

Considered a 2-level classifier approach for destination prediction

For more existing approaches documented here

b.First Cut approach to the problem:

Train and session data were merged and divided the merged data into 2 data frames one session and the other without session

Data analysis was done separately for both data sets of train and sessions data

Null, mean, median values imputations were done with the help of other columns/features

Extracted meaningful features like frequent action done by the user and average time spent by users

All categorical features were converted to one-hot encoding Features

Modeling has been done with an ensemble of High variance models(Random Forest) and also with an ensemble of High bias models(Xgboost classifier).

For detailed description, it is documented here

c.Final Improved Approach:

Train and session Data were used

Mean, Median imputations are not considered instead treated Null values as separate categories

Advance Feature engineering techniques like feature binning on sessions data to extract more meaning full features about 869 features are were extracted from sessions data

Applied Standard scaling on features to set each feature on the same range

Used Xgboost model for feature selection

For a detailed description, it is documented here

6.Exploratory Data Analysis:

a.Univariate Analysis

b.Bivariate Analysis

train.csv:

train.csv file

Here train data consists of 213451 users information/rows with 16 features/columns

Sessions.csv :

sessions.csv file

Here the sessions data consists of 10567737 user information and contains 6 features/columns

testdata.csv :

test data.csv file

Here test data consists of 62096 user information and 15 features

Here the features are similar to train data except for the true label-country destination column which has in train data

Check for Missing Values:

For train data:

Here date first booking, age and first affiliate tracked has missing values

For sessions data:

Here every feature has missing values except the device type feature

For test data:

Here data date_first_booking and age feature has missing values in the test data

To check percentage match on train data and test data with sessions data:

Train data with sessions data:

Only 34% of train data are matching with session data

Test data with sessions data:

99% of test data are matching with sessions data

So merging sessions data with train data and training with information from sessions data might be useful for prediction of test users

a.Univariate Analysis:

Country destination:

Here most of the data given are NDF(No destination Found) which constitutes about 58.3% and next to it is US country which constitutes about 29.2%

so the data is highly imbalanced

Age Feature:

To check outliers:

boxplot of age feature

Through the above plot, we can confirm outliers but for detailed observation, we will compute percentiles

Description of Age Feature:

Age column was right-skewed as mean>median and spread is more, outliers are affecting the distribution of age

so there is a need for replacing or removing outliers

check for percentiles:

Age column contains age-1 and 2014 as the age-1 person cannot travel and some users might fill a current year in the place of the age column

so these are outliers in the age of users in data

so here I am considering from age-19(1st percentile) to age-71 (97th percentile) and will observe the distributions within this age group

To check distributions of users with age between 19 to 71:

distplot of age feature

Here we can observe that most of the user’s age is between 25 to 40

Affiliate channel Feature:

value counts of the affiliate channel feature

Here 64.5% of user’s have visited the website directly without any paid marketing

Date first Booking Feature:

Booking month:

Here in 5th and 6th month, 11.8% of users have booked

Booking day:

Here the users have booked mostly on weekdays(Wednesday, Tuesday, Thursday) than weekends which constitutes about 16.2%,16.1%,15.7%

Insights about other categorical features :

Most of the user gender data provided are unknown and the next highest percentage are females

Most of the users have signed up using the basic method unlike Facebook or google

The majority percentage of users have signed up for the website directly using a web browser without any paid marketing

Most of the users have used Mac and Windows Desktop devices

Users have logged in to the website using chrome and safari browsers

Secs elapsed Feature:

To check outliers:

Through the above plot, we can confirm outliers but for detailed observation, we will compute percentiles

Check for percentiles:

Here I’ve taken from the first percentile to 90th percentile, beyond that can be taken as outliers as the user cannot spend 9hrs checking the website at one go

Description of secs elapsed feature excluding the outliers:

Here the mean time spent by the user for one-time usage was 3076 seconds and there is a lot of variation in data(4498 secs)

We need to normalize the column as it holds too much variance

b.Bivariate Analysis:

Age vs Country Destination:

boxplot of age vs country destination

Here in Great Britain(GB), data is positively skewed which implies users which have age 30–45 traveled more

for ES, young people have traveled more of age below 40

for all countries, users with age group 30–40 have traveled more (as median lies in this range)

Date Account Created vs Booking:

line plot

For every year, there has been an increasing trend in the first few months

No.of actions vs Users:

pdf of no.of unique actions done by a user

Here most of the users performed 5 to23 unique actions

Avg hrs vs country destination:

box plot for avg hrs vs country destination

Here users who have not booked spent avg_time(hrs) was <5 hrs and the users who have booked have spent some time>5hrs

Of all the countries, users who have booked for PT has spent a considerable time around 8 hrs

Overall observations from EDA:

Here the data is highly Imbalanced

Here all the Numerical features have outliers

Most of the bookings were done on the 5th and 6th month of every year on weekdays and there has been a continuous increase in bookings over the period of time

Here there are high null values so we cannot exactly fill the data using other imputations so they can be treated as other categories in the particular feature

7.Pre-Processing Data:

There are outliers in the age column, replaced outliers with the median

Rest all other categorical features missing values are filled with NAN as a separate category

sessions data categorical features missing values are filled with NAN as a separate category

secs elapsed feature is filled with zero

8.Feature Engineering:

a.Feature Extraction b.Feature Selection

a.Feature Extraction:

Extracted log values of min, max, mean, median of secs elapsed column

Extracted skewness, entropy, kurtosis of categorical features of sessions data

Extracted 5 groups/bins for age feature

Extracted difference between date account created and time stamp first active feature

Performed standardization on data

b.Feature Selection:

Here 70% of features are selected by training XGBoost Classifier

9.ML Modelling:

Modeling is done for identifying the patterns in data through some parameters and these parameters were learned through model evaluation by using a metric, and these learned parameters are used in predicting the required outcome

a.Logistic Regression Classifier:

As all features are standardized the outlier effect is reduced

With high dimensions, we might find a linear separating hyperplane which separates Data one vs rest is performed as data is multi-class and a regularizer is used to reduce overfitting

code snippet of Logistic Regression Classifier

b.Decision Tree classifier:

The decision tree can handle missing values, it doesn’t depend on the scale of data and considers randomness in data, and also separates data with Gini impurity

The runtime complexity of the decision tree is low.

code snippet for Decision Tree Classifier

c.Random Forest Classifier:

High variance models are initialized used for training and the variance is reduced by increasing the number of estimators and fitting with random subsamples and sub-columns

It performs well with high dimensional categorical data

code snippet for Random Forest Classifier

d.Stacking Classifier:

It acts as a calibrating classifier

Two staged model classifiers are applied on two different random data points(D1 and D2), where at the first stage, different XGBoost models were fitted on random data points, and in the second stage the first stage ensemble models are used for predicting the data points of the second stage and these predictions and D2 labels were used as data for training meta-model (stage 2 classifier).

At the prediction time, a data point is given to the first stage for predictions and these predictions were used as data points for the second stage and stage 2 classifier predicts final outcomes.

code snippet for stacking classifier

Best Estimator:0.830024

e.XGBoost Classifier:

Comprises of Very low variance and high biased models and bias is reduced by the additive combination (error fitting) by keeping variance low thereby no overfitting.

As no of estimators increases, the bias gets reduced

code snippet for XGB Classifier

10.Comparison of Models using Pretty Table:

pretty table

XGB classifier outperformed all other classifiers rather than depending on single model predictions.

XGB uses an ensemble of decision trees and every model gets different samples of data with different features(as we used subsample and column sample parameters), and also due to additive combination(error fitting) the model has learned the patterns in Data very well and no of estimators are high and regularization parameter helped to reduce overfitting which helped it to generalize very well

11.Kaggle Score:

kaggle score in the leaderboard

Hereby using the XGBoost classifier with 8000 estimators. I’ve secured top 6 percentile in the Kaggle competition with a private score of 0.88565 and a public score of 0.88026

12.Future Work:

Here instead of using ML models, we can use Neural Networks with a reasonable depth of layers which is useful in automatic feature extraction and selection.

We can try out different types of Stacking Classifiers, Cascading Classifiers in ML Modelling

For feature selection, we can try out Forward feature selection, Backward feature selection, select k-best

We can try out response encoding instead of using one-hot encoding for categorical features

we can also try out the clustering methods to get more advanced feature engineering techniques

13.References:

Major Credits: Applied ai course

Github Link:

check out my Github repository here

Linkedin profile:

check out my Linkedin profile here