Airbnb price prediction Across Different Cities
By: Kashish Narang
Tags: Deep Learning, Regression, Machine Learning

Dataset Information

The dataset has been taken from the website InsideAirbnb.com(http://insideairbnb.com/get-the-data.html) which is an archive of the Airbnb listings for different cities. Here, we have used the listings data of the cities: Paris, Berlin and Melbourne. There are 96 features in Paris and Berlin while 84 in the case of Melbourne. The count of listings in Paris, Berlin and Melbourne are 59881, 22552 and 22895 respectively.

Exploratory data analysis

We performed exploratory data analysis to see the distribution of prices with respect to various features. The list of such analysis is exhaustive (and we care for you), so just included some of the important findings.

i) We tried to find the price range and we found that the median price for each of the three cities was around $90 to $100. The maximum price was even greater than $3000, mainly all of them were in very less number. So, we took prices in the range $0 to $500.

ii) Then we tried to analyze the price variations with the number of bedrooms and we found that the median price was maximum if the number of bedrooms is 6. But with the increase in the number of bedrooms after 6 the price also gets decreased and this is obvious as its demand is less.

iii) We also tried to find the price according to the cancellation policy and we observed that the price was very high for the super_strict_30 cancellation policy.

iv) We then calculated the popularity among various types of room and we found that the most popular room type was the entire home/apartment.

Evaluation Metric

Since this is a regression task, we have currently used Root Mean Square Error and R2-score to evaluate the models. R-squared is a statistical measure of how close the data are to the fitted regression line. It is also known as the coefficient of determination, or the coefficient of multiple determination for multiple regression.

where RSS is the sum of squares of residuals and TSS is the total sum of squares.

After being one with the data, Let’s dive deep into the process ….

Baseline Implementation

We have developed our baseline models with the help of two different types of machine learning models i.e. tree-based and statistical-based models.

Decision tree regressor: Decision tree being the simplest, interpretable learning algorithm is used to benchmark all the tree-based models in our solution. After tuning the hyper-parameter of max depth, the best decision tree model is fit on the data and predictions obtained for the test data. Linear regression: Linear regression being the most basic model for regression tasks is an obvious choice as a baseline. Linear regression with L2-norm: It is well known that normally the linear regression weights become unconstrained and result into overfitting. So, we went with L2- norm based regularization to somehow overcome the problem of overfitting. The lambda value is tuned using GridSearchCV and the best model used further for obtaining predictions. Random forest regressor: We hope that you must also be familiar with this popular bagging ensemble technique which greatly helps deal with the overfitting problem of decision trees. The different hyper-parameters like the number of decision trees, depth of the tree, count of features used for splitting have been tuned with the standard GridSearchCV.

Among the different baseline implementations, the best results are obtained using the random forest regressor. Using a random forest, the RMSE value obtained is 1762.37 and R2 score is 0.664869.

Table 0: Baseline Results

Methodology

Let’s divide our methodology to improve the price prediction task in 3 broad categories. At first, we will consider the state of the art ensemble models and deep neural networks on the baseline dataset. We expect the reader to be well aware of all notations and nuances of ANN and CNN models. Second, we will take you through some of the text features and let them meet the continuous/categorical features. For this, we have extended the previous neural architecture. Finally, we have considered the pair of city data for training and prediction on the third city in order to make it interesting from a domain perspective and explore further. Let’s start!!

State of the art models

1.1 XGBoost: It is one of the best performing ensemble methods(results from various Kaggle competitions) based on gradient boosting technique. Real hurdle in applying XGboost is the tuning of a lot of hyperparameters such as estimators, depth etc. We used GridSearchCV for this task. The results given by XGboost are one of the best.

2 Stacking Ensemble: Stacking simply means take the output from different experts and then let one more independent expert decide the final outcome based on the outcome provided by all previous experts. In our case, we started with some weak learner of our application and then stepped up by adding different weak/strong learners to get the final combination giving the best results in this task. Final model’s first layer includes XGBoost, Random Forest, Linear Regression and K-NN model. While on the last layer, simple Linear regression has done pretty well.

1.3 Deep Neural network aka Feed Forward NN: In the modern era, deep learning models have performed really well but the real challenge is the design of architecture which works well in particular tasks. For neural network implementation, we have used the Keras library. We started with a basic 2 hidden layer model as it is capable of learning any continuous and discontinuous patterns. The first setting is to try a wider network by increasing the units in each layer. Next setting is to make this network deeper by adding more hidden layers. After trying these combinations and avoiding any overfitting issue, we finalised the architecture with 3 hidden layers with a number of units 128,64,32, a dropout layer with each hidden layer at a rate of 0.2, batch norm and ‘Relu’ activation function. We trained our model with an ‘Adam’ optimizer for 60 epochs. For each forward and backpropagation iteration, weights are updated by calculating the mean squared error as the loss function.

2. Text feature incorporation

A sample text in the form of description…

We offer comfortable accommodation in Inner Melbourne in Clement House, a two storey Victorian built in 1888. Large double bedroom with balcony, private bath and internet access; comfortable queen size bed.

There is one feature in the dataset named ‘Description’ of Airbnb listings. We used this feature to generate more features with the aim of performance gain over we already have with only categorical/continuous features. One can’t use the text in the raw form, so a very popular library for text processing called ‘nltk’(https://www.nltk.org/) is used to tokenize the description, stop words removal, convert to lower case, and numbers converted to words. Each word is represented by word embedding obtained from pretrained Glove50(https://nlp.stanford.edu/projects/glove/) where each word embedding has a length of 50 features.

We train a 5-layer neural network where convolution layer with output dimension 128, a max-pooling layer followed by two dense layers with output dimension 128 and 32 respectively and dropout rates of 0.3 and 0.2 with ReLU is used as the activation function. In order to use both the neural architectures i.e. one for categorical/continuous features created earlier and other for text features created just now, we will use ‘concatenation layer’ with the final output layer to combine both the networks.

3. Geography based analysis

Let’s try with another approach. We would like to see how the model learned on two cities data will perform on other city price prediction in terms of nearby to faraway cities. We have concatenated the data of two different cities and taken the data of the third city as the test data with different such combinations. We have performed a C(3,2) combination of train and test data for each city like Berlin and Paris as train data and Melbourne as the test data. We applied our baseline models on every combination of train and test data. We found that the best result we got on the training data was having Melbourne and Paris as train data and the Berlin data as test data for price predictions. We applied the best performing ensemble methods in the above approaches and tuned their hyperparameters.