Dissecting Review Scores: What makes a top-scoring Airbnb?
By: Madeline Kehl
Tags: Kaggle, Airbnb, Random Forest Regressor, Multilevel Modeling, Udacity Nanodegree

Credit for icons: Adrien Coquet, Srinivas Agra, Cattaleeya Thongsriphong, IconMark @ Noun Project

If you had a friend who wanted your advice on how to open a highly rated Airbnb, what would you suggest to them? Low prices? Specific neighborhood? Maybe writing compelling descriptions?

There are many intuitive suggestions, but in order to give our friend good advice, it would be nice to have some real data to back it up. We can use Airbnb data available here: https://www.kaggle.com/airbnb/boston

STEP 1: Which features might be important?

First, it would be a good idea to find which elements have a better than average shot at making an impact on your friend’s review scores. We can do this using a Random Forest Regressor.

One might ask, if we are going to have to check them all for significance anyway, why don’t we run all the variables? While this might make sense if we had only 5 or 10 variables, it starts to get really inefficient.

For example, in this case we have 126 possible predictors. Running 126 separate statistical models when most of them will be insignificant, doesn’t make sense. Using random forests, we can select features that are likely to be worth testing.

After cleaning our data, we find that the following features have greater than average importance in our models:

Top Features Predicting Review Scores

It looks like several variables relating to host experience made their way into the top features as well as several amenities and some text-related variables.

However, it is still too early to draw conclusions. This is because even though we know that these predictors are more relevant than the others in our dataset, we still don’t know their full statistical relationship to our outcome variable, review scores.

STEP 2: Which features are actually significant?

Because we have different types of id variables, multilevel models or Linear Mixed Effect models will best suit our data. For more information: https://www.statstutor.ac.uk/resources/uploaded/multilevelmodelling.pdf

We can run them on each of the features that might be important and get the following significance scores (p < 0.05):

Top Features P-Values (significant if pvalue < 0.05)

As we can see here, the vast majority of our top features are significant, however two (word count of comments, and adjective/adverb count of comments) don’t have a significant relationship to review score. We can eliminate them from our final analysis.

STEP 3: How are these significant and how impactful are they?

We should now examine or significant features and see:

the directionality of the effect (is it positively or negatively impacting our outcome variable) The size of the effect (is it relatively important or unimportant, even if it is significant.)

Coefficients and Confidence Intervals for Significant Top Features

As we can see, a lot of our significant effects have really small effect sizes. So, even though they are significant, maybe they aren’t the most important to focus on.

Even though we had separate neighborhoods encoded in our data, latitude and longitude were the only relevant geographic features. They also look a bit unreliable (large confidence intervals). If our friend is trying to open an Airbnb in Boston, maybe this is helpful, but they are hard to interpret without more information.

One finding that does seem clear, is that amenities are the most impactful (large effect sizes) and consistent (small error bars)predictor of review scores. WiFi, AC, fire extinguishers, hair dryer, and laptop friendly workspaces, are all associated with higher scores.

Conclusions

After our analysis, we can now tell our friend our recommendation for high ratings — invest in some amenities for their Airbnb space.

Of course, we don’t know any of the causal mechanisms here. Maybe people who include these amenities are more likely to have better decor, which is what is really driving higher ratings — so if our friend invests in amenities without having the decor, they won’t get a ratings boost. That being said, based on this information, amenities are our best bet.

If you have ideas for how to make a better recommendation, are interested in using the same process for a different recommendation, or just want to see the code it is available here: https://github.com/madkehl/Airbnb