Is there a correlation between a restaurants ratings and the income levels of a neighborhood ?
By: James Mbuthia
Tags: Machine Learning, Yelp Dataset, Capstone Project, Data Science, Toronto Dataset

This post is the capstone project of the Coursera IBM Data Science Professional Certificate. The full notebook and data are available on my github repo

Introduction

A few weeks ago, my girlfriend and her friends went out for a Saturday girls lunch at an upmarket and expensive restaurant in Johannesburg. The restaurant is set in an exclusive estate and boasts of very scenic views however the girls lunch was not a success because of the restaurant. The food and drinks were overpriced, the meal portions were little, the service was terrible and the food was mediocre at best. The ladies had to pass by a local KFC to get some chicken to soothe their appetite.

This experience made me wonder. Do we assume that expensive restaurants will have better service than less expensive restaurants. Is the service better in restaurants in higher income areas than in lower income areas? Is there a correlation between income levels and restaurant ratings? This is an interesting question because conventional opinion would say that restaurants in higher income areas have better service which leads to better ratings because the restaurant’s patrons will demand better service and/or the restaurant will offer better service when in a high income neighborhood because they can charge more. But can the data science prove that or is this biased thinking?

My study will look at the data and see whether we can confirm that restaurant service does differ depending on the income levels of the neighborhood. If the study shows there is a link between poor Yelp ratings and income levels, this would be of major interest to business owners, to restaurant patrons and to a city’s management team. For business owners running a restaurant, they might spot an opportunity to gain customers by improving their customer service especially in low income areas. Patrons living in low income neighborhoods would have the ammunition to demand better customer service in the restaurants they frequent. City hall officials might also see this as a form of income segregation and work harder to ensure restaurants in low income areas have the same amenities and service levels as those in middle or high income areas.

The city I have chosen for this study is Toronto, Canada. It’s a relatively wealthy city with a population of approximately 6.5 million people (2016 Canada census). It is also one of the most diverse, multicultural and multiracial cities in the world with 51% of the population made up of visible minorities according to the 2016 Canada census. The big and varied population has resulted in a large number of restaurants and fast food places offering all types of cuisine from Indian to Chinese to South Korean to Ethiopian. This is good for our study as it means we can get a diverse number of restaurants to sample.

Data acquisition and cleaning

Data Sources

We will use the following datasets to work on this study

A Toronto neighborhood dataset showing average income per neighborhood in Toronto. We have sourced this dataset from the Open Data website for the City of Toronto, http://map.toronto.ca/wellbeing A Toronto neighborhood restaurant dataset with average restaurant ratings. We will source this data from Yelp by collecting the ratings of restaurants in each Toronto neighborhood using the Toronto dataset above and the Yelp API. We initially wanted to use Foursquare however the Foursquare API has very little api calls when compared to Yelp. The Foursquare API allows 500 api calls per day for free while the Yelp api allows 5000 api calls per day for free. On top of that Yelp was built for reviews and ratings and has a much larger restaurant reviews database then Foursquare therefore it made more sense to use Yelp.

Methodology

I first downloaded a csv file from the map.toronto.ca website. The website is a data scientist heaven. It’s well built and enables a user to customise what metrics he wants in a dataset such as neighborhood population, income before/after tax, area, demographics, employment, health statistics etc. The dataset we downloaded had the following features; neighborhood, total population, total area, after tax household income.

This is my first dataframe

Next step was to pull the latitude and longitude of each neighborhood using Google Maps. We will use the coordinates to query the Yelp api and get neighborhood restaurant reviews and ratings. I initially tried to use Geopy to get the latitude and longitude but I kept on getting timeouts and it couldn’t find some coordinates so I dusted off my credit card and used Google Maps.

I created a new dataframe which had the neighborhood, restaurant review count, rating and specialization. The next step was to merge the two dataframe together. After merging the dataframes and deleting unnecessary columns, I got the dataframe below with approximately 6,700 rows.

Exploratory Analysis

We want to know if there is a relationship between the income levels of a neighborhood and the average ratings that restaurants in the neighborhood receive. If there is a correlation than as income levels rise than average ratings would also rise. There would be a linear relationship between income and ratings. So first thing we will do is visualize the data using a scatter plot to see if we can get a clue.

There’s definitely no linear relationship there. Lets try again using different parameters against ratings.

In the figure above, we can see there is no linear relationship between population/ratings and review count/ratings. Our initial analysis shows that a neighborhoods income, population, population density has no effect on a restaurants ratings. However, lets create a linear regression model and evaluate its accuracy to see what we can get.

Model creation and evaluation

This results in a high negative R square. Our model just isn’t working at all. Let's add in other features and see if it will make a difference

The R square value is better but still not acceptable. We can safely conclude that there is no correlation between income and restaurant ratings.

Discussion & Conclusion

It is good to know that the income of a neighborhood has nothing to do with the ratings of its restaurants. High income neighborhoods can have poorly rated restaurants and low income neighborhood can have highly rated restaurants. Also other metrics like the population density or the number of reviews a restaurant receives have no effect at all on a restaurant’s ratings. It would be more effective to try and use sentiment analysis to calculate the rating of a restaurant. This would involve analysing the text of the reviews to see if we can use that to calculate the rating a user would give a restaurant.

The combination of data from Yelp and other similar sources with a city’s data opens up vast opportunities for data scientists. For example, approximately 80–90% of restaurants fail in the first year. It would also be interesting to see whether the income levels of a neighborhood have an effect on a restaurant’s success or failure. Another idea is to find out which restaurant specialization e.g. Japanese, Chinese, pizza etc has a higher chance of success in a neighborhood e.g. is a fast food restaurant more likely to succeed in a low income neighborhood than a high income neighborhood?

There lots of other interesting studies that we can work on when we combine different sources of data together. As more and more cities and countries open up their dataset to the public there will be lots of opportunities to gain insights from the data by combining it will other available datasets, doesn’t matter whether public or private.