AI and Real State: Predicting Rental Prices in Amsterdam | Towards Data Science

Renting or buying a new house, whether you are a university student or a middle-class family, is always a daunting process that often seems impulsive or risky (a true economic market for lemons).
If the renting itself is already hard, doing it in Amsterdam doesn’t make it any easier. With increasing city regulations, long waitlists for student housing and overpopulation, renting an apartment in Amsterdam leaves many desperate for the first opportunity available, becoming vulnerable to scams and overpriced contracts.
In this tutorial, I will go through an entire Data Science project for the rental market of Amsterdam, from the basics of gathering data, data cleaning, visualization, up until using machine-learning and hyperparameter tuning to develop valuation models for the city’s houses. There are also a lot of discussions about interesting machine learning topics along the way, such as the curse of dimensionality and multicollinearity. Feel free to adapt the code and apply the project in your city to understand a bit more of where you stand as a renter/buyer!
The actual statistical methods you employ in your analysis should be your value-judgment and I will link deeper explanations to all the methods I apply, so make sure you check them out!
With (high-quality) data, it’s the more, the merrier. Machine-learning is a special field of statistics where we apply computer algorithms to very very large datasets. After you have established the questions you want answered (should I rent? should I buy? should I move to another city?) you can start looking for websites that will contain the data that you need to answer them.
In my case, I wanted to find a good-value apartment in Amsterdam. Therefore, I searched for rental websites in the city of Amsterdam. Simple right? However, always check their terms of service and robots.txt to make sure that you are allowed to scrape their data respectfully. (we will talk about what this means later on in the tutorial). For this project, I will use the Amsterdam rental website Pararius.
Find the listing page you want to get your data from. Click inspect to learn more about how data is structured in their website. Once you click, the sidebar will show the HTML storage where the data is located. In the following example, the data I need is mainly inside the box <li class=search-list__item search-list__item — listing> this means that I will refer to this data using this information later in the code. At this moment, we don’t need to specify if we want price, location, zipcodes...we only want to know the “box” where all this data is.
Now, we are going to get started with the coding! As with any web-scraping, we will be starting with making requests to the website we chose previously so that they can provide us with the data we want.
We should keep in mind that making a request to a website is similar to refreshing our browser on that specific page: it adds traffic to their servers and may overwhelm them if done in robot high rates! I added random sleep times to the code so the script will stop a few seconds before scraping more pages.
If you are particularly interested in how requests and headers work, you can also make your requests with fully randomised headers. This will not be necessary for most websites (and should not break their TOS) but if you would like to add security and anonymity into your code, this is my solution for (as much as possible) anonymous headers:
After scraping all the desired pages, you can run:
len(houses)
to find out exactly how many house ads were collected in total. Because we are training a model with this data later on, you should aim for at least 1000 housing ads.
After you have gathered all the data, you should do a few print commands to make sure that everything worked out:
print(response) will print if the request was successful (i.e. we were not blocked by the website)
len(houses) will print how many house ads you successfully scraped
print(house_data[1]) will print the second ad block you scraped in HTML format. I always prefer to look at the second one because the first might contain headers and confusing bits for the next part of our analysis: the data cleaning.
When I execute print(house_data[1]) I get this in my Jupyter Lab:
Ok, don’t run away from this tutorial just yet. What you’re seeing here is the beautiful HTML script that was scraped from your listing website! Look further down into it:
We can actually recognise a few things there! The apartment in this ad block seems to be located at 1078 RA Amsterdam, it costs 1500 euros a month and is 60m2! For the data cleaning, we need to find this information in the soup of HTML and write down where they are located (just like we did earlier with the website!)
For example: to get the price, your need to search for <span class =listing-search-item__price> and to get the location you need: <div class =listing-search-item__location>. You should do the same with all the information that you need for your analysis. However, just looking for the HTML tag and class can return more information than you’d like. Make sure to try adding [0], [1],[2], ...to test which parameter will give you exactly the line you are looking for.
A lot of times, even after finding the right line, there will be extra characters that you want to clean out of the data frame, such as letters in the rent price, or whitespaces in the postcode. Check out the amazing regex tester (https://regex101.com/) to know how to str.replace() your problem away. A few examples:
to delete non-digit characters:
df["column name"].str.replace("\d","")
to delete digit characters:
df["column name"].str.replace("\D","")
to delete the word “new” :
df["column name"].str.replace("new","")
This is how our data frame looks like after cleaning:
It is not only important to have the data, but also know how to use it. What is important for tenants? What can make the difference in rental prices? Depending on the data you scraped earlier, these variables might be available to you:
Surface Size
Number of Bedrooms
If the apartment comes furnished (binary)
If the price is inclusive of utility bills (binary)
Distance to City Center
Trendiness of the neighbourhood
Rental Agency (binary)
Temporary vs. Long term contracts. (binary)
On this tutorial, we will focus on how measure these variables. I won’t be able to use furniture, utility bills or length of contracts for this analysis as that data is not available from the website I am using, but if those are available to you I highly recommend including them.
Location, location, location. Does the location of apartments really matter that much when renting a house? Many people in Amsterdam would agree, as driving cars is impractical in the charming medieval streets, public transport is not very affordable and biking under windy rain just sucks.
But how do you differentiate that effect on rental prices? Geolocation! In this tutorial, we will use Nominatim. We will use this to get the coordinates of every apartment in our dataset and later compare them with a (desirable) point in the city.
Here’s the code:
The idea here is to get a latitude and longitude for each address in your data frame, so we can calculate the distances as precisely as possible. This is what you should end up with:
It should be noted that the columns point, location, and altitude will be deleted. Point is only needed to obtain the latitude and longitude points and altitude is not needed since we are researching houses in The Netherlands!(but would be an interesting factor to take into account if you live in Switzerland, for example).
Let’s calculate the distance: by now, you should have chosen a point in your city to calculate the distance between the apartments and this specific point. For Amsterdam, I chose the Amsterdam Centraal station with the following coordinates (centre point):
After choosing my point and getting its coordinates (on Nominatim as well), you will create two columns with the latitude (52.370216) and the longitude (4.895168) of that point.
If you are analysing a bigger city that has multiple locations that are considered desirable, you can also run this code as many times as needed with different geographical points. (Don’t forget to change the column names so you don’t overwrite the previous point!).
For example, there is a financial district close to the Amsterdam Zuid station that could be equally (or even more) relevant to working tenants than living close to the city center. Measuring these various scenarios is more important if you are using methods similar to multiple linear regressions rather than machine-learning statistical algorithms because they are inherently better at recognising non-linear relationships and clusters. For this reason, I won’t include it in this analysis but it is an interesting factor to weight in depending on the statistical method being used.
Now that we have the geolocations of all the apartments in our dataset, we can further visualize how their rental prices are placed geographically and spot any trends that might be relevant for our analysis. We will do this with Google Maps, with the package gmaps for Jupyter:
conda install gmaps ## to install the google maps package
You will also need a Google Maps API key that is easily requested (and free for most purposes). You can click here to request it.
With the simple code below, we can request an interactive google maps on a Jupyter Notebook (sometimes it does not work correctly with JupyterLab), and output a heat map layer that will tell us where the highest rents of the city are.
We should get this beautiful geographical representation of rental prices in Amsterdam:
Analysis of the heat map: As expected, most of the higher rent prices are concentrated around the city center, specifically De Wallen, and also around the outer neighbourhoods such De Pijp and around the city park, Vondelpark, where there is a concentration of luxury houses. The further out neighbourhoods of Nieuw-West, Zuidoost, Ijburg and Noord appear to have lower rental prices represented in green. Location seems to have a strong effect on rental prices, but it is definitely not the sole factor.
For more examples and tutorials on gmaps click here.
We should also take a look on how the variables in our data frame relate to house prices. This is the code to check the relationship between house prices and surface area:
import matplotlib.pyplot as pltplt.figure()plt.scatter(amsmodel1['surface'],amsmodel1['house_price'], s=20, edgecolor="black",c="darkorange", label="surface")plt.xlabel("Surface Area")plt.ylabel("House Price")plt.title("Surface Area vs. House Price")plt.legend()plt.show()
When it comes to the surface area of the apartment, there is a very clear upwards sloping trend relationship between the two! However it should be noted that:
As the house gets bigger, the marginal price of an additional square meter decreases dramatically. Thus, almost always going for a bigger house (if you can afford it) will actually get you more bang for your buck.
This is code to check the relationship between house prices and number of bedrooms:
import matplotlib.pyplot as pltplt.figure()plt.scatter(amsmodel1['bedrooms'],amsmodel1['house_price'], s=20, edgecolor="black",c="darkorange", label="bedrooms")plt.xlabel("Bedrooms")plt.ylabel("House Price")plt.title("Bedrooms vs. House Price")plt.legend()plt.show()
In a somewhat similar conclusion to the surface area, the more bedrooms a house has, the higher the rental price should be. However, that is also not the only factor as there are 5-room apartments going for as little as $3000 a month and as high as $10000.
If you look into your data frame now, you will notice a lot of columns that were “step” columns in order to get more information about the houses. We can delete all these columns as we will not be using them anymore.
del df5['address']del df5['address2']del df5['altitude']del df5['latitude']del df5['longitude']del df5['point']del df5['lat2']del df5['lon2']del df5['coord1']del df5['coord2']del df5['location']
If we were running multiple linear regressions (or variants of one), we would have to check for independent variables with high correlations with each other. One way we can do this is through a correlation matrix:
As we can see in the correlation matrix below, there are two independent variables that show some multicollinearity (r > 0.7) : bedrooms and surface area. This makes sense since the bigger a house is, the more bedrooms it will usually have.
One way to solve this issue is to combine the two variables into one, such as creating a new variable for the square space per bedroom:
amsmodel1['surface_per_bedroom'] = amsmodel1['surface']/amsmodel1['bedrooms']
After deleting both surface and bedrooms variables, this is what our correlation matrix looks like:
This process eliminated multicollinearity from the dataset successfully. However, for most machine-learning algorithms, we never need to check for high correlations between variables — multicollinearity does not affect the accuracy of machine learning models, especially ensemble learning models such as the random forest algorithm.
At every point of the decision tree, the algorithm will make the best split that will predict the target variable more accurately, regardless of how the independent variables correlate with other. Additionally, for regression models the purpose is to understand the effect of a specific variable, whereas in machine learning, we are much more interested in the predictive power of the model. If you would like to read more on this, here is a great discussion on the subject: https://stats.stackexchange.com/questions/168622/why-is-multicollinearity-not-checked-in-modern-statistics-machine-learning
In every large city, a few neighbourhoods seem to be very popular (and thus have exceptionally high rents) even though they are not close to the city center, or necessarily populated by bigger apartments. An example that can be found in our Google Maps heat map would be De Pijp, that is outside the Amsterdam ring and mainly offers small and non-renovated apartments, but has a higher rents average than the apartments on the right side of the Centraal Station.
How do we measure the effect of popularity in a quantitative way?
One way is by using Yelp! Popular neighbourhoods tend to have popular bars and restaurants, with high ratings and potentially also high prices. Most people that pay higher rents to live in De Pijp justify it with:
“that’s where the life of the city is!”
“that’s where all the cool bars and restaurants are”.
Yelp can help us by giving us hundreds of restaurants and telling us exactly how well-rated they are and how much they cost in a standardised and easily quantifiable way: $ are cheap eats, $$ and $$$ are in the middle, and $$$$ are expensive. Furthermore, this also helps us understand the neighbourhoods that may not be popular at the moment, but are traditionally richer areas (with a lot of $$$$ restaurants) that consequently charge higher rents.
You will need to register for the Yelp API, and replace my api_key with yours, and also replace location with the city you are analysing. It is also possible to build a second outer loop to get data from different cities. If you have a data frame with the column city, you can transform that column into a list and iterate the requests over that list!
After gathering the data from Yelp, we need to match that with the rental data that we already have. Here’s how to do it:
After mapping the yelp data into our data frame, dropping any empty rows and checking how many columns we have, this is how the data frame looks like:
Now that we have a dataset with more information on yelp prices and ratings, we can visualize their relationships with house prices to understand whether this metric will likely improve our model or not.
The following code scatter plot yelp ratings against house prices:
import matplotlib.pyplot as pltplt.figure()plt.scatter(amsmodel1['yelp_ratings'],amsmodel1['house_price'], s=20, edgecolor="black",c="darkorange", label = "yelp")plt.xlabel("Yelp Ratings")plt.ylabel("House Price")plt.title("Yelp Ratings vs. House Price")plt.legend()plt.show()
The following code scatter plot yelp prices against house prices:
import matplotlib.pyplot as pltplt.figure()plt.scatter(amsmodel1['yelp_prices'],amsmodel1['house_price'], s=20, edgecolor="black",c="darkorange", label="yelp")plt.xlabel("Yelp Prices")plt.ylabel("House Price")plt.title("Yelp Prices vs. House Price")plt.legend()plt.show()
There definitely does not seem to be a strong linear relationship here when inspecting only the relationship between the two variables. Hopefully, we will be able to judge with the machine-learning algorithm if adding these variables have a resulting effect, albeit more complex that linear, on house prices after all.
Are we finally ready for to train the model? Nope. Our model will only take values in integer type, and right now we have a few columns that are categorical (or in other words, don’t make sense numerically but as categories for our apartments). We need to create something called dummy variables to represent these categories in a way that the computer can understand numerically.
Here’s a quick (but great) explanation about dummy variables and what they do: https://medium.com/@brian.collins0409/dummy-variables-done-right-588f58596aea
Here’s the code:
Now we have a staggering amount of 327 columns! In data science projects, we should criticise every step of our analysis to prevent bias and other misinterpretations. With data frames with high-dimensionality, we need to consider the curse of dimensionality which can confuse machine-learning methods due to the points being so far apart that they all look the same (and thus no conclusion/differentiation can really be made from the analysis). If our dataset suffers from this problem, it could decrease our accuracy results in the random forest algorithm.
The most accepted rule of thumb is that we should have at least 5 training data points for every feature in our dataset. In this project, we have:
training data set = 3376*80% ration training points/features = 2700.8/327 = 8.26
which (fortunately) passes our 5 training data to feature ratio rule!
At this point, our data frame should be looking like this:
We have 3376 rows, 327 columns, and data about the house prices, number of bedrooms, surface area, distance from the city center, mean yelp prices and ratings of their areas, dummy variables for postcodes and dummy variables for rental agencies.
The Random Forest algorithm is an ensemble learning method that builds decision trees, splitting the data and testing each decision to understand the weight of each feature, hopefully achieving the true effects of the features of dataset against the target (in this case, the house prices). Here is a great article about this algorithm.
With random forest, I will also combine the K-Fold (with K = 10) Cross Validation approach, which means we will be slicing the data into ten parts, training 9 parts and testing it against the 10th part, and using a different slice of that ten-piece pie as the testing data in each iteration, maximising the size of our dataset to obtain better results. More about Cross Validation here.
For this project, I will be using the open source package sklearn to apply the algorithm to the dataset. Here’s the code:
After a few minutes, you should start getting a few results. These are the results I got for this dataset:
An average accuracy of 94.75% is not bad, but would it be possible to increase the efficacy of the model? We can try different Hyperparameter Tuning settings to possibly increase the accuracy rate by a few percentage points:
This part of the article is heavily based on this article: Hyperparameter Tuning the Random Forest in Python, you should definitely check it out for a deeper understanding of the code.
Sklearn has the great feature RandomizedSearchCV that tests different parameters and recommends which parameters are likely to maximize the accuracy results of the model.
The code first specifies what are the choices for each parameter that should be tested by the feature:
And we fit the Random Forest Regressor into the RandomizedSearchCV, and it searches across 100 combinations 3 times, and returns the best parameters.
These are the results for the best parameters we get for this model:
{'n_estimators': 400, 'min_samples_split': 2, 'min_samples_leaf': 1, 'max_features': 'sqrt', 'max_depth': None, 'bootstrap': False}
We go back to the original code and include these parameters:
Increasing our accuracy rates to 98.30%!
If you open the image tree.png, you should get something like this:
This is the graph representing all of the random forest’s algorithm decisions, including splits, branches, leaves, mean standard errors, number of samples and predicted values for each split. It’s not extremely useful (or time-efficient) to look through each tree split, but it very interesting to look at a few and see if they make sense to you.
The feature importance bar plot is an sklearn option that is a bit more efficient in understanding the algorithm’s findings. First, we define that we would like to see only the features that determine more than 0.5% of our prediction results, and we will also create a list of these features.
The list_of_index will return a list with the arrays of the column numbers that are the most important. Transforming the list correctly proved be a bit challenging due to the format of the output, so I just redefined the list with the corresponding correct columns.
list_of_index = [0,1,2,3,4,24,97,249,280,308]
Next we need to get the columns and column labels:
And finally this is code for the matplotlib bar plot:
This is the resulting graph:
Analysis of the Feature Importance graph: Contrary to popular belief, location (represented by the variable dist) is the second most important effect. Surface area, instead, is by far the most important factor into determining rental prices in Amsterdam. The number of bedrooms, yelp ratings, yelp prices and a few rental agencies also place a significant effect. Living in the specific postcode 1071 (which indeed is a luxurious postcode to have in Amsterdam, considering it is the neighbourhood of many celebrities) can add a significant amount to your rental bill, everything else being equal.
What if we analyse rental price of houses per square meter, instead of absolute prices? By adding the following code to the process immediately before the algorithm:
and running the Random Forest again, we obtain the following results:
With an average accuracy of 98.309%, measuring the price by square meter increases the model’s accuracy rate slightly from the 98.30% we obtained earlier. The feature list also changes:
It still features surface area and location as primary factors in determining the rental price, but decrease their relative importances and increase the relative importance of yelp ratings and prices, while also adding more rental agencies to the list of agencies that can influence (positively or negatively) the rental price of an Amsterdam home, everything else being equal.
We have the model, and we understand how it works. However, we already know the prices of the listed apartments and we have a fairly good idea about whether it is worth the price tag or not. How can this project be useful to someone who already knows the real state market in Amsterdam?
The true beauty of the model lies on its prediction power. While a real state developer may intuitively know the ballpark of how much an apartment should be worth, we now also have a quantitative tool that can analyse a sample of the market and output a precise price that could bring the expertise of a real state landlord or long-time Amsterdam resident to new renters and buyers.
Here’s how to make predictions with the trained model:
The prediction above is for an apartment with 1 bedroom, 45 sq. meters, that has the postcode 1018 and is being rented from the real state agency JLG Real Estate. We can query the yelp_prices and yelp_ratings data frames from the corresponding postcode 1018 to specify them in the prediction.
For this specific apartment, the predicted rental price is $1530, which is extremely close to the real rent price being paid for the apartment ($1538), even though the model has never this data point, and the apartment has not been listed for more than a year.
This project showed me that understanding the real state market in Amsterdam is definitely more complex than simply looking at postcodes and the square meterage of the apartment. Although these two features are the most important, you can easily be paying more money for the same apartment just because you used an specific rental agency, decided to live in a neighbourhood with good restaurants, or wanted to be neighbours with celebrities (ok, that one is not that shocking).
Hopefully, I have been able to shed a bit more light into the inner workings of this market and help a few people predict how much they should be paying for their new homes. Thank you for reading!
Webscraping Algorithm: https://towardsdatascience.com/looking-for-a-house-build-a-web-scraper-to-help-you-5ab25badc83e
Random Forest Visualization: https://towardsdatascience.com/how-to-visualize-a-decision-tree-from-a-random-forest-in-python-using-scikit-learn-38ad2d75f21c
Hyperparameter Tuning: https://towardsdatascience.com/hyperparameter-tuning-the-random-forest-in-python-using-scikit-learn-28d2aa77dd74