“ImmoScan” — Unravelling the real estate market
Tags: Real Estate, Web App Development, Data Science

1. Introduction

Nowadays, real estate plays a major role in individuals long-term investment decisions. Nonetheless, it’s still a market that exhibits significant information asymmetry between potential buyers and sellers. Especially the buy-side finds theirselve in decision situations of overwhelming complexity: hundreds of influence factors, which are merely all accessible, need to be considered in order to choose a suitable investment object. In addition to that, lacking trust towards mostly unknown real estate agents increases uncertainty. This is where ImmoScan comes into play: by increasing the transparency within this complex market, it aims to make investment decisions easier and more suitable for individual investors.

2. Methodology/ Project Parts

a. Web Scraper

As one part of our project was to predict apartment prices in different cities with an AI tabular learner, we needed data. Lots of data. The biggest real estate platform Immobilienscout24 offered a good start for this necessity. It had the most listings (i.e. up to 300 pages of apartments in Berlin) and a structured and accessible HTML code to extract all the key values like space, location or number of rooms for each apartment. For the scraper itself, we used the Beautiful Soup library to access the website source code and pandas to structure and clean up the resulting data into a large data frame with almost 8,000 apartments in all German cities with more than 200,000 inhabitants ready to be used as an AI training set.

b. Broad and granular filter

To get a full picture of the real estate market and simultaneously keep complexity at a user-acceptable level, we decided to split the analysis into a broad and a granular filter. The broad filter aims to show the overall attractivity on municipality level, including several relevant information like population development and available income. The aim of the granular filter is to go more into detail on a city level. This includes many other information like number of schools or share of publicly subsidized rental apartments.

According to the motto “start small grow big” we initially kept the scope small and agreed on NRW for the broad and Cologne for the granular filter. As the process of the analysis can be standardized, scaling up to other cities or regions is, assuming data availability, possible. We got the data as csv and json files from different open data providers (NRW, Cologne), which then had to be cleaned, filtered and merged. Therefore, a wide range of data processing commands were used, e.g. reading files, dropping columns or reindexing. Some parts of the data needed to be transformed using arithmetical operations, so that also defining functions and loops were used frequently. Most of the work was done with the packages pandas and numpy. Finally, the outcome of separate dataframes for NRW and Cologne including the most important information were obtained, which then could be used further to develop a scoring model that allows to calculate the attractiveness of each region or district. To do so, the information was 1) split into 4 categories (demographics, economics, infrastructure, housing), 2) normalized to make values comparable and 3) computed to an equally-weighted score for each category. These scores, together with the individual weights the user assigns for each category, then lead to the attractiveness score of the region or district.

c. AI model

The AI model we trained on the Immobilienscout24 data set was built on the fastai library for tabular data learning. Fastai is a beginner friendly library for training neural nets based on the PyTorch library with modern best practices as a default configuration. It is also used as the key example in the TechLabs AI Track to understand the basics of different AI applications like computer vision, language processing and tabular data analysis. This enabled us to train a model on our data set without having to configurate everything the hard way to increase our training success. After some fine tuning of the layer size and the dropout rate, we were able to obtain two models based on the same data set with a randomly chosen validation set. The models were able to predict the real estate prices with a Root Mean Square Percent Error of 18.7% and 22.6% and were averaged to put out a single prediction. While those deviations do not offer a serious deployment of the models in a commercial environment, it is still impressive to see it work, as the data set was between ten and hundred times smaller than the typical data set size to obtain satisfying predictions. A proof-of-concept was therefore demonstrated. In the context of real estate platforms only showing negotiation prices, the predictions might be a second orientation on where to start the price negotiations. As a final result, it is possible for the user to insert a link to an apartment on Immobilienscout24 to get a price prediction from our model.

d. Visualization

It was important to us to have an interactive and easy-to-understand surface for our data. Therefore, we chose Tableau as visualization tool to create a dashboard for ImmoScan. This tool enables the user to click through different maps and lists, drill-down and filter the dashboard according to his needs and even set his individual weights for the categories influencing the attractiveness score based on his preferences. Following you can find an example of the layout for the development of a worksheet:

These worksheets can then be combined in a dashboard to extend the functionalities. Within the dashboard, all relevant data is included: it connects the data from the Web Scraper which is then processed in the AI model together with data from the broad and the granular filter that are used to compute the attractiveness scores:

3. Conclusion

In the end, we were able to design a tool which is technically able to tackle the issues mentioned in the introduction: the user gets a quick overview of the attractiveness of various regions and is able to evaluate if a specific real estate display is fairly priced. We even managed to design the dashboard more interactively than initially planned. However, it has to be mentioned that the powerfulness as well as the usefulness of our tool strongly depends on the data that is given to it as an input. For example, it was not possible to obtain all the data necessary to fully evaluate the attractiveness of a location, simply because it is not provided. Nevertheless, ImmoScan offers a solid framework for above mentioned challenges, which can be enhanced quickly to an extremely powerful tool once it is provided a wider input base.

GitHub-Link: https://github.com/techlabsms/ms-st-20-05-ImmoScan