How can we predict house prices with Machine Learning?
By: Marcos Juncken
Tags: Real State, Data Science, Predictions, Machine Learning

Using Python to understand which variables affect the house rent and sales price in São Paulo.

Photo by Kaique Rocha from Pexels

Hello World! I'm learning more about data science and how we can predict values based on other variables. Since I work in a Real State Startup — Loft — I tried to apply some of the concepts and processes in data science to answer some interesting questions about this market!

The first question that I thought about is the main subject of this article — how to predict house prices? Is that possible using the concepts that I've already learned?

But following this question, which variables impact more in rent and in sales? Because probably they are different since the people that choose between these 2 different types of negotiation have different needs.

It's common sense that the region where is the property located affects the price — so, this is our third question, where in São Paulo is cheaper and where is more expensive?

So, these are our 3 questions to answer:

How can we get at least 90% accuracy to predict property price? Rent vs Sale: what is the difference between variables correlation with the price? What neighborhoods in São Paulo have the highest squared meter price? And the lowest?

And to answer those questions we need a good database, surfing in our World Wide Web, I founded this São Paulo properties database on Kaggle that was very useful to me! And was great because São Paulo is the city I live and work in. 😀

But before exploring the questions I will show some challenges that we had on the dataset.

*This project is available on GitHub here.

Preparing the Data

Random image from Word Wide Web

Before trying to analyze the data, I needed to check if everything there was making sense. And I saw three main things that were very important to fix before answering our questions.

To accomplish that I need to use the knowledge that Yoda’s Jedi Master taught me.

First I understood if we had values for all lines of the dataset and a sample of this data. Until here everything was looking good. But when I checked the types of values, I saw that Elevator, Furnished, Swimming Pool, and New were classified as integers instead of boolean — and all of these values were 0 or 1.

The second problem that I founded was when I check the value counts for Latitude and Longitude. A lot of them were 0?!? I'm not the smartest person on geography, but I know that São Paulo is not located in the Gulf of Guinea.

Latitude 0 and Longitude 0

So after that, I plot the 13640 lines of data that we have on the dataset and I faced this image.

Didn't look so good, maybe this is wrong. MAYBE.

So I searched for the maximum latitude and longitude for São Paulo and replaced all the values that were out of São Paulo with the values mean (excluding the out of location values, of course).

And the new plot was making a lot of sense now! After this part, I just created 2 other datasets to separate properties for sale and for rent, because the price between them is very different.

That was very helpful to answer our 3 questions later!

1 — How can we get at least 90% accuracy to predict property price?

The only way that I learned to predict values was with the Linear Regression Model. Before trying this model, I created a dummy variable for each categorical values in our dataset. Remembering that now we have 3 datasets:

df = all properties

df_rent = properties for rent

df_sale = properties for sale

But unfortunately, the results were not satisfactory:

So I needed to surf again on our World Wide Web and I find the Gradient Boost Model — for more information check this article.

And now we achieved our objective, we have an accuracy score of 0.93 — at the data frame that has both for sales and for rental properties.

This is great, and Gradient Boost can understand a lot better than Linear Regression the number of features that have an impact on the price and that the features have a correlation between than.

2 — What neighborhoods in São Paulo have the highest squared meter price? And the lowest?

To prepare the answer to this question we need to understand that the price for itself doesn't mean a lot comparing different regions. The squared meter is a variable that can explain better the differences in property valuation between neighborhoods.

So let's begin to analyze the neighborhoods with the highest square meter price in properties for sale and for rental:

Properties For Rental

Properties For Sale

Here we can already see the differences:

We have 9 districts that are in both lists: Iguatemi, Vila Olimpia, Itaim Bibi, Brooklin, Pinheiros, Vila Madalena, Moema, Campo Belo and Jardim Paulista.

Alto de Pinheiros is just in the 10 highest Squared Meter Price for sale.

Consolação is just in the 10 highest Squared Meter Price for rent.

And the lowest squared meter price:

Properties For Rental

Properties For Sale

Let’s understand this data:

We have 5 districts that are in both lists: Grajaú, Artur Alvim, Itaim Paulista, Lajeado and Guaianazes.

Capão Redondo, Medeiros, Itaquera, São Rafael and Cidade Tiradents are just in the 10 lowest Squared Meter Price for sale.

Ponte Rasa, Jardim Helena, Jaçanã, José Bonifácio and Jardim São Luis are just in the 10 lowest Squared Meter Price for rent.

3— Rent vs Sale: what is the difference between variables correlation with the price?

Now let’s see which variables impact more in price! First I will show the heatmap indicating the correlation of the variables with the price of the properties for rental and for sale.

Properties For Rental

Properties For Sale

At those heatmaps, we can see some difference.

Let’s begin with variables that impact more when the apartment is for sale:

Size, number of rooms, toilets, and parking spots.

And the variables that impact more when the apartment is for rent:

The condo price and if the apartment is furnished.

Bringing that to reality makes sense because the variables with a bigger correlation in the properties for sale are more about long-term characteristics and in the properties for rent is more about short-term characteristics.

Another way to understand that is to measure the coefficient weights in a Linear Regression model that we tried before:

Properties For Rent

Properties For Sale

In this case, we can understand that Geographic Location is the more important characteristic to define the price, which is very true. We can see this difference when we analyzed the Districts with the highest and lowest squared meter price.

Suites and Furnished have a bigger impact on price when we look at the properties for sale.

New and Toilets have a bigger impact on price when we look at the properties for rent.

Thanks for reading! Feedbacks are very welcomed! 😃