Can we use XGBoost to predict public housing price in Singapore? (Part 2)
By: Cheong Wei Si
Tags: Web Scraping, Selenium, R, Rvest, Docker

Can we use XGBoost to predict public housing price in Singapore? (Part 2)

Step-by-step guide to web-scraping with Rvest & RSelenium

Image Credits: David Kubovsky from Unsplash

Greetings! If you have made it here, I would assume that you may have read Part 1 of this 2-part series where we explored the use of eXtreme Gradient Boosting (XGBoost) to predict public housing (HDB) price in Singapore.

This post provides a step-by-step guide on how we derived the data (used in Part 1) from the property website 99.co using web-scraping techniques.

Web-scraping Ethics

Before we proceed any further, it is always prudent to check first if we are allowed to web-scrape from any website. You can use the robotstxt package in R with the command “paths_allowed( )” to check. If the output is TRUE, it means we are good to go! Do keep in mind that we are not allowed to scrape from sites containing personal data such as Facebook or Instagram.

Let’s begin!

The entire dataset of 7,620 HDB listings used in Part 1 was obtained from the property website 99.co using Rvest & Rselenium packages. If you clicked on the link here, it will bring you to this page as shown below.

Taking the example of the listing 107 Potong Pasir Ave 1 above, we can obtain quite a number of useful property-level data such as the district (D13), the price ($398,000), the flat type (HDB 3A), the year built (1984), the number of bedrooms (2 beds) and the square foot (797 sqft). As you can see from the image above, we can potentially obtain 228 pages worth of data.

The next step is to use SelectorGadget to extract the information you want into R. This handy tool helps to simplify web-scraping and can be downloaded from Chrome for free here.

Suppose we want the property names, we can simply activate SelectorGadget and click on any property name. All the property names (there are 35 of them on this page) will be highlighted in yellow. Then, copy the line: “#nintynine_page_container ._1vzK2” into the code block in R as shown below.

As you can see, with just a few lines of code, we have managed to obtain all the 35 property names in the first page of the HDB listings.

If you look at the URL above in the second line, you might see (in gray color) “page_num=1”. Suppose we wanted to view the HDB listings on the third page, we can simply change the “1” to a “3”. Now, we can exploit this by writing a simple loop that allows us to grab all the details of the HDB listings efficiently for all of the 228 pages.

Using the same idea of how we obtained the property names using SelectorGadget above, we could also obtain the other features such as the price and sqft. The code block below grabs all of the features at one go, assigns NAs if those values are missing and orders the output concisely in a data frame.

Do note that the code below only pulls the first 3 pages. If you want the listings in all of the 228 pages, simply change “c(1:3)” to “c(1:228)”.

Since each page has 35 listings, and I’ve only pulled 3 pages here (as a demo), so we have a dataset of 105 listings. For brevity, the first 10 rows of the data frame is shown below.