What’s the average rental price of a 2-bed in Dubai these days? (PART 2)
By: Silviu Matei
Tags: Dubai Real Estate, Python, Pandas, Beautifulsoup, Web Scraping Series

In more detail, these are the operations defined in the rentals function:

1. Declare the results dataframe: results = pd.DataFrame()

2. Send requests to the Property Finder website, 10 at a time:

rs = (grequests.get(url) for url in alist)

responses = grequests.imap(rs, size = 10)

3. Create a for loop to go through each batch of responses, parse the content with Beautifulsoup and find all the tags identifying the listed properties:

for response in responses:



soup = BeautifulSoup(response.text, 'lxml')



div_tag = soup.find_all('div', {'class':'card-list__item'})

In order to identify the property tag, we go on the Propertyfinder page -> right-click -> Inspect -> Click on the first button on the left side of the down tab which opens, and then we select the property tab. We copy the class of the div tag, namely ‘card-list__item’, and then paste it in the script. You can check, there are 25 such card-list items.

4. Next, we iterate through each of these cards to extract the data from all of them. A very important aspect that took me one day to figure out is how to deal with missing data. My solution was to use a try-except scheme for each variable extracted and to assign the None value in case the variable tag does not exist or is empty.

For instance, the script is going to look for the h2 tag with of class card__title card__title-link. If the tag does exist and is not empty, then, the script extracts the text attribute and deletes any extra empty spaces with the strip() function. If the tag does not exist, then it assigns value None to the title variable. The process is similar for all the other data points.

for div in div_tag:



try:

title = div.find('h2', {'class':'card__title card__title-link'}).text.strip()

except:

title = None



try:

ttype = div.find('p', {'class':'card__property-amenity card__property-amenity--property-type'}).text.strip()

except:

ttype = None



try:

bedrooms = div.find('p', {'class':'card__property-amenity card__property-amenity--bedrooms'}).text.strip()

except:

bedrooms = None



try:

bathrooms = div.find('p', {'class':'card__property-amenity--bathrooms'}).text.strip()

except:

bathrooms = None



try:

area = div.find('p', {'class':'card__property-amenity card__property-amenity--area'}).text.strip()

except:

area = None



try:

price = div.find('span', {'class':'card__price-value'}).text.strip()

except:

price = None



try:

frequency = div.find('p', {'class':'card__property-amenity--bathrooms'}).text.strip()

except:

frequency = None



try:

location = div.find('span', {'class':'card__location-text'}).text.strip()

except:

location = None



try:

link = div.find('a', {'class':'card card--clickable'})['href']

link = 'www.propertyfinder.ae' + link

except:

link = None

A special mention for the last variable, which is the direct link to the listed property page. Note that we extract the ‘href’ attribute rather than the text and, because we may want to use these links in order to extract some more data present on the individual-property pages, we concatenate the relative link of each property with the base website address, www.propertyfinder.ae.

5. The next step is to declare a temporary dataframe, temp_df, to store the output from each iteration by specifying the variables to store and names of the columns in the dataframe. Finally, we append the output of each iteration to the dataframe, called results.

temp_df = pd.DataFrame([[title, ttype, bedrooms, bathrooms, area, price, location, link]], columns=['title', 'type', 'bedrooms', 'bathrooms', 'area', 'price', 'location', 'link'])

results = results.append(temp_df, sort=False).reset_index(drop=True)

6. Scraping can be tricky. Different websites have different limitations in terms of the number of requests they accept. It will be your job to find the best way to deal with it. For the purpose of this exercise, I am using a lag of 2–3 seconds between requests, by using a randomly generated series of 2 and 3 seconds and telling the system to add that lag between iterations. The important thing to remember is that if you notice that your requests do not pass through it may be because the website blocked you for making too many requests in a very short period of time.

In this script, one can try and reduce the number of pages you ask every time — for instance, change 10 to 5 in the line above: responses = grequests.imap(rs, size = 5). In addition, you may want to increase the time between requests i.e. change the below to something like sleep(randint(4,12)) or bigger numbers. However, this will increase the time needed for the scraping.