Building a dataset for the São Paulo Subway operation
By: Douglas Navarro
Tags: Towards Data Science, Python, Data Science, Web Scraping

For all other lines, the data is grouped in three tags <div class="linhas"> , one for each company that participates in the urban rail system. Inside each one of those divs we have an unordered list tag, where each item contains the line number, its name and current operation status inside another div that looks like <div class="info"> .

In order to extract all of this we will iterate over the <div class="linhas"> divs and then iterate over each <div class="info"> div to get what we really want: operation status and line name:

we will iterate over the <div class="linhas"> divs and then iterate over each <div class="info"> div to get what we really want

>> lines_containers = operation_column .find_all(class_ = "linhas") >> for container in lines_containers:

line_info_divs = container.find_all(class_ = "info")

for div in line_info_divs:

line_title = ''

line_status = ''

spans = div.find_all("span")

line_title = spans[0].text.lower()

line_status = spans[1].text.lower()

extracted_status[line_title] = line_status >> extracted_status

{'safira': 'normal', 'prata': 'normal', 'rubi': 'normal', 'azul': 'normal', 'turquesa': 'normal', 'amarela': 'Operaç\xe3o Normal', 'lilás': 'normal', 'esmeralda': 'normal', 'coral': 'normal', 'verde': 'normal', 'diamante': 'normal', 'jade': 'normal', 'vermelha': 'normal'}

Now we have the operation status for all the lines inside our dictionary! Another crucial information is the timestamp for this data. It is shown in the ‘OPERATION’ panel, right by the title. It is the only <time> element in the page. Lets pull it out of there:

>> time_data = soup.find('time').text

>> time_data

'18/08/2018 23:11'

Finally, we have all the data! I confess I needed a moment to appreciate this.

Web scraping is so cool! Source

At this point we can simply write extracted_status and time_data to a file in our local system or store it basically anywhere using the magic of the internet.

But lets go the extra mile and use Google Sheets as it is quite popular, accessible for non-developers and provided with a well documented API. On top of that, the gspread package wraps all of the requests for us and oauth2client takes care of the token acquisition procedure using our API credentials.

I know there is a lot of hidden stuff going on but being able to do so much with so little code is one of the beauties of programming in a language like Python.

In a nutshell, what we need to do is:

Get our google credentials to use them later for authorization (this is manual and only needs to be done once)

Obtain authorization using the credentials through our code

Write data to a public spreadsheet through our code

To get the credentials, we can follow this step-by-step guide provided by the gspread documentation. By the end of this procedure what you get is a client_secret.json file that looks like this:

{

"private_key_id": "2cd … ba4",

"private_key": "-----BEGIN PRIVATE KEY-----

NrDyLw … jINQh/9

-----END PRIVATE KEY-----

",

"client_email": "473000000000-yoursisdifferent@developer.gserviceaccount.com",

"client_id": "473 … hd.apps.googleusercontent.com",

"type": "service_account"

}

This is important: we need to share the spreadsheet we will be using with our API user. Otherwise we get a denied access error when trying to write the data. This user is represented by the client_email field from the json file.

This is important: we need to share the spreadsheet we will be using with our API user

Back to the code, lets import gspread and oauth2client and load the credentials from the .json file to our environment:

>> import gspread

>> from oauth2client.service_account import ServiceAccountCredentials >> scope = ['https://spreadsheets.google.com/feeds',

'https://www.googleapis.com/auth/drive']

>> creds = ServiceAccountCredentials.from_json_keyfile_name(

'client_secret.json',

scope)

Now lets get authorized using our credentials and initialize the spreadsheet. To initialize the spreadsheet we need its ID, which is shown in the URL we use to access it in the browser:

https://docs.google.com/spreadsheets/d/1WiKE6SUCqAaUF9qOJOg_tMeOSA_lm_kjm83qwXF9dSg

>> client = gspread.authorize(creds) >> SPREADSHEET_ID = "1WiKE6SUCqAaUF9qOJOg_tMeOSA_lm_kjm83qwXF9dSg" >> data_sheet = client.open_by_key(SPREADSHEET_ID).worksheet("data")

Note that we do not write using the spreadsheet object directly, but a worksheet inside it, so it makes sense that we need to specify its name.

Lets write our data! The append_row method does exactly what it sounds like it does, we just need to pass a list, where each element will be written in a different column of that row.

>> for line in all_lines:

data_sheet.append_row([time_data, line, extracted_status[line]])

We can actually see the data being stored in real time!

In order to have an autonomous scraper that does not keep your computer turned on all the time we must organize this code and have it running on a cloud service. Some protection against exceptions is also important, but these are beyond the scope of this article. Feel free to take a look at the github repo in case you want more details!

I’ll be analyzing this data in a future article! We’ll see what comes out of it!

[Edit] An awesome analysis has been made by Paulo! Read it here!