Scraping Live Traffic Data in 3 Lines of Code, Step-by-Step | by Nathan Thomas | Towards Data Science

In light of the recent pandemic, lockdown measures imposed by policymakers have halted the majority of economic activity. Due to the lagging nature of economic data, it’s not always easy to gauge in the data how consumers are behaving in real-time. In fact, data often has at least a one-month time lag, making it difficult to assess if people are actually moving around or not. Therefore, governments, scientists and economists alike have been looking at high-frequency data, which provides a real-time view of how movement in the global economy has changed since restrictions have been imposed. A popular gauge of how activity has fared has been traffic data, provided by TomTom. This data covers 416 cities across 57 countries and calculates travel times of all vehicles on the entire road network. On TomTom’s website, the average congestion level is shown as a live traffic chart. I’ve chosen to look at Milan:
We’re able to hover over the chart to get each individual data point, but there’s no way of downloading the data from the website. So there are two options:
Spend hours hovering over each point and recording each value...Scrape it!
Spend hours hovering over each point and recording each value...
Scrape it!
Inspecting the data
On any website, it’s common for the site to load in a range of different information when loading the page. This may include cookies, data files, images and more. We are able to see what’s under the hood by inspecting the page. In Chrome, this is made easy by pressing ‘Ctrl+shift+I’. A window will come up which will show all the data that has been loaded into the page. The tab that we want is the Network tab, shown at the top of the pop-up window.
If you can’t see all the data, hit ‘Ctrl+R’ to reload the page and watch as all the data loads in. On the Name tab, you’ll see many different file types: .js, .json, .svg, .otf and more. So how do we know which one contains the underlying data for our chart? Firstly, we need to filter by XHR. This is short for XML HTTP request. It’s common for data on a webpage to be loaded using this format. We can then filter by size — underlying data is usually one of the largest files contained in the page (likely to be in kilobytes rather than bytes). Finally, more often than not, source data will come in the form of a .json file. This stands for Javascript Object Notation, and it’s a common way for large amounts of data to be stored.
However, looking at our dataset, it’s not immediately obvious where the json file is. As a next step, let’s look at the type column. Since we’ve filtered by xhr, we should only see xhr types in this column. Having said that, when we look down the column, we can see both xhr and fetch. The fetch type implies that this data is being loaded from an API, or Automated Processing Interface. Looking at the data with the fetch type, the filename for this specific object is “ITA_milan”. Looks like we’ve got our data! We can check it’s correct by opening it in a new tab. When we open the new tab, we see that this indeed the data we want, under the link: https://api.midway.tomtom.com/ranking/live/ITA_milan, which is being retrieved from an API. Great — now let’s load this data into python and start cleaning it to replicate the live chart.
Retrieving and cleaning the data
First of all, requests is the module that we’ll use to load the data. Using requests.get() on the url we found earlier, we can show this file as a json and unpack it using the json module in python (yes, using only three lines of code):
%matplotlib inlineimport matplotlib.pyplot as pltimport pandas as pdimport jsonimport requests# retrieve json fileurl_ = "https://api.midway.tomtom.com/ranking/live/ITA_milan"italy_req = requests.get(url)italy_json = italy_req.json()
This gives us the following output:
{'data': [{'JamsDelay': 13.5,'TrafficIndexLive': 6,'TrafficIndexHistoric': 17,'UpdateTime': 1586608290000,'JamsLength': 1.4,'JamsCount': 4},{'JamsDelay': 4.2,'TrafficIndexLive': 6,'TrafficIndexHistoric': 17,'UpdateTime': 1586609190000,'JamsLength': 0.4,'JamsCount': 2},{'JamsDelay': 5.8,'TrafficIndexLive': 6,'TrafficIndexHistoric': 18,'UpdateTime': 1586610090000,'JamsLength': 0.4,'JamsCount': 2},
This file may look slightly confusing, with a lot of different punctuation and parentheses. It’s simpler than it looks — the json file is actually a dictionary called ‘data’, with a list of dictionaries contained within it. So to get to the information that we want; (“Time” and “TrafficIndexLive”), we have to call the keys for each point we want to retrieve. For example, to get the first value of the live traffic index, we would enter the following:
italy_json["data"][0]["TrafficIndexLive"]
We can then append each item in this dictionary to a list, converting each list to a column in a pandas DataFrame:
Using dictionaries and a little bit of data cleaning, and we’ve got it! Let’s validate that the data matches the live chart by plotting it:
And there we have it! In a few lines of code, we were able to retrieve data from a chart that we couldn’t have otherwise accessed. Turns out what they say is true: If you can see it, you can scrape it!
Thanks for reading! Please feel free to leave any comments for any insights you may have. The full Jupyter Notebook which contains the source code I used to do this project can be found on my Github repository.
References:
[1] TomTom, 2020. “Milan traffic report”. Available at: https://www.tomtom.com/en_gb/traffic-index/milan-traffic/
Disclaimer: All views expressed in this article are my own, and are not in any way associated with Vanguard or any other financial entity. I am not a trader and am not making any money from the methods used in this article. This is not financial advice.