Mapping messy addresses part 1: getting latitude and longitude | by Mark Ryan | Towards Data Science

My home city of Toronto is blessed with the only major streetcar network in North America that survived the postwar vendetta on street railways. Toronto’s streetcars are an essential part of the overall rapid transit system. However, they have one significant vulnerability — if a streetcar gets blocked it can be impossible for other streetcars to get around it, so streetcar delays have the potential to trigger gridlock. I have been working on a model that uses a publicly available dataset of information on streetcar delays to predict and help prevent such delays.
The streetcar delay dataset includes details about every delay in the system since January, 2014, including the time, duration and location of the delay.
The location field is completely unstructured — junctions are expressed in a variety of formats (“Queen and Sherbourne”, “queen/sherbourne”) and streets and landmarks are indicated inconsistently.
For example, a single landmark can appear in the dataset with multiple, distinct location values:
Roncesvalles Yard
Roncy Yard
Ronc. Carhouse.
I need to visualize the locations of delays to understand which parts of the network are most prone to gridlock. To visualize the locations I need to convert them into latitude and longitude values. In this article I describe how I got latitude and longitude values from the messy locations in the input dataset. In the next article in this series I describe how I used these latitude and longitude values to generate maps to visualize the delay patterns.
Before attempting to get latitude and longitude values I started with some old-fashioned cleanup of the location values, including:
Set all location values to lowercase
Replace common values that were represented with multiple strings, including inconsistent street names, and use a consistent conjunction for all junctions:
Apply a function to give a consistent order to street names in junctions to avoid redundancy like “Queen and Broadview” / “Broadview and Queen”:
These simple clean ups reduced the number of unique location by 35%, from 15.6 k to a little over 10 k. As we’ll see, reducing the number of unique locations means fewer calls to the API to convert locations to longitude and latitude values. The Google Geocoding API cost $5.00 / k calls, so I’ve saved $25 / batch run by reducing the number of unique locations.
I decided to use the Google Geocoding API to get latitude and longitude values. This ended up being a less straightforward process than I anticipated — I hope that if you use the Geocoding API you can benefit from the lessons I learned, as described below.
Here are the steps I had to take before invoking the Geocoding API from Python:
Set up a project in Google Cloud. I followed the instructions here.
Review the Geocoding API introduction material and follow instructions there to (1) activate the Geocoding API for your Google Cloud project and (2) get an API key for the Geocoding API
Review the Python Client for Google Maps Services readme for instructions on how to invoke the Geocoding API from Python
To prepare to invoke the Geocoding API from Python:
Install the client
Install the client
! pip install -U googlemaps
2. Associate your API key with the Geocoding API client and invoke the Geocoding API for a sample address from the cleaned up dataset: “lake shore blvd. and superior st.” Note that the address passed to the Geocoding API includes both the location from the dataset and the city (which is “Toronto” for all locations in the dataset).
3. check the latitude and longitude returned to confirm that it matches the input address:
Now that we have validated the roundtrip from location value to latitude / longitude and back to address, there are a couple of snags to be overcome before we can convert the whole batch of location values.
The Geocoding API choked on some location values, but not the ones I expected. I naively tried testing the API by sending a junk address “asdfasdfjjjj” and got non-empty JSON back:
However, when I tried to convert a batch of locations, it failed on a location value that looked fine: “roncesvalles to neville park”
To get a batch of locations converted reliably, I had to wrap the Geocoding API call in a function that checks if the returned list is empty, and if so, returns placeholder values:
With the get_geocode_result function defined to call the Geocoding API reliably, I was ready to do a batch run to convert the location values. To minimize the calls to the API I defined a new dataframe df_unique containing only the unique location values:
However, when I called the get_geocode_result function to add latitude and longitude values to the df_unique dataframe:
I got the following error message:
Checking the quota page for my project in the Google Cloud console, I could see that my daily limit for invocations of the Geocoding API was only 1,400. That’s why I was getting the OVER_QUERY_LIMIT error when I tried to invoke the API for the df_unique dataframe with over 10k values.
To increase my daily limit for geocoding API calls for this project I had to open a ticket with Google Cloud support asking for my daily limit for the geocoding API to be raised:
With my daily Geocoding API limit raised I was able to invoke the API on the df_unique dataframe without errors. After 1.5 hours (indicating about 110 API calls /minute) I had a dataframe including the latitude and longitude values for all the distinct locations:
Next I created distinct longitude and latitude columns in the df_unique dataframe and then joined the original dataframe with df_unique:
Finally, I have a dataframe containing all the original data plus latitude and longitude values corresponding to the location values:
Here’s a summary of the steps required to get latitude and longitude values to correspond with all the messy locations in the original dataset:
Clean up the original dataset to remove redundant locations and reduce the number of unique locationsSet up Python access to the Google Geocoding API by creating a project in Google Cloud, getting an API key, and setting up the Python Client for Google Maps ServicesCall the Geocoding API with the address (location and city) and parse the returned JSON to get the latitude and longitude. Check for the API returning an empty list, and open a ticket with Google Cloud support to get an increased daily API limit if the number of distinct locations that you want to convert is bigger than the default daily limit.
Clean up the original dataset to remove redundant locations and reduce the number of unique locations
Set up Python access to the Google Geocoding API by creating a project in Google Cloud, getting an API key, and setting up the Python Client for Google Maps Services
Call the Geocoding API with the address (location and city) and parse the returned JSON to get the latitude and longitude. Check for the API returning an empty list, and open a ticket with Google Cloud support to get an increased daily API limit if the number of distinct locations that you want to convert is bigger than the default daily limit.
In the next article in this series I describe how I used these latitude and longitude values to generate maps to visualize the delay patterns from the original dataset.
If you want to try out the code described in this article yourself:
The primary notebook for converting locations to latitude and longitude is here. You will need to get your own API key to run it.
An example input dataframe that you can use for this notebook is here. Note that the location values in this dataframe have already been cleaned up (lowercased, street names in consistent order) as described in The solution part 1: clean up the location values to reduce redundancy section above.