Using Python and Selenium to get coordinates from street addresses | by Khalid El Mouloudi | Towards Data Science

This is a step by step article on how to use Python and Selenium to scrape coordinates data (Latitude and Longitude values) from Google Maps based on street addresses.
In this case example, I’m going to work with an official dataset containing the street addresses of all Australian charities and nonprofits. At the end, I’ll proceed to map all the charities and nonprofits in the city of Melbourne using Folium, to demonstrate what can be done with the newly acquired coordinates data.
To follow along, you can download the Jupyter Notebook from my GitHub repository.
We’ll need the Selenium python package to perform the data scraping. If you don’t have it yet, you can install it using pip: pip install selenium .
We’ll also need a WebDriver in order to interact with the browser, so you have to go here and download it to your machine (make sure it’s compatible with your current Chrome version).
Now let’s do the import :
from selenium import webdriver
Let’s also get tqdm, an essential progressbar python package. It is very useful to estimate how much time the web scraping part of your code will take (i’m using tqdm_notebook because I’m working in a Jupyter Notebook) :
from tqdm import tqdm_notebook as tqdmn
In addition to that, we’ll need Pandas to read and manipulate the dataset :
import pandas as pd
Finally, we’ll get Folium to map coordinates data on a map (it can be installed using pip install folium):
import folium
I’ll be working in a Jupyter Notebook (you can download it here). All these steps still apply if you’re working in a python IDE instead.
Australia is ranked 4th worldwide by the CAF World Giving Index in its 10th edition that covers the last 10 years. By visiting the ACNC’s website, you’ll quickly notice how easy they make it for researchers to access all the data about charities and nonprofits in Australia.
The dataset we’ll be using can be downloaded here (along with a useful user notes document explaining the variables ):
ACNC Charity Register dataset [Excel file XLSX | 7.7 Mb | 74155 rows and 60 columns as of 23/01/2020]
The dataset presents many interesting aspects about the charities and nonprofits in Australia: unique identifier, legal name, address (for correspondence), registration date, size, purpose, beneficiaries and more (read the user notes for more info).
Here, we’re mainly interested in the street address features which span several columns: Address_Line_1, Address_Line_2, Address_Line_3, Town_City, State, Postcode and Country. Also, for simplicity, we will only look into charities and nonprofits in the city of Melbourne.
First, let’s read in the dataset we previously downloaded from the ACNC website using Pandas :
Make sure to supply the path where the Excel file resides on your machine. The keep_default_na parameter is set to False so we’ll get empty values instead of NaN when certain values are missing. This will be useful later when we’ll combine all the street address variables into one variable.
Now, let’s make a new dataframe mel as a copy of acnc after filtering it by the variable Town_City to only select the charities and nonprofits in the city of Melbourne :
mel = acnc[acnc.Town_City.str.contains('melbourne', case=False)][['ABN', 'Charity_Legal_Name', 'Address_Line_1', 'Address_Line_2', 'Address_Line_3', 'Town_City', 'State', 'Postcode', 'Country', 'Date_Organisation_Established', 'Charity_Size']].copy()
I did two things here: I filtered the acnc dataframe by the Town_City variable, and then I only selected 11 useful columns out of the 60 we initially had. copy() makes sure we made a proper copy of the filtered acnc dataframe.
I didn’t use acnc[acnc.Town_City == 'Melbourne'] here because I suspected it may have been written in different ways. To make sure this was necessary :
mel.Town_City.value_counts()
As we can see above, the column contained different ways of indicating that a charity is indeed situated in Melbourne, some of which using suburbs or even very specific places in the city like The University of Melbourne. By using acnc.Town_City.str.contains('melbourne', case=False) , we insured all of the above charities are accounted for (otherwise we would only get the 1779 ones correctly labeled).
Let’s see how our new mel dataframe looks like :
mel.head()
And now, let’s add a new column Full_Address containing the full address :
mel['Full_Address'] = mel['Address_Line_1'].str.cat( mel[['Address_Line_2', 'Address_Line_3', 'Town_City']], sep=' ')
str.cat() works here because all of these columns are of type object or string.
Here’s an example of the very first full address in mel :
mel.Full_Address.iloc[0]Output:'G Se 11 431 St Kilda Rd   Melbourne'
One more thing: some of these full addresses only contain the Post Office Box number (mentioned as GPO Box or Po Box). These addresses are completely useless to us because they don’t refer to an existing place. Here’s an example :
mel[mel.Full_Address.str.contains('po box', case=False)].Full_Address.iloc[0]Output:'GPO Box 2307   Melbourne VIC 3001 AUSTRALIA'
We need to remove these records (or rows) before proceeding :
mel = mel[~mel.Full_Address.str.contains('po box', case=False)].copy()
One last thing: some addresses contain the character / which can break any URL. We need to substitute any slash with a space :
mel.Full_Address = mel.Full_Address.str.replace('/', ' ')
Before any web scraping job, it is essential to explore the website you want to extract data from. In our case, it’s Google Maps.
First, let’s study how searching for a full address using the search bar inside Google Maps affects the URL of the result page. For this, I’ll go with the fictitious address Grinch house mount crumpit whoville because I want Google Maps to return no results :
As you can see above, we get www.google.com/maps/search/followed by the address we searched for. In other words, if we want to search for an address XYZ within Google Maps, all we have to do is use the URL www.google.com/maps/search/XYZ, without having to interact with the search bar itself.
The idea here is to generate a new column within mel where we combine www.google.com/maps/search/ with every Full_Address we have in our dataframe mel, and then have Selenium iterate over them visiting the URLs one after the other.
Let’s create that new Url column :
mel['Url'] = ['https://www.google.com/maps/search/' + i for i in mel['Full_Address'] ]
Now that we have a column containing all the URLs we want to crawl, let’s take a look at the address G Se 11 431 St Kilda Rd Melbourne for example :
www.google.com/maps/search/G Se 11 431 St Kilda Rd Melbourne
The above link gives us :
The above address corresponds to the charity Australian Nurses Memorial Centre. Let’s search for it on Google Maps by name :
We get the exact same spot, but not the same coordinates in the URL. That’s because the coordinates in the URL are linked to how the map is centered and not to the marker (they change if you zoom in or out). That’s why we’re going to extract the coordinates directly from the source code of the page itself.
To view the source code, right click on a blank space within the page (outside of the map) and choose View Page Source (CTRL+U or Command+U in Mac). Now search for -37.8 or 144.9 within the source page :
You’ll find the coordinates we are seeking in many places throughout the hot mess that is the source code. But they are mostly useful to us if they are comprised within an HTML tag we can target. Luckily, there is one meta tag we can make useful here :
For now, let’s note that it’s a meta tag with an attribute content containing the URL we want to extract, and an attribute itemprop with the value image which can be used to identify and target this particular meta tag.
Now all we have to do is use Selenium to visit each URL in mel.Url and target this meta tag in order to extract the value of its attribute content.
Here is the code we’ll use to extract the URLs containing the coordinates from Google Maps :
Line 1: we make an empty list called Url_With_Coordinates, which we will fill later on with (you guessed it) the URLs we want to extract ;
Lines 3 to 5 : prefs to run the Webdriver without javascript and images. This way the code will take much less time to load webpages. Obviously, this isn’t a good choice if what you want to extract relies on javascript. By removing 'images':2, 'javascript':2, the web pages will load images and javascript normally ;
Line 7: make sure to specify where you put the chromedriver.exe file in your machine. In my case I put it in the C drive for simplicity. Note that backlashes \ in the path need to be doubled \\ for the path to be recognized ;
Line 9: this for loop is iterating over the mel.Url series. The tqdmn() wrapping our iterable adds a progress bar right after the cell is executed. Its parameter leave=False makes sure the bar goes away once the operation is finished ;
Line 10: for each URL in mel.Url, the webdriver opens that URL (for the first URL you’ll see a chrome window open, and then you’ll notice it going from URL to URL until mel.Url is finished) ;
Line 11: first, we search for our meta tag using driver.find_element_by_css_selector and we identify the tag by meta[itemprop=image]. After that, we extract the value of the attribute content using .get_attribute('content'). The result of this operation (the URL containing the coordinates) is then added to the Url_With_Coordinates list using append().
Line 13: we close the webdriver (chrome window) after the script is finished (this is a good practice).
Here’s the script and the tqdm progress bar in action (or tqdmn because I’m using the tqdm_notebook submodule) :
NB 1: the next time you run the notebook, you don’t have to rerun the web scraping code all over again because we have saved the result in a CSV file called Url_With_Coordinates.csv. Let's read in that file instead :
import csvwith open('Url_With_Coordinates.csv', 'r') as f:    reader = csv.reader(f, delimiter=',')    for i in reader:        Url_With_Coordinates = i        break
NB 2: in your tests, you don’t want to make the for loop iterate over thousands of addresses only for it to throw you an error at the end. You need to test it first on a couple of values before properly executing the script. In our case, the test code will be like this to only go over the first 10 values of mel.Url :
for url in tqdmn(mel.Url[:10], leave=False):    driver.get(url)    ......
Now let’s see what the Url_With_Coordinates list looks like :
Let’s add this list as a column in our mel dataframe :
mel['Url_With_Coordinates'] = Url_With_Coordinates
But how are we supposed to extract just the coordinates from these URLs? Here’s a visual explanation on how to use Python’s split() method for this purpose :
Which translates in code to the following (this code won’t run because url is not defined, it’s just to show how the above solution would work):
url.split('?center=')[1].split('&zoom=')[0].split('%2C')Output:[-37.8386737, 144.97706]
Now using the code above, we’re going to add two new columns to our mel dataframe: lat for latitudes and long for longitudes :
mel['lat'] = [ url.split('?center=')[1].split('&zoom=')[0].split('%2C')[0] for url in mel['Url_With_Coordinates'] ]mel['long'] = [url.split('?center=')[1].split('&zoom=')[0].split('%2C')[1] for url in mel['Url_With_Coordinates'] ]
Most likely, the above code gave you an error list index out of range :
What this error means is that the split() method didn't work as intended with some URLs in the column Url_With_Coordinates. Perhaps some URLs didn't have the keywords we used for the split() method. Let's look for the URLs that lack &zoom= for example :
mel[~mel.Url_With_Coordinates.str.contains('&zoom=')]
As we can see here, we have 5 instances where the extracted URL starts with //www.gstatic.com/images ... (hence the error we got) :
list(mel[~mel.Url_With_Coordinates.str.contains('&zoom=')].Url_With_Coordinates)Output:['//www.gstatic.com/images/branding/product/1x/maps_round_512dp.png', '//www.gstatic.com/images/branding/product/1x/maps_round_512dp.png', '//www.gstatic.com/images/branding/product/1x/maps_round_512dp.png', '//www.gstatic.com/images/branding/product/1x/maps_round_512dp.png', '//www.gstatic.com/images/branding/product/1x/maps_round_512dp.png']
For simplicity, and because 5 isn’t a big number, we’ll remove these instances from mel :
mel = mel[mel.Url_With_Coordinates.str.contains('&zoom=')].copy()
Now, let’s rerun the code that adds two columns lat and long to our mel dataframe :
mel['lat'] = [ url.split('?center=')[1].split('&zoom=')[0].split('%2C')[0] for url in mel['Url_With_Coordinates'] ]mel['long'] = [url.split('?center=')[1].split('&zoom=')[0].split('%2C')[1] for url in mel['Url_With_Coordinates'] ]
It worked! Here’s how our mel dataframe looks like, where each charity or nonprofit gets it latitude and longitude values (some columns here are masked) :
mel.head()
Let’s map these coordinates to see how accurate they can be.
Color coding (charity size depends on annual income) :
Red: Large charities (1 Million AUD or more) ;
Purple: Medium charities (between 250.000 AUD and 1 Million AUD) ;
Orange: Small charities (less than 250.000 AUD);
Grey: no data.
Here’s the code we’ll use to map these coordinates :
I’m not going into great detail on how to use Folium, but I’ll just clarify some points here :
I used CartoDB positron for this map because it provides very low contrast against the colored markers (it makes them more visible). Using the default tiles OpenStreetMap makes it difficult to see the markers ;
I changed the size of the marker based on the size of the charity using the parameter icon_size=(..,..) of folium.CustomIcon. The reason behind this is to prevent charities that reside in the same building from masking each other. Because Large markers are drawn first, smaller markers are drawn on top of them so that even overlapping charities can still be distinguished ;
I used custom markers (hosted on imgur) because the default one tends to slow navigation down a lot since we have around 2000 markers on the map. For custom markers, you can provide a URL to the image you want to use, or the path to the image file on your machine ;
If you click on a marker, it gives you the name of the charity and its address, so you can verify if the positioning is correct ;
In the code above, I could have used just mel_map instead of mel_map.save('mel_map.html') followed by IFrame(src='mel_map.html', width='100%', height=500), but when the number of markers is big, it's better to save the map as an HTML file and then open it using IFrame() (otherwise you'll get a blank map).
This is a very legitimate question. Obviously, the best way to get coordinates out of street addresses is from a reputable API like Google Maps’s or Bing’s, but these options can cost money.
The accuracy of this method is heavily influenced by the precision and correctness of the street addresses provided. For instance, in our example above, you’ll notice one marker thrown away in the middle of the Indian Ocean. Upon examination, the address 65 Macarae Road Melbourne is actually supposed to be 65 Mcrae Road Melbourne, hence the error.
To empirically test our method, we are going to use a dataset containing both the street addresses and the coordinates of thousands of businesses in Washington DC. We’ll proceed to take a random sample of 500 business addresses and then we’ll use our method to generate coordinates out of them. After that, we’ll compare them to the actual coordinates listed in the dataset. The dataset we’ll use for our test can be downloaded here :
Basic Business License in Last 30 Days: opendata.dc.gov dataset, 6418 records (as of 26/01/2020), 2.64 Mb
And the results are in (the test details are available in the Jupyter Notebook) :
As we can see above, the generated coordinates are very close to the actual coordinates in this random sample of 500 businesses in Washington DC. I made the orange marker smaller than the red one so they’ll both be visible when they overlap perfectly (that’s why it looks like the orange markers have a red border when zooming out).