Batch Geocoding on the Cheap
By: Jennifer Livingstone
Tags: Tableau Prep, Data, Tableau, Geocoding, Customer Data

Batch Geocoding the Cheap, Manual Way…With a Data Key

Photo by Calvin Hanson on Unsplash

So you have addresses. Thousands, perhaps tens of thousands-perhaps hundreds of thousands of addresses. Customer data, origin/destination data- it doesn’t matter the context. The addresses are effectively meaningless (data-wise) unless they are geocoded. An address is geocoded once it is assigned a latitude and longitude. And once you have a latitude and longitude, you can map the data, conduct spatial analyses, calculate distances, assign them to census tracts- the possibilities are (almost) endless.

But the task of geocoding itself can pose a significant hurdle. If you have a handful of addresses or even a few thousand, you have a lot of options. For these (smaller) quantities, there are many tools and services available for free or for cheap. I won’t address (pun intended) those here, but here are links to articles written by others that cover the options well.

https://researchguides.uoregon.edu/geocoding/free-batch

The exercise becomes decidedly more tricky the more addresses you have. In my particular situation, I had 600,000+ addresses that needed to be geocoded on a regular (quarterly, or ideally monthly) basis. Most of the tools listed above have limits on the numbers of addresses that can be geocoded freely. These limits usually range from 1,000 to 10,000 addresses a day. Even with 10,000 addresses a day, it would take me over 60 days to run through all my addresses. Paid geocoding (where I wouldn’t be constrained by daily limits) for this volume of addresses varies (as does the quality) but the cost is around $1–3k.

In my search for an affordable solution, I ran across a couple of promising leads. One is QGIS’s MMQGIS product. QGIS itself is a great, free, open-source GIS tool. I use it often and highly recommend it. But I ran into limitations with MMQGIS. Although no data limits are specified, I found that the program crashed when I tried to run more than a few thousand addresses at a time. Although it looked like I could run segments of a dataset back to back (and thereby cumulatively geocode more than 10,000 addresses a day), it was a very manual and time-consuming process. Worse, I was getting an extremely high error rate. Less than a third of my addresses were being geocoded successfully. In terms of this error rate, I can’t entirely fault MMQGIS. This dataset in particular had extremely messy address data. I had done some work to clean it, but clearly not enough. If you have clean address data, MMQGIS might be a good option for you. If not, keep reading…

There is another way.

Geocoding with a Data Key

You first need to find yourself a Data Key.

You have Your Dataset with your addresses. But what if you could find a dataset that had your addresses but with spatial information? If you were able to find a dataset like that, all you would have to do is match the addresses in Your Data to the addresses in your Data Key…and then you’d have your spatial information matched to your addresses. Easy, yes? Sort of.

But where would you find a Data Key? A dataset like that sounds like it could be expensive. And could you trust it? How would you know it had all the data you needed?

For many regions, you should be able to find a Data Key for free. Government agencies often keep and share this information. There are also organizations that collect and share address information. I found my particular dataset on Open Addresses. Open Addresses is a project aiming to collect and make publicly, and freely available, all addresses globally. Its coverage for different regions vary. As you can see here, data coverage is better in some countries than others- many countries have complete coverage while other countries have minimal data and many are missing from the list entirely.

Do some research and check to see what Data Key would be right for your project. Map your Data Key. Does it look like it reflects your region accurately? How many addresses are listed? Does this match with what you know about your community independently, from census data and from other sources?

Matching Your Data to the Data Key

Your next step is to match the addresses in Your Data Set to the addresses in Your Data Key. Once you’ve done so, you will have effectively matched spatial data from your Data Key to Your Data Set.

For smaller sets of data and for where minimal address cleaning is needed, this could be done in Excel. You would want to review the format the addresses from your data is in against the format the data key addresses are in. Based on the differences you see, you would then clean up either the addresses in Your Data or the addresses in your Data Key - or both until you had matching data formats. You could then use a simple vlookup to pair the addresses together. If you had addresses from Your Data without matches, you could filter for those and see if you could find new differences in the formats of your addresses and the format of the Data Key addresses. Some of the errors might also come from typos and manual data entry errors.

However, past a certain volume of data and a certain level of required data cleansing, this task is near impossible, or at least very frustrating, to perform in Excel. For my data at least and the amount of cleansing required, it was near impossible.

If your data is fairly clean, you may be able to get away with working in Excel.

If you are accessing your data directly from a database, you could try joining your data to the Data Key there, with some SQL reformatting.

If you’ve successfully matched Your Data to the Data Key and have a decent match rate, your job here is done. You now have latitudes and longitudes and whatever other spatial data was in your Data Key matched to your addresses. Go forth and map!

With very messy data though (as mine was), I would recommend an additional tool for data cleansing…if this is you, continue on to my second post, Using Tableau Prep to Clean Your Address Data.