Using Geotab’s Open Datasets— Visualizing Results Using Python and Colab Notebooks
By: Zachary Manesiotis
Tags: Environment, Geotab, Data Science, Matplotlib, Python

Geotab is a world leader in vehicle telematics, processing petabytes of data each day. To give us a glimpse into what we can do they have released some open datasets for us to explore, from hyper-local temperature data to urban infrastructure data that allows us to examine areas of idling and road impediments. Check out their website for more awesome examples.

First let’s register an account and see how we can use Python and various other data science tools to extract visualizations and meaningful results. Once you’re done registering, open up a Google Colab Notebook and let’s get started.

Next, we’ll import some relevant libraries that will enable us to retrieve, process, and visualize our data.

from google.cloud import bigquery

from google.colab import auth import matplotlib.pyplot as plt

import matplotlib.dates as mdates import IPython

import datetime

import numpy as np

import pandas as pd !pip install datalab import datalab.bigquery as bq

%matplotlib inline

plt.rcParams['figure.figsize'] = [12, 10] !pip install geopandas

import geopandas as gpd

import requests

import zipfile

import io !pip install us

import us

!pip install descartes

from descartes.patch import PolygonPatch

That’s a lot of libraries! We’ll see why we need all of them as we go along. Now let’s authenticate so that we can access the data from data.geotab.com.

_ = auth.authenticate_user()

A URL will come up; click on it and give Google the relevant permissions. Then, copy and paste the alphanumeric key and paste it in the white box below. Let’s continue accessing our data:

client = bigquery.Client(project='geotab-intelligence')

job_config = bigquery.QueryJobConfig()

job_config.use_legacy_sql = False

Now all that’s left is to query our data and process it with Python.

For this example, we’ll examine the average idle time throughout New York State, segmented by County. First, let’s get our raw data.

SQL = """ select distinct County, AverageIdleTime

from `geotab-intelligence.UrbanInfrastructure.IdlingAreas`

where State = "New York"

limit 70; """ df = client.query(SQL).to_dataframe()

df.head()

Here’s a glimpse at the dataframe we just accessed.

What exactly does AverageIdleTime mean? The table’s schema says “The average number of minutes that vehicles idle in this geohash (per idling instance).” Clearly, this is a good way of measuring our carbon footprint and environmental impact.

Now, how exactly are we going to segment New York State by County? Well, we’ll make use of shapefiles, specifically from an open-source library called “us.” Since we are using a Google Colab Notebook, we are not going to download any of the files to our local storage; instead, we are going to access the data via a zip file that the library provides a URL to:

url = us.states.NY.shapefile_urls('county') print('Downloading shapefile...')

!wget $url

z = zipfile.ZipFile('/content/tl_2010_36_county10.zip')

print("Done") filenames = [y for y in sorted(z.namelist()) for ending in ['dbf', 'prj', 'shp', 'shx'] if y.endswith(ending)]

print(filenames) !unzip /content/tl_2010_36_county10.zip dbf, prj, shp, shx = [filename for filename in filenames]

NY = gpd.read_file(shp)

print("Shape of the dataframe: {}".format(NY.shape))

print("Projection of dataframe: {}".format(NY.crs))

NY.head()

The resulting table from our preceding code.

Awesome, so we now have our shapefile data for each County of New York State. Now we need to combine both dataframes so that the shapefile data is associated with the AverageIdleTime data for each corresponding County. To do this, I made a helper method:

def countyToIndex(county):

for i in range(len(df['County'])):

if county == df['County'][i]:

return i

else:

print("The index " + str(i) + " does not correspond to the County " + county)

countyToIndex() takes a County Name (as a string) as input, and it returns the corresponding index to the row which contains the specified County in Geotab’s dataframe “df”.

Making use of our helper method, let’s make our final dataframe:

NY['AverageIdleTime'] = np.zeros(len(NY))

for i in range(len(NY)):

NY['AverageIdleTime'][i] = df['AverageIdleTime'][countyToIndex(NY['NAME10'][i])] NY.head()

If we scroll to the right, we can see the newly added column “AverageIdleTime” which corresponds to each County.

Finally, let’s visualize our data using Matplotlib:

fig, ax = plt.subplots(1, 1)

NY.plot(column= NY['AverageIdleTime'], ax=ax, legend=True, cmap='Reds')

Here we can see NY State segmented by County, with the shade of red indicating the amount of average idle time.

There you have it! Try playing around with what metrics you’re measuring. For example, what about road impediments? Temperature?

Geotab provides many datasets which are perfect for beginners and seasoned data scientists alike. Feel free to examine their website for further in-depth examples of how you can use this data and why it matters.

Use your imagination and always consider realistic use-cases. In the end, your graphs can be pretty but if they offer no valuable insights, who cares?