3.6 million points, 1 GIF — Visualise big data in Python | by LeAnne Chan | Towards Data Science

Hello! This article will walk you through step-by-step to creating a GIF to visualise big data in Python. GIFs are a great way to show changes over time, especially with large datasets. This article uses the New York City construction permits from the past 30 years, published by the Department of Buildings, taken from NYC Open Data. This GIF was created as part of a project I did with my good friend Yap Yun Qi on Gentrification in NYC in Fall 2019.
The IPython Notebook and environment file can be found here: https://github.com/LeanneChan/create-a-GIF. To download the notebook and open it with Jupyter Notebook, you can follow my other article here. You may also run the code in GoogleColab, but based on past experiences, it takes extra steps to load the geopandas library in GoogleColab.
A GIF is a collection of images stitched together. Hence our approach to creating the GIF is to create separate plots for each different time frame (years in this case), then stitch them together. If our dataset was small enough, we could use matplotlib to plot the points. But, with larger datasets, we will use Datashader, a library for visualising large datasets. To work with the large dataset, we will use Dask, a library for flexible parallel computing in Python.
Datashader is part of the HoloViz family (other packages include HoloViews, hvPlot), and is used to create representations of large datasets. It is able to visualise large dataset by rasterising (breaking a plot into pixels) the plotting area and aggregating the value of the points that fall in each pixel. The pixels are then shaded according to this aggregated value.
Dask allows us to store data that is large than the memory on our laptops. Furthermore, Dask is not too intimidating to use as Dask DataFrames mirror Pandas and Dask Arrays mirror NumPy. Dask splits up the partitions that are Pandas DataFrame objects, and loads the data into our computer’s memory only when necessary.
dask, datashader, colorcet, matplotlib, numpy, imageio, geopandas
The dataset has shape (3616003, 60), meaning there are 60 columns/features and 3.6 million rows/points. This is a geospatial dataset and visualisation, so each row represents a point in NYC, which we want to plot. Since there are 3.6M rows, we will need the dask.dataframe library. import is used to load in libraries into your current workspace/session, which allows us to use the methods of the library.
import dask.dataframe as dd
If you’ve used pandas before, the syntax is really similar, calling read_csv to read in the dataframe, which I downloaded from NYC open data and saved in the same folder as my Jupyter Notebook as ‘DOB_permit_Issuance.csv’.
permits = dd.read_csv(‘./data/DOB_Permit_Issuance.csv’) permits.head()
However, if you get the following error, you will also need to address columns with mixed types.
In this example, I had to specify the dtype for certain columns as they had mixed types, and this threw an error when trying to call the dataframe. The dtype specifies the format of data in the column, so there is an error when pandas/dask had to guess too many column types. I also selected only three columns using usecols rather than working with 60 columns, to make future processes faster. These columns indicated the date the permit was filed (to later extract the year from), and the lat/lng of that permit. Another way around the ‘mixed dtypes’ problem is to just select only useful columns which, fingers-crossed, do not have mixed dtypes. On hindsight, this is something I could have done instead to save myself much time!
permits = dd.read_csv('./data/DOB_Permit_Issuance.csv',                     dtype={                         'Lot': 'int64',                         'Block': 'int64',                         "Owner's Phone # ": 'float64',                         "Owner’s House Zip Code": 'float64',                         "Permittee's License #": 'float64',                         "Zip Code":'int64'                      },                     usecols=['Filing Date', 'LATITUDE', 'LONGITUDE'])
Calling permits.head() show the first 5 rows of our dataframe. (Fyi, calling dataframe.head(x) where x is any integer , will print out the first x rows of your dataframe!)
First, we want to extract the year of each permit. I used the to_datetime(), a pandas function, to convert the ‘filing date’ column to date-time object. Since the column is now a datetime object, I can use another the function .dt.strftime(“%Y”) to access the years of each point to create a new column.
# change the column typepermits['Filing Date'] = dd.to_datetime(permits['Filing Date'])# create new column for the yearpermits=permits.assign(year= permits['Filing Date'].dt.strftime("%Y"))
Secondly, we need to decide on a standardised projection for our coordinates. This is a very important point in geospatial data science, because if your basemap is in a different projection and coordinate system from your points, chaos will ensue. A Coordinate Reference System (CRS), is a code (epsg code) that refers to how spatial data (inherently round because flat earth is a hoax) is projected onto a flat 2D screen. It also specifies the units of measurement, which are usually feet, meters or decimal degrees.
For this example, I used a Web-Mercator projection (epsg: 3857), a meter based projection. Though not done in this examlpe, this projection is especially useful if you want to use basemaps from OpenStreetMap or GoogleMaps, which are in epsg:3857 projection. Datashader has a useful function lnglat_to_meters that converts longitudes and latitudes (epsg:4326, decimal degrees) to web-mercator (meters). It is super important to ensure that you know which is your lat and lng after conversion, and that you code them in the correct order required in each function.
# conversion to web mercator from datashader.utils import lnglat_to_meters# create new columns 'x' (longitude) and 'y' (latitude)permits['x'], permits['y'] = lnglat_to_meters(permits['LONGITUDE'], permits['LATITUDE'])# keep a clean dataframe permits = permits.drop(['LONGITUDE', 'LATITUDE'], axis=1)
We follow some Dask best practices, and set the index to be the column we want to subset by, the year. Even though using set_index takes about 8 minutes, this makes subsetting later a looot faster (0.001 seconds!).
# set index so that it is arranged by year # drop na, if not will not be able to set index permits = permits.dropna()# dataframe will be arranged by yearpermits_indexed = permits.set_index('year')#  set_index takes about 8-9 minutes 
Partitioning: Dask operates by splitting the large dataset into partitions. One of the best practices mentioned is to set your partitions by the subsets you are interested in (years in our example).
# create list of strings of yearsdateRange = list(range(1989,2020))years = []for year in dateRange:     str_year = str(year)    years.append(str_year)# repartition the data - takes 0.001 secondspermits_repartitioned = permits_indexed.repartition(divisions=years)
Compute: .compute() executes all the above processes on our Dask dataframe and returns an ordinary dataframe. Doing so will significantly increase the time needed to create our GIF. If compute() is not called here, all the above data wrangling will be recomputed on each call of the dataframe, taking each frame 10minutes. That’s 300 minutes for 30 frames!
permits_computed = permits_repartitioned.compute()
This takes 9 minutes, but as mentioned above, saves us much more time later on.
From the example GIF at the start, the plots included an outline of New York City’s boroughs — Manhattan, Brooklyn, Queens, The Bronx and Staten Island. I downloaded the shapefile ‘borough boundaries’ from NYC OpenData. We will need the geopandas library to read in the shapefile. Don’t forget to set the CRS to the same as what you are using for data points in step 2.
# add NYC borough boundaries (coords must be in 3857)import geopandas as gpdNYCBoroughs= gpd.read_file("./data/borough boundaries")NYCBoroughs = NYCBoroughs.to_crs({'init': 'epsg:3857'})
As we wish for the plot to fit perfectly around the city as well, we need to set the range of x and y coordinates on the plot to be the NYC limits, which I got from dropping points in GoogleMaps. Once again, not forgetting to convert the coordinates to epsg: 3857. As for plot dimensions, I would suggest playing around with those figures and plotting the canvas (see next section) to see what works best for you.
# NYC limits - lng(east/west) and lat(north/south)NYC        = (( -74.25,  -73.7), (40.50, 40.92)) x_range, y_range = [list(r) for r in lnglat_to_meters(NYC[0], NYC[1])]# plot dimensionsplot_width  = int(750)plot_height = int(plot_width//1.2)
At this point, we have all the pieces needed to make the GIF, we just need to put them together! First, we need to import a few libraries to help us with plotting. As mentioned in ‘Approach’, we will be using datashader to aggregate the points before plotting. We don’t just want points on a map, but for the map to show us where there is a higher frequency of points. Datashader will help us with colouring the points based on this.
# plotting functionsfrom colorcet import fire import datashader as dsimport datashader.transfer_functions as tf
Let’s create a helper function to plot the construction permits that we can use later on in plotting different points for each year.
The function takes in the parameters: df — data,x_range , y_range— the range of x/y coordinates on the base (see step 3), w/h — width and heigh of plot (see step 3), cmap — the colour map used to colour the points.
def create_image(df, x_range, y_range, w=plot_width, h=plot_height, cmap=fire):        # create the canvas    cvs = ds.Canvas(plot_width=w, plot_height=h, x_range=x_range, y_range=y_range)        # the construction permits, aggregated by their count    agg = cvs.points(df, 'x', 'y')        # shade the pixels     img = tf.shade(agg, cmap=cmap, how='eq_hist')        # return a PIL image    return tf.set_background(img, "black").to_pil()
The function uses datashader’s canvas function to represent the space in which we want to plot the points. First, we create the canvas, specifying the dimensions. Next, we feed in the data using cvs.points() which takes in the data, ‘x’ — the converted longitude, and ‘y’ — the converted latitude. cvsis the canvas object created in the first line.
Aggregating dataAnother parameter you can specify in cvs.points() is agg which will inform datashader how to aggregate your data. Suppose you have a third column called ‘z’, if you wanted to plot the average of z in that aggregated space, you can include agg=ds.mean('z'). If not specified, the default of aggis ‘count’, which is what we want in this example, thus it is left unspecified.
Finally, the datashader.transfer_functions.shade specifies how to colour each pixel. The colormap we use is fire, from the colorcet library. how specifies the scale of the colormap. Some options are: ‘eq_hist’ [default], ‘cbrt’ (cube root), ‘log’ (logarithmic), and ‘linear’. For more information, see the datashader documentation linked at the end. Then we specify the background to be black, and convert the image to the Python Imaging Library (PIL) format.
We are almost done! We don’t want to be manually creating new images for each time frame, so we package this in another function. For this function, we will need matplotlib’s pyplot to set the axes, and numpy to transform the figure values into range 0–255 (dtype=uint8) to create the final image.
from matplotlib import pyplot as pltimport numpy as np 
The function takes in the parameters: fig— from pyplot,all_data — our dataframe,year — the specific year to be plotted,city_limits — the shapefile of boundaries to be plotted,x_range,y_range — the range of x/y coordinates on the base (see step 3).
def plot_permits_by_year(fig, all_data, year, city_limits, x_range, y_range):    # trim to the specific year    df_this_year = all_data.loc[year]    # create the datashaded image    img = create_image(df_this_year, x_range, y_range)    # plot the image on a matplotlib axes    plt.clf() # this clears current figure    ax = fig.gca() # this gets the axes from fig    ax.imshow(img, extent=[x_range[0], x_range[1], y_range[0], y_range[1]]) # show the image on the axes    ax.set_axis_off()         # plot the city limits (NYC borough boundaries)    city_limits.plot(ax=ax, facecolor="none", edgecolor="white")    # add a text label for the year    ax.text(        0.0,        0.9,        "Yearly Construction Permits\nFiled in NYC",        color="white",        fontsize=20,        ha="left",        transform=ax.transAxes,    )    ax.text(        0.7,        0.1,        year,        color="white",        fontsize=40,        ha="left",        transform=ax.transAxes,    )    # draw the figure and return the image    fig.canvas.draw()    image = np.frombuffer(fig.canvas.tostring_rgb(), dtype="uint8")    image = image.reshape(fig.canvas.get_width_height()[::-1] + (3,))    return image
Using loc to subset our data takes just 0.001 seconds.
The function also specifies text to be shown on each frame, done so using the matplotlib function ax.text() where ax refers to the image’s axes.
We’re right at the end, we just need to use theimageio library to convert the array of images into a gif. To create the GIF, we create images using the function in step 5, append them to an array, then stitch them into a GIF using image.io.
# library to create GIF import imageio
Run the following block code to create the full GIF! If you’re not satisfied with the design, change the parameters (e.g. location/colour/size of labels) in the function in step 5. (see matplotlib documentation)Note: the ‘years’ array was created in step 3 during the partitioning.
# create a figurefig, ax = plt.subplots(figsize=(10,10), facecolor='black')# Create an image for each hourimgs = []for year in years:    print('processing ' + year + ' ...')    img = plot_permits_by_year(fig, permits_computed, year, NYCBoroughs, x_range=x_range, y_range=y_range)    imgs.append(img)      # Combining the images for each hour into a single GIFimageio.mimsave('1989_2019_permits.gif', imgs, fps=1)
This block of code that creates 30 frames took just 9.56463 seconds to run!
That’s it! Thanks for following this and let me know if I can clarify anything!
Past edits and additional information about time complexity30 May 2020 — Changed process to make subsetting faster and called .compute() to reduce Dask operations. (Current article reflects correct code.)
In the initial process (code no longer shown), I skipped step 3 and subsetted the dataframe inside the plot_by_year function. As a result, printing the frame for each year took about 10 minutes. This is also because the data wrangling on the dataset was repeated each time the dataframe was called because of the nature of Dask. My hunch is processes of converting the coordinates into mercator, transforming the ‘Filing Date’ column and subsetting the dataframe had to be done again in each call of the plot_by_year function. Calling compute() on the Dask dataframe resolved this issue of repetition.
To check the time taken, I used the time library.
Time taken to subset: Old method
start=time.time()df_this_year = permits.loc[permits["Filing Date"].dt.strftime("%Y") == '2009'].compute()print(time.time()-start)# 616.4123058319092 seconds
Time taken to subset: New method
start = time.time()permits_2009 = computed_repartitioned.loc['2009']print(time.time()-start)# 0.0010952949523925781 seconds
This new process is such a huge improvement from my original code that took me a couple of hours to create the GIF! Thank you to @james.a.bednar for the little extra push to improve the original code!
Acknowledgements: Nicholas Hand, MUSA 620 Professor, who taught me the bulk of my python data visualisation skills!
Data sources:Construction permits, NYC Open Data, Department of Buildings( DOB) https://data.cityofnewyork.us/Housing-Development/DOB-Permit-Issuance/ipu4-2q9a
NYC borough boundaries, NYC Open Data, Department of City Planning https://data.cityofnewyork.us/City-Government/Borough-Boundaries/tqmj-j8zm
Datashader documentation: https://readthedocs.org/projects/datashader/downloads/pdf/stable/