Handling Environmental Big Data: Introduction to NetCDF and CartoPY
By: Juan Nathaniel
Tags: Data Science, Machine Learning, Climate, Data Visualization, Environment

Handling Environmental Big Data: Introduction to NetCDF and CartoPY

Photo by NOAA on Unsplash

Environmental data are getting increasingly abundant with the proliferation of remote sensing apparatus, IoT sensors, and weather stations. As a result, an appropriate data format is required to efficiently encode the spatio-temporal properties of climate information so as to promote transferability and generalizability.

Within the climate circle, NetCDF is often used to store these big climate datasets. NetCDF is described as machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data.

NetCDF: Load & Analyze

Let’s get started by installing NetCDF either using pip or conda. We will be using pip for this demonstration.

pip install netCDF4

Next we are going to import the relevant library.

from netCDF4 import Dataset

import numpy as np

import matplotlib.pyplot as plt

We are going to use a NetCDF climate data called CHIRPS which models precipitation events around the globe. You can download any single file that ends with .nc file extension here. Let’s load the data.

file = '<file_name>.nc' # replace <file_name> with your data path

data = Dataset(file, mode='r') # read the data print(type(data)) # print the type of the data

print(data.variables.keys()) # print the variables in the data

The above logs will give us the output which describes the object instance’s type and the data’s available columns or keys.

<class 'netCDF4._netCDF4.Dataset'>

odict_keys(['longitude', 'latitude', 'time', 'u10', 'v10', 'd2m', 't2m', 'tp'])

Let’s extract the relevant columns (ie. spatial coordinates, time, and climate variable which in this case is total precipitation) to separate out variables that we will further manipulate.

# Spatial information

lats = data.variables['latitude'][:]

longs = data.variables['longitude'][:] # Temporal information

time = data.variables['time'][:] # Climate variable of interest (Total Precipitation)

tp = data.variables['tp'][:]

CartoPY: Visualize

Sample output from our visualization exercise using CartoPY

Now that we have the relevant datasets loaded, we need to visualize how the climate variable varies across time and space. For this purpose we are going to use CartoPY, a handy library for geospatial plotting that interfaces well with matplotlib. First, we need to install CartoPY using pip.

pip install cartopy

Next, import the library into our notebook.

import cartopy.crs as ccrs

We are going to generate a continuous frame for all the time interval of our dataset, which in this case is daily over a year.

# loop for 365 days

days = np.arange(0,365) # creating meshgrid from our list of spatial coordinates

lon, lat = np.meshgrid(longs,lats) # generate a frame for each day for i in days:

# set the axes' spatial projection, styling, and title

ax = plt.axes(projection=ccrs.PlateCarree())

ax.coastlines()

ax.set_title('Precipitation for day '+ str(i+1))



# generate the contour map

plt.contourf(longs, lats, tp[i,], 60,

transform=ccrs.PlateCarree())



# save the figure and give a pause for visualization

plt.savefig(f’./output/{i+1}.jpg)

plt.pause(1) plt.show()

Now that we have one frame of climate observation per day, we can create a GIF for visualization.

import PIL image_frames = []

days = np.arange(20,80) # sub-sample day number 20 to 80 for k in days:

new_frame = PIL.Image.open(f’./output/{k}.jpg’)

image_frames.append(new_frame) # save as GIF

image_frames[0].save(‘tp_timelapse.gif’,

format=’GIF’,

append_images = image_frames[1: ],

save_all = True,

duration = 100,

loop = 0)

That’s it! We have successfully loaded, analyzed, and visualized climate big data using tools that are well-maintained and robust. Climate big data can seem daunting at first due to its sheer volume and complexity. However, they essentially contain three basic components: spatial, temporal, and a collection of climate variables. Once you are able to grasp this concept, you can decompose other forms of climate datasets that are available out there. If you have any feedback or comment, please do not hesitate to let me know!