Interactive Weather Data Visualizations with Plotly | by Will Norris | Towards Data Science

The Global Surface Summary of the Day (GSOD) is a weather data set from over 9,000 weather stations dating back to 1929. It was created and is maintained by the National Oceanic and Atmospheric Administration (NOAA) and generates a daily summary of hourly surface measurements for 18 climate variables like mean dew point, mean wind speed, and min/max temperate.
This is a great data set to use for practicing visualizing spatial data since it has many stations across the globe with a relatively large time series to play around with. It also is updated daily, which makes it a great candidate for practicing with dashboards.
While Python might not be the first language or tool that comes to mind for visualizing spatial data, there are several libraries that are capable of creating fantastic geographic visualizations. Plotly has become a very well rounded interactive plotting library for Python, and its geographic plotting capabilities have come a long way over the last few years. Because of its excellent documentation and popularity I will focus only on Plotly in this post, but keep in mind that there are other libraries available.
This data set is somewhat large at 4.2 GB so loading it into Python will take a little effort; it is not a massive data set by any means, but it is larger than the average “practice” data set we encounter. We will start by walking through how to access the data and then explore visualizing the processed data on interactive maps with Plotly. Make a cup of coffee, get comfortable, and let’s work with a larger weather data set and visualize our findings with Plotly.
There are multiple ways to obtain this data set. An easy to download version of it is available from Kaggle, but it isn’t updated daily like the NOAA repositories and the Kaggle version is more difficult to format in Python. This is how the NOAA repository is structured:
1929.tar.gz/1930.tar.gz/    03005099999.csv    03026099999.csv    ......
The data is stored in yearly compressed tarballs that contain .csv files for each station active that year. Every .csv file is named after its station ID, which is made up of the station’s WMO and WBAN numbers.
The World Meteorological Organization (WMO) is an agency of the United Nations responsible for weather, climate, and water resources. The first two digits are the country code, the next three digits are set by the National Meteorological Service (NMS). If the last number is a zero that means the WMO number is, or was in the past, official.
WBAN stands for Weather-Buereau-Army-Navy. The WBAN number is an identifier for digital data storage that started as a way for the US Weather Bureau, Canada’s Transportation Department, and the three major military branches to share data with each other. It later expanded to include some German and Korean stations.
Note: The important piece to remember is that the WMO and WBAN numbers create a unique station ID that is our primary means of referencing each weather station once loaded into pandas.
The best way to get this data is directly from NOAA where you can download a tarball for each year that contain a csv file for each station active during that year. I have included a simple Python module for downloading and processing all of the data that you can find in the documentation section at the end.
To download the NOAA data I decided to make a simple scraping function using requests. It starts on the home directory of the data set and uses regex to find all of the years available from the text of the base URL page. With each year in a list it is easy to construct the final URLs and download each tarball. Remember that all of this code is available on Github and can be imported as a local module in Python.
The most important question to ask yourself when loading in large datasets is what data you want to have access to once it is loaded into a dataframe. There were over 11,000 stations in 2020, which means ~10 years of daily data would be quite a lot of information to work with. Instead you may want to do some aggregation. For this article I decided to aggregate the data by month and only save the daily data for one day of each year. Aggregating the data by month will greatly reduce the size of our final dataframe and still let us find trends over a decently long time series.
It may seem weird to keep one specific day un-aggregated, and it is! The real reason we want to save a single day’s data is so that we can load in the most recent day to a live dashboard with Dash in a future post. For now it will still be useful for creating similar plots to what we will explore with Dash later.
Since processing this data is a little more involved than most practice datasets, I have included a GitHub repository with a local module that can be imported locally into your Python environment.
import gsodpy.gsodProcess as gsodimport gsodpy.gsodDownloader as gsodDownimport datetime
In order to import modules locally like this, the “gsodpy” directory must be in your project directory.
The first thing we need to do is decide how many years to include and which day we will store un-aggregated. We can then use regex to create our desired file list.
num_years = 10target_day = datetime.datetime(2020,3,20)years, files = get_years_files(num_years)
Once we have defined our time series, we are ready to start cracking open tarballs and processing .csv files! Here are the functions I wrote to unpack our data.
To save you some time, here is the basic psuedo-code for this workflow:
- Loop through each tarball (years)    - Loop through each csv file stored in tarball (stations)        - Open .csv file for current station/year with pandas         - Call process_df() to clean the dataframe        - Append all data for this station on target_day to df_day        - Aggregate data by month        - Append monthly aggregate to df- Call add_meta() to create metadata column on df and df_day- Return completed df and df_day 
Overall the process is not complex, and I’m sure I could make this more concise given more time. Remember that with the large number of stations in each year it takes quite a while to load in all of this data.
We can go ahead and call our processing function to get our dataframes:
df, df_day = gsod.process_all_years(files, target_day)
Note: If you notice that your machine is struggling to process all of the data try starting with a short time series (2–3 years). If you want a longer time series you can also select a random sample of stations in the tarball to reduce the number of station files you need to process.
I highly recommend pickling your finished dataframes so that you can load them in much faster later.
import pandas as pd# Save DataFrames to pickle in current directorydf.to_pickle("df_monthly.pkl")df_day.to_pickle("df_daily.pkl")# Read them back in later:df = pd.read_pickle("df_monthly.pkl")df_day = pd.read_pickle("df_daily.pkl")
Now that we have some meaningful data loaded into Python we can get to the fun part and start making visualizations with Plotly!
If you use the free version of Plotly you are limited to uploads of 512kb. This isn’t a problem if you are making your plots locally. However, for me to share interactive plots here I need to use Plotly’s Chart Studio to upload them to their cloud. This means I am limited to roughly 300 weather stations that I have chosen randomly with this function:
To start off, lets take a look at one month in our time series to see where some of our stations are on the map. This is about as basic of a map we can create with Plotly and this data; considering how little configuration it took, I think it looks pretty good.
Plotly’s main data structure is the figure , which is an instance of the plotly.graph_objects.Figure class. The figure is rendered via Plotly.js using JavaScript, which is why we are able to access quality looking maps in Python.
I love plotting spatial data with Plotly because of its snappy and responsive interface that can be embedded almost anywhere. Quality base maps have always been difficult for me to find in Python packages, but Plotly’s are great.
While this plot may be fine, there are absolutely several stylistic changes I would make to it. The coastlines are obtrusive and the color scale could use adjusting. Overall it could be altered to make our data points stand out more. Luckily all of these things are easily accomplished with Plotly!
One reason I chose to keep a single day of the data instead of only taking monthly aggregates was so that we could look at the warmest and coldest places on a given day.
First we need to find our extreme temperatures:
This plot uses the same structure as our first one, but I spend more time customizing the marker and figure properties. You will notice that I am able to pass a dictionary to certain parameters like marker to customize them further. In most cases, these parameters are objects that are being initialized by the information in the dictionary. For example, the marker parameter initializes a plotly.graph_objects.scattergeo.Marker object. While this increases complexity, you can see from the scattergeo.Marker documentation how much control this gives us over the markers on our plot.
I have always been a sucker for Mapbox. It has great base maps and is super responsive. Plotly has limited Mapbox integration that can enhance your geographical plots. Especially when zooming in you will notice a pleasing amount of detail in the base map compared to the base Plotly version. Mapbox has many effects on the way Plotly maps are rendered, so I will be covering Mapbox integration in an entire post.
One feature that I really like about Plotly is how easy it makes adding a slider to view how data changes over time. As an example, lets take a look at the average monthly temperatures in 2012.
You will notice that I am using a slightly different syntax to format this plot. This is technically the “old” method of plotting with Plotly. But for more complex plots like this it is nice to be able to format the data separately.
Creating this plot looks more complex than the others. But really the bulk of the code is simply restructuring data into monthly groupings for Plotly to easily send to its JavaScript library. If you have ever worked with json data, you will probably see how a lists of dictionaries can easily be converted to json.
Spatial data is never the easiest to work with in Python. It is multidimensional, often quite large, and normal 2d plotting spaces are not well suited to plotting geographical coordinates. Plotly removes a lot of the headache by providing easy to use functionality with quality base maps that I generally had to use javascript to access a couple years ago. When it comes to spatial plotting in Python, Ploty is generally my first choice.
There is so much more to learn with plotting spatial data with Plotly so don’t be afraid to sift through the documentation and examples online. The same developers also make Dash, which is a great tool for building live dashboards.
Github Repo for Processing GSOD Data
GSOD Data Repository (NOAA)
Kaggle GSOD Data