Acquire and Analyze Weather and Climate Data with Python and Tableau
By: Randy Runtsch
Tags: Data Science, Weather, Climate, Data Analysis, Python

Weather and Climate data are valuable resources, not only to forecast the weather but for dozens of purposes across industries, governments, and in personal life. Opportunities to apply data analytics and data science techniques to these data to inform and support decision-making are almost endless.

NOAA’s National Centers for Environmental Information (NCEI) is the world’s most extensive weather and climate data collection. Its databases contain over 37 petabytes of atmospheric, coastal, oceanic, and geophysical data. This article describes the data, how to download its data files, and instructions to acquire it with a Python program that calls its Data Service API. It also demonstrates a Tableau Public dashboard to visualize a sample set of wind data retrieved with the API.

Weather and Climate Data Uses

Weather and climate data serve a multitude of needs in society. Here are some examples:

Agriculture — Farmers can use weather data to determine when to plant and harvest crops, apply fertilizer and herbicides, and so on.

— Farmers can use weather data to determine when to plant and harvest crops, apply fertilizer and herbicides, and so on. Allergies — Weather plays a significant role in allergy seasons, especially for those related to pollen and other plant materials.

— Weather plays a significant role in allergy seasons, especially for those related to pollen and other plant materials. Building and road construction — Weather may impact outdoor construction work.

— Weather may impact outdoor construction work. Civil Engineering — Engineers who design structures such as bridges, roads, and tunnels, must be aware of weather effects on them and their materials over short and long periods.

— Engineers who design structures such as bridges, roads, and tunnels, must be aware of weather effects on them and their materials over short and long periods. Climatology — Climate scientists use weather and climate data to understand past, present, and potential future climates.

— Climate scientists use weather and climate data to understand past, present, and potential future climates. Clothing and fashion — The types of clothing we wear vary with the season and weather conditions. Some people choose their style and colors by season.

— The types of clothing we wear vary with the season and weather conditions. Some people choose their style and colors by season. Emergency management — Emergency managers use weather and climate data to determine the potential risks of catastrophic events, such as wildfires, tornadoes, hurricanes, and floods.

— Emergency managers use weather and climate data to determine the potential risks of catastrophic events, such as wildfires, tornadoes, hurricanes, and floods. Forensic meteorology — Forensic meteorologists analyze weather data to piece together past weather events.

— Forensic meteorologists analyze weather data to piece together past weather events. Forestry and Land Management — Foresters use weather and climate data to manage forests, understand the effects of droughts, mitigate the risks of fires, and more. The U.S. Forest Service, for example, operates an extensive network of automated weather stations.

— Foresters use weather and climate data to manage forests, understand the effects of droughts, mitigate the risks of fires, and more. The U.S. Forest Service, for example, operates an extensive network of automated weather stations. Migration of People — Weather and climate are factors that people use to help them determine where to live.

— Weather and climate are factors that people use to help them determine where to live. Outdoor recreation — Outdoor activities such as hiking, sailing, skiing, camping, fishing, and hunting vary by season and weather. People evaluate historical climate and weather data and current conditions and forecasts to choose good times for their outdoor activities.

— Outdoor activities such as hiking, sailing, skiing, camping, fishing, and hunting vary by season and weather. People evaluate historical climate and weather data and current conditions and forecasts to choose good times for their outdoor activities. Product sales and marketing — Retailers, for example, use weather data to determine when to stock certain types of goods, such as bathing suits or snow skis, in their stores.

— Retailers, for example, use weather data to determine when to stock certain types of goods, such as bathing suits or snow skis, in their stores. Shipping — Shipping may be affected by the weather. For example, the shipment of grain and other bulk goods in barges on inland waterways that may freeze in the winter is affected by the weather.

— Shipping may be affected by the weather. For example, the shipment of grain and other bulk goods in barges on inland waterways that may freeze in the winter is affected by the weather. Tourism and travel — People rely on weather and climate data to plan short and long trips to their chosen destinations.

— People rely on weather and climate data to plan short and long trips to their chosen destinations. Wave activity and hydrodynamic impact — Data from weather instruments mounted on buoys can help project the effects of waves and weather on coastlines and beaches.

— Data from weather instruments mounted on buoys can help project the effects of waves and weather on coastlines and beaches. Weather forecasting — Meteorologists use climate and weather data to help them predict future weather.

Data Overview

NCEI is the leading authority for environmental data in the U.S. It manages extensive archives of atmospheric, coastal, geophysical, and oceanic research data.

NCEI manages a variety of data products, many of which are available to the public. Too numerous to mention, and beyond the scope of this article, here are some examples of its data product categories:

Air temperature and atmospheric properties

Arctic and sea ice

Ecosystems and natural resources

Geomagnetism

Natural hazards, disasters, and severe weather

Ocean climate

Precipitation

Space weather

U.S. and regional climate

Download Weather and Climate Data

In addition to application programming interfaces (APIs), NCEI provides tools to find, request, and download weather and climate data.

Find a Station and Order and Download its Data

Weather observing stations located in the United States and worldwide capture much of the weather data managed by NCEI. Navigate to the Data Tools: Find a Station page to search for stations by city, county, state, ZIP code, or country.

In the example shown below, I use the Find a Station tool to search a location of San Jose, California, the Daily Summaries category, a date range 2000–01–01 through 2020–12–31, and the Wind data category. Note the observation station symbols in Fremont and San Jose, California, on the map.

Data Tools: Find a Station. Image captured by the author.

When I click on the station symbol in San Jose, a STATION DETAILS window pops up, as shown below. Note the component of the ID value to the right of “GHCND:.” This value, “USW00023293,” is the observing station’s unique identifier at the San Jose International Airport. This ID will be used later in a call to the NCEI Data Service API to retrieve weather data captured by this station programmatically.

Weather observation station details. Image captured by the author.

To start the process to retrieve this data, click on the [ADD TO CART] button. Then, click on the [Cart (Free Data)…] button in the screen’s upper-right corner.

Data shopping cart button. Image captured by the author.

On the Cart: Daily Summaries page, select the “Custom GHCN-Daily CSV” format. This selection will result in the download of wind data in a comma-separated value (CSV) file. The file format will be similar to the CSV file that a Python program will save after calling the Data Service API.

Review the search values, including the “Date Range.” When ready, click the [CONTINUE] button.

Cart: Daily Summaries page. Image captured by the author.

On the Custom Options: Daily Summaries page, make the selections shown in the screenshot below. Note that a Units value of “Standard” will return wind speed values in miles per hour (MPH). Selecting “Metric” would return wind speed values in meters per second.

The “Average wind speed (AWND)” data type will also be used later in the article to demonstrate visualizing the data in Tableau Public. The “Fastest 2-minute wind speed (WSF2)” and “Fastest 5-minute wind speed (WSF5)” data types have been selected only for demonstration.

The “AWND,” “WSF2,” and “WSF5” data type values will be used later in a call to the Data Service API.

Custom Options: Daily Summaries page. Image captured by the author.

When presented with the Review Order page, review your selections and enter your email address twice. When ready, click on the [SUBMIT ORDER] button. Within seconds or minutes, NCEI will send you an initial email to confirm the order submission and a second email to confirm its completion. The completion email contains a link to download a data file that meets your selection criteria.

Climate Data online request email subject lines. Image captured by the author.

Open the second “Order Complete…” email. Click on the [Download] link to download the data file.

Partial Climate Data Online request completion email. Click on the [Download] button to download the data file. Image captured by the author.

Open the downloaded CSV file in a text editor. It should look similar to the file shown below. Note that the column names (called “data types” by NCEI) in the first row correspond with comma-separated data values in subsequent rows.

Sample of downloaded daily wind data CSV file. Image captured by the author.

Other Data Search Toolsets

Searching for observation stations and selecting and downloading their data is one of many ways to retrieve data from NCEI datasets. Use the links on the Climate Data Online page to explore other methods.

Climate Data Online page. Image by the author.

NOAA also offers the OneStop data search platform to find and view a variety of data interactively. Some data are available for download.

NOAA OneStop Data Search Platform home page. Image captured by the author.

Calling the Access Data Service API with Python

NCEI provides four RESTful APIs to acquire weather and climate data. This article describes how to use the Data Service API to obtain daily summary data for several weather stations.

The Data Service API retrieves data based on parameters sent to a web server with a predefined URL. See the brief Data Service API documentation for an overview of how it works.

URL and Parameters

To use the Data Service API, append parameters to this predefined URL:

https://www.ncei.noaa.gov/access/services/data/v1

The sample code described in the next section will query the daily-summaries dataset to retrieve the daily average wind (‘AWND’) speed value for several U.S. locations for each day between January 1, 2000, and December 31, 2020. The API returns AWND and other wind values in miles per hour (MPH). To retrieve the desired data, append these parameters to the URL shown above:

dataset — The ‘daily-summaries’ dataset provides summarized daily weather data, including wind data.

— The ‘daily-summaries’ dataset provides summarized daily weather data, including wind data. dataTypes — We are interested in daily average wind speed (‘AWND’) in this example. But for possible future use, the parameter value is set to obtain the fastest 2-minute windspeed (‘WSF2’) and the fastest 5-second windspeed (‘WSF5’). To get these three types, set the dataTypes parameter value to ‘AWND,WSF2,WSF5.’

— We are interested in daily average wind speed (‘AWND’) in this example. But for possible future use, the parameter value is set to obtain the fastest 2-minute windspeed (‘WSF2’) and the fastest 5-second windspeed (‘WSF5’). To get these three types, set the dataTypes parameter value to ‘AWND,WSF2,WSF5.’ stations — In this example, we are interested in the daily average wind speed for these stations: Chicago O’Hare International Airport, Rochester International Airport (Minnesota), and San Jose International Airport. Provide these stations’ identifiers to the API in this comma-delimited list: ‘USW00094846,USW00014925,USW00023293.’ Search for station IDs in the Data Tools: Find a Station page as described in a previous section of this article.

— In this example, we are interested in the daily average wind speed for these stations: Chicago O’Hare International Airport, Rochester International Airport (Minnesota), and San Jose International Airport. Provide these stations’ identifiers to the API in this comma-delimited list: ‘USW00094846,USW00014925,USW00023293.’ Search for station IDs in the Data Tools: Find a Station page as described in a previous section of this article. startDate — Records from January 1, 2000, through December 31, 2020, will be retrieved. So, in this example, a value of ‘2000–01–01’ will be set.

— Records from January 1, 2000, through December 31, 2020, will be retrieved. So, in this example, a value of ‘2000–01–01’ will be set. endDate — Since we’re interested in data through 2020, set this value to ‘2020–12–31’.

— Since we’re interested in data through 2020, set this value to ‘2020–12–31’. boundingBox — Set the bounding box values to select data from a geographic location contained within the coordinates. It includes four comma-separated numbers. North and South range from -90 to 90, and East and West range from -180 to 180. For this example, set boundingBox to ‘90,-180,-90,180’.

— Set the bounding box values to select data from a geographic location contained within the coordinates. It includes four comma-separated numbers. North and South range from -90 to 90, and East and West range from -180 to 180. For this example, set boundingBox to ‘90,-180,-90,180’. units — Values are “metric” or “standard.” For the average wind speed data type, the “standard” value will return the wind speed in miles per hour.

Here is the full URL that resulted from appending the parameters to the base URL:

https://www.ncei.noaa.gov/access/services/data/v1/?dataset=daily-summaries&dataTypes=AWND,WSF2,WSF5&stations=USW00094846,USW00014925,USW00023293&startDate=2000-01-01&endDate=2020-12-31&boundingBox=90,-180,-90,180&units=standard

Python Code

Class c_ncei_data_service_api

The file c_ncei_data_service_api.py defines the c_ncei_data_service_api class. The following sections describe its functions.

__init__() constructor function

The __init__() constructor function is called when a program creates an instance of the c_ncei_data_service_api class. It takes these input parameters:

dataset — The name of the NCEI dataset that contains the desired data.

— The name of the NCEI dataset that contains the desired data. data_types — The fields to return from the dataset. It consists of a comma-delimited list of field names.

— The fields to return from the dataset. It consists of a comma-delimited list of field names. stations — These are unique identifiers of observation stations of interest. The parameter consists of a comma-delimited list of station IDs.

— These are unique identifiers of observation stations of interest. The parameter consists of a comma-delimited list of station IDs. start_date_time — The first date of the data to retrieve.

— The first date of the data to retrieve. end_date_time — The last date of the data to retrieve.

— The last date of the data to retrieve. bounding_box — See the description for boundingBox in the previous section.

The function performs these steps:

Set variable self._base_api_url to the value of the base URL of the NCEI Service Data API. Call self.call_api() with the input parameters to retrieve the requested data.

Definition of __init__() function at the top of the c_ncei_data_service_api class module. Note the “import requests” directive, which is required to use requests.get() to submit the request to the NCEI Data Service API.

Note: By default, a call to the URL will return data in CSV format. See documentation for the parameter to set to retrieve the data in JSON and other types.

call_api() function

The call_api() function retrieves data from the specified NCEI dataset. It performs these steps:

Append parameters to the base URL to create the full_url variable. Note that it sets the API’s ‘units’ parameter in addition to the function’s input parameters. Submit the URL with the requests.get() function. The resulting dataset, if any, will be returned into the ‘response’ variable. Return the ‘response.text’ variable to the constructor.

call_api() function within the c_ncei_data_service_api class module. This function uses requests.data() to submit the full URL of the NCEI Data Service API request.

get_data() function

The get_data() function returns the dataset data to its caller.

get_data() function in the c_ncei_data_service_api class module.

write_data_file() function

The write_data_file() function writes the data returned from the URL call and stored in the self._dataset variable to a text file. It performs these steps:

Open the file specified in the parameter in write mode. Write the value in the self._dataset variable to the file. Close the file.

write_data_file() function within the c_nei_data_service_api class module.

Controller Module

The controller module imports the c_ncei_data_service_api module and performs these steps:

Create an instance of the c_ncei_data_service_api class with the parameters described in the __init__() constructor description. This step will retrieve data from the NCEI dataset. Print the retrieved data to the display. Write the retrieved data to the specified file.

The controller module creates and calls an instance of the c_ncei_data_service_api class. It then prints the data returned from the API to the screen and writes it to a file.

Possible Class and Program Improvements

I developed the c_ncei_data_service_api class and controller module for demonstration and learning purposes. Here are some ideas to add features and make the modules more robust and production-ready:

Add error-handling to gracefully recover from errors and retry parts of the code if needed.

to gracefully recover from errors and retry parts of the code if needed. Write information about the program’s operations, such as the date and time when it starts and finishes, error information, input parameters, and the number of records retrieved, to a log file .

. Modify the call_api() function to handle more input parameters, such as the desired output data format (CSV, JSON, PDF, and others).

Running the Program

Run the controller module to create an instance of the c_ncei_data_service_api, retreive the desired data from the NCEI Data Service API, display the data, and write it to a file. The output file should appear similar to the sample in the output weather_data.csv file shown below. The first row contains column headers, while subsequent rows contain data.

CSV file retrieved with call to NCEI Data Service API. Image captured by the author.

Other NCEI APIs

While this article explores the use of the Data Service API, NCEI has provided these APIs, too:

Search Service API — Discover datasets and data based on a set of parameters.

— Discover datasets and data based on a set of parameters. Access Support Service API — Discover metadata and attributes about datasets based on a set of parameters.

— Discover metadata and attributes about datasets based on a set of parameters. Access Order Service API — Retrieve information about previous orders based on a set of parameters.

Learn more about NCEI datasets on the Access page. It provides links to data discovery tools, data access information, and developer tools, including the APIs, map service directories, climate data web services, and the official NOAA GitHub repository.

Analyzing Data in Tableau Public

Chicago, Illinois, is known as “The windy city.” I live in southeastern Minnesota, which is a windy place, too. To visually compare the Chicago areas’ daily average wind speeds with those of Rochester, the largest city in southeastern Minnesota, I loaded the data retrieved with the Data Service API in Tableau Public. There, I created a Daily Average Wind Speed dashboard. My son lives near San Jose, California, so wind speeds at San Jose International Airport are shown, too.

The results surprised me. Wind speeds at Rochester International Airport (Minnesota) are consistently higher than at Chicago O’Hare International Airport. Both have winds much greater than San Jose International Airport.

From 2000 through 2020, the locations have these daily average wind speeds in miles per hour:

Rochester International Airport (Minnesota) — 11.31 MPH

Chicago O’Hare International Airport — 9.52 MPH

San Jose International Airport — 6.44 MPH

Daily average windspeed for three locations from 2000 to 2020. Image captured by the author.

The dashboard also shows daily average wind speed by year, month, month by year, and season. Meteorological seasons in the northern hemisphere are defined as follows.

Spring — March, April, and May.

Summer — June, July, and August.

Fall — September, October, and November.

Winter — December, January, and February.

According to the data, spring and winter are the windiest seasons in Rochester and Chicago, and summer is the calmest. San Jose has relatively light winds year-round.