Weather Model Processing With FOSS4G and JavaScript: Best Practices
By: Thomas Horner
Tags: Weather, Weather Data, Foss4g, GIS, JavaScript

Weather Model Processing With FOSS4G and JavaScript: Best Practices

Tips and tricks for retrieving, processing, and managing weather model data with free and open source software for geography (FOSS4G) and/or JavaScript.

A year ago I wrote Automated Weather Model Processing With FOSS4G: Lessons Learned, which outlined my journey in creating a system that automatically retrieved, processed, and managed weather model data using open source geography software. I have now worked extensively with this system for a year, including the release of a cutting-edge weather forecasting website for the state of Colorado, and had the opportunity to learn from key players in the commercial and academic weather forecasting fields.

The findings in the original article are still useful as an introduction to common paradigms of distributing and using raw weather model data. This follow-up expands upon that and covers best practices and techniques for dealing with this data, using only free and open source software for geography (FOSS4G). There is also some tips and tricks for working with this data using JavaScript, which is becoming a more ubiquitous language, including in the field of Geographic Information Systems (GIS).

This article assumes you have GDAL installed, and you are already aware of where to download the weather model files and how they are generally structured. “Index files” will be referenced often, which are the .idx files you will often see accompanying grib2 files:

These files are incredibly useful for managing huge rasters and variable band ordering, and you should utilize them as much as possible.

Image by author

1. Use GRaDS

The reason this isn’t recommended outright is it gets away from the point of the article — there is already a large field of people doing incredible things with GIS and geospatial rasters, and the goal is to help make weather model data more easily accessible in this field and leverage that existing talent.

However, if your focus is purely on meteorology or a more scientific discipline, you may want to just use the industry standard for working with weather model and other scientific data — GRaDS. This software has its own scripting language, and it requires a lot of work to make maps that look anything like that quality you can easily produce in common desktop GIS applications, so it represents a use case that may be outside of the discipline of GIS.

2. Don’t reproject the data, unless you need to draw it on a web map

Weather models have a variety of projections (usually some form of Lambert Conformal Conic) that don’t easily reproject to EPSG:4326 or Web Mercator without losing data or forcing interpolation. If you query reprojected data, you will likely get a forecast that is inaccurate when compared to the raw model data.

The top raster (NAM 3km model) is in its original projection. The bottom raster is a result of reprojecting to EPSG:4326. Ignoring the skew (the bottom image is still drawn on a map with the top image’s projection), note how the original data is not adequately represented, such as the circled cluster of pixels. Image by author.

3. JavaScript-based projection libraries (like Proj4js) don’t reproject coordinates for most weather models properly

If you’re using Node.js to query geospatial rasters (with something like Geotiff.js) and you want to keep them in their original projection when querying a latitude/longitude coordinate, the Proj4 and WKT strings provided from GDAL can be passed to proj4js — but you’ll notice some pretty serious spatial inaccuracies after the reprojection (many miles, not a few feet). There are certain kinds of transformations that proj4js cannot do and it seems like most weather model projections fall into this category.

You’ll need to either use the GDAL bindings (gdal-js — not ideal, as they require an old version of GDAL) or invoke a shell to call GDAL’s gdaltransform for a more accurate reprojection — or another reprojection library, outside of JavaScript. To keep things easy, you can load a grib2 file into QGIS and use the “Extract Projection” tool to export a .prj file which can be passed to gdaltransform .

4. Use PostGIS out-db rasters to manage a slew of files

If you process weather model data with any degree of regularity, you will have a massive amount of geospatial rasters that have dozens of bands in them. PostGIS for PostgreSQL is a good tool to keep this data organized, but is too slow at reading/writing the data itself — however, you can just keep the raster as a GeoTIFF (or whatever) on the filesystem and reference it from PostGIS as an out-db raster.

5. Node.js is an okay choice for making an API that handles geospatial weather model data

There advantages to using other languages for your backend or API (such as python, with its GDAL bindings), but Node.js is becoming more popular, and it is a viable choice for working with this data. If you’re using Postgres and GeoTIFF files, this is an acceptable solution, though if you expect to have huge concurrent demand, you’ll probably want to go with a different approach using raw binary files or extreme caching techniques. For the JS-based approach, you can load weather model GeoTIFF files using Geotiff.js and query them with decent performance. I wrote an article that covers specifics here.

6. Identifying unknown bands

NOAA’s NCEP (the U.S. agency that is generally responsible for development and dissemination of weather models) has not been entirely consistent with the usage of the GRIB2 format, which trips up a variety of GRIB decoders including some created by NOAA themselves! At the time of writing, the NBMv4 model shows unknown bands, even in NCEP’s own index lookup files:

GDAL, however, generally seems to struggle with band identification more than some other libraries, so you may have a number of “unknown” bands in your raster. Here’s the workflow for trying to figure out what a band pertains to, if the metadata cannot seem to describe the weather model parameter and/or level:

Try to find it in the accompanying .idx file for the grib2 file. Use NASA’s Panoply Data Viewer, which seems to be more successful than other GRIB decoders such as degrib. Use wgrib2, which is the standard (and an option to use in GRaDS), but can be difficult to use.

Grib2 files have been historically a little neglected by the hard working GDAL contributors, but they have been great at fixing some recent errors I discovered with newer weather models, such as the NBMv4. These models should work completely in the GDAL >3.1, but older versions such as the current upstream version in some Linux distributions (3.0.4) choke when processing some parameters (like snow accumulation). If you notice other issues, reach out to the gdal listserv!

7. Subscribe to the NOMADS listserv

The primary distribution server for weather models — NCEP NOMADS — tends to experience outages at a fairly alarming frequency, but are usually precluded or shortly followed by an outage announcement on the listserv. Other important notifications include changes to models and additions or deletions of them, which may break your scripts.

8. Optimize read/write speed and RAM on your servers, as opposed to CPU speed

In general, if you’re just performing basic transformation of weather model data and simple analysis or querying of it, you’ll be limited by your ability to read/write data on the disk and load large raster files into RAM, while your CPU sits mostly idle. You should definitely be using SSD drives. If you’re doing a lot of reprojection of raster files, CPU will be more important. If you’re going to be storing global weather models or some of the more advanced ones (like the NAM or HRRR) back a large number of runs, you are going to need a few terabytes of hard drive space.

9. The “surface” elevation is entirely dependent on the model’s resolution

Comparing two values at the surface (such as temperature) between models of different spatial resolutions is not a fair comparison, especially in rugged terrain. Each model has an underlying DEM, which you can generally extract from the “analysis” forecast hour (aka zero). For instance, this data is available in the NAM model at fh00 / analysis, according to the corresponding index file:

10. Index files may consider two variables to be part of the same byte range

You’ll likely encounter this with the ugrd and vgrd components of wind.

You can only get both of these variables at the same time, which means if your retrieval/processing script is entirely dependent on one variable per row in the index file, things will break. What happens when you retrieve this byte range is that you will get a two band raster, with the first band pertaining to ugrd and the second pertaining to vgrd .

11. Some models have a projection with longitude extents of 0 to 360