DATA ANALYTICS APPLIED TO SATELLITE-DERIVEDPRECIPITATION ESTIMATES AND HIGHRESOLUTION MODEL OUTPUT
Taylor A. Gowan
Ph.D. Dissertation Defense

Committee: John Horel (Chair), Erik Crosman,
George Huffman, Jim Steenburgh, and Court Strong

ARCHIVAL AND ANALYSIS OF HIGH-RESOLUTIONRAPID REFRESH MODEL OUTPUT USINGZARR FILES IN THE CLOUD

2

The Big DataProblem

Satellites

Radar

Numerical
WeatherModelsO(10 TB) data produced day-1
https://www.noaa.gov/organization/information-technology/big-data-program

3

OVERVIEWZARR FORMATHRRR-ZARR

APPLICATIONSSUMMARYThe Response:
NOAA Big Data Program
2015

Infrastructure as a Service (IaaS) providers have the capacity tostore the increasing volume of data available and provide publicaccess and computer resources to end users
4OVERVIEW

ZARR FORMATHRRR-ZARRAPPLICATIONSSUMMARY

The Response:
NOAA Big Data Program
HRRR
Lightning

NEXRAD

GOES-16/17

Global Relief
ModelSpace WeatherForecasts
GEFSGFS

Most of these high-volume datasets
are stored in hypercube formats
e.g., GRIB2, netCDF45

OVERVIEWZARR FORMATHRRR-ZARRAPPLICATIONSSUMMARYGRIB2 Format:
GRIdded Binary Second Edition• Self-describing file format used to efficiently store and transmit twodimensional data fields (Wang 2014)• Data is compressed using a similar method to JPEG image compression(Silver and Zander 2017)• GRIB2 files are large when decompressed and are difficult to readefficiently, even when using compatible Python libraries
6OVERVIEW

ZARR FORMATHRRR-ZARRAPPLICATIONSSUMMARY

High-Resolution Rapid
Refresh version 4 (HRRR)
RapidRefresh

HRRR• 3-km convection-permitting numerical
weather model – 1.9 million grid
spaces over CONUS domain• Run hourly by the National Center
for Environmental Prediction (NCEP)• Output used for forecasting, research,development
7OVERVIEW

ZARR FORMATHRRR-ZARRAPPLICATIONSSUMMARY

High-resolution model, high-volume output…
HRRR
CONUS

Forecast Lengthfor Initialization Times

Number of Output Files

v. First Date 0, 6, 12, 18 UTC Other Hours
HRRR
CONUS
v. First Date

1 9/30/2014
2 8/23/20161

9/30/2014

28/23/2016

Forecast Lengthfor Initialization Times0, 6, 12, 18 UTC
Other Hours

151815

Number of Output Files
Surface102659

PressureNative151518102659
1357781818135

687

1110

151

151
701

173

3 7/12/20183

367/12/2018

3618
18

4 12/2/20204

12/2/2020

48

1848Surface Pressure Native

18711
173

687

778
1110

-1576
files
day
701
1126
1126
1136

711

1136

8OVERVIEW

ZARR FORMATHRRR-ZARRAPPLICATIONSSUMMARYUtah HRRR Archive
2016-2020 Pando was the only HRRR archive

No longer able tomaintain dataset size
Convert archive toalternate format: Zarr

Blaylock et al. (2017)
OVERVIEWZARR FORMATHRRR-ZARRAPPLICATIONSSUMMARY

9An alternative model output format – Zarr
• Inspired by HDF5
• Creates N-dimensional arrayswith any NumPy dtype• Arrays can be chunked in anydimension
Developed by Alistair Miles (2016) and supported by theMRC Centre for Genomics and Global Healthhttps://zarr.readthedocs.io/• Cloud-compatible• Back-end compatibility
with xarray, dask, iris
10OVERVIEWZARR FORMATHRRR-ZARRAPPLICATIONSSUMMARY

https://zarr-developers.github.io/

What exactly is a chunk?

An N-dimensional subset of a Zarr array (zarray) whose shape,
data type, and memory specifications are based on the userdefined selections for the application
Say we have an array of size (1500,1500)
But we want our chunks to be a moremanageable size of (500,500), or 9
equal chunks

Zarr Array
1500 x 1500

11OVERVIEWZARR FORMATHRRR-ZARRAPPLICATIONSSUMMARY

https://zarr-developers.github.io/

What exactly is a chunk?

A user-defined N-dimensional subset of a Zarr array (zarray)
whose shape, data type, and memory specifications arepredefined based on the application
Say we have an array of size (1500,1500)

(0,0)

(0,1)

(0,2)

But we want our chunks to be a moremanageable size of (500,500), or 9
equal chunks

(1,0)

(1,1)

(1,2)

Each resulting chunk will be labeledwith numbers corresponding to its
location in the zarr array

(2,0)

(2,1)

(2,2)
12OVERVIEWZARR FORMATHRRR-ZARRAPPLICATIONSSUMMARY

https://zarr-developers.github.io/

What exactly is a chunk?

A user-defined N-dimensional subset of a Zarr array (zarray)
whose shape, data type, and memory specifications arepredefined based on the application
Each chunk can be encoded andcompressed based on the user specificationEncoding: Byte order, character code,
byte length
Compression: LZMA, Blosc, LZ4,Zstandard, Zlib

(0,0)

(0,1)

(0,2)

(1,0)

(1,1)

(1,2)

(2,0)

(2,1)

(2,2)
13

OVERVIEWZARR FORMATHRRR-ZARRAPPLICATIONSSUMMARY

So, does anyone use it?

14OVERVIEWZARR FORMATHRRR-ZARRAPPLICATIONSSUMMARYHRRR-Zarr Dataset

HRRR Surface
GRIB2

HRRR Surface
Zarr

 Keep Climate and Forecast
naming conventions
 Store in Amazon Web ServiceCloud
15

OVERVIEWZARR FORMATHRRR-ZARRAPPLICATIONSSUMMARYHRRR-Zarr Workflow
49 surface files with173 data fields

500mb/HGT

Compression: LZ4
Level: 900 UTC HRRR
Surface Forecasts

All files loaded
into memoryand organizedby level/variable

Arrays encoded,
chunked, andcompressed

20201015_00z_anl.zarr

F00

20201015_00z_fcst.zarr

F01 – F48
16OVERVIEWZARR FORMATHRRR-ZARRAPPLICATIONSSUMMARY

Consider 20201015_00z_fcst.zarr• Each data array (variable) isof size (48,1799,1059)• We want to subset the array
along the time dimension,
into chunks of size(48,150,150) for easy timeseries construction• Produces 96 chunks acrossthe HRRR domain~500 kB – 1 MB
17

OVERVIEWZARR FORMATHRRR-ZARRAPPLICATIONSSUMMARYAWS Simple Storage Service (S3)S3 object store

Zarr organization mimics ahierarchical data structure
18

OVERVIEWZARR FORMATHRRR-ZARRAPPLICATIONSSUMMARYPublicly available throughthe Registry of Open Data
on AWShttps://registry.opendata.aws/noaahrrr-pds/

19

OVERVIEWZARR FORMATHRRR-ZARRAPPLICATIONSSUMMARYData Application: Labor Day Weather Event
7-9 September 2020
Downslope
winds andwildfires

Downslope
winds
Snow

https://worldview.earthdata.nasa.gov/
20

OVERVIEWZARR FORMATHRRR-ZARRAPPLICATIONSSUMMARYHRRR-Zarr Use Cases
PortlandMt. Hood(e.g., operational forecasting, machine learning)

Salem

EugeneUsing HRRR-Zarr data for examples ofhigh-throughput applications• Time series

Bend Forecast spread for a point• Spatial plots
Crater Lake

 Time-lagged Ensemble (Probabilities)
 Empirical Cumulative Distributions andexceedance values

21

OVERVIEWZARR FORMATHRRR-ZARRAPPLICATIONSSUMMARYForecast Spread near Beechie Creek Fire
21 forecasts valid for each time 96 Zarr chunks (500 kB)
 1260 GRIB2 files (150 MB)
 3.3 seconds on CHPC’s
Kingspeak34
No multiprocessing
Station shelteredby forest

22

OVERVIEWZARR FORMATHRRR-ZARRAPPLICATIONSSUMMARY

𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇 𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹
𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹 > 10 𝑚𝑚 𝑠𝑠 −1

Probabilistic Guidance

23

OVERVIEWZARR FORMATHRRR-ZARRAPPLICATIONSSUMMARY

Empirical Cumulative Distributions
Goal is to compare data to a “climatology”
distribution to identify outlier or extreme eventsBlaylock et al. (2018) used the Open Science Gridto calculate HRRR distributions
Use all HRRR wind gust analyses (F00) from themonth of September 2016-2019 to create adistribution for each grid point in the chunk24

OVERVIEWZARR FORMATHRRR-ZARRAPPLICATIONSSUMMARY

25

OVERVIEWZARR FORMATHRRR-ZARRAPPLICATIONSSUMMARY

Problem: Efficiently accessing and processing high-volume model
output for machine learning and forecasting applications
Proposed Solution: Convert the High-Resolution Rapid Refresh(HRRR) GRIB2 model output archive to Zarr format• We converted the HRRR archive to an alternative format, Zarr, forits compatibility with cloud environments and its flexibility• Each HRRR model run forecasts are condensed into two Zarr files (aforecast and analysis) which contain all data fields and are named using
CF conventions• Each Zarr array is subset into 96 chunks for a more efficient andcustomizable user experience
26

OVERVIEWZARR FORMATHRRR-ZARRAPPLICATIONSSUMMARY

Problem: Efficiently accessing and processing high-volume model
output for machine learning and forecasting applications
Proposed Solution: Convert the High-Resolution Rapid Refresh(HRRR) GRIB2 model output archive to Zarr format• Caveat: Zarr is optimal for many applications, but GRIB2 files are bestfor running model simulations and other tasks that require full-domaingrids• The HRRR-Zarr dataset is now stored on AWS, thanks to the Amazon
Sustainability Initiative and Open Data Program

27

OVERVIEWZARR FORMATHRRR-ZARRAPPLICATIONSSUMMARY