DOI: 10.5061/dryad.h9w0vt4nq
title: GREMLIN CONUS3 Dataset for 2020

description: Geostationary Operational Environmental Satellite (GOES) Radar Estimation
 via Machine Learning to Inform NWP (GREMLIN) is a machine learning model
 that produces composite radar reflectivity using data from the Advanced
 Baseline Imager (ABI) and Geostationary Lightning Mapper (GLM). GREMLIN is
 useful for observing severe weather and providing information during
 convective initialization especially over regions without ground-based
 radars. Previous research found good skill compared to ground-based radar
 products, however, the analysis was done over a dataset with similar
 climatic and precipitation characteristics as the training dataset: warm
 season Eastern CONUS in 2019. This study expands the analysis to the
 entire contiguous United States, during all seasons, and covering the
 period 2020-2022. Several validation metrics including root-mean-square
 difference (RMSD), probability of detection (POD), and false alarm ratio
 (FAR) are plotted over CONUS by season, day-of-year, and time-of-day, and
 the regional and temporal variations are examined. GREMLIN skill is
 highest in summer and spring, with lower skill in winter due to cold
 surfaces frequently mistaken as precipitating clouds. In summer, diurnal
 patterns of RMSD in different longitude regions follow diurnal patterns of
 precipitation occurrence. GREMLIN’s accuracy is the best over the Central
 to Eastern United States where it has been trained. Over New England,
 GREMLIN POD is lower due to different brightness temperature distributions
 and low frequency of lightning compared to the training data. Over
 Florida, GREMLIN FAR is higher due to the high frequency of lightning.
 Overall, GREMLIN has reliable skill over CONUS in spring, summer, and
 fall, while winter needs more improvements.
description: The methodology is described in detail by Hilburn et al. (2021).
 The ABI, GLM, and MRMS data sets were resampled to a common 3 km grid. A
 cloud height of 10 km was used for removing parallax displacements.
 Satellite and radar samples were matched in time with a maximum time
 difference of 2.5 minutes. GLM lightning groups were accumulated over
 15-minute time periods.
description: Code for reading the data: The data can be read using the
 provided Python code ("read_conus3_file.py" and
 "test_read.py") or the provided Fortran code
 ("gzmodule.f90", "test_read.f90",
 "compile.sh"). Note that this code reads the data without having
 to unzip the files. The “test_read.py” and
 “test_read.f90” use one sample file (ABI_C13_202001010000.bin.gz) to
 verify the code is reading correctly by checking against seven points
 within the image and against the minimum and maximum over the full image.
 This file has been included in the software.tar package.
 To use the Python code, at the command line simply
 invoke: <pre> python test_read.py </pre>
 The function that actually reads a data file, in
 “read_conus3_file.py”, makes use of the gzip module, which is part of the
 Python Standard Library. If your code is reading correctly, you should see
 this output: testfile=
 ABI_C13_202001010000.bin.gz<br>min data, expected, isclose= -1e+30
 -1e+30 True<br>max data, expected, isclose= 296.77463 296.77463
 True<br>data[   0,   0], expected, isclose= -1e+30 -1e+30
 True<br>data[ 529,   0], expected, isclose= -1e+30 -1e+30
 True<br>data[1058,   0], expected, isclose= -1e+30 -1e+30
 True<br>data[   0, 899], expected, isclose= 254.55347 254.55347
 True<br>data[   0,1798], expected, isclose= 295.85233 295.85233
 True<br>data[ 529, 899], expected, isclose= 267.37692 267.37692
 True<br>data[1058,1798], expected, isclose= -1e+30 -1e+30
 True The important thing is that each line ends with
 “True”, which confirms that the values returned by the code match the
 expected values to a precision of 1.E-07. To use the
 Fortran code, first you must compile the code using “compile.sh”, at the
 command line: <pre> ./compile.sh</pre> Note
 that this script specifies a typical path to the zlib library on a Linux
 machine: <pre> /lib64/libz.so.1</pre> If
 your zlib library is in a different location, you must modify this
 path.  Once compiled, you can use the Fortran code by
 invoking at the command line: <pre>
 ./test_read.exe</pre> If your code is reading correctly,
 you should see the output: testfile=ABI_C13_202001010000.bin.gz<br>min gzdata, expected, isclose=  -1.00000002E+30  -1.00000002E+30 T<br>max gzdata, expected, isclose=   296.774628       296.774628     T<br>gzdata(   1,   1), expected, isclose=  -1.00000002E+30  -1.00000002E+30 T<br>gzdata(   1, 530), expected, isclose=  -1.00000002E+30  -1.00000002E+30 T<br>gzdata(   1,1059), expected, isclose=  -1.00000002E+30  -1.00000002E+30 T<br>gzdata( 900,   1), expected, isclose=   254.553467       254.553467     T<br>gzdata(1799,   1), expected, isclose=   295.852325       295.852325     T<br>gzdata( 900, 530), expected, isclose=   267.376923       267.376923     T<br>gzdata(1799,1059), expected, isclose=  -1.00000002E+30  -1.00000002E+30 T The important thing is that each line ends with “T”, which confirms that the values returned by the code match the expected values to a precision of 1.E-07. Once you have confirmed that the code is reading correctly for you, then you can use it in your own code. For Python, this means adding the lines to your code: <pre> from read_conus3_file import read_conus3_file<br> data = read_conus3_file(testfile)</pre> The data are accessible through the NumPy array “data”. For Fortran, this means adding the lines to your code: <pre> include 'gzmodule.f90'<br> use gzmodule, only: read_gzip, gzdata<br> logical :: success<br> call read_gzip(testfile,success)<br> if (.not.success) stop 'error reading testfile'</pre> The data are accessible through the “gzdata” array. Please refer to the “test_read.f90” code for more information about where these statements belong in a Fortran program.