DOI: 10.5061/dryad.jm63xsjmd
title: Supplementary data for: Nowcasting 3D cloud fields using forward warping optical flow

description: Large- to global-domain short-term prediction of clouds (0-3 hours), or
 cloud nowcasting, remains relevant to civilian and military applications
 ranging from solar energy production to intelligence gathering. 
 Despite the capabilities of contemporary numerical weather prediction
 models, nowcasting methods based on near real-time observations (i.e.
 satellite imagery) hold operational value due to their relative
 computational efficiency and accuracy for short-term
 applications.  A commonly used nowcasting approach involves using
 two or more images to retrieve the apparent motions of features, or
 optical flow, which can be used to extrapolate the future location of
 those features.  However, such approaches generally assume that
 the optical flow field remains unchanged with respect to time which is
 challenging to apply to piecewise cloud fields from satellite
 imagery.  Here, we propose a method to nowcast clouds that adapts
 a computer vision technique for image interpolation, commonly referred to
 as warping, to account for temporal changes to optical flow fields derived
 from infrared satellite imagery. We evaluate the proposed method for 991
 randomly selected regional cases from 2024 and perform a detailed analysis
 on three specific cases. Applying a dense (every image pixel) optical flow
 retrieval technique to full-disk GOES infrared imagery, we demonstrate
 that forward warping of the optical flow field when coupled with simple
 occlusion reasoning, improves skill in cloud nowcasting.
description: Explanation of code: Nowcast_Example.py provides an example of
 how to run the I-NOW nowcasting method on GOES-16 Band 13 data. Code
 downloads GOES-16 Band 13 data from AWS, runs a python version of OCTANE
 optical flow code to calculate the optical flow using the values contained
 in the JTEC article. Rather than linear extrapolation, the code uses a
 warping method formed from computer vision temporal interpolation methods
 to account for occlusions and time related changes to the optical flow
 field in the forward time direction rather than a time that is
 intermediate of two images. The code provides nowcasts of GOES-16 10.3
 micron brightness temperature,CLAVR-x derived cloud top heights (CTH), and
 CLAVR-x derived cloud base heights (CBH). Observed brightness temperature
 data and CLAVR-x data are plotted alongside the nowcast results in
 10-minute intervals. The code also creates animated gifs of the nowcasts
 to easier visualize the results through time. The code is designed to be
 run on a GPU with CUDA support, and the user can define the save directory
 for the results to be saved and data that is downloaded. CLAVR-x data is
 not downloadable from the internet,so the repository contains a folder
 with CLAVR-x data for the example date and time used in the code. The
 CLAVR-x files have been reduced to contain only the required CTH and CBH
 data instead of full CLAVR-x retrieval data to save on storage space in
 the repository.
description: # Supplementary data for: Nowcasting 3D cloud fields using forward warping
 optical flow DOI:
 [https://doi.org/10.5061/dryad.jm63xsjmd](https://doi.org/10.5061/dryad.jm63xsjmd) Dryad data contains only the observed CLAVR-x (Clouds from AVHRR Extended) cloud property retrievals, namely cloud top height (CTH) and cloud base height (CBH) derived from GOES-16 ABI (Advanced Baseline Imager) imagery. These data are utilized in the example script (contained in Zenodo:[https://doi.org/10.5281/zenodo.16740427](https://doi.org/10.5281/zenodo.16740427)) that produces full-disk nowcasts (short-term forecasts 0 - 3 hours) using the I-NOW (Improved Nowcasting via Optical flow Warping) technique presented in "Nowcasting 3D Cloud Fields Using Forward Warping Optical Flow" by Matthew King, Jason Apke, Steven Miller, Katherine Haynes, Yoo-Jeong Noh, and John Haynes, 2025: J. Atmos Ocea. Tech. ## Description of the data and file structure There are 19 .hdf files that all have the same naming convention. The files start with "clavrx_goes16_" which references that they are CLAVR-x retrieval files for the GOES-16 Geostationary Satellite. The next set of of numbers is the 4 digit year (e.g. 2023). The next set of numbers is the 3 digit Julian day of the year (e.g. 016). The next set of numbers represents the time of the satellite imagery scan (e.g. 160020), where the first two digits represents the hour, the next two digits represents the minute, and the last two digits represent milliseconds. The file then ends with ".level2.hdf" which indicates that is level 2 data and are in .hdf file format. The following files are contained in clavrx_files.zip: clavrx_goes16_2023_016_160020.level2.hdf clavrx_goes16_2023_016_161020.level2.hdf clavrx_goes16_2023_016_162020.level2.hdf clavrx_goes16_2023_016_163020.level2.hdf clavrx_goes16_2023_016_164020.level2.hdf clavrx_goes16_2023_016_165020.level2.hdf clavrx_goes16_2023_016_170020.level2.hdf clavrx_goes16_2023_016_171020.level2.hdf clavrx_goes16_2023_016_172020.level2.hdf clavrx_goes16_2023_016_173020.level2.hdf clavrx_goes16_2023_016_174020.level2.hdf clavrx_goes16_2023_016_175020.level2.hdf clavrx_goes16_2023_016_180020.level2.hdf clavrx_goes16_2023_016_181020.level2.hdf clavrx_goes16_2023_016_182020.level2.hdf clavrx_goes16_2023_016_183020.level2.hdf clavrx_goes16_2023_016_184020.level2.hdf clavrx_goes16_2023_016_185020.level2.hdf clavrx_goes16_2023_016_190020.level2.hdf To access the contents of a file, xarray may be used in Python to open the file in the following manner: import xarray as xr file_dir = 'clavrx_goes16_2023_016_160020.level2.hdf' Clavrx_xarray = xr.open_dataset(file_dir, engine='netcdf4) CBH_values = CLavrx_xarray['cld_height_base'].data CTH_values = Clavrx_xarray['cld_height_acha'].data ## Sharing/Access information The CLAVR-x software, which was utilized to create the retrieval data contained in the Dryad repository, is available for download from: [https://cimss.ssec.wisc.edu/cspp/clavrx_v3.0.shtml](https://cimss.ssec.wisc.edu/cspp/clavrx_v3.0.shtml) GOES imagery files, which are required for CLAVR-x to retrieve cloud properties, are available from NOAA's Registry of Open Data on AWS: [https://registry.opendata.aws/noaa-goes/](https://registry.opendata.aws/noaa-goes/) ## Code/Software The script (and relvant python files) which produces the I-NOW nowcasts and utilizes the CLAVR-x data are available both on Zenodo ([https://doi.org/10.5281/zenodo.16740427](https://doi.org/10.5281/zenodo.16740427)) and GitHub ([https://github.com/AFwxking/I-NOW](https://github.com/AFwxking/I-NOW)). CLAVR-x files contained in Dryad are also available on Zenodo as well as GitHub. The following is information for the aforementioned script: Requires: ``` - Python 3.12.2 (version used for development, other versions may work) - Data files which include CLAVRx_data and GOES Full Disk Band 13 data (code to download GOES data from AWS is included) - Python scripts contained in OCTOPY folder (contains scripts to run python version of OCTANE optical flow code and the I-NOW nowcast method which requires temporal interpolation scripts) - Code requires GPU with support for CUDA (developed on NVIDIA RTX A6000) - Data files within the DOI in data availability statement of manuscript (CLAVRx CTH/CBH data saved in clavrx_data folder on GitHub Repo) ``` Dependencies: netcdf4, numba, cudatookit, s3fs, matplotlib, cartopy, xarray, metpy ``` - netcdf4, numba, and cudatoolkit required to run nowcasts - s3fs required to download GOES data from AWS - xarray, matplotlib, cartopy, and metpy required in code to plot results ``` Recommend using a virtual environment to run this code, such as conda environment.yml file provided in directory for easy installation of all dependencies ``` $ conda env create -f environment.yml ``` Should the environment.yml file fail to install workable environment, environment can be installed using the commands below in the order they are provided. ``` create an anaconda environment: $ conda create -n INOW_env python=3.12.2 $ conda activate INOW_env Install Netcdf4 first, this one does not like to play nice with the others: $ conda install -c conda-forge netcdf4 Install numba $ conda install -c numba numba Install the cudatoolkit, use the anaconda distribution, the conda-forge one has a driver issue $ conda install -c anaconda cudatoolkit Now matplotlib, use the pip-install for this $ python -m pip install -U matplotlib Now cartopy $ conda install -c conda-forge cartopy Now xarray $ conda install -c conda-forge xarray Now metpy $ conda install -c conda-forge metpy Now s3fs $ conda install -c conda-forge s3fs ``` Example run commnand (unix): ``` $ python Nowcast_Example.py ``` Code Author: Matthew P. King, Lt Col, USAF Date: 27 June 2025 Version: 1.0 Explanation of code: Nowcast_Example.py provides an example of how to run the I-NOW nowcasting method on GOES-16 Band 13 data. Code downloads GOES-16 Band 13 data from AWS, runs a python version of OCTANE optical flow code to calculate the optical flow using the values contained in the JTEC article. Rather than linear extrapolation, the code uses a warping method formed from computer vision temporal interpolation methods to account for occlusions and time related changes to the optical flow field in the forward time direction rather than a time that is intermediate of two images. The code provides nowcasts of GOES-16 10.3 micron brightness temperature,CLAVR-x derived cloud top heights (CTH), and CLAVR-x derived cloud base heights (CBH). Observed brightness temperature data and CLAVR-x data are plotted alongside the nowcast results in 10-minute intervals. The code also creates animated gifs of the nowcaststo easier visualize the results through time. The code is designed to be run on a GPU with CUDA support, and the user can define the save directory for the results to be saved and data that is downloaded. CLAVR-x data is not downloadable from the internet,so the GitHub repository contains a folder with CLAVR-x data for the example date and time used in the code that has been reduced to contain only the required CTH and CBH data instead of full CLAVR-x retrieval data.