Package: thyme
thymeTools forHydrodynamicModel OutputExtractionA Python package providing utilities for processing, interpolating, and
converting hydrodynamic ocean model NetCDF datasets.Why?There are many hydrodynamic modeling frameworks in use today by the
oceanographic community. All models are output to a NetCDF format, but
the structure, dimensions, variables, attributes, naming conventions,
coordinate systems, and masking rules can all vary significantly, making
it difficult for users to extract meaningful information from these
complex systems.This package attempts to support a common methodology for interacting
with these datasets by abstracting the nuances of each modeling system
into distinct modules and unifying them with a single API.Supported ModelsNOTE: To date, this package has been tested to work with National Ocean
Service (NOS) Operational Forecast Systems (OFS) (e.g. CBOFS, DBOFS,
NGOFS, etc.), however with minor tweaks it should also work with output
from non-NOS OFS models.The following ocean modeling frameworks are presently supported:Regional Ocean Modeling System (ROMS)"ROMS is a free-surface, terrain-following, primitive equations
ocean model widely used by the scientific community for a diverse
range of applications...In the horizontal, the primitive equations
are evaluated using boundary-fitted, orthogonal curvilinear
coordinates on a staggered Arakawa C-grid. The general formulation of
curvilinear coordinates includes both Cartesian (constant metrics) and
spherical (variable metrics) coordinates. Coastal boundaries can also
be specified as a finite-discretized grid via land/sea masking. As in
the vertical, the horizontal stencil utilizes a centered, second-order
finite differences."Description fromhttps://www.myroms.org/Finite-Volume Community Ocean Modeling System (FVCOM)"FVCOM is a prognostic, unstructured-grid, finite-volume,
free-surface, 3-D primitive equation coastal ocean circulation model
developed by UMASSD-WHOI joint efforts. The model consists of
momentum, continuity, temperature, salinity and density equations and
is closed physically and mathematically using turbulence closure
submodels. The horizontal grid is comprised of unstructured triangular
cells and the irregular bottom is presented using generalized
terrain-following coordinates."Description fromhttp://fvcom.smast.umassd.edu/fvcom/Princeton Ocean Model (POM)"POM is a sigma coordinate (terrain-following), free surface ocean
model with embedded turbulence and wave sub-models, and wet-dry
capability."Description fromhttp://www.ccpo.odu.edu/POMWEB/index.htmlHybrid Coordinate Ocean Model (HYCOM)"The HYbrid Coordinate Ocean Model is a primitive equation ocean
general circulation model that evolved from the Miami
Isopycnic-Coordinate Ocean Model (MICOM) developed by Rainer Bleck and
colleagues. Vertical coordinates in HYCOM remain isopycnic in the
open, stratified ocean. However, they smoothly transition to z
coordinates in the weakly-stratified upper-ocean mixed layer, to
terrain-following sigma coordinate in shallow water regions, and back
to level coordinates in very shallow water."Description fromhttps://www.hycom.org/attachments/067_overview.pdfAdditionally, a genericrectilinearmodule has been created to support
any model output whose coordinate system conforms to a rectilinear grid
and whose depth coordinates reflect standard depths/z-levels, however
this module does not support vertical interpolation.FeaturesSupport for ROMS, FVCOM, POM, HYCOM model outputInterpolate staggered horizontal coordinates (i.e. ROMS Arakawa-C
grid rho/eta/xi) to common coordinates before further processingApply spatially-varying rotation angle to u/v current components to
obtain true-north/true-east values before further processingInterpolate sigma (bathymetry-following) vertical coordinates to a
given depth-below-surface, respecting the appropriate vertical
transformation, if anyGiven an approximate target grid resolution, generate a regular grid
definition conforming to the model domain's bounding box (or
optionally to a predefined set of bounding rectangles), and output
regular grid definition to CF-compliant NetCDFModel ConsiderationsAll modules inthyme/models/are written specifically for
NOAA hydrodynamic ocean models and should be used with caution.
The modules were written by referencing NOS model metadata, NOS model
guidance, and roms, pom, fvcom, and hycom model documentation.To develop a custom module to support a new model, use one of the existing modules
(fvcom.py, hycom.py, pom.py, rectilinear.py, or roms,py) as a template, and place
the new python module file alongside the others under thethyme/model/folderDevelopment of a custom module may be required to support:Model output options specific to your organizationDifferent variable names and dimensionsDifferent vertical coordinatesDifferent variable masksDifferent vertical or time varying horizontal masksDifferent date and time formatIf the model has already been converted to a regular or rectilinear gridIf the model output's vertical coordinate system uses standard depth levels versus sigmaIndex (Grid Definition) FilesIn order to convert model output files to a regular grid, an index file
must be supplied at runtime. The purpose of the index file is to persist
information that does not change between model runs in order to reduce
the overall processing time per cycle run. The information stored in the
index file includes the output grid definition and general metadata
about the model itself. Index files for FVCOM-based models with a hybrid
(generalized) coordinate system additionally store interpolated vertical
coordinate values for each output grid cell.When generating a grid definition, the user has a choice between using
the full extent of the model's domain or supplying a shapefile
containing one or more polygons defining subgrids
(sub-regions/subdomains) to which the grid definition will conform. If
no subgrid shapefile is specified, the resulting index file will define
a regular grid (with NoData mask) matching the full model domain
(extent). Otherwise, if a subgrid shapefile is specified, the resulting
index file will define a regular grid (with NoData mask) matching the
unified extent of all subgrid polygons that intersect the model domain,
along with information identifying the grid cell ranges that correspond
with each subgrid. Note that when supplying a subgrid shapefile, all
subgrid polygons must be rectangular, congruent with each other, and
adjacent to one another.A subgrid index is intended to be used to subset the model output into
smaller geographic areas (i.e. tiles), which in turn results in smaller
output file sizes. Optionally, for a subgrid index file, the user may
specify the name of an attribute field within the subgrid shapefile that
uniquely identifies each subgrid (tile). If no field is specified, each
polygon's FID value is used as the identifier. This identifier can be
used to construct unique filenames.Additionally, a land mask polygon shapefile can be supplied when
generating an index file. If supplied, any output grid cells whose
centroid intersects a land polygon will be masked in the final grid
definition.One index file must be created per ocean forecast system for each
combination of target resolution and extent (whether using the model's
full domain extent or subgrid definition).When to Generate a New Index FileOnce a model index file is created, it can be reused indefinitely for
that model/resolution/extent/land mask until any of those properties
change. For example, if an FVCOM-based model has a hybrid (generalized)
vertical coordinate system that is modified at some point (i.e., sigma
values are changed), any associated index files will need to be
regenerated using a new model output file in the updated format.Generally, a new index file is required:For each new modelFor each desired output grid resolutionFor each desired set of subgridsIf the subgrid polygons changeIf the subgrid attribute identifier changesIf the land mask shapefile changesIf the underlining model changes (e.g., new geographic extent,
change to FVCOM sigma coordinates, etc.)RequirementsThis codebase is written for Python 3 and relies on the following python
packages:gdalnetCDF4numpyscipyshapelyInstallationThe GDAL Python bindings used by this package require system libraries to be
present, so it usually can't just be installed usingpip install gdal.
We recommend installing GDAL either through a package manager (e.g.conda,apt,yum,pacman) or by compiling from scratch.Minicondais probably
the easiest method.Oncegdalhas been installed, thyme can be installed usingpip:pipinstallthymeExample UsageTo generate a new index file for an FVCOM-based model using the default
grid extent, a ~500 meter target resolution, and a shoreline shapefile
defining land areas to be masked:fromthyme.modelimportfvcomnative_model_file=fvcom.FVCOMFile('/path/to/existing_fvcom_file.nc')model_index_file=fvcom.FVCOMIndexFile('/path/to/new_index_file.nc')try:native_model_file.open()model_index_file.open()model_index_file.init_nc(native_model_file,500,'my_fvcom_model','/path/to/shoreline_shapefile.shp')finally:model_index_file.close()native_model_file.close()To generate a new index file for a ROMS-based model using a subgrid
shapefile (with fieldname 'id' used to identify subgrid areas) and a
~300m target resolution (with no shoreline mask shapefile specified):fromthyme.modelimportromsnative_model_file=roms.ROMSFile('/path/to/existing_roms_file.nc')model_index_file=roms.ROMSIndexFile('/path/to/new_index_file.nc')try:native_model_file.open()model_index_file.open()model_index_file.init_nc(native_model_file,300,'my_roms_model',None,'/path/to/subgrid_shapefile.shp','subgrid_id_fieldname')finally:model_index_file.close()native_model_file.close()To interpolate u/v current components from a ROMS-based model to a
regular grid defined in an existing model index file, at a depth of 4.5
meters below surface, for time index 0, and store the resulting u/v
values in twonumpy.ma.masked_arrayobjects:fromthyme.modelimportromsnative_model_file=roms.ROMSFile('/path/to/existing_roms_file.nc')model_index_file=roms.ROMSIndexFile('/path/to/existing_index_file.nc')try:native_model_file.open()model_index_file.open()(u_with_mask,v_with_mask)=native_model_file.uv_to_regular_grid(model_index_file,0,4.5)# u_with_mask and v_with_mask now contain 2D numpy masked arraysfinally:model_index_file.close()native_model_file.close()Running TestsThis project usespytestfor unit testing.To run the test suite:pipinstallpytest
pytest-vvAuthorsErin Nagel (UCAR),erin.nagel@noaa.govJason Greenlaw (ERT),jason.greenlaw@noaa.govLicenseThis work, as a whole, is licensed under the BSD 2-Clause License (seeLICENSE), however it contains major contributions from the
U.S. National Oceanic and Atmospheric Administration (NOAA), 2017 -
2019, which are individually dedicated to the public domain.DisclaimerThis repository is a scientific product and is not official
communication of the National Oceanic and Atmospheric Administration, or
the United States Department of Commerce. All NOAA GitHub project code
is provided on an "as is" basis and the user assumes responsibility
for its use. Any claims against the Department of Commerce or Department
of Commerce bureaus stemming from the use of this GitHub project will be
governed by all applicable Federal law. Any reference to specific
commercial products, processes, or services by service mark, trademark,
manufacturer, or otherwise, does not constitute or imply their
endorsement, recommendation or favoring by the Department of Commerce.
The Department of Commerce seal and logo, or the seal and logo of a DOC
bureau, shall not be used in any manner to imply endorsement of any
commercial product or activity by DOC or the United States Government.AcknowledgmentsThis software has been developed by the National Oceanic and Atmospheric
Administration (NOAA)/National Ocean Service (NOS)/Office of Coast
Survey (OCS)/Coast Survey Development Lab (CSDL) for use by the
scientific and oceanographic communities.CSDL wishes to thank the following entities for their assistance:NOAA/NOS/Center for Operational Oceanographic Products and Services
(CO-OPS)Canadian Hydrographic Service (CHS)Teledyne CARIS