Package: hydrotools.nwm-client-new
OWPHydroTools :: NWM Client NewThis subpackage implements various interfaces to retrieve National Water Model (NWM) data from various sources including Google Cloud Platform, NOMADS, local directories, or a generic web server directory listing. The primary use for this tool is to populatepandas.Dataframeobjects with NWM streamflow data. See theNWM Client New Documentationfor a complete list and description of the currently available methods. To report bugs or request new features, submit an issue through theOWPHydroTools Issue Trackeron GitHub.InstallationIn accordance with the python community, we support and advise the usage of virtual
environments in any workflow using python. In the following installation guide, we
use python's built-invenvmodule to create a virtual environment in which the
tool will be installed. Note this is just personal preference, any python virtual
environment manager should work just fine (conda,pipenv, etc. ).# Create and activate python environment, requires python >= 3.8$python3-mvenvenv
$sourceenv/bin/activate
$python3-mpipinstall--upgradepipwheel# Install nwm_client$python3-mpipinstallhydrotools.nwm_client_newUsageThe following example demonstrates how one might usehydrotools.nwm_client_newto retrieve NWM streamflow forecasts.CodeRetrieving data from google cloud# Import the NWM Clientfromhydrotools.nwm_client_new.NWMFileClientimportNWMFileClient# Instantiate model data client#  By default, NWM values are in SI units#  If you prefer US standard units, nwm_client can return#  values in US standard units by setting the unit_system parameter#  to MeasurementUnitSystem.US## from hydrotools.nwm_client_new.NWMClientDefaults import MeasurementUnitSystem# model_data_client = NWMFileClient(unit_system=MeasurementUnitSystem.US)model_data_client=NWMFileClient()# Retrieve forecast dataforecast_data=model_data_client.get(configurations=["short_range"],reference_times=["20210101T01Z"],nwm_feature_ids=[724696])# Look at the dataprint(forecast_data.head())Example outputreference_time  nwm_feature_id          value_time      value measurement_unit variable_name configuration usgs_site_code0 2021-01-01 01:00:00          724696 2021-01-01 02:00:00  56.340000           m3 s-1    streamflow   short_range       010135001 2021-01-01 01:00:00          724696 2021-01-01 17:00:00  56.090000           m3 s-1    streamflow   short_range       010135002 2021-01-01 01:00:00          724696 2021-01-01 16:00:00  56.119999           m3 s-1    streamflow   short_range       010135003 2021-01-01 01:00:00          724696 2021-01-01 15:00:00  56.149998           m3 s-1    streamflow   short_range       010135004 2021-01-01 01:00:00          724696 2021-01-01 14:00:00  56.180000           m3 s-1    streamflow   short_range       01013500Retrieving data from Azure Blob Storage# Import the NWM Clientfromhydrotools.nwm_client_new.NWMFileClientimportNWMFileClientfromhydrotools.nwm_client_new.AzureFileCatalogimportAzureFileCatalogimportpandasaspd# Instantiate model data clientcatalog=AzureFileCatalog()model_data_client=NWMFileClient(catalog=catalog)# Set reference timeyesterday=pd.Timestamp.utcnow()-pd.Timedelta("1D")# Retrieve forecast dataforecast_data=model_data_client.get(configurations=["short_range"],reference_times=[yesterday],nwm_feature_ids=[724696])# Look at the dataprint(forecast_data.head())Example outputreference_time  nwm_feature_id          value_time      value measurement_unit variable_name configuration usgs_site_code0 2022-08-07 18:00:00          724696 2022-08-07 19:00:00  20.369999           m3 s-1    streamflow   short_range       010135001 2022-08-07 18:00:00          724696 2022-08-08 10:00:00  24.439999           m3 s-1    streamflow   short_range       010135002 2022-08-07 18:00:00          724696 2022-08-08 09:00:00  24.469999           m3 s-1    streamflow   short_range       010135003 2022-08-07 18:00:00          724696 2022-08-08 08:00:00  24.490000           m3 s-1    streamflow   short_range       010135004 2022-08-07 18:00:00          724696 2022-08-08 07:00:00  24.510000           m3 s-1    streamflow   short_range       01013500Retrieving data from Nomads# Import the NWM Clientfromhydrotools.nwm_client_new.NWMFileClientimportNWMFileClientfromhydrotools.nwm_client_new.HTTPFileCatalogimportHTTPFileCatalogimportpandasaspd# Instantiate model data clientcatalog=HTTPFileCatalog("https://nomads.ncep.noaa.gov/pub/data/nccf/com/nwm/prod/")model_data_client=NWMFileClient(catalog=catalog)# Set reference timeyesterday=pd.Timestamp.utcnow()-pd.Timedelta("1D")# Retrieve forecast dataforecast_data=model_data_client.get(configurations=["short_range"],reference_times=[yesterday],nwm_feature_ids=[724696])# Look at the dataprint(forecast_data.head())Example outputreference_time  nwm_feature_id          value_time      value measurement_unit variable_name configuration usgs_site_code0 2022-08-07 18:00:00          724696 2022-08-07 19:00:00  20.369999           m3 s-1    streamflow   short_range       010135001 2022-08-07 18:00:00          724696 2022-08-08 10:00:00  24.439999           m3 s-1    streamflow   short_range       010135002 2022-08-07 18:00:00          724696 2022-08-08 09:00:00  24.469999           m3 s-1    streamflow   short_range       010135003 2022-08-07 18:00:00          724696 2022-08-08 08:00:00  24.490000           m3 s-1    streamflow   short_range       010135004 2022-08-07 18:00:00          724696 2022-08-08 07:00:00  24.510000           m3 s-1    streamflow   short_range       01013500Retrieving data from a private file server# Import the NWM Clientfromhydrotools.nwm_client_new.NWMFileClientimportNWMFileClientfromhydrotools.nwm_client_new.HTTPFileCatalogimportHTTPFileCatalogfromhydrotools.nwm_client_new.NWMClientDefaultsimportMeasurementUnitSystemimportssl# Create ssl contextcontext=ssl.create_default_context(cafile="/path/to/my/ca-bundle.crt")# Instantiate model data clientcatalog=HTTPFileCatalog("https://path-to-my-private-server.com/nwm/2.2/",ssl_context=context)model_data_client=NWMFileClient(catalog=catalog,unit_system=MeasurementUnitSystem.US,ssl_context=context)# Retrieve forecast dataforecast_data=model_data_client.get(configurations=["short_range"],reference_times=["2022-06-01T13"],nwm_feature_ids=[724696])# Look at the dataprint(forecast_data.head())Example outputreference_time  nwm_feature_id          value_time        value measurement_unit variable_name configuration usgs_site_code0 2022-06-01 13:00:00          724696 2022-06-01 14:00:00  3586.910645           ft^3/s    streamflow   short_range       010135001 2022-06-01 13:00:00          724696 2022-06-02 05:00:00  2167.260986           ft^3/s    streamflow   short_range       010135002 2022-06-01 13:00:00          724696 2022-06-02 04:00:00  2168.673584           ft^3/s    streamflow   short_range       010135003 2022-06-01 13:00:00          724696 2022-06-02 03:00:00  2172.558350           ft^3/s    streamflow   short_range       010135004 2022-06-01 13:00:00          724696 2022-06-02 02:00:00  2177.855469           ft^3/s    streamflow   short_range       01013500System RequirementsWe employ several methods to make sure the resultingpandas.DataFrameproduced bynwm_clientare as efficient and manageable as possible. Nonetheless, this package can potentially use a large amount of memory.The National Water Model generates multiple forecasts per day at over 3.7 million locations across the United States. A single forecast could be spread across hundreds of files and require repeated calls to the data source. The intermediate steps of retrieving and processing these files into leanerDataFramemay use several GB of memory. As such, recommended minimum requirements to use this package are a 4-core consumer processor and 8 GB of RAM.DevelopmentThis package uses a setup configuration file (setup.cfg) and assumes use of thesetuptoolsbackend to build the package. To install the package for development use:$python3-mvenvenv
$sourceenv/bin/activate
$python3-mpipinstall-Upip
$python3-mpipinstall-Usetuptools
$python3-mpipinstall-e.[develop]To generate a source distribution:$python3-mpipinstall-Uwheelbuild
$python3-mbuildThe packages generated indist/can be installed directly withpipor uploaded to PyPI usingtwine.