Package: nwp-consumer
NWP CONSUMERMicroservice for consuming NWP data.A microservice for multi-source consumption of NWP data, storing it in a common format. Built with inspiration
from theHexagonal Architecturepattern, the nwp-consumer is
currently packaged with adapters for pulling and converting.gribdata from:MetOffice Atmospheric APICEDA Atmospheric ArchiveECMWF MARS APIDWD's ICON Model from the Opendata APICMC's GDPS Model from the Opendata APINOAA's GFS Model from AWS Open DataNOAA's GFS Model from NCAR's ArchiveSimilarly, the service can write to multiple sinks:Local filesystemAWS S3HuggingFace DatasetsIts modular nature enables straightforward extension to alternate future sources.Running the serviceThe service uses environment variables to configure sources and sinks in accordance with
theTwelve-Factor App methodology.
The program will inform you of missing env vars when using an adaptor, but you can also check theconfigfor the given module, or use theenvcommand.Using DockerThis service is designed to be run as a Docker container. TheContainerfileis the Dockerfile for the service.
It is recommended to run it this way due to the dependency on external non-python binaries, which at the moment
cannot be easily distributed in a PyPi package. To run, pull the latest version fromghcr.iovia:$dockerrun\-v/path/to/datadir:/data\-eENV_VAR=<value>\ghcr.io/openclimatefix/nwp-consumer:latest<command...>Using the Python PackageEnsure theexternal dependenciesare installed. Then, do one of the following:EitherInstall fromPyPIwith$pipinstallnwp-consumerorClone the repository and install the package via$gitclonegit@github.com:openclimatefix/nwp-consumer.git
$cdnwp-consumer
$pipinstall.Then run the service via$ENV_VAR=<value>nwp-consumer<command...>CLIWhether running via Docker or the Python package, available commands can be found with the commandhelpor the--helpflag. For example:$nwp-consumer--help# or$dockerrunghcr.io/openclimatefix/nwp-consumer:latest--helpUbiquitous LanguageThe following terms are used throughout the codebase and documentation. They are defined here to avoid ambiguity.InitTime- The time at which a forecast is initialised. For example, a forecast initialised at 12:00 on 1st
January.TargetTime- The time at which a predicted value is valid. For example, a forecast with InitTime 12:00 on 1st
January predicts that the temperature at TargetTime 12:00 on 2nd January at position x will be 10 degrees.Repository structureProduced usingexa:$exa--tree--git-ignore-F-I"*init*|test*.*"./
├── Containerfile # The Dockerfile for the service
├── pyproject.toml # The build configuration for the service
├── README.md
└── src/
   ├── nwp_consumer/ # The main library package
   │  ├── cmd/
   │  │  └── main.py # The entrypoint to the service
   │  └── internal/ # Packages internal to the service. Like the 'lib' folder
   │     ├── config/ 
   │     │  └── config.py # Contains the configuration specification for running the service
   │     ├── inputs/ # Holds subpackages for each incoming data source
   │     │  ├── ceda/
   │     │  │  ├── _models.py
   │     │  │  ├── client.py # Contains the client and functions to map CEDA data to the service model
   │     │  │  └── README.md # Info about the CEDA data source
   │     │  └── metoffice/
   │     │     ├── _models.py
   │     │     ├── client.py # # Contains the client and functions to map MetOffice data to the service model
   │     │     └── README.md # Info about the MetOffice data source
   │     ├── models.py # Describes the internal data models for the service
   │     ├── outputs/ # Holds subpackages for each data sink
   │     │  ├── localfs/
   │     │  │  └── client.py # Contains the client for storing data on the local filesystem
   │     │  └── s3/
   │     │     └── client.py # Contains the client for storing data on S3
   │     └── service/ # Contains the business logic and use-cases of the application
   │        └── service.py # Defines the service class for the application, whose methods are the use-cases
   └── test_integration/nwp-consumeris structured following principles from the hexagonal architecture pattern. In brief, this means a clear
separation between the application's business logic - it'sCore- and theActorsthat are external to it. In
this package, the core of the service is ininternal/service/and the actors are ininternal/inputs/andinternal/outputs/. The service logic has no knowledge of the external actors, instead defining interfaces that the
actors must implement. These are found ininternal/models.py. The actors are then responsible for implementing these
interfaces, and aredependency-injectedin at runtime. This allows the service to be easily tested and extended. Seefurther readingfor more information.Local developmentClone the repository, and create and activate a new python virtualenv for it.cdto the repository root.Install theExternalandPythondependencies as shown in the sections
below.TaskfileThis repository bundles often used commands into ataskfilefor convenience. To use these commands, ensurego-taskis installed, easily done viahomebrew.You can then see the available tasks using$task-lExternal dependenciesThecfgribpython library depends on the ECMWFcfgribbinary, which is a wrapper around the ECMWFecCodeslibrary.
One of these must be installed on the system and accessible as a shared library.On a MacOS with HomeBrew use$brewinstalleccodesOr if you manage binary packages withCondause$condainstall-cconda-forgecfgribAs an alternative you may install the official source distribution
by following the instructions athttps://confluence.ecmwf.int/display/ECC/ecCodes+installationYou may run a simple selfcheck command to ensure that your system is set up correctly:$python-m<eccodesORcfgrib>selfcheck
Found:ecCodesv2.27.0.
Yoursystemisready.Python requirementsInstall the required python dependencies and make it editable with$pipinstall-e.or use the taskfile$taskinstallThis looks for requirements specified in thepyproject.tomlfile.Note that these are the bare dependencies for running the application. If you want to run tests,
you need the development dependencies as well, which can be installed via$pipinstall-e.[dev]or$taskinstall-devWhere is the requirements.txt file?There is norequirements.txtfile. Instead, the project uses setuptool's pyproject.toml integration to specify
dependencies. This is a new feature of setuptools and pip, and is therecommended wayto specify dependencies.
Seethe setuptools guideandthe PEP621 specificationfor more information, as well asFurther Reading.Running testsEnsure you have installed thePython requirementsand theExternal dependencies.Run the unit tests with$python-munittestdiscover-ssrc/nwp_consumer-p"test_*.py"or$tasktest-unitand the integration tests with$python-munittestdiscover-stest_integration-p"test_*.py"or$tasktest-integrationSeefurther readingfor more information on thesrcdirectory structure.Further readingOn packaging a python project using setuptools and pyproject.toml:The officialPyPA packaging guide.Astep-by-step practical guideon thegodatadrivenblog.The pyproject.tomlmetadata specification.On hexagonal architecture:Aconcrete exampleusing Python.Anoverview of the fundamentalsincorporating TypescriptAnotherexampleusing Go.On the directory structure:The officialPyPA discussionon
src and flat layouts.Contributing and communitySee theOCF Organisation Repofor details on contributing.Find out more about OCF in theMeta Repo.Follow OCF onTwitter.Check out the OCF blog athttps://openclimatefix.org/blogfor updates.