Package: nowcastlib
Nowcast Library🧙‍♂️🔧 Utils that can be reused and shared across and beyond the ESO Nowcast
projectThis is a public repository hosted on GitHub via a push mirror setup in theinternal ESO GitLab repositoryInstallationSimply runpip install nowcastlibUsage and DocumentationNowcast Library (nowcastlib) consists in a collection of functions organized in
submodules (API) and a tool accessible via the command line (CLI). The latter is
primarily intended for accessing the Nowcast Library Pipeline, an opinionated
yet configurable set of processing steps for wrangling data and evaluating
models in a consistent and rigorous way. More information can be found on the
nowcastlib pipeline index page
(link to markdownandlink to hosted docs)Please refer to theexamples folderon GitHub for usage examples.APIHere is a quick example of how one may import nowcastlib and access to one of
the functions:"""Example showing how to access compute_trig_fields function"""importnowcastlibasnclimportpandasaspdimportnumpyasnpdata_df=pd.DataFrame(np.array([[0,3,4,np.NaN],[32,4,np.NaN,4],[56,8,0,np.NaN]]).T,columns=["A","B","C"],index=pd.date_range(start="1/1/2018",periods=4,freq="2min"),)result=ncl.datasets.compute_trig_fields(data_df,["A","C"])More in-depth API documentation can be foundhere.CLISome of the library's functionality is bundled in configurable subcommands
accessible via the terminal with the commandnowcastlib:usage: nowcastlib [-h] [-v]{triangulate,preprocess,sync,postprocess,datapipe} ...positional arguments:{triangulate,preprocess,sync,postprocess,datapipe}available commandstriangulate         Run `nowcastlib triangulate -h` for further helppreprocess          Run `nowcastlib preprocess -h` for further helpsync                Run `nowcastlib sync -h` for further helppostprocess         Run `nowcastlib postprocess -h` for further helpdatapipe            Run `nowcastlib datapipe -h` for further helpoptional arguments:-h, --help            show this help message and exit-v, --verbose         increase verbosity level from INFO to DEBUGRepository StructureThe following output is generated withtree . -I 'dist|docs|*.pyc|__pycache__'.
├──LICENSE
├──Makefile# currently used to build docs├──README.md
├──de421.bsp# not committed├──docs/# html files for the documentation static website├──examples
│├──README.md
│├──cli_triangulate_config.yaml
│├──data/# not committed│├──datasync.ipynb
│├──output/# not committed│├──pipeline_datapipe.json
│├──pipeline_preprocess.json
│├──pipeline_sync.json
│├──signals.ipynb
│└──triangulation.ipynb
├──images
│└──pipeline_flow.png
├──nowcastlib# the actual source code for the library│├──__init__.py
│├──cli
││├──__init__.py
││└──triangulate.py
│├──datasets.py
│├──dynlag.py
│├──gis.py
│├──pipeline
││├──README.md
││├──__init__.py
││├──cli.py
││├──process
│││├──__init__.py
│││├──postprocess
││││├──__init__.py
││││├──cli.py
││││└──generate.py
│││├──preprocess
││││├──__init__.py
││││└──cli.py
│││└──utils.py
││├──split
│││└──__init__.py
││├──structs.py
││├──sync
│││├──__init__.py
│││└──cli.py
││└──utils.py
│├──signals.py
│└──utils.py
├──poetry.lock# lock file generated by python poetry for dependency mgmt└──pyproject.toml# general information file, handled by python poetryDirectories and Files not CommittedThere are a number of files and folders that are not committed due to their
large and static nature that renders them inappropriate for git version control.
The following files and folder warrant a brief explanation.Certain functions (time since sunset, sun elevation) of the Nowcast Library
rely on the use of a .bsp file, containing information on the locations
through time of various celestial bodies in the sky. This file will be
automatically downloaded upon using one of these functions for the first
time.The examples scripts make use of adata/directory containing a series of
csv files. Most of the data used in the examples can be downloaded from theESO Ambient Condition Database.
Users can then change the paths set in the examples to fit their needs. For
users interested in replicating the exact structure and contents of the data
directory, a compressed copy of it (1.08 GB) is available to ESO members
throughthis Microsoft Sharepoint link.At times the examples show the serialization functionality of the nowcastlib
pipeline or need to output some data. In these situations theoutput/directory in the examples folder is used.Development SetupThis repository relies onPoetryfor tracking
dependencies, building and publishing. It is therefore recommended that
developersinstall poetryand
make use of it throughout their development of the project.DependenciesMake sure you are in the right Python environment and runpoetry installThis readspyproject.toml, resolves the dependencies, and
installs them.DeploymentThe repository is published toPyPi, so to make it
accessible via apip installcommand as mentionedearlier.To publish changes follow these steps. Ideally this process is automated via a
CI tool triggered by a push/merge to the master branch:Optionally runpoetry versionwith the
appropriate argument based onsemver guidelines.Update the documentation by runningmake documentPrepare the package by runningpoetry buildEnsure you haveTestPyPiand PyPi configured as
your poetry repositories:poetry config repositories.testpypi https://test.pypi.org/legacy/poetry config repositories.pypi https://pypi.org/Publish the repository to TestPyPi, to see that everything works as expected:poetry publish -r testpypiStage, commit and push your changes (to master) with git.Publish the repository to PyPi:poetry publish -r pypiUpon successful deployment, the library should be available for install viapip