Package: nowcasting-forecast
Nowcasting ForecastMaking live forecasts for the nowcasting project.
The aim is to produce Grid Supply Point (GSP) Solar generation Forecasts.
This is done by loading live PV results, Numerical Weather Predictions (NWPs) and Satellite images,
and running this data through various different ML models.You can run the application locally by:pythonnowcasting_forecast/app.py--db-url='sqlite:///test.db'Installation with condamambaenvcreate-fenvironment.yml
condaactivatenowcasting_forecast# Installing fastai inside environment.yml is currently broken. So install separately:mambainstall-cfastchanfastai# You need to manually install nowcasting_dataset, nowcasting_datamodel,# nowcasting_dataloader, and power_perceiver. This can be done via pypi (for all except# power_perceiver) or by git cloning the repo and using:pipinstall-e<path_to_directory># Install nowcasting_forecastpipinstall-e.# You may also want to install dev tools:mambainstallpytestflake8blackpre-commitpydocstyleisortmypy
pre-commitinstallDirectories and filesThe following shows the main files+-- nowcasting_forecast
|   +-- config
|       +-- mvp_v0.yaml
|       +-- mvp_v1.yaml
|   +-- models
|       +-- nwp_solar_simple.py
|       +-- nwp_solar_simple_trained
|           +-- model.py
|       +-- cnn
|           +-- cnn.py
|           +-- dataloader.py
|           +-- model.py
|   +-- app.py
|   +-- batch.py
|   +-- dataloader.py
+-- scripts
+-- tests☀️ nowcasting_forecastapp.pyis the main entry point for running the forecast module. It contains the following arguments:--db-url: the database url that the forecasts will be saved too--fake: Create fake forecast, mainly used for testingThe app has three main stepsMake batched dataRun forecasting model to make forecastsSave forecasts to the databasebatch.pyis used to load the data and create batches of data. This uses mainly usedManagerLivedataloader.pyis used to load the batched data in an efficient way. This is current a stripped down version of thisdataloader.Theconfigdirectory contains configurations that are used to load different data sources.
The configuration is used bybatch.pyto convert large amounts of data into batched data, which is then ready for ML models.Thedatabasedirectory contains database models and functions to interact with the database. Seebelowfor more informationThemodeldirectory contains ml models to make forecasts. Seebelowfor more informationModelsNWP SimpleThis takes the average 'dswrf' for each example and then divides this by 10 to give a rough model for MW per GSPconfiguration: mvp_v0.yamlNWP Simple trainedCNN model of 'dswrf' channel in NWP. 6 CNN layers then 4 fully connected layers.Training run:https://app.neptune.ai/o/OpenClimateFix/org/predict-pv-yield/e/PRED-951/chartsconfiguration: mvp_v1.yamlCNNThis model takes both satellite and NWP video data and puts them through
separate 3D convolutional neural networks. These are then connected with
a few fully connected layers, joined with some simple input data like
historic PV data.🩺 TestingTests are run withpytestThese sets uppostgresin a docker container.
This slightly more complicated testing framework is needed (compared to runningpytest)
as some queries can not be fully tested on asqlitedatabase🛠️ infrastructure.github/workflowscontains a number of CI actionslinters.yaml: Runs linting checks on the coderelease.yaml: Make and pushes docker files on a new code releasetest-docker.yaml': Runs tests on every pushThe docker file is in the folderinfrastructure/docker/The version is bumped automatically for any push tomain.Environmental VariablesDB_URL: The database url which the forecasts will be saved tooDB_URL_PV: The database url for the PV dataNWP_ZARR_PATH: Override NWP data path. This is useful when running this locally, and shows to get data from the cloud.SATELLITE_ZARR_PATH: Override Satellite data path. This is useful when running this locally, and shows to get data from the cloud.FAKE: Option to make fake/dummy forecastsMODEL_NAME: Optional of 'nwp_simple' or 'nwp_simple_trained'