Package: nowcasting-dataset
PLEASE NOTE THAT THIS IS DEPRECATED IN FAVOUR OFOCF_DATAPIPES!nowcasting_datasetPre-prepare batches of data for use in machine learning training.This code combines several data sources including:Satellite imagery (EUMETSAT SEVIRI RSS 5-minutely data of UK)Numerical Weather Predictions (NWPs.  UK Met Office UKV model from CEDA)Solar PV power timeseries data (from PVOutput.org, downloaded using
ourpvoutput Python code.)Estimated total solar PV generation for each of the ~350 "grid supply points"
(GSPs) in Britain fromSheffield Solar's PV Live Regional API.Topographic data.The Sun's azimuth and angle.This repo doesn't contain the ML models themselves.  Please seethis
page for an overviewof
the Open Climate Fix solar PV nowcasting project, and how our code
repositories fit together.User manualInstallationcondaFrom within the clonednowcasting_datasetdirectory:condaenvcreate-fenvironment.yml
condaactivatenowcasting_dataset
pipinstall-e.pipA (probably older) version is also available throughpip install nowcasting-datasetPV Live APIIf you want to also installPVLivethen usepip install git+https://github.com/SheffieldSolar/PV_Live-APIPre-commitA pre commit hook has been installed which makesblackrun with every commit. You need to installblackandpre-commit(these will be installed bycondaorpipwhen installingnowcasting_dataset) and runpre-commit installin this repo.TestingTo test using the small amount of data stored in this repo:py.test -sTo output debug logs while running the tests then runpy.test --log-cli-level=10To test using the full dataset on Google Cloud, add the--use_cloud_dataswitch.dockerTest using a docker file and databasedocker stop $(docker ps -a -q)
docker-compose -f test-docker-compose.yml build
docker-compose -f test-docker-compose.yml run datasetDownloading dataSatellite dataUseSatipto download
native EUMETSAT SEVIRI RSS data from EUMETSAT's API and then convert
to an intermediate file format.PV data from PVOutput.orgDownload PV timeseries data from PVOutput.org usingour PVOutput code.OCF uk_pv datasetPV solar generation data from the UK. This dataset contains data from 1311 PV systems from 2018-01-01 to 2021-10-27. The time series of solar generation is in 5 minutes chunks. This data is collected from live PV systems in the UK. We have obfuscated the location of the PV systems for privacy.Numerical weather predictions from the UK Met OfficePlease use ournwpcode to download UKV NWPs and convert to Zarr.GSP-level estimates of PV outturn from PV Live RegionalTODO - GSPTopographical dataMake an account at theUSGS EarthExplorerwebsiteCreate a region of the world to download data for, in our case, the spatial extant of the SEVIRI RSS imageSelect the data products you want, in this case SRTM elevation mapsDownload all the SRTM files that cover that areaThere does not seem to be an automated way to do this selecting and downloading, so this might take awhile.Configurenowcasting_datasetto point to the downloaded dataCopy and modify one of the config yaml files innowcasting_dataset/config/.Prepare ML batchesRunscripts/prepare_ml_data.py --helpto learn how to run theprepare_ml_data.pyscript.What exactly is in each batch?Please see thedata_sources/<modality>/<modality>_model.pyfiles
(where<modality>is one of {datetime, metadata, gsp, nwp, pv,
satellite, sun, topographic}) for documentation about the different
data fields in each example / batch.History of nowcasting_datasetWhen we first started writingnowcasting_dataset, our intention was
to load and align data from these three datasets on-the-fly during ML
training.  But it just isn't quite fast enough to keep a modern GPU constantly fed
with data when loading multiple satellite channels and multiple NWP
parameters.  So, now, this code is used to pre-prepare thousands of
batches, and save these batches to disk, each as a separate NetCDF
file.  These files can then be loaded super-quickly at training time.
The end result is a 12x speedup in training.Contributors ✨Thanks goes to these wonderful people (emoji key):Jack Kelly💻Jacob Bieker💻Peter Dudfield💻Flo💻Rohan Nuttall💻Nasser Benabderrazik💻Shanmukh Chava💻Rishi Kumar Ray💻JanEbbing💻This project follows theall-contributorsspecification. Contributions of any kind welcome!