Package: nowcasting-datamodel
nowcasting_datamodelDatamodel for the nowcasting projectThe data model has been made usingsqlalchemywith a mirrored model inpydantic.⚠️ Database tables are currently made automatically,
but in the future there should be a migration processFuture: The data model could be moved, to be a more modular solution.nowcasting_datamodelmodels.pyAll models are innowcasting_datamodel.models.py.
The diagram below shows how the different tables are connected.connection.pynowcasting_datamodel.connection.pycontains a connection class which can be used to make a sqlalchemy session.fromnowcasting_datamodel.connectionimportDatabaseConnection# make connection objectdb_connection=DatabaseConnection(url='sqlite:///test.db')# make sessionswithdb_connection.get_session()assession:# do something with the databasepass👓 read.pynowcasting_datamodel.read.pycontains functions to read the database.
The idea is that these are easy to use functions that query the database in an efficient and easy way.get_latest_forecast: Get the latestForecastfor a specific GSP.get_all_gsp_ids_latest_forecast: Get the latestForecastfor all GSPs.get_forecast_values: Gets the latestForecastValuefor a specific GSPget_latest_national_forecast: Returns the latest national forecastget_location: Gets aLocationobjectfromnowcasting_datamodel.connectionimportDatabaseConnectionfromnowcasting_datamodel.readimportget_latest_forecast# make connection objectdb_connection=DatabaseConnection(url='sqlite:///test.db')# make sessionswithdb_connection.get_session()assession:f=get_latest_forecast(session=session,gsp_id=1)💾 save.pynowcasting_datamodel.save.pyhas one functions to save a list ofForecastto the database🇬🇧 national.pynowcasting_datamodel.fake.pyhas a useful function for adding up forecasts for all GSPs into a national Forecast.fake.pynowcasting_datamodel.fake.pyFunctions used to make fake model data.🩺 TestingTests are run by using the following commanddockerstop$(dockerps-a-q)docker-compose-ftest-docker-compose.ymlbuild
docker-compose-ftest-docker-compose.ymlruntestsThese sets uppostgresin a docker container and runs the tests in another docker container.
This slightly more complicated testing framework is needed (compared to runningpytest)
as some queries can not be fully tested on asqlitedatabaseMac M1 usersAn upstream builds issue of libgp may cause the following error:sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SCRAM authentication requires libpq version 10 or aboveAs suggested in thisthread, a temporary fix is to set the env variableDOCKER_DEFAULT_PLATFORM=linux/amd64prior to building the test images - although this reportedly comes with performance penalties.🛠️ infrastructure.github/workflowscontains a number of CI actionslinters.yaml: Runs linting checks on the coderelease.yaml: Make and pushes docker files on a new code releasetest-docker.yaml': Runs tests on every pushThe docker file is in the folderinfrastructure/docker/The version is bumped automatically for any push tomain.Environmental VariablesDB_URL: The database url which the forecasts will be saved tooContributors ✨Thanks goes to these wonderful people (emoji key):Brandon Ly💻Chris Lucas💻James Fulton💻This project follows theall-contributorsspecification. Contributions of any kind welcome!