Package: ternyxmimosa
mimosaThe ELT part of a modern data stack with practical data pipelines using cloud functionality.Explore the docs »Report Bug·Request FeatureTable of ContentsAbout the ProjectGetting StartedPrerequisitesAPI KeysInstallationUsageRoadmapContributingContact(back to top)About the ProjectThe ELT part of a modern data stack with practical data pipelines and reporting using cloud functionality. This is similar in concept tomimodastusing alternative software options and cloud functionality.Mimosa encompasses the ELT (extract load transform) components necessary to generate the webpage found atgas.aspireto.win, providing detailed reports on natural gas storage volumes within the European Union. This process involves retrieving data from a REST API, transforming it, and storing it in a database tailored for reporting purposes.The source data is published byGas Infrastructure Europeand exposed in a REST API.Beyond gas storage data, Mimosa offers a hands-on experience with essential tools:🚀dltfor smooth data loading.🔍dbtfor powerful data transformation.☁️MotherDuckfor storing the data in a cloud basedDuckDBdatabase.Further the full tech stack used to create thegas.aspireto.winpages is detailedbelow.(back to top)Getting StartedPrerequisitesSetup a Python development environment.API KeysEnsure the following sensitive information is securely stored in environment variables or within a .env file:To access theGIE Gas InventoryREST API, an API key is necessary. Quickly obtain your API key by signing up for a freeGIE account. Once acquired, expose it using the following environment variable:ENV_GIE_XKEY = "YOUR-API-KEY"ForMotherDuck, you'll need theservice tokenand the database name. Set up the following environment variables to establish the connection:DESTINATION__MOTHERDUCK__CREDENTIALS = "md:///YOUR-DATABASE-NAME?token=YOUR-SERVICE-TOKEN"Please note that theMotherDuckpage utilizes a different format, whereas the above format is specifically required fordlt.(back to top)InstallationExecute the following command. Consider using a venv.pipinstallternyxmimosaAlternatively clone this repository and usepoetry install. Or pip install from GitHub.(back to top)UsageCommand LineNot currently supported.As a Python PackageThe following sample obtains the storage data for the last available date and stores it in MotherDuck.importmimosa.cliasGEIGEI.main()Tech StackThese are the technologies driving the content ongas.aspireto.win:Google cloudfunctionfor the ELT component:The function is a bare bones wrapper around themimosaPython package (the current repository). The function is in thisrepository.It is scheduled to run the ELT twice daily (using Google Scheduler andPub/Submessage).The result is updated data in MotherDuck.Reporting notebookBuilt using theevidencereporting tool, defined in this GitHubrepository.Rebuild and published to a web host using a GitHub workflow.Run on a twice daily schedule. The workflow is defined in the notebookrepository.NOTE: As of November 2023 it is possible to fully deploy this stack without breaking the bank (using free tiers of the cloud services used). Dive into our GitHub repository and the linked ones for the Google Function and Evidence notebook, where all the code awaits. 🚀(back to top)NOTE: For some reason the environment variable DESTINATION__MOTHERDUCK__CREDENTIALS is oftentimes incorrectly set between runs when using the dev container. Useunset DESTINATION__MOTHERDUCK__CREDENTIALSto clear the environment variable.RoadmapConsider:Get source data (Using REST API)Transform data, possibly SQL Mesh or dbt.Create data vault transformations (https://automate-dv.readthedocs.io/en/latest/).dlt update/error messages using SlackStorage (currently local DuckDB, maybe consider some cloud alternative. Though that would stray from the data stack in a Docker concept.) (MotherDuck)Scheduling Tool (Google Cloud Scheduler)Reporting tool (Metabase?) (Evidence.dev in separate repository)Bare bones CLI(back to top)ContributingAny contributions you make aregreatly appreciated.If you have a suggestion that would make this better, please fork the repo and create a pull request. You can alsoopena feature request or bug report.
Don't forget to give the project a star! Thanks again!Fork the ProjectCreate your Feature Branch (git checkout -b feature/AmazingFeature)Commit your Changes (git commit -m 'Add some AmazingFeature')Push to the Branch (git push origin feature/AmazingFeature)Open a Pull Request(back to top)ContactProject Link:mimosa(back to top)