Package: pystatis
pystatispystatisis a Python wrapper for the different GENESIS web service interfaces (API). Currently we are supporting the following databases:GENESIS-OnlineRegionaldatenbankZensus DatenbankThe main features are:Simplified accessto all supported API. No more need to write cumbersome API calls or switch between databases.Credential managementremoves the need to manually add credentials. We handle all your credentials for you.Database managementhandles different databases and lets you switch easily between them.Integrated workflowenables an end-to-end process from finding the relevant data to download it.Pandas supportinstead of manually parsing results.Cachingto enable productive work despite strict query limits.Starting and handling background jobsfor datasets that are to big to be downloaded directly from GENESIS.To learn more about GENESIS refer to the official documentationhere.InstallationYou can install the package viapipinstallpystatisIf everything worked out correctly, you should be able to importpystatislike thisimportpystatisprint("Version:",pystatis.__version__)Getting startedTo be able to use the web service/API of either GENESIS-Online, Regionaldatenbank or Zensus, you have to be a registered user. You can create your userhere,here, orhere.Once you have a registered user, you can use your username and password as credentials for authentication against the web service/API.You can usepystatiswith only one of the supported database or with all, it is simply about providing the right credentials.pystatiswill only use databases for which you have provided credentials.Please follow thisguideto set uppystatiscorrectly.All APIs provide ahelloworldendpoint that can be used to check your credentials.frompystatis.helloworldimportloginchecklogincheck("genesis")If everything worked out, your setup is complete and you can start downloading data.For more details, please study the provided examplesnotebooks.How to useThe GENESIS data modelThe Genesis data structure consists of multiple elements as summarized in the image below.This package currently supports retrieving the following data types:Tables: Main statistical objects provided by the Genesis databases.Find the right datapystatisoffers theFindclass to search for any piece of information with GENESIS. Behind the scene it's using thefindendpoint.Example:frompystatisimportFindresults=Find("Rohöl")# Initiates object that contains all variables, statistics, tables and cubesresults.run()# Runs the queryresults.tables.df# Results for tablesresults.tables.get_code([1,2,3])# Gets the table codes, e.g. for downloading the tableresults.tables.get_metadata([1,2])# Gets the metadata for the tableA complete overview of all use cases is provided in the example notebook forfind.Download dataAt the moment, the package only supports the download of Tables.Example for downloading a Table:frompystatisimportTablet=Table(name="21311-0001")# data is not yet downloadedt.get_data()# Only now the data is either fetched from GENESIS or loaded from cache. If the data is downloaded from online, it will be also cached, so next time the data is loaded from cache.t.data#prettified data stored as pandas data frameFor more details, please study the provided sample notebook fortables.Clear CacheWhen a table is queried, it will be put into cache automatically. The cache can be cleared using the following function:frompystatisimportclear_cacheclear_cache("21311-0001")# only deletes the data for the object with the specified nameclear_cache()# deletes the complete cacheFull documentationThe full documentation of the main and dev branches are hosted viaGitHub Pages (main)andGitHub Pages (dev). It can also be built locally by runningcddocs&&makeclean&&makehtmlfrom the project root directory. Besides providing parsed docstrings of the individual package modules, the full documentation currently mirrors most of the readme, like installation and usage. The mirroring crucially relies on the names of the section headers in the ReadMe, so change them with care!More information on how to use sphinx is providedhere.LicenseDistributed under the MIT License. SeeLICENSE.txtfor more information.RoadmapA few ideas we should implement in the maybe-near future:Mechanism to download data that is newer than the cached version. Right now, once data is cached, it is always retrieved from cache no matter if there is a newer version online. However, this could be quite challenging as the GENESIS API is really bad in providing a good and consistent field for the last update datetime.Improve Table metadata so the user can look up the variables contained in the dataset and for each variable the values that this variable can have.Understand and support time series.How to contribute?Contributions to this project are highly appreciated! You can either contact the maintainers or directly create a pull request for your proposed changes:Fork the ProjectCreate your Feature Branch (git checkout -b feature/<descriptive-name>)Commit your changes (git commit -m 'Added NewFeature')Push to remote (git push origin feature/<descriptive-name>)Open a Pull Request to be merged with devContributor informationTo contribute to this project, please follow these steps:Dev env setupInstallminiforge.Create a new virtual environment usingconda: Runconda create -n pystatis python=3.11. You can choose another Python version as long as it is supported by this package, see the pyproject.toml for supported Python versions.Installpoetryinside your conda environment: Runconda install poetry.Clone the repository via git.Change into the project root directory.Runpoetry installto install all dependencies into the current conda environment (runpoetry env infoto see the details of the current environment). Runpoetry install --with devto receive all additional developer dependencies.poetryhas installed all dependencies for you, as well as the packagepystatisitself.Install pre-commit: Runpoetry run pre-commit install. This will activate the pre-commit hooks that will run prior every commit to ensure code quality.WorkflowCheck out thedevbranch and make sure it is up to date by runninggit pull.Create a new branch by runninggit checkout -b <new-branch>orgit switch -c <new-branch>. If possible, add an issue number to the branch name.Do your changes.Runpoetry run pytestto see if all existing tests still run through. It is important to usepoetry runto callpytestso thatpoetryuses the created virtual environment and not the system's default Python interpreter. Alternatively, you can runpoetry shellto letpoetryactivate the virtual environment for the current session. Afterwards, you can runpytestas usual without any prefix. You can leave the poetry shell with theexitcommand.Add new tests depending on your changes.Runpoetry run pytestagain to make sure your tests are also passed.Commit your changes. This will trigger all pre-commit hooks as defined in.pre-commit-config.yaml. If any of these pre-hooks fails, your commit is declined and you have to fix the issues first.Before you create a PR make sure that you have the latest changes from dev. Rungit switch dev, rungit pull, switch back to your branch withgit switch -and either do agit rebase -i devorgit merge devto get the latest changes in your current working branch. Solve all merge conflicts.Push your final changes.Create a new PR, always againstdevas target.To learn more aboutpoetry, seeDependency Management With Python Poetryby realpython.com.