Package: pmps
python ml skeleton projectgeneric skeleton for machine learning project with python, hydra, pytest, sphinx, github actions, etc.
with dummy functionalities!
It is mostly oriented geospatial projectsWhy this project?The goal of this project is to present a standard architecture of python repository/package
including a full CiCd pipeline to document/test/deploy your project with standard methods
of 2022. It can be used as starting point for any project without reinventing the wheel.The code has no interest!The code of this project is totally dummy: it makes simple
mathematics operations like addition and subtration!
The next iteration will make the opetations more interesting by
using multi-layers perceptron! It will try to add a complete example of Hydra
configuration.In a close future, it will serve as a demonstrator by the example
of a standard ML pipeline for experimentation and productionInstallationInstall requirementsAs Gdal dependencies are presents it's preferable to
install dependencis via conda before installing the package:gitclonehttps://github.com/samysung/python_ml_project_skeletoncdpython_ml_project_skeleton/packagingcondaenvcreate-fpackage_env.ymlFrom pip:pipinstallpmps
orpipinstallpmps==vx.x# for a specific versionOther installation optionsFrom source:pythonsetup.pyinstallFrom source with symbolic links:pipinstall-e.From source using pip:pipinstallgit+https://github.com/samysung/python_ml_project_skeletonProject Architecture├──CHANGELOG.rst
├──.codecov.yml
├──deploy
│└──dockerfile
├──docs
│├──add.rst
│├──build.sh
│├──changelog.rst
│├──conf.py
│├──deploy.sh
│├──index.rst
│├──Makefile
│├──readme_link.md
│└──_static
│└──img
├──.github
│└──workflows
│├──publish.yml
│├──test_code.yml
│├──test_docs.yml
│├──test_packaging.yml
│└──test_publish.yml
├──.gitignore
├──LICENSE
├──packaging
│├──doc_env.yml
│├──doc_requirements.txt
│├──package_env.yml
│├──requirements.txt
│├──test_env.yml
│└──test_requirements.txt
├──pmps
│├──api
││├──add.py
││├──__init__.py
││└──subtract.py
│├──core
││├──add.py
││├──__init__.py
││└──subtract.py
│└──__init__.py
├──.pre-commit-config.yaml
├──.pylintrc
├──README.md
├──readthedocs.yml
├──setup.cfg
├──setup.py
├──tests
│├──api
││├──__init__.py
││├──test_add.py
││└──test_subtract.py
│└──__init__.py
└──VERSIONArchitecture component overviewComponentPathDescriptionPython Packagepmps/where the python executable code is localized. It is your root package as it's the first directory to contain ainit.py and its name is generally the one you choose for your publishing package (the one build and published on forge like pypi conda, etc. Don't forget for any subpackage to add aninit.py module to declare it as python package. NB: separate core and api in different sub package is a design choice not standard, it comes from java world but a lot of python project prefers declaring private python modules.Documentationdocs/the source code of your documentation: conf.py is where you configure your sphinx doc, _static/ for your additional statis files (img, text, icon, video, etc.), doc is built under docs/_build/html but can be modified in maekfile.Tests Packagetests/where you organize the test code of your executable code. Your unit tests (pytest is the library used) should at least test what you expose to your clients, you can add static analysis of your tests code with extentions like mypy and flake8. Use the pytest-cov extension to produce test cover reporting.Python Envpackaging/Place for your conda environment files and requirement files.Deploymentdeploy/Place for Dockerfiles or any other deployment solutionCI/CD workflows.github/github workflows configuration files (details below)CD (Documentation publishing).readthedocs.ymlconfiguration of the documentation publication on readthedocs (seereadthedocs link)CI (tests covering publishing).codecov.ymlconfiguration of the code covering pubication on codecov (seecodecov)CI (static analysis publishing).pre-commit.ymlconfiguration of the pre-commit publication (seepre-commit)CD (packaging)setup.cfg and setup.pyconfiguration files for packaging on pipy, local, etc (seepython doc)CI/CD pipelineThe first and essential goal is to have a skeleton quickly editable for a lot of use case projects
with a big emphasis on continuous integration and continuous deployment.
Here is a schematic view of the Ci/Cd pipeline targeted for
open source python project, largely inspired by others well
known projects:DIAGRAMGithub Workflowstest code worflow (.github/workflows/test_code.yml):Used to run unit test (and functionnal if implemented) tests on pull request events
or push on main branch. It publishes coverage results on codecov.io.
Use the packaging/test_env.yml conda environment file, github cache action and codecov/codecov-actiontest docs workflow (.github/workflows/test_docs.yml):Used to test the build of sphinx documentation. Run on pull request events
or push on main branch.
Use the packaging/doc_env.yml conda environment file, and the github cache action.publish workflow (.github/workflows/publish.yml):Used to publish the package on pypi, when a new tagged version or release is published.
Use the packaging/package_env.yml conda environment file, github cache action github download
and upload artifacts, and gh-action-pypi-publish.test publish workflow (.github/workflows/test_publish.yml):Same worflow as above, but on a test branch and test.pypi forge,
for testing deployment improvement recipestest packaging workflow (.github/workflows/test_packaging.yml):Worflow actioned by CRON event (seecrontab-guru), every n hours. Used
to test that the package has been published and the lasted version is working.Github workflows based on github webhooks or githu AppsSome workflow works are handled by third party applications, like the readthe docs publication or the online
pre-commit static analysis.Pre-commitPre-commit action is launched via a github app (pre-commit.ci) on every commit made on remote.
it's configured via the file pre-commit-config.yamlRead-the-docs publicationReadthedocs publish new documentation version via a webhook subscribed for push and commit event.
You can configure the type of push trigering the process in the readthedocs.org configuration section.
See read the docs documentation for more detail.