Package: mhered-test-pkg
mhered-test-pkgA simple demo package to practice creating python packages.Inspired in this article:https://mathspp.com/blog/how-to-create-a-python-package-in-2022The code implements a simple Rock, Paper, Scissors text-based game, loosely inspired in the one Al Sweigart wrote for Chapter 2 ofAutomate the boring stuff with Python.Installation:$pipinstallmhered-test-pkgUsage:$rpsAlternatively$python3-mmhered_test_pkgCreating a python package: HowtoPick a nameCheck that the name is available inPyPIInitializepoetryInstallpoetry(I started fromherethen modified the instructions):$curl-sSLhttps://install.python-poetry.org/|python3-Create a local folder for the project and initializepoetryinside$cd~
$mkdirmhered-test-pkg
$cdmhered-test-pkg

$poetrynew.
Createdpackagemhered_test_pkgin.

$tree
.
├──mhered_test_pkg
│└──__init__.py
├──pyproject.toml
├──README.rst
└──tests├──__init__.py└──test_mhered_test_pkg.pyNote: I renamedREADME.rsttoREADME.mdto work inMarkdown.$mvREADME.rstREADME.mdRunpoetry installto create a filepoetry.lockwith the dependencies:$poetryinstallInitialize git in the local folderCreate an empty github repo:mhered-test-pkgand follow the instructions to set it as the remote and push a first commit with the local file structure:$gitinit
$gitadd*
$gitcommit-m"First commit"$gitbranch-Mmain
$gitremoteaddoriginhttps://github.com/mhered/mhered-test-pkg.git
$gitpush-uoriginmainSet uppre-commithooksAddpre-commitas a development dependency, then commit updates:$poetryadd-Dpre-commit

$gitaddpoetry.lockpyproject.toml
$gitcommit-m"Add pre-commit devt dependency."Create a file.pre-commit-config.yamlin the root:# See https://pre-commit.com for more information# See https://pre-commit.com/hooks.html for more hooksrepos:-repo:https://github.com/pre-commit/pre-commit-hooksrev:v4.0.1hooks:-id:check-toml-id:check-yaml-id:end-of-file-fixer-id:mixed-line-ending-repo:https://github.com/psf/blackrev:22.3.0hooks:-id:black-repo:https://github.com/PyCQA/isortrev:5.10.1hooks:-id:isortargs:["--profile","black"]Activate thepoetryvirtual environment to be able to usepre-committhen install the hooks and run them once:$poetryshell

$pre-commitinstall
$pre-commitrun--all-filesNote:pre-commitis not found unless run from inside the shell - or you can usepoetry run pre-commitCommit the changes (including the updates toREADME.md):$gitadd*
$gitcommit-m"Run all pre-commits."$gitpushNote: If some test fails the commit is cancelled and the files are automatically modified, so you need to add again the modified files and repeat the git commit - which is fine. However there is a strange behavior if the file was open: it seems to revert to an older version?Add a licenseAdd a license from the github repothen pull changes to local.This will addLICENSE.mdUpload the TestPyPIDeclare the test repositoryhttps://test.pypi.orginpoetryand name ittestpypi:$poetryconfigrepositories.testpypihttps://test.pypi.org/legacy/Create an account on TestPyPI, go to Account Settings to get an API token and then configurepoetryto use it:$poetryconfighttp-basic.testpypi__token__pypi-YOUR-TESTPYPI-API-TOKENNote: Be careful not to expose your API token, e.g. I wrote it to asecrets.mdfile then used.gitignoreso as not to commit and publish it publicly.Build and upload the package:$poetrybuild
$poetrypublish-rtestpypiWith this our package is live in TestPyPI:https://test.pypi.org/project/mhered-test-pkg/Note: Build creates thedist/folder that should be added to.gitignore$echodist/>>.gitignore

$gitadd.
$gitcommit-m"Publish to TestPyPI"$gitpushPopulate the package with codeFor this example I wrote a simple Rock, Paper, Scissors game inspired and slightly refactored from the example proposed by Al Sweigart in his great bookAutomate the boring stuff with Python. The code goes inmhered-test-pkg/__init__.py.Changelog managementAddscrivfor changelog management, as a development dependency with the[toml]extra :$ poetry add -D scriv[toml]Configurescrivto useMarkdownand add version numbering in the title by adding the following lines to thepyproject.tomlfile, refer toscriv's readthedocs:[tool.scriv]format="md"version="literal: pyproject.toml: tool.poetry.version"Then create the default directory for changelog fragmentschangelog.d/, and add an empty.gitkeepfile so that git tracks the empty folder.$mkdirchangelog.d
$touchchangelog.d/.gitkeep

$gitaddpyproject.tomlpoetry.lockchangelog.d/.gitkeep
$gitcommit-m"Add scriv as devt dependency."Create a new.mdfragment file in thechangelog.dfolder:$scrivcreateEdit it to add a description of the changes:### Added-A first simple implementation of Rock Paper ScissorsUpdateREADME.mdand commit everything:$gitaddREADME.mdchangelog.d/*__init__.py
$gitcommit-m"Simple Rock Paper Scissors game"Publish the package to PyPICreate aPyPIaccount and API token, and configurepoetryto use it:$poetryconfigpypi-token.pipypypi-YOUR-PYPI-API-TOKENBuild and publish:$poetrypublish--buildDo a victory lapInstall, import and uninstall the package (outside of the shell) to check it works. Note: the hyphens (-) in the package name turn into underscores (_) in the module name.$pipinstallmhered-test-pkg
Defaultingtouserinstallationbecausenormalsite-packagesisnotwriteable
Collectingmhered-test-pkgUsingcachedmhered_test_pkg-0.1.0-py3-none-any.whl(2.5kB)Installingcollectedpackages:mhered-test-pkg
Successfullyinstalledmhered-test-pkg-0.1.0

$python3-mmhered_test_pkg
ROCK,PAPER,SCISSORS0Wins,0Losses,0Ties
Enteryourmove:(r)ock(p)aper(s)cissorsor(q)uit
r
ROCKversus...SCISSORS
Youwin!1Wins,0Losses,0Ties
Enteryourmove:(r)ock(p)aper(s)cissorsor(q)uit
q
Bye!

$pipuninstallmhered-test-pkgPublish a releaseAdd a description, installation and usage instructions in theREADME.mdand declare it inpyproject.toml:readme="README.md"Makescrivcollect the previously created changelog fragment to a newCHANGELOG.mdfile with:$scrivcollectLets commit:$gitaddCHANGELOG.mdREADME.mdpyproject.toml
$gitcommit-m"Prepare release 0.1.0"Tag the commit, and push the tag to the remote (seenhere):$gittag-av0.1.0-m"Initial version"$gitpushoriginv0.1.0I discovered I hadn't properly configured version numbering for scriv ( had forgotten to addversion = "literal: pyproject.toml: tool.poetry.version"under[tool.scriv]inpyproject.toml), so I did it now, and added a new release to test it. I bumped the version inpyproject.tomlto0.1.1with:$poetryversionpatch
Bumpingversionfrom0.1.0to0.1.1Next add a changelog fragment:$scrivcreateand edit it to describe the change### Fixed-Configure`scriv`to get version number from`pyproject.toml`Then I manually increased the version inmhered-test-pkg/__init__.py__version__ = "0.1.1"and added a test to check it is always in sync withtool.poetry.versioninpyproject.toml(there seems to be no better way)importtomlfrompathlibimportPathimportmhered_test_pkgdeftest_versions_are_in_sync():""" Checks if tool.poetry.version in pyproject.toml and__version__ in mhered_test_pkg.__init__.py are in sync."""path=Path(__file__).resolve().parents[2]/"pyproject.toml"pyproject=toml.loads(open(str(path)).read())pyproject_version=pyproject["tool"]["poetry"]["version"]init_py_version=mhered_test_pkg.__version__assertinit_py_version==pyproject_versionAdd a new changelog fragment and edit it to describe the change$scrivcreate--edit## Added-Test to check that versions defined in`pyproject.py`and`__init__.py`are in syncUpdate the Changelog:$scrivcollectCommit and push, tag and push:$gitaddpyproject.tomlmhered_test_pkg/__init__.pytests/test_mhered_test_pkg.pyCHANGELOG.mdREADME.md

$gitcommit-m"Configure versions in scriv"$gitpush

$gittag-av0.1.1-m"Configure versions in scriv"$gitpushoriginv0.1.1

$scrivgithub-release-vDEBUG
debug:Runningcommand'git tag'debug:Commandexitedwith0status.Output:'v0.1.0\nv0.1.1\n'debug:Runningcommand['git','config','--get-regex','remote[.].*[.]url']debug:Commandexitedwith0status.Output:'remote.origin.url https://github.com/mhered/mhered-test-pkg.git\n'debug:StartingnewHTTPSconnection(1):api.github.com:443
debug:https://api.github.com:443"GET /repos/mhered/mhered-test-pkg/releases HTTP/1.1"200600warning:Version0.1.1hasnotag.Noreleasewillbemade.
warning:Version0.1.0hasnotag.Noreleasewillbemade.It still does not work... to be continued.TroubleshootingThere is something off with the tagv0.1.0. Was linked to8ad878fwhich does not show here!$gitlog--oneline5979855(HEAD->main,origin/main)UpdateREADME.md
fa7a843(tag:v0.1.1)Configureversionsinscriv
74b8fcbPreparerelease0.1.09519229AddvictorylaptoREADME.md
c22f560Move__init__.pytomhered_test_pkg/
066172eMinorupdatestoREADME.md
e438080SimpleRockPaperScissorsgame
2c93077Addscrivasdevtdependency
262bb9ePublishtoTestPyPI
dd9cfd9Add.gitignoretoprotectAPItokens
81fa08dMinorchangeofREADME.md
4d8b935UpdateLicenseinREADME.md
e18df98CreateLICENSE.md9400227UpdateREADME.md
3d3a9d8Runallpre-commits.
8de2b5aAddpre-commitdevtdependency
9743aefFirstcommitlets repair the tagv0.1.0:$gittag-dv0.1.0# delete the old tag locallyDeletedtag'v0.1.0'(was8ad878f)$gitpushorigin:refs/tags/v0.1.0# delete the old tag remotelyTohttps://github.com/mhered/mhered-test-pkg.git-[deleted]v0.1.0
$gittag-av0.1.074b8fcb# make a new tag locally$gitpushoriginv0.1.0# push the new local tag to the remoteVoilá:$gitlog--oneline5979855(HEAD->main,origin/main)UpdateREADME.md
fa7a843(tag:v0.1.1)Configureversionsinscriv
74b8fcb(tag:v0.1.0)Preparerelease0.1.09519229AddvictorylaptoREADME.md
c22f560Move__init__.pytomhered_test_pkg/
066172eMinorupdatestoREADME.md
e438080SimpleRockPaperScissorsgame
2c93077Addscrivasdevtdependency
262bb9ePublishtoTestPyPI
dd9cfd9Add.gitignoretoprotectAPItokens
81fa08dMinorchangeofREADME.md
4d8b935UpdateLicenseinREADME.md
e18df98CreateLICENSE.md9400227UpdateREADME.md
3d3a9d8Runallpre-commits.
8de2b5aAddpre-commitdevtdependency
9743aefFirstcommitWrite testsI added a couple of tests, refactored a bit the code and committed the changes.Then, to create a release there are a few steps:bump the  versionedit__init__.pymanually to sync version numberrun the testscreate a fragment - with--editoption to launch the editor directlycollect all fragments toCHANGELOG.mdcommit changes topyproject.toml README.md CHANGELOG.md mhered_test_pkg/__init__.pycreate tagpush commit and tag$gitcommit-a-m"Add tests"$poetryversionpatch
$atommhered_test_pkg/__init__.py
$pytest=============================testsessionstarts=============================platformlinux--Python3.8.10,pytest-5.4.3,py-1.11.0,pluggy-0.13.1
rootdir:/home/mhered/mhered-test-pkg
collected4itemstests/test_mhered_test_pkg.py....[100%]==============================4passedin0.02s==============================$scrivcreate--edit
$scrivcollect

$gitcommit-m"Prepare release 0.1.2"$gitpush

$gittag-a0.1.2-m"Add tests"$gitpushorigin0.1.2try to create a release withscriv github-releaseThis time$ scriv github-release -v  DEBUGgives an error message so I rename tag0.1.2tov0.1.2:$gittagv0.1.20.1.2^{}$gittag-d0.1.2
$gitpushorigin:refs/tags/0.1.2
$gitlog--oneline
aa3e44a(HEAD->main,tag:v0.1.2,origin/main)Preparerelease0.1.2
b03efa8Addtests
d688927Passesbasictests5979855UpdateREADME.md
fa7a843(tag:v0.1.1)Configureversionsinscriv
74b8fcb(tag:v0.1.0)Preparerelease0.1.09519229AddvictorylaptoREADME.md
...Back to normal:$scrivgithub-release-vDEBUG
debug:Runningcommand'git tag'debug:Commandexitedwith0status.Output:'v0.1.0\nv0.1.1\nv0.1.2\n'debug:Runningcommand['git','config','--get-regex','remote[.].*[.]url']debug:Commandexitedwith0status.Output:'remote.origin.url https://github.com/mhered/mhered-test-pkg.git\n'debug:StartingnewHTTPSconnection(1):api.github.com:443
debug:https://api.github.com:443"GET /repos/mhered/mhered-test-pkg/releases HTTP/1.1"200717warning:Version0.1.2hasnotag.Noreleasewillbemade.
warning:Version0.1.1hasnotag.Noreleasewillbemade.
warning:Version0.1.0hasnotag.Noreleasewillbemade.Apparently this is because I need to add my PAT as environment variableGITHUB_TOKEN, seescriv docs. I tried though, and it does not work... I create the release manually in github.Automating withtoxtoxautomates testing, linting, formatting, test coverage, documentation, etc.We add tox as a development dependency:$poetryadd-DtoxConfiguration is done in atox.inifile intomlformat, which can be initiated in its simplest form running$ tox-quickstartand answering a few questions.# tox (https://tox.readthedocs.io/) is a tool for running tests# in multiple virtualenvs. This configuration file will run the# test suite on all supported python versions. To use it, "pip install tox"# and then run "tox" from this directory.[tox]envlist=py37[testenv]deps=pytestcommands=pytestHowever this simplest version did not work. I had to do a few iteratons:Addedisolated_build = Trueto work withpoetryAddedtomlto dependencies otherwisepytestcannot import it. It seems thattoxcreates its own virtual environments and you need to add all dependencies again...Added formatting withblack, linting withflake8,pylint,mccabe, sorting of imports withisort, and testing withpytest.mccabeis aflake8plugin to check the code's McCabe complexity (should be <10).black --checkdoes not modify the files, only exits with an error if the check is not passed. What is the point?Result:# tox (https://tox.readthedocs.io/) is a tool for running tests# in multiple virtualenvs. This configuration file will run the# test suite on all supported python versions. To use it, "pip install tox"# and then run "tox" from this directory.[tox]isolated_build=Trueenvlist=py38[testenv]deps=tomlblackflake8isortmccabepylintpytestcommands=black--checkmhered_test_pkgisort--checkmhered_test_pkgflake8mhered_test_pkg--max-complexity10pylintmhered_test_pkgpytest.Execute with:$toxQuestion: why usetoxwhen I can use apoetryscript or even better apre-commithook?Check coverageInstallation and basic execution ofcoverage:$poetryadd-Dcoverage
$coveragerun-mpytest
$coveragereport
NameStmtsMissCover
---------------------------------------------------
mhered_test_pkg/__init__.py493822%
tests/__init__.py00100%
tests/test_mhered_test_pkg.py150100%
---------------------------------------------------
TOTAL643841%We can execute the checks in a more nuanced way including all files in the package and all branching paths, and we can also generate nicer HTML reportssuch as this oneas follows:$coveragerun--source=mhered_test_pkg--branch-mpytest.
$coveragehtmlTo addcoveragetotoxmodifytox.inito add the relevant lines:# tox (https://tox.readthedocs.io/) is a tool for running tests# in multiple virtualenvs. This configuration file will run the# test suite on all supported python versions. To use it, "pip install tox"# and then run "tox" from this directory.[tox]isolated_build=Trueenvlist=py38[testenv]deps=tomlblackflake8isortmccabepylintpytestcoverage# development dependencycommands=black--checkmhered_test_pkgisort--checkmhered_test_pkgflake8mhered_test_pkg--max-complexity10pylintmhered_test_pkgcoveragerun--source=mhered_test_pkg--branch-mpytest.# executecoveragereport-m--fail-under90# report & fail below 90%Added tests to increase coverage up to 97% - i.e. all lines of code covered except the case"__name__" == "__main__:"because tests import as module.In the process I learned about monkeypatching user input, using iterators to simulate a sequence of inputs, or testing for sys exit, see references in./tests/test_mhered_test_pkg.pyUsual ritual to create a release. As some of the steps modify files that need to be committed this process ends up being iterative. I try to keep it clean using profusely$ git commit --amendalong the way:$scrivcreate--edit# describe changes in a fragment$poetryversionpatch# bump version$atommhered_test_pkg/__init__.py# sync _version__$scrivcollect# update CHANGELOG.md$gitadd.$gitcommit
$gittag-av0.1.3-m"97% test coverage"$gitpush
$gitlog--oneline
91c4aa1(HEAD->main,tag:v0.1.3,origin/main)0.1.3automatedwithtoxand97%coverage
d6670c4Automatingwithtox
028028aUpdateREADME.md
aa3e44a(tag:v0.1.2)Preparerelease0.1.2
b03efa8Addtests
d688927Passesbasictests5979855UpdateREADME.md
fa7a843(tag:v0.1.1)Configureversionsinscriv
74b8fcb(tag:v0.1.0)Preparerelease0.1.0
...
$gitpushoriginv0.1.3CI/CD with GitHub ActionsGitHub Actionsallow automating workflows of actions that are triggered by certain events e.g. a commit pushed to the repo, a pull request or a release. They are defined in YAML files that live in the directory.github/workflows.The CI runner spawns the full environment in a Github server, including setting up the OS withruns-on:, installing and activating python withuses: actions/setup-python@v2...with:...python-version:..."3.8", installing dependencies (viatoxorpoetrycommands) and downloading our repository withuses: actions/checkout@v2A nice intro tutorial in Youtube:https://www.youtube.com/watch?v=R8_veQiYBjI# .github/workflows/CI.yamlname:mhered-test-pkg CIon:push:branches:[main]pull_request:branches:[main]workflow_dispatch:jobs:build:runs-on:ubuntu-lateststrategy:matrix:python-version:["3.8","3.9","3.10"]steps:-name:Checkout sourcesuses:actions/checkout@v2-name:Setup Pythonuses:actions/setup-python@v2with:python-version:${{ matrix.python-version }}-name:Install dependenciesrun:|python -m pip install --upgrade pippython -m pip install tox tox-gh-actions-name:Run toxrun:tox-name:Upload coverage to Codecovuses:codecov/codecov-action@v2with:fail_ci_if_error:trueEverything seems to be working including Codecov integration except PR comments.Automatic publishing of releases to PyPII wrote a newPyPI_publish.yamlfile with the following steps:Checkout the repoSet up Python 3.8Install Poetry and dependenciesConfigure Poetry with a PyPI tokenBuild and publish the packageA more straighforward solution using a pre-made GH action from PyPA to upload both to Test PyPI and PyPI is here:https://packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/Note: add the Test PyPI and PyPI credentials created earlier as repository secrets in Github asPYPI_TOKENandTEST_PYPI_TOKEN:Settings-->Secrets-->Actions-->Add new repository secret.# .github/workflows/PyPI_publish.yamlname:publish mhered-test-pkg to PyPIon:release:types:[published]branches:[main]workflow_dispatch:jobs:build-and-publish:runs-on:ubuntu-lateststeps:-name:Checkout sourcesuses:actions/checkout@v2-name:Setup Pythonuses:actions/setup-python@v2with:python-version:"3.8"-name:Install poetry and dependenciesrun:|python -m pip install --upgrade pippython -m pip install poetry-name:Configure poetryenv:pypi_token:${{ secrets.PYPI_TOKEN }}run:poetry config pypi-token.pypi $pypi_token-name:Build and publishrun:poetry publish --buildNew release:$poetryversionpatch
$atommhered_test_pkg/__init__.py
$pytest
$scrivcreate--edit
$scrivcollect
$gitadd.
$gitcommit-m"Add CI/CD and Codecov - release 0.1.4"$gitpush

$gittag-av0.1.4-m"Add GH actions for CI/CD and Codecov integration"$gitpushoriginv0.1.4Did not work initially. I found three typos  inPyPI_publish.yaml, should be:secrets.PYPI_TOKENinstead ofsecrets.PyPI_TOKENrun: poetry config pypi-token.pypi $pypi_tokeninstead ofpypi_token.pypirun: poetry publish --buildinstead ofuses:I also added minor updates toREADME.mdand I created a few releases in the process.$poetryversionpatch
$atommhered_test_pkg/__init__.py
$pytest
$scrivcreate--edit
$scrivcollect
$gitadd.
$gitcommit-m"Commit Message - release 0.1.X"$gitpush

$gittag-av0.1.X-m"Release Message"$gitpushoriginv0.1.XFinally succeeded withv0.1.7Add badges to README.mdPiece of cake following the instructions inhttps://shields.io/Move code to/srcAd adapt everywhere:in tests change the import statement:fromsrc.mhered_test_pkgimport...intox.inireplacemhered_test_pkgbysrc...commands=black--checksrcisort--checksrcflake8src--max-complexity10pylintsrccoveragerun--source=src--branch-mpytest.coveragereport-m--fail-under60coveragexml...Add entry pointsIn development, as an alternative to calling:$python3./src/mhered_test_pkg/__init__.pyWe can create a__main__.pyfile:""" __main__ entry point """frommhered_test_pkgimportrock_paper_scissorsif__name__=="__main__":rock_paper_scissors()to allow calling the package as a module:$python3-msrc.mhered_test_pkgAnd/oreditpyproject.pyto add the following line:[tool.poetry.scripts]rps="mhered_test_pkg.__init__:rock_paper_scissorsto create a shortcut to execute the game writing:$rpsNote that when the app is distributed the methods that work are different! - cfr. the ones described at the beginning of theREADME.mdTime to make a new release 0.1.8...Tidy upI noticed some issues due to discrepancies between the linting in pre-commit and the one in tox. For instance isort and black seem to be doing incompatible changes in one of the improts in tests.To add all the linting to pre-commit, we edit.pre-commit-config.yaml:# See https://pre-commit.com for more information# See https://pre-commit.com/hooks.html for more hooksrepos:-repo:https://github.com/pre-commit/pre-commit-hooksrev:v4.0.1hooks:-id:check-toml-id:check-yaml-id:end-of-file-fixer-id:mixed-line-ending-repo:https://github.com/psf/blackrev:22.3.0hooks:-id:black-repo:https://github.com/PyCQA/isortrev:5.10.1hooks:-id:isortargs:["--profile","black"]-repo:https://github.com/PyCQA/flake8rev:4.0.1hooks:-id:flake8additional_dependencies:[mccabe]args:["--max-line-length","88","--max-complexity","10"]-repo:https://github.com/PyCQA/pylint/rev:v2.14.5hooks:-id:pylintexclude:tests/# Prevent files in tests/ to be passed in to pylint.Modifytox.inito create dedicated environments for linting and coverage and execute them only with python 3.8 to avoid running them multiple times. The linting one simple calls pre-commit:# tox (https://tox.readthedocs.io/) is a tool for running tests# in multiple virtualenvs. This configuration file will run the# test suite on all supported python versions. To use it, "pip install tox"# and then run "tox" from this directory.[tox]isolated_build=Trueenvlist=py38,py39,py310,linting,coverage,[testenv]deps=tomlpytestchangedir={envtmpdir}commands=pytest{toxinidir}[testenv:linting]deps=pre-commitcommands=pre-commitrun--all-files--show-diff-on-failure[testenv:coverage]deps=tomlpytestcoveragecommands=coveragerun--source=src--branch-mpytest{toxinidir}coveragereport-m--fail-under90coveragexml-o{toxinidir}/coverage.xml[gh-actions]python=3.8:py38,linting,coverage3.9:py393.10:py310Finally modify the GH actionCI.yamlto upload to codecov only when running Python 3.8. In this case we use a premade GH action instead of running manually the commands:...- name:Upload coverage to Codecov# Only generate the coverage report in Python 3.8if:"matrix.python-version=='3.8'"uses:codecov/codecov-action@v2with:fail_ci_if_error:trueAnd with this we are ready to publish the final release 0.1.10.