Package: catalystcoop.cheshire
Cheshire: a Python Template Repository for CatalystThis template repository helps make new Python projects easier to set up and more
uniform. It contains a lot of infrastructure surrounding a minimal Python package namedcheshire(the cat who isn’t entirely there…).Create a new repository from this templateChoose a name for the new package that you are creating.The name of the repository should be the same as the name of the new Python package
you are going to create. E.g. a repository atcatalyst-cooperative/cheshireshould
be used to define a package namedcheshire.Fork this template repository to create a new Python project repo.See these instructions.Clone the new repository to your development machine.Runpre-commitinstallin the newly clone repository to install thepre-commit hooksdefined in.pre-commit-config.yamlCreate thecheshireconda environment by runningconda env createor
(preferably)mamba env createin the top level of the repository.Activate the new conda environment withconda activate cheshire.Runtoxfrom the top level of the repository to verify that everything is working
correctly.Rename the package and distributionOnce you know that your forked version of thecheshirepackage is working as
expected, you should update the package and distribution names in your new repo to
reflect the name of your new package. Thepackage nameis determined by the name of
the directory undersrc/which contains the source code, and is the name you’ll use
to import the package for use in a program, script, or notebook. E.g.:importcheshireThedistribution nameis the name that is used to install the software using a
program likepip,conda, ormamba. It is often identical to the package
name, but can also contain a prefix namespace that indicates the individual or
organization responsible for maintaining the pacakge. SeePEP 423PEP 423for more on Python package naming
conventions. We are using thecatalystcoopnamespace for the packages that we
publish, so ourpudlpackage becomescatalystcoop.pudlin the
Python Package Index (PyPI) or onconda-forge. Similarly thecheshirepackage
becomes thecatalystcoop.cheshiredistribution. The distribution name is determined
by theproject.namedefined inpyproject.tomlpipinstallcatalystcoop.cheshireThe package and distribution names are referenced in many of the files in the template
repository, and they all need to be replaced with the name of your new package. You can
usegrep-rto search recursively through all of the files for the wordcheshireat the command line, or use the search-and-replace functionality of your IDE / text
editor. The name of the package directory undersrc/will also need to be changed.Supply any required tokens, e.g. for CodeCovRename thesrc/cheshiredirectory to reflect the new package name.Search forcheshireand replace it as appropriate everywhere. Sometimes
this will be with a distribution name likecatalystcoop.cheshire(the package as it appears forpiporPyPI) and sometimes this will be the
importable package name (the name of the directory undersrce.g.cheshire)Create the new project / package at Read The Docs.What this template providesPython Package SkeletonThesrcdirectory contains the code that will be packaged and deployed on the user
system. That code is in a directory with the same name as the package.Using a separatesrcdirectory helps avoid accidentally importing the package when
you’re working in the top level directory of the repository.A simple python module (dummy.py), and a separate module providing a command line
interface to that module (cli.py) are included as examples.Any files in thesrc/package_data/directory will also be packaged and deployed.What files are included in or excluded from the package on the user’s system is
controlled by theMANIFEST.infile and some options in the call tosetup()insetup.py.The CLI is deployed usingproject.scriptsdefined inpyproject.toml.We usesetuptools_scmto obtain the package’s version directly fromgittags,
rather than storing it in the repository and manually updating it.README.rstis read in and used for the pacakge’slong_description. This is
what is displayed on the PyPI page for the package.By default we create at least three sets of “extras” – additional optional package
dependencies that can be installed in special circumstances:dev,docs`, andtests. The packages listed there are used in development, building the docs, and
running the tests (respectively) but aren’t required for a normal user who is just
installing the package frompiporconda. These are defined under theproject.optional-dependenciessection ofpyproject.tomlPython has recently evolved a more diverse community of build and packaging tools.
Which flavor is being used by a given package is indicated by the contents ofpyproject.toml. That file also contains configuration for a few other tools,
includingblackandruff.Pytest Testing FrameworkA skeletonpytesttesting setup is included in thetests/directory.Tests are split intounitandintegrationcategories.Session-wide test fixtures, additional command line options, and other pytest
configuration can be added totests/conftest.pyExactly what pytest commands are run during continuous integration controlled by Tox.Pytest can also be run manually without using Tox, but will use whatever your
personal python environment happens to be, rather than the one specified by the
package. Running pytest on its own is a good way to debug new or failing tests
quickly, but we should always use Tox and its virtual environment for actual testing.Test Coordination with ToxWe define several different test environments for use with Tox intox.iniToxis used to run pytest in an isolated Python
virtual environment.We also use Tox to coordinate running the code linters, building the documentation,
and releasing the software to PyPI.The default Tox environment is namedciand it will run the linters, build the
documentation, run all the tests, and generate test coverage statistics.Git Pre-commit HooksA variety of sanity checks are defined as git pre-commit hooks – they run any time
you try to make a commit, to catch common issues before they are saved. Many of these
hooks are taken from the excellentpre-commit project.The hooks are configured in.pre-commit-config.yamlFor them to run automatically when you try to make a commit, youmustinstall the
pre-commit hooks in your cloned repository first. This only has to be done once.These checks are run as part of our CI, and the CI will fail if the pre-commit hooks
fail.We also use thepre-commit.ciservice to run the same
checks on any code that is pushed to GitHub, and to apply standard code formatting
to the PR in case it hasn’t been run locally prior to being committed.Code Formatting & LintingTo avoid the tedium of meticulously formatting all the code ourselves, and to ensure as
standard style of formatting and sytactical idioms across the codebase, we use theblackandruffcode formatters, which run as pre-commit hooks. These can be
integrated directly into your text editor or IDE with the appropriate plugins. The
formatters are included in.pre-commit-config.yaml. Therufflinter / formatter
has a huge array of configuration options and different kinds of checks it can run,
which are defined under thetool.ruffsection ofpyproject.toml.We also have a custom hook that clears Jupyter notebook outputs prior to committing.Code & Documentation LintersTo catch errors before commits are made, and to ensure uniform formatting across the
codebase, we also use linters outside ofruff. They don’t change the code or
documentation files, but they will raise an error or warning when something doesn’t
look right so you can fix it.doc8mypyDoes static type checking,
and ensures that our code uses type annotations.pre-commithas a collection of built-in checks thatuse
pygrep to search Python filesfor
common problems like blanket# noqaannotations, as well aslanguage agnostic
problemslike accidentally
checking large binary files into the repository or having unresolved merge conflicts.hadolintchecks Dockerfiles for errors
and violations of best practices. It runs as a pre-commit hook.Test CoverageWe use Tox and a the pytestcoverageplugin to measure and record what percentage of our codebase is being tested, and to
identify which modules, functions, and individual lines of code are not being
exercised by the tests.When you runtoxortox-eci(which is equivalent) a summary of the test
coverage will be printed at the end of the tests (assuming they succeed). The full
details of the test coverage is written tocoverage.xml.There are some configuration options for this process set in the.coveragercfile
in the top level directory of the repository.When the tests are run via thetox-pytestworkflow in GitHub Actions, the test
coverage data from thecoverage.xmloutput is uploaded to a service calledCodeCovthat saves historical data about our test
coverage, and provides a nice visual representation of the data – identifying which
subpackages, modules, and individual lines of are being tested. For example, here are
the resultsfor the cheshire repo.The connection to CodeCov is configured in the.codecov.ymlYAML file.In theory, we should be able to automatically turn CodeCov on for all of our GitHub
repos, and it just Just Work, but in practice we’ve had to turn it on in the GitHub
configuration one-by-one. Open source repositories are also supposed to be able to
upload to the CodeCov site without requiring authentication, but this also hasn’t
worked, so thus far we’ve needed to request a new token for each repository. This
token is stored in.codecov.yml.Once it’s enabled, CodeCov also adds a couple of test coverage checks to any pull
request, to alert us if a PR reduces overall test coverage (which we would like to
avoid).Documentation BuildsWe build our documentation usingSphinx.Standalone docs files are stored under thedocs/directory, and the Sphinx
configuration is there inconf.pyas well.We useSphinx AutoAPIto
convert the docstrings embedded in the python modules undersrc/into additional
documentation automatically.The top level documentation index simply includes thisREADME.rst, theLICENSE.txtandCODE_OF_CONDUCT.mdfiles are similarly referenced. The only
standalone documentation file underdocs/right now is therelease_notes.rst.Unless you’re debugging something specific, the docs should always be built usingtox-edocsas that will lint the source files usingdoc8andrstcheck,
and wipe previously generated documentation to build everything from scratch. The docs
are also rebuilt as part of the normal Tox run (equivalent totox-eci).If you add something to the documentation generation process that needs to be cleaned
up after, it should be integrated with the Sphinx hooks. There are some examples of
how to do this at the bottom ofdocs/conf.pyin the “custom build operations”
section. For example, this is how we automatically regenerate the data dictionaries
based on the PUDL metadata whenever the docs are built, ensuring that the docs stay up
to date.Documentation PublishingWe use the popularRead the Docsservice to host our
documentation.When you open a PR, push todevormain, or tag a release, the associated
documentation is automatically built on Read the Docs.There’s some minimal configuration stored in the.readthedocs.ymlfile, but
setting up this integration for a new repository requires some setup on the Read the
Docs site.Create an account on Read the Docs using your GitHub identity, go to “My Projects”
under the dropdown menu in the upper righthand corner, and click on “Import a
Project.” It should list the repositories that you have access to on GitHub. You may
need to click on the Catalyst Cooperative logo in the right hand sidebar.It will ask you for a project name – this will become part of the domain name for the
documentation page on RTD and should be the same as the distribution name, but with
dots and underscores replaced with dashes. E.g.catalystcoop-cheshireorcatalystcoop-pudl-catalog.Under Advanced Settings, make sure youenable builds on PRs.
This will add a check ensuring that the documentation has built successfully on RTD
for any PR in the repo.Under the Builds section for the new project (repo) you’ll need to tell it which
branches you want it to build, beyond the defaultmainbranch.Once the repository is connected to Read the Docs, an initial build of the
documentation from themainbranch should start.DependabotWe use GitHub’sDependabotto automatically update the allowable versions of packages we depend on. This applies
to both the Python dependencies specified insetup.pyand to the versions of theGitHub Actionsthat we employ. The dependabot
behavior is configured in.github/dependabot.ymlGitHub ActionsUnder.github/workflowsare YAML files that configure theGitHub Actionsassociated with the repository. We use GitHub
Actions to:Run continuous integration usingtoxon several different
versions of Python.Build a Docker container directly and push it to Docker Hub using thedocker-build-push action.Release a new version of the package on PyPI when a version tag is pushed.Automatically merge bot PRs from pre-commit.ci and the dependabot.About Catalyst CooperativeCatalyst Cooperativeis a small group of data
wranglers and policy wonks organized as a worker-owned cooperative consultancy.
Our goal is a more just, livable, and sustainable world. We integrate public
data and perform custom analyses to inform public policy (Hire us!). Our focus is primarily on mitigating
climate change and improving electric utility regulation in the United States.Contact UsFor general support, questions, or other conversations around the project
that might be of interest to others, check out theGitHub DiscussionsIf you’d like to get occasional updates about our projectssign up for our email list.Want to schedule a time to chat with us one-on-one? Join us forOffice HoursFollow us on Twitter:@CatalystCoopMore info on our website:https://catalyst.coopFor private communication about the project or to hire us to provide customized data
extraction and analysis, you can email the maintainers:pudl@catalyst.coop