Package: tox-gh-matrix
tox-gh-matrixAtoxplugin that generates a JSON version of your tox.ini envlist,
which can be used in a GitHub workflowbuild matrix(or potentially with other CI systems).This plugin is useful when:Your tox.ini envlist covers a complex set of factors
(e.g., all supported combinations of Django and Python versions).You use GitHub actions and want to run each tox testenv
in a separate workflow job, so that tests run in parallel and so
GitHub's actions log breaks out the result for each testenv.You're tired of manually syncing your workflow build matrix and
your tox.ini envlist.tox-gh-matrix adds a newtox --gh-matrixcommand line option that
outputs a JSON representation of your tox envlist:[
  {
    "name": "py35-django22",
    "factors":  ["py35", "django22"],
    "python": { "version": "3.5", "spec": "3.5.0-alpha - 3.5" }
  },
  {
    "name": "py36-django22",
    "factors":  ["py36", "django22"],
    "python": { "version": "3.6", "spec": "3.6.0-alpha - 3.6" }
  },
  // ...
  {
    "name": "py310-django40",
    "factors":  ["py310", "django40"],
    "python": { "version": "3.10", "spec": "3.10.0-alpha - 3.10", "installed": "3.10.2" }
  },
  {
    "name": "py311-django40",
    "factors":  ["py311", "django40"],
    "python": { "version": "3.11", "spec": "3.11.0-alpha - 3.11" },
    "ignore_outcome": true
  },
  // ...
  { "name": "docs", "factors": ["docs"] },
  { "name": "lint", "factors": ["lint"] }
]Your workflow can use this to define a build matrix from the tox envlist:jobs:get-envlist:outputs:envlist:${{ steps.generate-envlist.outputs.envlist }}steps:# ... (details omitted; see complete example below)-id:generate-envlistrun:python -m tox --gh-matrixtest:needs:get-envliststrategy:matrix:tox:${{ fromJSON(needs.get-envlist.outputs.envlist) }}steps:# ... (details omitted; see complete example below)-uses:actions/setup-python@v4with:python-version:${{ matrix.tox.python.spec }}-run:python -m tox -e ${{ matrix.tox.name }}See theusagesection below for a complete annotated example
and other variations.ContentsUsageComplete exampleInstallationHandling ignore outcomeUsing setup-pythonTesting PyPy and older Python versionsFiltering the tox envlistExamining tox factorsMatrix output names and multiple envlistsAdditional build matrix dimensionsDebugging the matrixContributing, issues, helpSimilar projectsLicenseUsageThe basic approach to running GitHub workflow jobs based on
your tox envlist is:Runtox --gh-matrixin a preliminary job, to generate a
JSON version of your tox envlist.In your main test job, define a workflowbuild matrixproperty that iterates that list, usingfromJSON().Complete exampleHere's a complete, annotated example workflow:name:teston:pushjobs:# First, use tox-gh-matrix to construct a build matrix# from your tox.ini:get-envlist:runs-on:ubuntu-latest# Make the JSON envlist available to the test job:outputs:envlist:${{ steps.generate-envlist.outputs.envlist }}steps:# Checkout project code to get tox.ini:-uses:actions/checkout@v3# Install tox and tox-gh-matrix:-run:python -m pip install tox tox-gh-matrix# Run `tox --gh-matrix` to generate the JSON list:-id:generate-envlistrun:python -m tox --gh-matrix# Now run your tests using that matrix:test:# Pull in the JSON generated in the previous job:needs:get-envliststrategy:# Define a build matrix property `tox` that iterates# the envlist:matrix:tox:${{ fromJSON(needs.get-envlist.outputs.envlist) }}# Run all matrix jobs, even if some fail:fail-fast:false# The workflow treats everything below as a template# to run a separate job for each build matrix item.name:Test ${{ matrix.tox.name }}runs-on:ubuntu-lateststeps:-uses:actions/checkout@v3# Install the required Python version if necessary:-name:Setup Python ${{ matrix.tox.python.version }}if:matrix.tox.python.spec && ! matrix.tox.python.installeduses:actions/setup-python@v4with:python-version:${{ matrix.tox.python.spec }}# Install tox (you don't need tox-gh-matrix at this point):-run:python -m pip install tox# Run `tox -e {name}` to test the single tox environment# for this matrix entry:-run:python -m tox -e ${{ matrix.tox.name }}Some other variations are covered below. Also, see this project's
ownworkflow definition.InstallationInstall tox-gh-matrix from PyPI using pip:python -m pip install tox-gh-matrix(Typically you'd do this in a GitHub workflow as shown above,
but that's not required. You can run tox-gh-matrix in your local
development environment to examine its output.)tox-gh-matrix requires Python 3.6 or later and tox 3.5.12 or later.
(It is not currently compatible with tox 4 alpha.)The minimum Python version only applies to running thetox --gh-matrixcommand itself. (tox can generate virtual environments withanyversion
of Python to run your tests.)Handling ignore outcomeIf you use tox'signore_outcomesetting to
allow failures in certain testenvs, those jobs will always appear
successful in GitHub's actions log.You may prefer to hoist the failure handling up to the workflow level,
so you can see which environments have failed in the actions log.tox-gh-matrix adds"ignore_outcome": trueto each matrix
item where your tox.ini specifies that option. You can check this
in the workflow'scontinue-on-errorjob step setting.You'll also need prevent tox fromactuallyignoring failures during
those workflow runs. Tox doesn't have a built-in way to "ignoreignore_outcome", but we can simulate it with an environment variable.
This example calls itTOX_OVERRIDE_IGNORE_OUTCOME(but the exact name
doesn't matter).First, update your workflow to set the workflow'scontinue-on-errorfrom the matrix and set the environment variable tofalse(meaning
we want tox to pretendignore_outcome = falseeverywhere, regardless
of what tox.ini says):jobs:test:steps:# ...-run:python -m tox -e ${{ matrix.tox.name }}continue-on-error:${{ matrix.tox.ignore_outcome == true }}env:TOX_OVERRIDE_IGNORE_OUTCOME:false(Only add this variable in thetestjob, not theget-envlistjob.)Then, in your tox.ini change everyignore_outcome = trueto use the
environment variable (using tox'senvironment variable substitutionsyntax with a default value oftrue):[testenv:experimental]ignore_outcome={env:TOX_OVERRIDE_IGNORE_OUTCOME:true}# This also works with factor-conditional settings:[testenv]ignore_outcome=djangoDev={env:TOX_OVERRIDE_IGNORE_OUTCOME:true}py322={env:TOX_OVERRIDE_IGNORE_OUTCOME:true}Now when tox is run from the GitHub workflow, itwon'tignore failures,
so the workflow will catch and report them (and then continue). But when
you run tox locally (or anywhere the environment variable isn't set),
toxwillignore failures in those testenvs.Using setup-pythonThe tox-gh-matrix JSON includespythonobjects providing version
data that works with GitHub'sactions/setup-python.For example, if your tox.ini hasenvlist = py36,py38,docs,
thetox --gh-matrixJSON might look something like this:[
  {
    "name": "py36",
    "python": {
      "version": "3.6",
      "spec": "3.6.0-alpha - 3.6",
    },
  },
  {
    "name": "py38",
    "python": {
      "version": "3.8",
      "spec": "3.8.0-alpha - 3.8",
      "installed": "3.8.10"
    },
  },
  { "name": "docs" }
]Thepythonfield is only present if the tox testenv calls for
a specific Python version–either implied with apy*factor or
explicitly via the testenv'sbasepythonsetting. (So in this example, there's nopythonfield
for the "docs" testenv. If you wanted a specific Python there,
you could add something likebasepython = py310in your
tox.ini's[testenv:docs]section .)When present,pythonis an object with two or three fields:python.versionis a Python version compatible with setup-python'spython-versionparameter. E.g.,"3.10"or"2.7"or"pypy-3.8".python.specis a versionrangespecifier, also compatible
with setup-python'spython-versionparameter, which allows
pre-release versions. E.g.,"3.22.0-alpha - 3.22".python.installedis only present if tox found a compatible
Python version already available on the runner instance. If so, it is
the actual version reported by that interpreter. (This is useful for
skipping setup-python if tox can already find a compatible interpreter
on the runner.)Putting this all together, the recommended way to call setup-python
for most workflows is:steps:-name:Setup Python ${{ matrix.tox.python.version }}if:matrix.tox.python.spec && ! matrix.tox.python.installeduses:actions/setup-python@v4with:python-version:${{ matrix.tox.python.spec }}If you don't want to use pre-release Python interpreters, changepython.spectopython.versionin both places.If yourget-envlistjob (that runstox --gh-matrix) has a differentruns-onrunner type than thetestjob (tox -e ${{matrix.tox.name}}),
you will have different Python versions available on your test runners.
In that case, you should ignorepython.installedchange the check
to justif: matrix.tox.python.spec.Testing PyPy and older Python versionsIf your tox envlist includes PyPy or outdated Python versions, you may need
an extra build step to ensure tox runs under a Python version it supports.For example, say your tox envlist includespy34. Tox dropped Python 3.4
support in 2019. It can still generate a Python 3.4 virtualenv and run
tests in it, but you must run tox itself on a newer version of Python.To make that work, runactions/setup-pythontwice: once to install
the (possibly old) version of Python needed for the test environment,
and a second time to change back to a newer version of Python to run tox:steps:-uses:actions/checkout@v3# Install the required Python version if necessary:-name:Setup Python ${{ matrix.tox.python.version }}if:matrix.tox.python.spec && ! matrix.tox.python.installeduses:actions/setup-python@v4with:python-version:${{ matrix.tox.python.spec }}# Now restore the default Python to something newer for tox:-name:Restore modern Pythonuses:actions/setup-python@v4with:python-version:"3.8"# Install and run tox with that newer Python:-run:python -m pip install tox-run:python -m tox -e ${{ matrix.tox.name }}You can choose anypython-versionthat makes sense for the second setup-python.
(If you use the version that comes pre-installed on your runner platform,
it will be nearly instantaneous because there's nothing to install;
it's just changing some paths to alter the default.)Filtering the tox envlistYour tox envlist may include environments you don't want to test
in your workflow. You can either restrict the envlist when
you calltox --gh-matrixto generate it, or you can use workflow
conditionals to skip jobs based on tox factors or other information.By default, tox-gh-matrix includes your entire tox.inienvlistin its JSON output. You can limit this with tox command line options
or environment variables thatfilter the envlist,
such as-e envlist,TOXENVorTOX_SKIP_ENV.For example, if you wanted the matrix to omit all tox testenvs
containingwinormac, you could use:get-envlist:steps:# ...-id:generate-envlistenv:# (TOX_SKIP_ENV is a Python regular expression)TOX_SKIP_ENV:".*(win|mac).*"run:python -m tox --gh-matrixtox-gh-matrix should also work with other tox plugins that
manipulate the envlist, such astox-factorandtox-envlist.Examining tox factorsThe tox-gh-matrix JSON includes a list of toxfactorsfor each tox
environment. You can use this with GitHub workflowconditional executionto skip or include steps for certain factors.For example, you might useif: contains(matrix.tox.factors, "pre")to only execute a particular job step for tox environments containing a "pre"
factor. Contrast that withcontains(matrix.tox.name, "pre")which would do something similar but also match environments containing
factors like "prep" or "present", which may or may not be what you want.(factorsis a list of strings;nameis a single string. In workflow
expression syntax,matrix.tox.nameis equivalent tojoin(matrix.tox.factors, '-').)Matrix output names and multiple envlistsRunningtox --gh-matrixsets a GitHub workflowoutput parameterto the JSON build matrix. (It appends to theGITHUB_OUTPUTfile;
you'll get an error if that environment variable isn't set properly.)The default output name isenvlist, but you change this withtox --gh-matrix=VAR.
You can use this to (in combination with filtering) to create multiple matrices.Here's an example that uses custom output names, along with thetox-factorfiltering plugin, to construct separate matrices for Mac- and Windows specific
tests (environments withmacorwinfactors, respectively):jobs:get-envlist:runs-on:ubuntu-latestoutputs:mac-envlist:${{ steps.generate-envlist.outputs.mac-envlist }}win-envlist:${{ steps.generate-envlist.outputs.win-envlist }}steps:-uses:actions/checkout@v3# Also install the tox-factor plugin:-run:python -m pip install tox tox-factor tox-gh-matrix# Run --gh-matrix twice with different filters and output names:-id:generate-envlistrun:|python -m tox -f mac --gh-matrix=mac-envlistpython -m tox -f win --gh-matrix=win-envlisttest-mac:runs-on:macos-latestneeds:get-envliststrategy:matrix:tox:${{ fromJSON(needs.get-envlist.outputs.mac-envlist) }}# ...test-win:runs-on:windows-latestneeds:get-envliststrategy:matrix:tox:${{ fromJSON(needs.get-envlist.outputs.win-envlist) }}# ...Additional build matrix dimensionsYour workflow can define additionalbuild matrixproperties alongside the tox envlist. GitHub will run all combinations
of properties.For example, your workflow could repeat the entire tox envlist
on macOS, Windows, and Ubuntu by adding in anosproperty:jobs:# ...test:strategy:matrix:os:[macos-latest,windows-latest,ubuntu-latest]tox:${{ fromJSON(needs.get-envlist.outputs.envlist) }}name:Test ${{ matrix.tox.name }} on ${{ matrix.os }}runs-on:${{ matrix.os }}steps:-uses:actions/checkout@v3-name:Setup Python ${{ matrix.tox.python.version }}if:matrix.tox.python.spec && ! matrix.tox.python.installeduses:actions/setup-python@v4with:python-version:${{ matrix.tox.python.spec }}-run:python -m pip install tox-run:python -m tox -e ${{ matrix.tox.name }}Debugging the matrixRuntox --gh-matrix-dumpto display a nicely formatted (multiline,
indented) JSON build matrix, without any GitHub-specific output parameter
syntax.This can be helpful for debugging the generated matrix (either run in your
local development environment, or as a step in yourget-envlistjob).It could also be useful for integrating tox with other (non-GitHub) CI systems.(Or perhaps you were interested indebuggingThe Matrix?)Contributing, issues, helpContributions of all types are very welcome, including bug reports, fixes,
documentation corrections and improvements, and new features.If you have any questions or need help with tox-gh-matrix, pleaseask in the discussions.If you encounter any problems, pleasefile an issuealong with as much detail as possible to help reproduce the problem.For bug fixes and other code changes, tests can be run withtox(naturally).
We try to keep test coverage high before merging new code, but please don't
let incomplete tests keep you from opening a PR. (We'll be happy to work with
you to add tests, etc.)To propose a new feature, it's often helpful to open adiscussionbefore
investing significant time or effort in code.Similar projectsOther tox + GitHub integrations tend to take the opposite approach:
you fully declare the build matrix in your GitHub workflow definition,
and the plugin then simplifies running the correct tox environment(s)
for each matrix job.tox-gh-actionsdetects which tox environments to run based on the
current active Python version, platform, environment variables, and other
context, with flexible mapping to tox factors. It also improves integration
with GitHub's actions logging.tox-ghis a newer project with goals similar to tox-gh-actions,
but some different design philosophies.LicenseDistributed under the terms of the MIT License, tox-gh-matrix is
free and open source software.