Package: pytest-replay
Saves previous test runs and allow re-execute previous pytest runs to reproduce crashes or flaky testsThispytestplugin was generated withCookiecutteralong with@hackebrot’sCookiecutter-pytest-plugintemplate.FeaturesThis plugin helps to reproduce random or flaky behavior when running tests with xdist.pytest-xdistexecutes tests
in a non-predictable order, making it hard to reproduce a behavior seen in CI locally because there’s no convenient way
to track which test executed in which worker.This plugin records the executed node ids by each worker in the directory given by--replay-record-dir=<dir>flag,
and a--replay=<file>can be used to re-run the tests from a previous run. For example:$ pytest -n auto --replay-record-dir=build/tests/replayThis will generate files with each line being ajsonwith the following content:
node identification, start time, end time and outcome. It is interesting to note
that usually the node id is repeated twice, that is necessary in case of a test
suddenly crashes we will still have the record of that test started. After the
test finishes,pytest-replaywill add anotherjsonline with the
complete information.
That is also useful to analyze concurrent tests which might have some kind of
race condition and interfere in each other.For example workergw1will generate a file.pytest-replay-gw1.txtwith contents like this:{"nodeid": "test_foo.py::test[1]", "start": 0.000}
{"nodeid": "test_foo.py::test[1]", "start": 0.000, "finish": 1.5, "outcome": "passed"}
{"nodeid": "test_foo.py::test[3]", "start": 1.5}
{"nodeid": "test_foo.py::test[3]", "start": 1.5, "finish": 2.5, "outcome": "passed"}
{"nodeid": "test_foo.py::test[5]", "start": 2.5}
{"nodeid": "test_foo.py::test[5]", "start": 2.5, "finish": 3.5, "outcome": "passed"}
{"nodeid": "test_foo.py::test[7]", "start": 3.5}
{"nodeid": "test_foo.py::test[7]", "start": 3.5, "finish": 4.5, "outcome": "passed"}
{"nodeid": "test_foo.py::test[8]", "start": 4.5}
{"nodeid": "test_foo.py::test[8]", "start": 4.5, "finish": 5.5, "outcome": "passed"}If there is a crash or a flaky failure in the tests of the workergw1, one can take that file from the CI server and
execute the tests in the same order with:$ pytest --replay=.pytest-replay-gw1.txtHopefully this will make it easier to reproduce the problem and fix it.FAQpytesthas its owncache, why use a different mechanism?The internal cache saves its data usingjson, which is not suitable in the advent of a crash because the file
will not be readable.Shouldn’t the ability of selecting tests from a file be part of thepytestcore?Sure, but let’s try to use this a bit as a separate plugin before proposing
its inclusion into the core.InstallationYou can installpytest-replayviapipfromPyPI:$ pip install pytest-replayOr with conda:$ conda install -c conda-forge pytest-replayContributingContributions are very welcome.Tests can be run withtoxif you are using a native Python installation.To run tests withconda, first create a virtual environment and execute tests from there
(conda with Python 3.5+ in the root environment):$ python -m venv .env
$ .env\scripts\activate
$ pip install -e . pytest-xdist
$ pytest testsReleasesFollow these steps to make a new release:Create a new branchrelease-X.Y.Zfrommaster;UpdateCHANGELOG.rst;Open a PR;After it isgreenandapproved, push a new tag in the formatX.Y.Z;GitHub Actions will deploy to PyPI automatically.Afterwards, update the recipe inconda-forge/pytest-replay-feedstock.LicenseDistributed under the terms of theMITlicense.IssuesIf you encounter any problems, pleasefile an issuealong with a detailed description.