Package: pytest-rerunfailures
Contentspytest-rerunfailuresRequirementsInstallationRecover from hard crashesRe-run all failuresRe-run all failures matching certain expressionsRe-run all failures other than matching certain expressionsRe-run individual failuresOutputCompatibilityResourcesDevelopmentChangelog13.0 (2023-11-22)Breaking changesFeaturesBug fixes12.0 (2023-07-05)Breaking changesFeaturesBug fixes11.1.2 (2023-03-09)11.1.1 (2023-02-17)11.1 (2023-02-09)Bug fixesFeatures11.0 (2023-01-12)Breaking changesBug fixesFeatures10.2 (2021-09-17)10.1 (2021-07-02)10.0 (2021-05-26)Backwards incompatible changesFeaturesOther changes9.1.1 (2020-09-29)9.1 (2020-08-26)FeaturesOther changes9.0 (2020-03-18)Backwards incompatible changesFeatures8.0 (2019-11-18)Backwards incompatible changesFeaturesBug fixes7.0 (2019-03-28)Backwards incompatible changesFeaturesBug fixes6.0 (2019-01-08)Backwards incompatible changesFeaturesBug fixes5.0 (2018-11-06)4.2 (2018-10-04)4.1 (2018-05-23)4.0 (2017-12-23)3.1 (2017-08-29)3.0 (2017-08-17)2.2 (2017-06-23)2.1.0 (2016-11-01)2.0.1 (2016-08-10)2.0.0 (2016-04-06)1.0.2 (2016-03-29)1.0.1 (2016-02-02)1.0.0 (2016-02-02)pytest-rerunfailurespytest-rerunfailures is a plugin forpytestthat
re-runs tests to eliminate intermittent failures.RequirementsYou will need the following prerequisites in order to use pytest-rerunfailures:Python 3.7, up to 3.12, or PyPy3pytest 7.0 or newerThis plugin can recover from a hard crash with the following optional
prerequisites:pytest-xdist 2.3.0 or newerThis package is currently tested against the last 5 minor pytest releases. In
case you work with an older version of pytest you should consider updating or
use one of the earlier versions of this package.InstallationTo install pytest-rerunfailures:$pipinstallpytest-rerunfailuresRecover from hard crashesIf one or more tests trigger a hard crash (for example: segfault), this plugin
will ordinarily be unable to rerun the test. However, if a compatible version of
pytest-xdist is installed, and the tests are run within pytest-xdist using the-nflag, this plugin will be able to rerun crashed tests, assuming the workers and
controller are on the same LAN (this assumption is valid for almost all cases
because most of the time the workers and controller are on the same computer).
If this assumption is not the case, then this functionality may not operate.Re-run all failuresTo re-run all test failures, use the--rerunscommand line option with the
maximum number of times you’d like the tests to run:$pytest--reruns5Failed fixture or setup_class will also be re-executed.To add a delay time between re-runs use the--reruns-delaycommand line
option with the amount of seconds that you would like wait before the next
test re-run is launched:$pytest--reruns5--reruns-delay1Re-run all failures matching certain expressionsTo re-run only those failures that match a certain list of expressions, use the--only-rerunflag and pass it a regular expression. For example,
the following would only rerun those errors that matchAssertionError:$pytest--reruns5--only-rerunAssertionErrorPassing the flag multiple times accumulates the arguments, so the following
would only rerun those errors that matchAssertionErrororValueError:$pytest--reruns5--only-rerunAssertionError--only-rerunValueErrorRe-run all failures other than matching certain expressionsTo re-run only those failures that do not match a certain list of expressions, use the--rerun-exceptflag and pass it a regular expression. For example,
the following would only rerun errors other than that matchAssertionError:$pytest--reruns5--rerun-exceptAssertionErrorPassing the flag multiple times accumulates the arguments, so the following
would only rerun those errors that does not match withAssertionErrororOSError:$pytest--reruns5--rerun-exceptAssertionError--rerun-exceptOSErrorNoteWhen the`AssertionError`comes from the use of theassertkeyword,
use--rerun-exceptassertinstead:$ pytest --reruns 5 --rerun-except assertRe-run individual failuresTo mark individual tests as flaky, and have them automatically re-run when they
fail, add theflakymark with the maximum number of times you’d like the
test to run:@pytest.mark.flaky(reruns=5)deftest_example():importrandomassertrandom.choice([True,False])Note that when teardown fails, two reports are generated for the case, one for
the test case and the other for the teardown error.You can also specify the re-run delay time in the marker:@pytest.mark.flaky(reruns=5,reruns_delay=2)deftest_example():importrandomassertrandom.choice([True,False])You can also specify an optionalconditionin the re-run marker:@pytest.mark.flaky(reruns=5,condition=sys.platform.startswith("win32"))deftest_example():importrandomassertrandom.choice([True,False])Exception filtering can be accomplished by specifying regular expressions foronly_rerunandrerun_except. They override the--only-rerunand--rerun-exceptcommand line arguments, respectively.Arguments can be a single string:@pytest.mark.flaky(rerun_except="AssertionError")deftest_example():raiseAssertionError()Or a list of strings:@pytest.mark.flaky(only_rerun=["AssertionError","ValueError"])deftest_example():raiseAssertionError()You can use@pytest.mark.flaky(condition)similarly as@pytest.mark.skipif(condition), seepytest-mark-skipif@pytest.mark.flaky(reruns=2,condition="sys.platform.startswith('win32')")deftest_example():importrandomassertrandom.choice([True,False])# totally same as the above@pytest.mark.flaky(reruns=2,condition=sys.platform.startswith("win32"))deftest_example():importrandomassertrandom.choice([True,False])Note that the test will re-run for anyconditionthat is truthy.OutputHere’s an example of the output provided by the plugin when run with--reruns2and-raR:test_report.py RRF

================================== FAILURES ==================================
__________________________________ test_fail _________________________________

    def test_fail():
>       assert False
E       assert False

test_report.py:9: AssertionError
============================ rerun test summary info =========================
RERUN test_report.py::test_fail
RERUN test_report.py::test_fail
============================ short test summary info =========================
FAIL test_report.py::test_fail
======================= 1 failed, 2 rerun in 0.02 seconds ====================Note that output will show all re-runs. Tests that fail on all the re-runs will
be marked as failed.CompatibilityThis plugin maynotbe used with class, module, and package level fixtures.This plugin isnotcompatible with pytest-xdist’s –looponfail flag.This plugin isnotcompatible with the core –pdb flag.This plugin isnotcompatible with the pluginflaky, you can only havepytest-rerunfailuresorflakybut not both.ResourcesIssue TrackerCodeDevelopmentTest execution count can be retrieved from theexecution_countattribute
in testitem’s object. Example:@hookimpl(tryfirst=True)defpytest_runtest_makereport(item,call):print(item.execution_count)Changelog13.0 (2023-11-22)Breaking changesDrop support for pytest < 7.0.FeaturesAdd support for Python 3.12.Bug fixesFix crashitem names mismatch between client and server.
(#172)Fix crash when setup fails with –rerun-except flag.
(#230)12.0 (2023-07-05)Breaking changesDrop support for pytest < 6.2.FeaturesAddonly_rerunandrerun_exceptarguments to@pytest.mark.flakymarker.Add support for pytest 7.3, 7.4.Bug fixesFailures are now rerun only if they match at least one--only-rerunpattern (if given) and none of the--rerun-exceptpatterns. Previously,
using both--only-rerunand--rerun-excepttogether could cause
failures to be rerun even if they did not match any--only-rerunpattern, and when using multiple--rerun-exceptpatterns, all failures
would be rerun unless they matched every pattern.
(#225)11.1.2 (2023-03-09)Bug fixesExecute teardown when test was skipped in setup phase of a fixture.11.1.1 (2023-02-17)Bug fixesFix crash during teardown when runtest protocol hook is overwritten by
another plugin.Fix crash during teardown when TestCase class is used as base class.11.1 (2023-02-09)Bug fixesRun teardown of session, class, … scoped fixtures only once after rerunning testsFeaturesExposererunsandreruns_delaythroughpytest.inifile.11.0 (2023-01-12)Breaking changesDrop support for Python 3.6.Drop support for pytest < 6.Bug fixesFix crash when pytest-xdist is installed but disabled.
(Thanks to@mgornyfor the PR.)Fix crash when xfail(strict=True) mark is used with –rerun-only flag.FeaturesAdded option–rerun-exceptto rerun failed tests those are other than the mentioned Error.Add support for Python 3.11.Add support for pytest 7.0, 7.1, 7.2.10.2 (2021-09-17)FeaturesAllow recovery from crashed tests with pytest-xdist.Add support for Python 3.10 (as of Python 3.10.rc2).
(Thanks to@hugovkfor the PR.)10.1 (2021-07-02)FeaturesAllows using astras condition for@pytest.mark.flaky(condition)which gets evaluated dynamically similarly to@pytest.mark.skipif(condition).
(#162provided by@15klli)10.0 (2021-05-26)Backwards incompatible changesDrop support for Python 3.5.Drop support for pytest < 5.3.FeaturesAddconditionkeyword argument to the re-run marker.
(Thanks to@BeyondEvilfor the PR.)Add support for Python 3.9.
(Thanks to@digitronikfor the PR.)Add support for pytest 6.3.
(Thanks to@bluetechfor the PR.)Add compatibility withpytest-xdist>= 2.0.
(Thanks to@bluetechfor the PR.)Other changesCheck for the resultlog by feature and not by version as pytest master does
not provide a consistent version.9.1.1 (2020-09-29)Compatibility fix.Ignore--result-logcommand line option when used together withpytest >= 6.1.0, as it was removed there. This is a quick fix, use an older
version of pytest, if you want to keep this feature for now.
(Thanks to@ntessorefor the PR)Support up to pytest 6.1.0.9.1 (2020-08-26)FeaturesAdd a new flag--only-rerunto allow for users to rerun only certain
errors.Other changesDrop dependency onmock.Add support for pre-commit and add a linting tox target.
(#117)
(PR from@gnikonorov)9.0 (2020-03-18)Backwards incompatible changesDrop support for pytest version 4.4, 4.5 and 4.6.Drop support for Python 2.7.FeaturesAdd support for pytest 5.4.Add support for Python 3.8.8.0 (2019-11-18)Backwards incompatible changesDrop support for pytest version 3.10, 4.0, 4.1, 4.2 and 4.3Drop support for Python 3.4.FeaturesAdd support for pytest version 4.4, 4.5, 4.6, 5.0, 5.1 and 5.2.Bug fixesExplicitly depend on setuptools to ensure installation when working in
environments without it.
(#98)
(PR from@Eric-Arellano)7.0 (2019-03-28)Backwards incompatible changesDrop support for pytest version 3.8 and 3.9.FeaturesAdd support for pytest version 4.2 and 4.3.Bug fixesFixed #83 issue about ignoredpytest_runtest_logfinishhooks.
(#83)
(PR from@KillAChicken)6.0 (2019-01-08)Backwards incompatible changesDrop support for pytest version 3.6 and 3.7.FeaturesAdd support for pytest version 4.0 and 4.1.Bug fixesFixed #77 regression issue introduced in 4.2 related to thererunattribute on the test report.
(#77)
(Thanks to@RibeiroAnafor the PR).5.0 (2018-11-06)Drop support for pytest versions < 3.6 to reduce the maintenance burden.Add support up to pytest version 3.10. Thus supporting the newest 5 pytest
releases.Add support for Python 3.7.Fix issue can occur when used together withpytest-flake8(#73)4.2 (2018-10-04)Fixed #64 issue related tosetup_classandfixtureexecutions on
rerun (Thanks to@OlegKuzovkovfor the PR).Added newexecution_countattribute to reflect the number of test case
executions according to #67 issue. (Thanks to@OlegKuzovkovfor the PR).4.1 (2018-05-23)Add support for pytest 3.6 by usingNode.get_closest_marker()(Thanks to@The-Compilerfor the PR).4.0 (2017-12-23)Added option to add a delay time between test re-runs (Thanks to@Kangurosfor the PR).Added support for pytest >= 3.3.Drop support for pytest < 2.8.7.3.1 (2017-08-29)Restored compatibility with pytest-xdist. (Thanks to@davehuntfor the PR)3.0 (2017-08-17)Add support for Python 3.6.Add support for pytest 2.9 up to 3.2Drop support for Python 2.6 and 3.3.Drop support for pytest < 2.7.2.2 (2017-06-23)Ensure that other plugins can run after this one, in case of a global setting--rerun=0. (Thanks to@subleefor the PR)2.1.0 (2016-11-01)Add default value ofreruns=1ifpytest.mark.flaky()is called
without arguments.Also offer a distribution as universal wheel. (Thanks to@tltxfor the PR)2.0.1 (2016-08-10)Prepare CLI options to pytest 3.0, to avoid a deprecation warning.Fix error due to missing CHANGES.rst when creating the source distribution
by adding a MANIFEST.in.2.0.0 (2016-04-06)Drop support for Python 3.2, since supporting it became too much of a hassle.
(Reason: Virtualenv 14+ / PIP 8+ do not support Python 3.2 anymore.)1.0.2 (2016-03-29)Add support for–resultlogoption by parsing reruns accordingly. (#28)1.0.1 (2016-02-02)Improve package description and include CHANGELOG into description.1.0.0 (2016-02-02)Rewrite to use newer API of pytest >= 2.3.0Improve support for pytest-xdist by only logging the final result.
(Logging intermediate results will finish the test rather rerunning it.)