Package: detect-test-pollution
detect-test-pollutiona tool to detect test pollutioninstallationpipinstalldetect-test-pollutionwhat is test pollution?test pollution is where a test fails due to the side-effects of some other test
in the test suite.it usually appears as a "test flake" something where the test fails
mysteriously but passes when run by itself.a simple example of this is the following python code:k=1deftest_k():assertk==1deftest_k2():globalkk=2assertk==2now this example is a little bit silly, you probably wouldn't write code this
poorly but helps us demonstrate the problem here.when run normally -- these tests pass:$pytest-qt.py..                                                                       [100%]2 passed in 0.00sbut, if the tests were run in some other order (due to something likepytest-randomlyorpytest-xdist) then the pollution would be apparent:$pytest-qt.py::test_k2t.py::test_k.F                                                                       [100%]=================================== FAILURES ===================================____________________________________ test_k ____________________________________def test_k():>       assert k == 1E       assert 2 == 1t.py:4: AssertionError=========================== short test summary info ============================FAILED t.py::test_k - assert 2 == 11 failed, 1 passed in 0.03soften this flake happens in a codebase with hundreds or thousands of tests
and it's difficult to track down which test is causing the global side-effects.that's where this tool comes in handy!  it helps you find the pair of tests
which error when run in order.usageonce you have identified a failing test, you'll be able to feed it intodetect-test-pollutionto find the causal test.the basic mode is to run:detect-test-pollution\--failing-testtest.py::test_id_here\--tests./testswheretest.py::test_id_hereis the identifier of the failing test and./testsis the directory where your testsuite lives.if you've already narrowed down the list of testids further than that, you
can specify a--testids-fileinstead of--teststo speed up discovery:detect-test-pollution\--failing-testtest.py::test_id_here\--testids-file./testidsyou can usually get a list of testids viapytest --collect-only -q(though
you'll need to strip some unrelated lines at the end, such as timing and
warning info).thendetect-test-pollutionwill bisect the list of tests to find the failing
one.  here's an example bisection from abug in pytest$detect-test-pollution--tests./testing--failing-testtesting/io/test_terminalwriter.py::test_should_do_markup_FORCE_COLORdiscovering all tests...-> discovered 3140 tests!ensuring test passes by itself...-> OK!ensuring test fails with test group...-> OK!running step 1:- 3139 tests remaining (about 12 steps)running step 2:- 1570 tests remaining (about 11 steps)running step 3:- 785 tests remaining (about 10 steps)running step 4:- 393 tests remaining (about 9 steps)running step 5:- 197 tests remaining (about 8 steps)running step 6:- 99 tests remaining (about 7 steps)running step 7:- 50 tests remaining (about 6 steps)running step 8:- 25 tests remaining (about 5 steps)running step 9:- 12 tests remaining (about 4 steps)running step 10:- 6 tests remaining (about 3 steps)running step 11:- 3 tests remaining (about 2 steps)double checking we found it...-> the polluting test is: testing/test_terminal.py::TestTerminal::test_report_teststatus_explicit_markupfuzzingdetect-test-pollutioncan also be used to "fuzz" out failing tests.it does this by shuffling the test ids and running the testsuite until it
fails.here's an example execution on a silly testsuite:$detect-test-pollution--fuzz--testst.pydiscovering all tests...-> discovered 1002 tests!run 1...-> OK!run 2...-> found failing test!try `detect-test-pollution --failing-test t.py::test_k --tests t.py`!afterwards you can use the normal mode ofdetect-test-pollutionto find the
failing pair.supported test runnersat the moment onlypytestis supported -- though in theory the tool could
be adapted to support other python test runners, or even other languages.