Package: pytest-aviator
Aviator's Pytest PluginAboutAviator's pytest plugin will automatically rerun flaky tests to keep your builds green. By analyzing the test data we have in Flakybot, we determine how many times we should rerun a specific flaky test and the threshold for passing.In the example output below,test_sample_randommust pass at least 1 time, with a maximum of 5 runs. It fails on the first run but passes on the second run, sotest_sample_randomsuccessfully passes on this CI run.============================= test session starts ==============================
platform linux -- Python 3.6.2, pytest-7.0.1, pluggy-1.0.0 -- /home/circleci/repo/venv/bin/python
cachedir: .pytest_cache
rootdir: /home/circleci/repo
plugins: flakybot-0.1.1
collecting ... collected 6 items                                                              

src/test_sample.py::TestSample::test_sample_dict PASSED                  [ 16%]
src/test_sample.py::TestSample::test_sample_list PASSED                  [ 33%]
src/test_sample.py::TestSample::test_sample_number PASSED                [ 50%]
src/test_sample.py::TestSample::test_sample_random PASSED                [ 66%]
src/test_sample.py::TestSample::test_sample_text PASSED                  [ 83%]
src/test_sample.py::TestSample::test_sample_tuple PASSED                 [100%]

------- generated xml file: /home/circleci/repo/test_results/output.xml --------
===FlakyBot Test Report===

test_sample_random
    FAILED: (4 runs remaining out of 5).
	<class 'AssertionError'>: 2 != 1
	[<TracebackEntry /home/circleci/repo/src/test_sample.py:35>]
	
test_sample_random passed 1 out of the required 1 times.

===End FlakyBot Test Report===
============================== 6 passed in 0.08s ===============================Here's another example run, wheretest_sample_randommust pass at least 2 times, with a maximum of 3 runs. It passes on the first run but fails on the next two runs, sotest_sample_randomfails on this CI run.============================= test session starts ==============================
platform linux -- Python 3.6.2, pytest-7.0.1, pluggy-1.0.0 -- /home/circleci/repo/venv/bin/python
cachedir: .pytest_cache
rootdir: /home/circleci/repo
plugins: flakybot-0.1.1
collecting ... collected 6 items                                                              

src/test_sample.py::TestSample::test_sample_dict PASSED                  [ 16%]
src/test_sample.py::TestSample::test_sample_list PASSED                  [ 33%]
src/test_sample.py::TestSample::test_sample_number PASSED                [ 50%]
src/test_sample.py::TestSample::test_sample_random FAILED                [ 66%]
src/test_sample.py::TestSample::test_sample_text PASSED                  [ 83%]
src/test_sample.py::TestSample::test_sample_tuple PASSED                 [100%]

=================================== FAILURES ===================================
________________________ TestSample.test_sample_random _________________________
 
self = <src.test_sample.TestSample testMethod=test_sample_random>
 
    def test_sample_random(self):
        a = random.randint(0, 2)
        # a = 1
>       self.assertEqual(a, 1)
E       AssertionError: 0 != 1
 
src/test_sample.py:44: AssertionError
- generated xml file: /opt/homebrew/var/buildkite-agent/builds/Doras-MacBook-Pro-local-1/aviator-demo/sample-test/test_results/output.xml -===FlakyBot Test Report===
===FlakyBot Test Report===

test_sample_random passed 1 out of the required 2 times.
Running test again until it passes 2 times.
 
test_sample_random
	FAILED: (1 runs remaining out of 3).
	<class 'AssertionError'>: 2 != 1
	[<TracebackEntry /opt/homebrew/var/buildkite-agent/builds/Doras-MacBook-Pro-local-1/aviator-demo/sample-test/src/test_sample.py:44>]
 
test_sample_random: FAILED
	It passed 1 out of the required 2 times.

===End FlakyBot Test Report===
============================== 1 failed, 5 passed in 0.08s ===============================InstallInstall the plugin before running tests with pytest.pip install pytest-aviatorStore Test ArtifactsIn order to correctly process and rerun the tests, we need the test results. You'll need to use pytest'sjunitxmlflag to report results in JUnit-XML format..pytest src/ -vv --junitxml="test_results/result.xml"BuildkiteIf the test results are stored intest_results/result.xml, you can upload the artifacts using the buildkite agent.buildkite-agent artifact upload test_results/output.xmlSee theBuildkite docsfor other ways to upload build artifacts.CircleCIMake sure to include thestore_artifactsstep in your config.yml.- store_artifacts:
      path: ./test_results/output.xml
      destination: output.xmlSee theCircleCI docsfor more information.Aviator API TokenMake sure to set your account's Aviator API token as an environment variableAVIATOR_API_URL. You can find the token athttps://app.aviator.co/account/api.CircleCI- set the variable under the Project Settings page.Buildkite- manage your API token using a storage service or environment hooks.