Repository: pytest-dev/pytest-randomly (language: python)

Issue:
Fix working with --stepwise
### Python Version

_No response_

### Package Version

_No response_

### Description

https://docs.pytest.org/en/6.2.x/cache.html#stepwise

Using `--stepwise` doesn't seem to work with pytest-randomly at current.

If I have tests A through D that all fail with 'assert 0', but a seed that runs them with D first, it stops after the failure of D but assumes that A through C passed even though they didn't run.

It seems to base its success/fail tracking on the test order pre-shuffle.


Discussion / hints:

Fix patch:
diff --git a/HISTORY.rst b/HISTORY.rst
index 8970414..84173fa 100644
--- a/HISTORY.rst
+++ b/HISTORY.rst
@@ -9,6 +9,15 @@ History
   Thanks to Tom Grainger for the suggestion in `Issue #210
   <https://github.com/pytest-dev/pytest-randomly/issues/210>`__.
 
+* Shuffle before other test collection hooks. This allows
+  pytest’s `--stepwise flag
+  <https://docs.pytest.org/en/latest/cache.html#stepwise>`__ to work, among
+  other things.
+
+  Thanks to Tom Grainger for the suggestion to try with ``--stepwise``. Fixes
+  `Issue #376
+  <https://github.com/pytest-dev/pytest-randomly/issues/376>`__.
+
 3.9.0 (2021-08-12)
 ------------------
 


Test patch:
diff --git a/src/pytest_randomly/__init__.py b/src/pytest_randomly/__init__.py
index d80fbc5..9f98f3a 100644
--- a/src/pytest_randomly/__init__.py
+++ b/src/pytest_randomly/__init__.py
@@ -9,7 +9,7 @@
 from _pytest.config import Config
 from _pytest.config.argparsing import Parser
 from _pytest.nodes import Item
-from pytest import Collector, fixture
+from pytest import Collector, fixture, hookimpl
 
 if sys.version_info < (3, 10):
     from importlib_metadata import entry_points
@@ -197,6 +197,7 @@ def pytest_runtest_teardown(item: Item) -> None:
         _reseed(item.config, 1)
 
 
+@hookimpl(tryfirst=True)
 def pytest_collection_modifyitems(config: Config, items: List[Item]) -> None:
     if not config.getoption("randomly_reorganize"):
         return
diff --git a/tests/test_pytest_randomly.py b/tests/test_pytest_randomly.py
index 7f2b195..2178597 100644
--- a/tests/test_pytest_randomly.py
+++ b/tests/test_pytest_randomly.py
@@ -505,6 +505,38 @@ def test_doctests_in_txt_files_reordered(ourtestdir):
     ]
 
 
+def test_it_runs_before_stepwise(ourtestdir):
+    ourtestdir.makepyfile(
+        test_one="""
+        def test_a():
+            assert 0
+
+
+        def test_b():
+            assert 0
+        """
+    )
+    out = ourtestdir.runpytest("-v", "--randomly-seed=1", "--stepwise")
+    out.assert_outcomes(failed=1)
+
+    # Now make test_b pass
+    ourtestdir.makepyfile(
+        test_one="""
+        def test_a():
+            assert 0
+
+
+        def test_b():
+            assert 1
+        """
+    )
+    ourtestdir.tmpdir.join("__pycache__").remove()
+    out = ourtestdir.runpytest("-v", "--randomly-seed=1", "--stepwise")
+    out.assert_outcomes(passed=1, failed=1)
+    out = ourtestdir.runpytest("-v", "--randomly-seed=1", "--stepwise")
+    out.assert_outcomes(failed=1)
+
+
 def test_fixtures_get_different_random_state_to_tests(ourtestdir):
     ourtestdir.makepyfile(
         test_one="""