Repository: pytest-dev/pytest
Pull request: #9062 - Allow `--stepwise-skip` to implicitly enable `--stepwise`
The ``StepwisePlugin`` plugin instance was only registered with pluggy when ``--stepwise`` was explicitly provided, this could be confusing where users would / could pass ``--stepwise-skip`` in isolation and observe no expected behaviour.

 * ``--stepwise-skip`` can now work alone, implicitly enabling ``--stepwise``

closes #9062 





Problem statement:
`--stepwise-skip` has no effect without `--stepwise`
- [x] a detailed description of the bug or problem you are having

The flag `--stepwise` will "exit on test failure and continue from last failing test".
The flag `--stepwise-skip` is supposed to "ignore the first failing test but stop on the next failing test".
However, it does nothing unless used in conjunction with `--stepwise`. The combo requirement is not mentioned in the help text, seems redundant, and is surprising behavior.

I recommend that `--stepwise-skip` should act the same as `--stepwise --stepwise-skip`.

- [x] output of `pip list` from the virtual environment you are using
```shell
$ pip list
Package    Version
---------- -------
attrs      21.2.0
iniconfig  1.1.1
packaging  21.0
pip        21.2.4
pluggy     0.13.1
py         1.10.0
pyparsing  2.4.7
pytest     6.2.4
setuptools 46.4.0
toml       0.10.2
wheel      0.34.2
```

- [x] pytest and operating system versions

MacOS, Python 3.9.5, pytest-6.2.4

- [x] minimal example if possible

Example **test_step.py**:
```python
def test_one():
  assert False

def test_two():
  assert False

def test_three():
  assert False
```

three failures:
```shell
$ pytest --tb=no test_step.py                
========================= test session starts ==========================
platform darwin -- Python 3.9.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /Users/okken/projects/sw
collected 3 items                                                      

test_step.py FFF                                                 [100%]

======================= short test summary info ========================
FAILED test_step.py::test_one - assert False
FAILED test_step.py::test_two - assert False
FAILED test_step.py::test_three - assert False
========================== 3 failed in 0.01s ===========================

```

`--stepwise-skip` has no effect:
```shell
$ pytest --tb=no --stepwise-skip test_step.py
========================= test session starts ==========================
platform darwin -- Python 3.9.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /Users/okken/projects/sw
collected 3 items                                                      

test_step.py FFF                                                 [100%]

======================= short test summary info ========================
FAILED test_step.py::test_one - assert False
FAILED test_step.py::test_two - assert False
FAILED test_step.py::test_three - assert False
========================== 3 failed in 0.01s ===========================
```

`--stepwise` works as expected, stopping after first failure:
```shell
$ pytest --tb=no --stepwise test_step.py
========================= test session starts ==========================
platform darwin -- Python 3.9.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /Users/okken/projects/sw
collected 3 items                                                      
stepwise: no previously failed tests, not skipping.

test_step.py F

======================= short test summary info ========================
FAILED test_step.py::test_one - assert False
!!!! Interrupted: Test failed, continuing from this test next run. !!!!!
========================== 1 failed in 0.07s ===========================
```

`--stepwise-skip` only works with `--stepwise` to stop after second failure:
```shell

$ pytest --tb=no --stepwise --stepwise-skip test_step.py
========================= test session starts ==========================
platform darwin -- Python 3.9.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /Users/okken/projects/sw
collected 3 items                                                      
stepwise: skipping 0 already passed items.

test_step.py FF

======================= short test summary info ========================
FAILED test_step.py::test_one - assert False
FAILED test_step.py::test_two - assert False
!!!! Interrupted: Test failed, continuing from this test next run. !!!!!
========================== 2 failed in 0.07s ===========================
```

I believe the above behavior, the combo of the flags, should work like that even if only `--stepwise-skip` is used.

Hints from issue discussion:
Agreed, thanks for catching and reporting it. 👍 
Thanks, i'll fix this up (I was in and around that plugin a while ago)

Commit message: #9062 - Allow `--stepwise-skip` to implicitly enable `--stepwise`

Patch to review:
diff --git a/changelog/9062.improvement.rst b/changelog/9062.improvement.rst
new file mode 100644
index 00000000000..7fcaf0408a1
--- /dev/null
+++ b/changelog/9062.improvement.rst
@@ -0,0 +1 @@
+``--stepwise-skip`` now implicitly enables ``--stepwise`` and can be used on it's own.
diff --git a/doc/en/how-to/cache.rst b/doc/en/how-to/cache.rst
index ae865fe280e..b5558eb6bbc 100644
--- a/doc/en/how-to/cache.rst
+++ b/doc/en/how-to/cache.rst
@@ -386,4 +386,4 @@ than speed.
 Stepwise
 --------
 
-As an alternative to ``--lf -x``, especially for cases where you expect a large part of the test suite will fail, ``--sw``, ``--stepwise`` allows you to fix them one at a time. The test suite will run until the first failure and then stop. At the next invocation, tests will continue from the last failing test and then run until the next failing test. You may use the ``--stepwise-skip`` option to ignore one failing test and stop the test execution on the second failing test instead. This is useful if you get stuck on a failing test and just want to ignore it until later.
+As an alternative to ``--lf -x``, especially for cases where you expect a large part of the test suite will fail, ``--sw``, ``--stepwise`` allows you to fix them one at a time. The test suite will run until the first failure and then stop. At the next invocation, tests will continue from the last failing test and then run until the next failing test. You may use the ``--stepwise-skip`` option to ignore one failing test and stop the test execution on the second failing test instead. This is useful if you get stuck on a failing test and just want to ignore it until later.  Providing ``--stepwise-skip`` will also enable ``--stepwise`` implicitly.
diff --git a/src/_pytest/stepwise.py b/src/_pytest/stepwise.py
index 197577c790f..4d95a96b872 100644
--- a/src/_pytest/stepwise.py
+++ b/src/_pytest/stepwise.py
@@ -31,13 +31,16 @@ def pytest_addoption(parser: Parser) -> None:
         action="store_true",
         default=False,
         dest="stepwise_skip",
-        help="ignore the first failing test but stop on the next failing test",
+        help="ignore the first failing test but stop on the next failing test.\n"
+        "implicitly enables --stepwise.",
     )
 
 
 @pytest.hookimpl
 def pytest_configure(config: Config) -> None:
-    # We should always have a cache as cache provider plugin uses tryfirst=True
+    if config.option.stepwise_skip:
+        # allow --stepwise-skip to work on it's own merits.
+        config.option.stepwise = True
     if config.getoption("stepwise"):
         config.pluginmanager.register(StepwisePlugin(config), "stepwiseplugin")
 
diff --git a/testing/test_stepwise.py b/testing/test_stepwise.py
index 85489fce803..63d29d6241f 100644
--- a/testing/test_stepwise.py
+++ b/testing/test_stepwise.py
@@ -248,3 +248,33 @@ def test_d(): pass
             "* 2 passed, 1 deselected, 1 xfailed in *",
         ]
     )
+
+
+def test_stepwise_skip_is_independent(pytester: Pytester) -> None:
+    pytester.makepyfile(
+        """
+        def test_one():
+            assert False
+
+        def test_two():
+            assert False
+
+        def test_three():
+            assert False
+
+        """
+    )
+    result = pytester.runpytest("--tb", "no", "--stepwise-skip")
+    result.assert_outcomes(failed=2)
+    result.stdout.fnmatch_lines(
+        [
+            "FAILED test_stepwise_skip_is_independent.py::test_one - assert False",
+            "FAILED test_stepwise_skip_is_independent.py::test_two - assert False",
+            "*Interrupted: Test failed, continuing from this test next run.*",
+        ]
+    )
+
+
+def test_sw_skip_help(pytester: Pytester) -> None:
+    result = pytester.runpytest("-h")
+    result.stdout.fnmatch_lines("*implicitly enables --stepwise.")


Code review comments:
diff_hunk: @@ -0,0 +1 @@
+``--stepwise-skip`` now implicitly enables ``--stepwise`` and can be used on it's own.
path: changelog/9062.improvement.rst
text: @user1:
```suggestion
``--stepwise-skip`` now implicitly enables ``--stepwise`` and can be used on its own.
```