Repository: pytest-dev/pytest-order (language: python)

Issue:
Optionally fail tests if order cannot be achieved [enhancement]
At present, users can create logically incompatible orderings (e.g. A after B, A before C, but C before B).  This results in silent skipping of the offending test with the warning log
```
WARNING: cannot execute 'test_a' relative to others: 'testb.py::test_b'
```

Unfortunately this can be quite a silent/subtle error.  It would be awesome if there were a flag we could add to ensure that an error is raised (e.g. tests fail) if not all tests can be ran.


Discussion / hints:
>  This results in silent skipping of the offending test

I guess you are using `pytest-dependency` markers, so the test will be skipped if it cannot be ordered. If using `pytest-order` to order the tests, it would not be skipped.

So, in your example, would you expect 'test_a' to fail if it would be skipped (e.g. for a dependency marker), or also if the ordering could not be done using `order` markers? The latter will not skip any tests, but might result in failing tests if the order matters for test execution, of course...
Hi @mrbean-bremen - we aren't using `pytest-dependency`, just `pytest-order`, so our tests are decorated like
```
import pytest

@pytest.mark.order(1)
def test_foo():
   ...
```

In the example I provided, I'd expect there to be some way to (optionally perhaps?) fail loudly if not all tests can be executed in accordance with the `order` markers provided.  This will just help prevent silent failures.
Ok, so you are using relative order markers (e.g. `before` or `after` markers), right? I'm still unclear why you write that tests are skipped in this case.
I understand that you want the tests to fail, my question was if you want the test that could not be ordered to fail, or something else (like failing all tests).
Yes, we are using relative order markers @mrbean-bremen.  I think tests are skipped because the relative order we've specified isn't possible.

I think probably having the tests that could not be ordered fail would make the most sense 🤔 but open to having the whole process fail during setup if that makes more sense.
Ok - tests are not skipped by pytest-order if they cannot be ordered (this is what pytest-dependency does), but I will have a look if I can make them fail as an option.
Huge, thank you so much @mrbean-bremen 

Fix patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d796cc5..3d7e5eb 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,10 @@
 
 ## Unreleased
 
+### New features
+- added option `--error-on-failed-ordering` to make tests that cannot be ordered fail
+  (see [#140](https://github.com/pytest-dev/pytest-order/issues/140))
+
 ## Infrastructure
 - added missing documentation examples, structured examples according to documentation structure
 
diff --git a/docs/source/configuration.rst b/docs/source/configuration.rst
index a9714f6..275762b 100644
--- a/docs/source/configuration.rst
+++ b/docs/source/configuration.rst
@@ -603,6 +603,49 @@ tests first from the start, then from the end if there are negative numbers,
 and the rest will be in between (e.g. between positive and negative numbers),
 as it is without this option.
 
+``--error-on-failed-ordering``
+------------------------------
+Relative ordering of tests my fail under some circumstances. Mostly this happens if the related marker
+is not found, or if the tests have a cyclic dependency.
+The default behavior in this case is not to order the test in question, issue a warning during test
+collection and execute the test as usual. If you want to make sure that your relative markers work
+without checking all warning messages, you can also make the tests that cannot be ordered fail, so that
+they show up as errored in the report:
+
+.. code:: python
+
+ import pytest
+
+
+ def test_one():
+     assert True
+
+
+ @pytest.mark.order(before="test_three")
+ def test_two():
+     assert True
+
+
+In this example, the test "test_three" on which "test_two" depends, does not exist.
+If you use the option `--error-on-failed-ordering`, "test_two" will now error:
+
+    $ pytest tests -vv --error-on-failed-ordering
+    ============================= test session starts ==============================
+    ...
+    WARNING: cannot execute 'test_two' relative to others: 'test_three'
+
+    test_failed_ordering.py::test_one PASSED
+    test_failed_ordering.py::test_two ERROR
+
+    =================================== ERRORS ====================================
+    __________________________ ERROR at setup of test_two__________________________
+    ...
+    =========================== short test summary info ===========================
+    ERROR test_failed_ordering.py::test_two - Failed: The test could not be ordered
+    ========================= 1 passed, 1 error in 0.75s ==========================
+
+
+
 .. _`pytest-dependency`: https://pypi.org/project/pytest-dependency/
 .. _`dynamic compilation of marked parameters`: https://pytest-dependency.readthedocs.io/en/stable/advanced.html#dynamic-compilation-of-marked-parameters
 .. _`add dependencies at runtime`: https://pytest-dependency.readthedocs.io/en/stable/usage.html#marking-dependencies-at-runtime


Test patch:
diff --git a/perf_tests/test_dependencies.py b/perf_tests/test_dependencies.py
index a70bcc9..37dd527 100644
--- a/perf_tests/test_dependencies.py
+++ b/perf_tests/test_dependencies.py
@@ -10,24 +10,24 @@
 @pytest.fixture
 def fixture_path_relative(testdir):
     for i_mod in range(10):
-        test_name = testdir.tmpdir.join("test_dep_perf{}.py".format(i_mod))
+        test_name = testdir.tmpdir.join(f"test_dep_perf{i_mod}.py")
         test_contents = "import pytest\n"
         for i in range(40):
             test_contents += dedent(
-                """
-                @pytest.mark.dependency(depends=["test_{}"])
-                def test_{}():
+                f"""
+                @pytest.mark.dependency(depends=["test_{i + 50}"])
+                def test_{i}():
                     assert True
                 """
-            ).format(i + 50, i)
+            )
         for i in range(60):
             test_contents += dedent(
-                """
+                f"""
                 @pytest.mark.dependency
-                def test_{}():
+                def test_{i + 40}():
                     assert True
                 """
-            ).format(i + 40)
+            )
         test_name.write(test_contents)
     yield testdir
 
diff --git a/perf_tests/test_ordinal.py b/perf_tests/test_ordinal.py
index ff18e1b..a269bb2 100644
--- a/perf_tests/test_ordinal.py
+++ b/perf_tests/test_ordinal.py
@@ -10,16 +10,16 @@
 @pytest.fixture
 def fixture_path_ordinal(testdir):
     for i_mod in range(10):
-        test_name = testdir.tmpdir.join("test_performance{}.py".format(i_mod))
+        test_name = testdir.tmpdir.join(f"test_performance{i_mod}.py")
         test_contents = "import pytest\n"
         for i in range(100):
             test_contents += dedent(
-                """
-                @pytest.mark.order({})
-                def test_{}():
+                f"""
+                @pytest.mark.order({50 - i})
+                def test_{i}():
                     assert True
                 """
-            ).format(50 - i, i)
+            )
         test_name.write(test_contents)
     yield testdir
 
diff --git a/perf_tests/test_relative.py b/perf_tests/test_relative.py
index 7a922be..f70e3dd 100644
--- a/perf_tests/test_relative.py
+++ b/perf_tests/test_relative.py
@@ -10,23 +10,23 @@
 @pytest.fixture
 def fixture_path_relative(testdir):
     for i_mod in range(10):
-        test_name = testdir.tmpdir.join("test_relative_perf{}.py".format(i_mod))
+        test_name = testdir.tmpdir.join(f"test_relative_perf{i_mod}.py")
         test_contents = "import pytest\n"
         for i in range(40):
             test_contents += dedent(
-                """
-                @pytest.mark.order(after="test_{}")
-                def test_{}():
+                f"""
+                @pytest.mark.order(after="test_{i + 50}")
+                def test_{i}():
                     assert True
                 """
-            ).format(i + 50, i)
+            )
         for i in range(60):
             test_contents += dedent(
-                """
-                def test_{}():
+                f"""
+                def test_{i + 40}():
                     assert True
                 """
-            ).format(i + 40)
+            )
         test_name.write(test_contents)
     yield testdir
 
diff --git a/perf_tests/test_relative_dense.py b/perf_tests/test_relative_dense.py
index 3b99bfe..54158b0 100644
--- a/perf_tests/test_relative_dense.py
+++ b/perf_tests/test_relative_dense.py
@@ -10,23 +10,23 @@
 @pytest.fixture
 def fixture_path_relative_dense(testdir):
     for i_mod in range(10):
-        test_name = testdir.tmpdir.join("test_relative_dense_perf{}.py".format(i_mod))
+        test_name = testdir.tmpdir.join(f"test_relative_dense_perf{i_mod}.py")
         test_contents = "import pytest\n"
         for i in range(90):
             test_contents += dedent(
-                """
-                @pytest.mark.order(after="test_{}")
-                def test_{}():
+                f"""
+                @pytest.mark.order(after="test_{i + 10}")
+                def test_{i}():
                     assert True
                 """
-            ).format(i + 10, i)
+            )
         for i in range(10):
             test_contents += dedent(
-                """
-                def test_{}():
+                f"""
+                def test_{i + 90}():
                     assert True
                 """
-            ).format(i + 90)
+            )
         test_name.write(test_contents)
     yield testdir
 
diff --git a/perf_tests/util.py b/perf_tests/util.py
index 7296b94..8384d22 100644
--- a/perf_tests/util.py
+++ b/perf_tests/util.py
@@ -12,5 +12,5 @@ def sort_items(self):
         start_time = time.time()
         items = super().sort_items()
         self.__class__.elapsed = (time.time() - start_time) / self.nr_marks * 1000
-        print("\nTime per test: {:.3f} ms".format(self.__class__.elapsed))
+        print(f"\nTime per test: {self.__class__.elapsed:.3f} ms")
         return items
diff --git a/pytest_order/item.py b/pytest_order/item.py
index 73950e7..1364f39 100644
--- a/pytest_order/item.py
+++ b/pytest_order/item.py
@@ -102,15 +102,20 @@ def sort_numbered_items(self) -> List[Item]:
         return sorted_list
 
     def print_unhandled_items(self) -> None:
-        msg = " ".join(
-            [mark.item.node_id for mark in self.rel_marks]
-            + [mark.item.node_id for mark in self.dep_marks]
-        )
-        if msg:
-            sys.stdout.write("\nWARNING: cannot execute test relative to others: ")
-            sys.stdout.write(msg)
+        failed_items = [mark.item for mark in self.rel_marks] + [
+            mark.item for mark in self.dep_marks
+        ]
+        msg = " ".join([item.node_id for item in failed_items])
+        sys.stdout.write("\nWARNING: cannot execute test relative to others: ")
+        sys.stdout.write(msg)
+        if self.settings.error_on_failed_ordering:
             sys.stdout.write(" - ignoring the marker.\n")
-            sys.stdout.flush()
+        else:
+            sys.stdout.write(".\n")
+        sys.stdout.flush()
+        if self.settings.error_on_failed_ordering:
+            for item in failed_items:
+                item.item.fixturenames.insert(0, "fail_after_cannot_order")
 
     def number_of_rel_groups(self) -> int:
         return len(self.rel_marks) + len(self.dep_marks)
diff --git a/pytest_order/plugin.py b/pytest_order/plugin.py
index 93dbd79..0be654d 100644
--- a/pytest_order/plugin.py
+++ b/pytest_order/plugin.py
@@ -110,6 +110,15 @@ def pytest_addoption(parser: Parser) -> None:
             "are handled like order markers with an index."
         ),
     )
+    group.addoption(
+        "--error-on-failed-ordering",
+        action="store_true",
+        dest="error_on_failed_ordering",
+        help=(
+            "If set, tests with relative markers that could not be ordered "
+            "will error instead of generating only a warning."
+        ),
+    )
 
 
 def _get_mark_description(mark: Mark):
@@ -146,6 +155,11 @@ def pytest_generate_tests(metafunc):
                 metafunc.parametrize("order", args)
 
 
+@pytest.fixture
+def fail_after_cannot_order():
+    pytest.fail("The test could not be ordered")
+
+
 class OrderingPlugin:
     """
     Plugin implementation.
diff --git a/pytest_order/settings.py b/pytest_order/settings.py
index 2da6eb8..e020adf 100644
--- a/pytest_order/settings.py
+++ b/pytest_order/settings.py
@@ -23,21 +23,23 @@ def __init__(self, config: Config) -> None:
         self.sparse_ordering: bool = config.getoption("sparse_ordering")
         self.order_dependencies: bool = config.getoption("order_dependencies")
         self.marker_prefix: str = config.getoption("order_marker_prefix")
+        self.error_on_failed_ordering: str = config.getoption(
+            "error_on_failed_ordering"
+        )
         scope: str = config.getoption("order_scope")
         if scope in self.valid_scopes:
             self.scope: Scope = self.valid_scopes[scope]
         else:
             if scope is not None:
                 warn(
-                    "Unknown order scope '{}', ignoring it. "
-                    "Valid scopes are 'session', 'module' and 'class'.".format(scope)
+                    f"Unknown order scope '{scope}', ignoring it. "
+                    "Valid scopes are 'session', 'module' and 'class'."
                 )
             self.scope = Scope.SESSION
         scope_level: int = config.getoption("order_scope_level") or 0
         if scope_level != 0 and self.scope != Scope.SESSION:
             warn(
-                "order-scope-level cannot be used together with "
-                "--order-scope={}".format(scope)
+                f"order-scope-level cannot be used together with --order-scope={scope}"
             )
             scope_level = 0
         self.scope_level: int = scope_level
@@ -47,10 +49,8 @@ def __init__(self, config: Config) -> None:
         else:
             if group_scope is not None:
                 warn(
-                    "Unknown order group scope '{}', ignoring it. "
-                    "Valid scopes are 'session', 'module' and 'class'.".format(
-                        group_scope
-                    )
+                    f"Unknown order group scope '{group_scope}', ignoring it. "
+                    "Valid scopes are 'session', 'module' and 'class'."
                 )
             self.group_scope = self.scope
         if self.group_scope.value > self.scope.value:
diff --git a/pytest_order/sorter.py b/pytest_order/sorter.py
index 1c67305..d5e7053 100644
--- a/pytest_order/sorter.py
+++ b/pytest_order/sorter.py
@@ -162,7 +162,7 @@ def handle_order_mark(self, item: Item, mark: Mark) -> None:
             elif order in orders_map:
                 order = orders_map[order]
             else:
-                warn("Unknown order attribute:'{}'".format(order))
+                warn(f"Unknown order attribute:'{order}'")
                 order = None
         if item.order is None:
             item.order = order
@@ -269,11 +269,15 @@ def handle_relative_marks(self, item: Item, mark: Mark) -> bool:
                 self.warn_about_unknown_test(item, after_mark)
         return has_relative_marks
 
-    @staticmethod
-    def warn_about_unknown_test(item: Item, rel_mark: str) -> None:
+    def warn_about_unknown_test(self, item: Item, rel_mark: str) -> None:
+        if self.settings.error_on_failed_ordering:
+            item.item.fixturenames.insert(0, "fail_after_cannot_order")
+            ignore_msg = ""
+        else:
+            ignore_msg = " - ignoring the marker"
         sys.stdout.write(
-            "\nWARNING: cannot execute '{}' relative to others: "
-            "'{}' - ignoring the marker.".format(item.item.name, rel_mark)
+            f"\nWARNING: cannot execute '{item.item.name}' relative to others: "
+            f"'{rel_mark}'{ignore_msg}."
         )
 
     def collect_markers(self) -> None:
@@ -303,8 +307,8 @@ def resolve_dependency_markers(
                         )
                 else:
                     sys.stdout.write(
-                        "\nWARNING: Cannot resolve the dependency marker '{}' "
-                        "- ignoring it.".format(name)
+                        f"\nWARNING: Cannot resolve the dependency marker '{name}' "
+                        "- ignoring it."
                     )
 
     @staticmethod
diff --git a/tests/test_order_scope_level.py b/tests/test_order_scope_level.py
index 6a19472..624811b 100644
--- a/tests/test_order_scope_level.py
+++ b/tests/test_order_scope_level.py
@@ -28,7 +28,7 @@ def test_one(): pass
         """
     )
     for i in range(3):
-        sub_path = "feature{}".format(i)
+        sub_path = f"feature{i}"
         test_path.mkpydir(sub_path)
         path = test_path.tmpdir.join(sub_path, "test_a.py")
         path.write(test_a_contents)
diff --git a/tests/test_relative_ordering.py b/tests/test_relative_ordering.py
index bf7ea84..095ca39 100644
--- a/tests/test_relative_ordering.py
+++ b/tests/test_relative_ordering.py
@@ -419,6 +419,36 @@ def test_3():
     assert warning in out
 
 
+def test_failing_dependency_fails_test(test_path):
+    test_path.makepyfile(
+        test_failed_ordering="""
+        import pytest
+
+        def test_1():
+            pass
+
+        @pytest.mark.order(before="test_4")
+        def test_2():
+            pass
+
+        def test_3():
+            pass
+        """
+    )
+    result = test_path.runpytest("-v", "--error-on-failed-ordering")
+    if int(pytest.__version__.split(".")[0]) < 6:
+        result.assert_outcomes(passed=2, error=1)
+    else:
+        result.assert_outcomes(passed=2, errors=1)
+    result.stdout.fnmatch_lines(
+        [
+            "test_failed_ordering.py::test_1 PASSED",
+            "test_failed_ordering.py::test_2 ERROR",
+            "test_failed_ordering.py::test_3 PASSED",
+        ]
+    )
+
+
 def test_dependency_in_class_before_unknown_test(item_names_for, capsys):
     test_content = """
         import pytest
@@ -470,6 +500,38 @@ def test_3():
     assert warning in out
 
 
+def test_failed_tests_after_dependency_loop(test_path):
+    test_path.makepyfile(
+        test_failed_ordering="""
+        import pytest
+
+        @pytest.mark.order(after="test_3")
+        def test_1():
+            pass
+
+        @pytest.mark.order(1)
+        def test_2():
+            pass
+
+        @pytest.mark.order(before="test_1")
+        def test_3():
+            pass
+        """
+    )
+    result = test_path.runpytest("-v", "--error-on-failed-ordering")
+    if int(pytest.__version__.split(".")[0]) < 6:
+        result.assert_outcomes(passed=1, error=2)
+    else:
+        result.assert_outcomes(passed=1, errors=2)
+    result.stdout.fnmatch_lines(
+        [
+            "test_failed_ordering.py::test_2 PASSED",
+            "test_failed_ordering.py::test_1 ERROR",
+            "test_failed_ordering.py::test_3 ERROR",
+        ]
+    )
+
+
 def test_dependency_on_parametrized_test(item_names_for):
     test_content = """
         import pytest