Repository: jwodder/pytest-fail-slow (language: python)

Issue:
consider supporting also setup time?
this plugin is not useful if you want to keep `setup` time under control.
I have fast tests, but with slow setup time.
Would you consider adding support for `setup` as well? maybe with a parameter to control the behavior 


Discussion / hints:

Fix patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d3d2e65..fc73d95 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,8 @@ v0.4.0 (in development)
 -----------------------
 - Drop support for Python 3.6
 - Support Python 3.11 and 3.12
+- Added `@pytest.mark.fail_slow_setup()` marker and `--fail-slow-setup`
+  command-line option for failing tests whose setups take too long to run
 
 v0.3.0 (2022-08-12)
 -------------------
diff --git a/README.rst b/README.rst
index 7488b30..0903844 100644
--- a/README.rst
+++ b/README.rst
@@ -26,14 +26,10 @@
 | `Issues <https://github.com/jwodder/pytest-fail-slow/issues>`_
 | `Changelog <https://github.com/jwodder/pytest-fail-slow/blob/master/CHANGELOG.md>`_
 
-``pytest-fail-slow`` is a pytest_ plugin for making tests fail that take too
-long to run.  It adds a ``--fail-slow DURATION`` command-line option to pytest
-that causes any & all otherwise-passing tests that run for longer than the
-given duration to be marked as failures, and it adds a
-``@pytest.mark.fail_slow(DURATION)`` marker for making an individual test fail
-if it runs for longer than the given duration.  If ``--fail-slow`` is given and
-a test has the ``@fail_slow()`` marker, the duration given by the marker takes
-precedence for that test.
+``pytest-fail-slow`` is a pytest_ plugin for treating tests as failed if they
+took too long to run.  It adds markers for failing tests if they or their setup
+stages run for longer than a given duration, along with command-line options
+for applying the same cutoff to all tests.
 
 Note that slow tests will still be run to completion; if you want them to
 instead be stopped early, use pytest-timeout_.
@@ -41,36 +37,98 @@ instead be stopped early, use pytest-timeout_.
 .. _pytest: https://docs.pytest.org
 .. _pytest-timeout: https://github.com/pytest-dev/pytest-timeout
 
-A duration can be supplied to the ``--fail-slow`` option as either a bare
-floating-point number of seconds or as a floating-point number followed by one
-of the following units (case insensitive):
 
-- ``h``, ``hour``, ``hours``
-- ``m``, ``min``, ``mins``, ``minute``, ``minutes``
-- ``s``, ``sec``, ``secs``, ``second``, ``seconds``
-- ``ms``, ``milli``, ``millisec``, ``milliseconds``
-- ``us``, ``μs``, ``micro``, ``microsec``, ``microseconds``
+Installation
+============
+``pytest-fail-slow`` requires Python 3.7 or higher and pytest 6.0 or higher.
+Just use `pip <https://pip.pypa.io>`_ for Python 3 (You have pip, right?) to
+install it::
+
+    python3 -m pip install pytest-fail-slow
+
+
+Usage
+=====
+
+Failing Slow Tests
+------------------
+
+To cause a specific test to fail if it takes too long to run, apply the
+``fail_slow`` marker to it, with the desired cutoff time as the argument:
+
+.. code:: python
 
-Durations passed to the ``@pytest.mark.fail_slow()`` marker can be either
-ints/floats (for a number of seconds) or strings in the same format as passed
-to ``--fail-slow``.
+    import pytest
 
-If ``pytest-fail-slow`` marks a test as a failure, the output will include the
-test's duration and the duration threshold, like so::
+    @pytest.mark.fail_slow("5s")
+    def test_something_sluggish():
+        ...
+
+In addition, the ``--fail-slow DURATION`` option can be passed to the
+``pytest`` command to affect all tests in that run.  If ``--fail-slow`` is
+given and a test has the ``fail_slow`` marker, the duration given by the marker
+takes precedence for that test.
+
+If a test fails due to being slow, pytest's output will include the test's
+duration and the duration threshold, like so::
 
     ________________________________ test_func ________________________________
     Test passed but took too long to run: Duration 123.0s > 5.0s
 
-**Note:** Only the durations for tests themselves are taken into consideration.
-If a test passes in less than the specified duration, but one or more fixture
-setups/teardowns take longer than the duration, the test will still be marked
-as passing.
+**Note:** This feature only takes the durations for tests themselves into
+consideration.  If a test passes in less than the specified duration, but one
+or more fixture setups/teardowns take longer than the duration, the test will
+still be marked as passing.  To fail a test if the setup takes too long, see
+below.
 
 
-Installation
-============
-``pytest-fail-slow`` requires Python 3.7 or higher and pytest 6.0 or higher.
-Just use `pip <https://pip.pypa.io>`_ for Python 3 (You have pip, right?) to
-install it::
+Failing Slow Setups
+-------------------
 
-    python3 -m pip install pytest-fail-slow
+*New in version 0.4.0*
+
+To cause a specific test to fail if the setup steps for all of its fixtures
+combined take too long to run, apply the ``fail_slow_setup`` marker to it, with
+the desired cutoff time as the argument:
+
+.. code:: python
+
+    import pytest
+
+    @pytest.mark.fail_slow_setup("5s")
+    def test_costly_resource(slow_to_create):
+        ...
+
+Do not apply the marker to the test's fixtures; markers have no effect on
+fixtures.
+
+In addition, the ``--fail-slow-setup DURATION`` option can be passed to the
+``pytest`` command to affect all tests in that run.  If ``--fail-slow-setup``
+is given and a test has the ``fail_slow_setupresou`` marker, the duration given
+by the marker takes precedence for that test.
+
+If the setup for a test takes too long to run, the test will be marked as
+"errored," the test itself will not be run, and pytest's output will include
+the setup stage's duration and the duration threshold, like so::
+
+    _______________________ ERROR at setup of test_func _______________________
+    Setup passed but took too long to run: Duration 123.0s > 5.0s
+
+**Note:** If a test depends on multiple fixtures and just one of them exceeds
+the given duration on its own, the remaining fixtures will still have their
+setup steps run.  Also, all fixture teardowns will still be run after the test
+would have run.
+
+
+Specifying Durations
+--------------------
+
+A duration passed to a marker or command-line option can be either a bare
+number of seconds or else a floating-point number followed by one of the
+following units (case insensitive):
+
+- ``h``, ``hour``, ``hours``
+- ``m``, ``min``, ``mins``, ``minute``, ``minutes``
+- ``s``, ``sec``, ``secs``, ``second``, ``seconds``
+- ``ms``, ``milli``, ``millisec``, ``milliseconds``
+- ``us``, ``μs``, ``micro``, ``microsec``, ``microseconds``


Test patch:
diff --git a/src/pytest_fail_slow.py b/src/pytest_fail_slow.py
index 3c78d2a..a3e97bd 100644
--- a/src/pytest_fail_slow.py
+++ b/src/pytest_fail_slow.py
@@ -1,14 +1,10 @@
 """
 Fail tests that take too long to run
 
-``pytest-fail-slow`` is a pytest_ plugin for making tests fail that take too
-long to run.  It adds a ``--fail-slow DURATION`` command-line option to pytest
-that causes any & all otherwise-passing tests that run for longer than the
-given duration to be marked as failures, and it adds a
-``@pytest.mark.fail_slow(DURATION)`` marker for making an individual test fail
-if it runs for longer than the given duration.  If ``--fail-slow`` is given and
-a test has the ``@fail_slow()`` marker, the duration given by the marker takes
-precedence for that test.
+``pytest-fail-slow`` is a pytest_ plugin for treating tests as failed if they
+took too long to run.  It adds markers for failing tests if they or their setup
+stages run for longer than a given duration, along with command-line options
+for applying the same cutoff to all tests.
 
 Note that slow tests will still be run to completion; if you want them to
 instead be stopped early, use pytest-timeout_.
@@ -16,31 +12,6 @@
 .. _pytest: https://docs.pytest.org
 .. _pytest-timeout: https://github.com/pytest-dev/pytest-timeout
 
-A duration can be supplied to the ``--fail-slow`` option as either a bare
-floating-point number of seconds or as a floating-point number followed by one
-of the following units (case insensitive):
-
-- ``h``, ``hour``, ``hours``
-- ``m``, ``min``, ``mins``, ``minute``, ``minutes``
-- ``s``, ``sec``, ``secs``, ``second``, ``seconds``
-- ``ms``, ``milli``, ``millisec``, ``milliseconds``
-- ``us``, ``μs``, ``micro``, ``microsec``, ``microseconds``
-
-Durations passed to the ``@pytest.mark.fail_slow()`` marker can be either
-ints/floats (for a number of seconds) or strings in the same format as passed
-to ``--fail-slow``.
-
-If ``pytest-fail-slow`` marks a test as a failure, the output will include the
-test's duration and the duration threshold, like so::
-
-    ________________________________ test_func ________________________________
-    Test passed but took too long to run: Duration 123.0s > 5.0s
-
-**Note:** Only the durations for tests themselves are taken into consideration.
-If a test passes in less than the specified duration, but one or more fixture
-setups/teardowns take longer than the duration, the test will still be marked
-as passing.
-
 Visit <https://github.com/jwodder/pytest-fail-slow> for more information.
 """
 
@@ -94,6 +65,13 @@ def pytest_configure(config) -> None:
         "markers",
         "fail_slow(duration): Fail test if it takes more than this long to run",
     )
+    config.addinivalue_line(
+        "markers",
+        (
+            "fail_slow_setup(duration):"
+            " Fail test if it takes more than this long to set up"
+        ),
+    )
 
 
 def pytest_addoption(parser) -> None:
@@ -103,29 +81,48 @@ def pytest_addoption(parser) -> None:
         metavar="DURATION",
         help="Fail tests that take more than this long to run",
     )
+    parser.addoption(
+        "--fail-slow-setup",
+        type=parse_duration,
+        metavar="DURATION",
+        help="Fail tests that take more than this long to set up",
+    )
 
 
 @pytest.hookimpl(hookwrapper=True)
 def pytest_runtest_makereport(item, call):
     report = (yield).get_result()
-    mark = item.get_closest_marker("fail_slow")
-    if mark is not None:
-        if len(mark.args) != 1:
-            raise pytest.UsageError(
-                "@pytest.mark.fail_slow() takes exactly one argument"
+    if report is None or report.outcome != "passed":
+        return
+    if report.when == "setup":
+        mark = item.get_closest_marker("fail_slow_setup")
+        if mark is not None:
+            if len(mark.args) != 1:
+                raise pytest.UsageError(
+                    "@pytest.mark.fail_slow_setup() takes exactly one argument"
+                )
+            timeout = parse_duration(mark.args[0])
+        else:
+            timeout = item.config.getoption("--fail-slow-setup")
+        if timeout is not None and call.duration > timeout:
+            report.outcome = "failed"
+            report.longrepr = (
+                "Setup passed but took too long to run:"
+                f" Duration {call.duration}s > {timeout}s"
+            )
+    elif report.when == "call":
+        mark = item.get_closest_marker("fail_slow")
+        if mark is not None:
+            if len(mark.args) != 1:
+                raise pytest.UsageError(
+                    "@pytest.mark.fail_slow() takes exactly one argument"
+                )
+            timeout = parse_duration(mark.args[0])
+        else:
+            timeout = item.config.getoption("--fail-slow")
+        if timeout is not None and call.duration > timeout:
+            report.outcome = "failed"
+            report.longrepr = (
+                "Test passed but took too long to run:"
+                f" Duration {call.duration}s > {timeout}s"
             )
-        timeout = parse_duration(mark.args[0])
-    else:
-        timeout = item.config.getoption("--fail-slow")
-    if (
-        report is not None
-        and timeout is not None
-        and report.when == "call"
-        and report.outcome == "passed"
-        and call.duration > timeout
-    ):
-        report.outcome = "failed"
-        report.longrepr = (
-            f"Test passed but took too long to run: Duration {call.duration}s >"
-            f" {timeout}s"
-        )
diff --git a/test/test_fail_slow.py b/test/test_fail_slow.py
index 6187989..ec2db9f 100644
--- a/test/test_fail_slow.py
+++ b/test/test_fail_slow.py
@@ -1,4 +1,4 @@
-from typing import List
+from __future__ import annotations
 import pytest
 
 CASES = [
@@ -78,7 +78,7 @@ def test_fail_slow_threshold(
     src: str,
     success: bool,
     slow: bool,
-    args: List[str],
+    args: list[str],
     decor: str,
     limitrgx: str,
 ) -> None:
diff --git a/test/test_fail_slow_setup.py b/test/test_fail_slow_setup.py
new file mode 100644
index 0000000..2c3c8a1
--- /dev/null
+++ b/test/test_fail_slow_setup.py
@@ -0,0 +1,181 @@
+from __future__ import annotations
+import pytest
+
+
+@pytest.mark.parametrize(
+    "args,decor,limitrgx",
+    [
+        (["--fail-slow-setup=2"], "", r"2\.\d+s"),
+        (["--fail-slow-setup=2.0"], "", r"2\.\d+s"),
+        (["--fail-slow-setup=0.0333m"], "", r"1\.9\d+s"),
+        (["--fail-slow-setup=10"], "", None),
+        ([], "@pytest.mark.fail_slow_setup(2)\n", r"2s"),
+        ([], "@pytest.mark.fail_slow_setup(2.0)\n", r"2\.\d+s"),
+        ([], "@pytest.mark.fail_slow_setup('0.0333m')\n", r"1\.9\d+s"),
+        ([], "@pytest.mark.fail_slow_setup(10)\n", None),
+        (["--fail-slow-setup=30"], "@pytest.mark.fail_slow_setup(2)\n", r"2s"),
+        (["--fail-slow-setup=0.01"], "@pytest.mark.fail_slow_setup(2)\n", r"2s"),
+        (["--fail-slow-setup=0.01"], "@pytest.mark.fail_slow_setup(10)\n", None),
+    ],
+)
+def test_fail_slow_setup_threshold(
+    pytester, args: list[str], decor: str, limitrgx: str | None
+) -> None:
+    pytester.makepyfile(
+        test_func=(
+            "from time import sleep\n"
+            "import pytest\n"
+            "\n"
+            "@pytest.fixture\n"
+            "def slow_setup():\n"
+            "    sleep(3)\n"
+            "    yield\n"
+            "    sleep(3)\n"
+            "\n"
+            f"{decor}def test_func(slow_setup):\n"
+            "    assert 2 + 2 == 4\n"
+        )
+    )
+    result = pytester.runpytest(*args)
+    if limitrgx is None:
+        result.assert_outcomes(passed=1)
+        result.stdout.no_fnmatch_line("*Setup passed but took too long to run*")
+    else:
+        result.assert_outcomes(errors=1)
+        result.stdout.re_match_lines(
+            [
+                r"_+ ERROR at setup of test_func _+$",
+                "Setup passed but took too long to run:"
+                rf" Duration \d+\.\d+s > {limitrgx}$",
+            ],
+            consecutive=True,
+        )
+
+
+@pytest.mark.parametrize("threshold,limitrgx", [(4, "4s"), (10, None)])
+def test_fail_slow_multi_setup(pytester, threshold: str, limitrgx: str | None) -> None:
+    pytester.makepyfile(
+        test_func=(
+            "from time import sleep\n"
+            "import pytest\n"
+            "\n"
+            "@pytest.fixture\n"
+            "def slow_setup():\n"
+            "    sleep(2)\n"
+            "\n"
+            "@pytest.fixture\n"
+            "def slower_setup():\n"
+            "    sleep(3)\n"
+            "\n"
+            f"@pytest.mark.fail_slow_setup({threshold})\n"
+            "def test_func(slow_setup, slower_setup):\n"
+            "    assert 2 + 2 == 4\n"
+        )
+    )
+    result = pytester.runpytest()
+    if limitrgx is None:
+        result.assert_outcomes(passed=1)
+        result.stdout.no_fnmatch_line("*Setup passed but took too long to run*")
+    else:
+        result.assert_outcomes(errors=1)
+        result.stdout.re_match_lines(
+            [
+                r"_+ ERROR at setup of test_func _+$",
+                "Setup passed but took too long to run:"
+                rf" Duration \d+\.\d+s > {limitrgx}$",
+            ],
+            consecutive=True,
+        )
+
+
+@pytest.mark.parametrize("threshold,success", [(2, False), (5, True)])
+def test_fail_slow_setup_skips_test(pytester, threshold: str, success: bool) -> None:
+    pytester.makepyfile(
+        test_func=(
+            "from pathlib import Path\n"
+            "from time import sleep\n"
+            "import pytest\n"
+            "\n"
+            "@pytest.fixture\n"
+            "def slow_setup():\n"
+            "    sleep(3)\n"
+            "\n"
+            f"@pytest.mark.fail_slow_setup({threshold})\n"
+            "def test_func(slow_setup):\n"
+            '    Path("test.txt").write_text("Tested\\n")\n'
+        )
+    )
+    result = pytester.runpytest()
+    if success:
+        result.assert_outcomes(passed=1)
+        assert (pytester.path / "test.txt").read_text() == "Tested\n"
+    else:
+        result.assert_outcomes(errors=1)
+        assert not (pytester.path / "test.txt").exists()
+
+
+def test_fail_slow_setup_teardown_still_run(pytester) -> None:
+    pytester.makepyfile(
+        test_func=(
+            "from pathlib import Path\n"
+            "from time import sleep\n"
+            "import pytest\n"
+            "\n"
+            "@pytest.fixture\n"
+            "def slow_setup():\n"
+            "    sleep(3)\n"
+            "    yield\n"
+            '    Path("teardown.txt").write_text("Torn down\\n")\n'
+            "\n"
+            "@pytest.mark.fail_slow_setup(2)\n"
+            "def test_func(slow_setup):\n"
+            "    assert 2 + 2 == 4\n"
+        )
+    )
+    result = pytester.runpytest()
+    result.assert_outcomes(errors=1)
+    assert (pytester.path / "teardown.txt").read_text() == "Torn down\n"
+
+
+def test_fail_slow_setup_all_run(pytester) -> None:
+    pytester.makepyfile(
+        test_func=(
+            "from pathlib import Path\n"
+            "from time import sleep\n"
+            "import pytest\n"
+            "\n"
+            "@pytest.fixture\n"
+            "def slow_setup():\n"
+            "    sleep(3)\n"
+            "\n"
+            "@pytest.fixture\n"
+            "def quick_setup():\n"
+            '    Path("quick.txt").write_text("Set up\\n")\n'
+            "\n"
+            "@pytest.mark.fail_slow_setup(2)\n"
+            "def test_func(slow_setup, quick_setup):\n"
+            "    assert 2 + 2 == 4\n"
+        )
+    )
+    result = pytester.runpytest()
+    result.assert_outcomes(errors=1)
+    assert (pytester.path / "quick.txt").read_text() == "Set up\n"
+
+
+@pytest.mark.parametrize("args", ["", "42, 'foo'"])
+def test_fail_slow_setup_marker_bad_args(pytester, args: str) -> None:
+    pytester.makepyfile(
+        test_func=(
+            "import pytest\n"
+            "\n"
+            f"@pytest.mark.fail_slow_setup({args})\n"
+            "def test_func():\n"
+            "    assert 2 + 2 == 4\n"
+        )
+    )
+    result = pytester.runpytest()
+    result.assert_outcomes()
+    result.stdout.no_fnmatch_line("?")
+    result.stderr.fnmatch_lines(
+        ["ERROR: @pytest.mark.fail_slow_setup() takes exactly one argument"]
+    )