Repository: okken/pytest-check (language: python)

Issue:
pytest-check fails with pytest's  --maxfail option
pytest-check can not be used in combination with the `--maxfail` option because `@check_func` seems to have no effect and `AssertionError`s are not caught.

### minimal example
Use example from readme as `test_check.py`:
```python
def test_example(check):
    a = 1
    b = 2
    c = [2, 4, 6]
    check.greater(a, b)
    check.less_equal(b, a)
    check.is_in(a, c, "Is 1 in the list")
    check.is_not_in(b, c, "make sure 2 isn't in list")
```
start with `pytest.exe --maxfail=10  test_check.py`

### result
```
================================================ test session starts =================================================
platform win32 -- Python 3.7.9, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: XXXX
plugins: pytest_check-1.0.1, cov-2.12.1
collected 1 item

test_check.py F                                                                                                [100%]

====================================================== FAILURES ====================================================== 
____________________________________________________ test_example ____________________________________________________ 

check = <module 'pytest_check.check_methods' from 'XXXX\\lib\\site-packages\\pytest_check\\check_methods.py'>

    def test_example(check):
        a = 1
        b = 2
        c = [2, 4, 6]
>       check.greater(a, b)

test_check.py:5:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

a = 1, b = 2, msg = ''

    @check_func
    def greater(a, b, msg=""):
>       assert a > b, msg
E       AssertionError: 
E       assert 1 > 2

XXXX\lib\site-packages\pytest_check\check_methods.py:175: AssertionError
============================================== short test summary info ============================================== 
FAILED test_check.py::test_example - AssertionError:
================================================= 1 failed in 0.18s ================================================= 
```
Same results with linux (docker) with python 3.9.



Discussion / hints:
It currently works as:
- if maxfail == 1 (or -x), then only one failure is allowed. First failed check stops testing.
- if maxfail >= 2, then the maxfail count refers to the failing test functions, not the failing checks.

In other words, one test with 5 failed checks still counts as one failed test.
The exception to this rule is for -x or --maxfail=1, which then stops on the first failed check

I could see the argument in both directions of whether maxfail should count checks or functions.
Right now it most closely aligns with functions.
Changing to counting checks would be possible, but would change the behavior of the plugin.
At this point in it's adoption, I don't think I want to change the behavior.

However, documenting this behavior would be good.
So thank you for submitting this issue.
closing this and opening #65 
@okken Unfortunately, I cannot understand your answer.
But maybe I didn't make my ticket clear enough, I thought that would be reflected in the console output (With `--maxfail=10` the tests fails with the **first** `assert`).
To make my description a little clearer:
1. Without `--maxfail` the outputs are as expected and multiple asserts can fail within one test.
2. With `--maxfail` pytest-check doesn't seem to have any effect. Obviously, the PlugIn cannot be used in combination, the test fails with the first failed assert.
Ok. Thanks for being persistent.
I can reproduce the error now.

Fix patch:
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
deleted file mode 100644
index 98e70e4..0000000
--- a/.pre-commit-config.yaml
+++ /dev/null
@@ -1,10 +0,0 @@
-repos:
--   repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.0.1
-    hooks:
-    -   id: end-of-file-fixer
-    -   id: trailing-whitespace
--   repo: https://github.com/psf/black
-    rev: 21.7b0
-    hooks:
-    -   id: black
diff --git a/README.md b/README.md
index c211dc6..89abc50 100644
--- a/README.md
+++ b/README.md
@@ -164,6 +164,16 @@ so you will need to use multiple `with check:` blocks for multiple asserts:
 
 ```
 
+## maxfail behavior
+
+Setting `-x` or `--maxfail=1` will cause this plugin to abort testing after the first failed check.
+
+Setting `-maxfail=2` or greater will turn off any handling of maxfail within this plugin and the behavior is controlled by pytest.
+
+In other words, the `maxfail` count is counting tests, not checks.
+The exception is the case of `1`, where we want to stop on the very first failed check.
+
+
 ## Contributing
 
 Contributions are very welcome. Tests can be run with [tox](https://tox.readthedocs.io/en/latest/).
@@ -181,6 +191,11 @@ If you encounter any problems, please [file an issue](https://github.com/okken/p
 
 ## Changelog
 
+* 1.0.3
+    * Fix #64, modifications to maxfail behavior.
+* 1.0.2
+    * Add `excinfo` to call object in `pytest_runtest_makereport()`.
+    * Intended to help with some report systems.
 * 1.0.1
     * Remove Beta Classifier
     * Status is now "Development Status :: 5 - Production/Stable"


Test patch:
diff --git a/src/pytest_check/__init__.py b/src/pytest_check/__init__.py
index 817fc84..8122d37 100644
--- a/src/pytest_check/__init__.py
+++ b/src/pytest_check/__init__.py
@@ -4,4 +4,4 @@
 
 pytest.register_assert_rewrite("pytest_check.check")
 
-__version__ = "1.0.2"
+__version__ = "1.0.3"
diff --git a/src/pytest_check/plugin.py b/src/pytest_check/plugin.py
index a6fe968..d710a7e 100644
--- a/src/pytest_check/plugin.py
+++ b/src/pytest_check/plugin.py
@@ -43,13 +43,13 @@ def pytest_runtest_makereport(item, call):
 
 
 def pytest_configure(config):
-    check_methods.set_stop_on_fail(config.getoption("-x"))
+    maxfail = config.getvalue("maxfail")
+    stop_on_fail = maxfail == 1
+    # If -x or --maxfail=1, then stop on the first failed check
+    # Otherwise, let pytest stop on the maxfail-th test function failure
+    check_methods.set_stop_on_fail(stop_on_fail)
 
 
 @pytest.fixture(name="check")
 def check_fixture():
     return check_methods
-
-
-# add --log-check-level flag
-# and log_check_level setting
diff --git a/tests/test_maxfail.py b/tests/test_maxfail.py
new file mode 100644
index 0000000..c43366b
--- /dev/null
+++ b/tests/test_maxfail.py
@@ -0,0 +1,48 @@
+import pytest
+
+# flake8: noqa
+
+
+@pytest.fixture()
+def example(testdir):
+    testdir.makepyfile(
+        """
+        from pytest_check import check
+
+        def test_a():
+          with check:
+            assert False, "one"
+          with check:
+            assert False, "two"
+          with check:
+            assert False, "three"
+
+        def test_b():
+            pass
+        """
+    )
+
+
+def test_check_maxfail_1(testdir, example):
+    """
+    Should stop after first failed check
+    """
+    result = testdir.runpytest("--maxfail=1")
+    result.assert_outcomes(failed=1, passed=0)
+    result.stdout.fnmatch_lines(["*AssertionError: one*"])
+
+
+def test_check_maxfail_2(testdir, example):
+    """
+    Should not stop on checks.
+    """
+    result = testdir.runpytest("--maxfail=2")
+    result.assert_outcomes(failed=1, passed=1)
+    result.stdout.fnmatch_lines(
+        [
+            "*FAILURE: one",
+            "*FAILURE: two",
+            "*FAILURE: three",
+            "*Failed Checks: 3*",
+        ]
+    )