Repository: HypothesisWorks/hypothesis

Issue:
example.xfail() defaults don't match pytest's defaults, no option to set strict parameter
Caveat: I'm new to both `pytest` and `hypothesis`. As such, it's possible that this is the expected behaviour, but it is simply unintuitive to me. So far my searches in issues and documentation have not yielded any meaningful results, so I'm hoping to be illuminated if this is indeed the expected behaviour. Any pointers to the relevant sections in the documentation are much appreciated.

Consider the following test suite, `test_xfail.py`:

```python
import pytest
from hypothesis import given, strategies as st, example

@given(x=st.integers())
@example(x=0).xfail()
def test_zero_hy(x):
    assert x != 0

@given(x=st.integers())
@example(x=0).xfail()
def test_one_hy(x):
    assert True

@pytest.mark.xfail
@given(x=st.integers())
@example(x=0)
def test_zero_pt(x):
    assert x != 0

@pytest.mark.xfail
@given(x=st.integers())
@example(x=0)
def test_one_pt(x):
    assert True
```

with the following versions installed:
```
platform linux -- Python 3.9.18, pytest-7.4.2, pluggy-1.3.0
hypothesis profile 'default-with-verbose-verbosity' -> verbosity=Verbosity.verbose, database=DirectoryBasedExampleDatabase('/home/nfraser/workspace/hyp_xfail_test/.hypothesis/examples')
rootdir: /home/nfraser/workspace/hyp_xfail_test
plugins: hypothesis-6.84.3
```

Note, I also tried python versions `3.10.12` & `3.11.5` with the same results.

Running the following (with nothing else in the directory except the above file):

```bash
rm -rf .hypothesis/ .pytest_cache/ __pycache__/ && pytest | tail -n 4
```

Note, I removed all other folders to prevent what appears to be another issue from occurring (I'll save that for a separate issue).

```bash
=========================== short test summary info ============================
FAILED test_xfail.py::test_zero_hy - assert 0 != 0
FAILED test_xfail.py::test_one_hy - AssertionError: Expected an exception fro...
=================== 2 failed, 1 xfailed, 1 xpassed in 0.19s ====================
```

I would've expected the results to be `2 xfailed, 2 xpassed`. As far as I can tell, the unexpected pass from `test_one_hy` results in a fail (as if `strict=True`, [ref](https://docs.pytest.org/en/7.4.x/reference/reference.html#pytest.mark.xfail)), but I'm not sure why the expected fail, fails. Finally, it would be nice to support the `strict` argument for example.xfail for `test_one_hy`.

Note, for brevity I've not provided a full log, but I can if that would be useful.


Discussion:

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/core.py b/hypothesis-python/src/hypothesis/core.py
--- a/hypothesis-python/src/hypothesis/core.py
+++ b/hypothesis-python/src/hypothesis/core.py
@@ -173,7 +173,8 @@ def xfail(
             Type[BaseException], Tuple[Type[BaseException], ...]
         ] = BaseException,
     ) -> "example":
-        """Mark this example as an expected failure, like pytest.mark.xfail().
+        """Mark this example as an expected failure, similarly to
+        :obj:`pytest.mark.xfail(strict=True) <pytest.mark.xfail>`.
 
         Expected-failing examples allow you to check that your test does fail on
         some examples, and therefore build confidence that *passing* tests are
@@ -188,6 +189,37 @@ def xfail(
             @example(...).xfail(condition=sys.platform != "linux", raises=OSError)
             def test(x):
                 pass
+
+        .. note::
+
+            Expected-failing examples are handled separately from those generated
+            by strategies, so you should usually ensure that there is no overlap.
+
+            .. code-block:: python
+
+                @example(x=1, y=0).xfail(raises=ZeroDivisionError)
+                @given(x=st.just(1), y=st.integers())  # Missing `.filter(bool)`!
+                def test_fraction(x, y):
+                    # This test will try the explicit example and see it fail as
+                    # expected, then go on to generate more examples from the
+                    # strategy.  If we happen to generate y=0, the test will fail
+                    # because only the explicit example is treated as xfailing.
+                    x / y
+
+        Note that this "method chaining" syntax requires Python 3.9 or later, for
+        :pep:`614` relaxing grammar restrictions on decorators.  If you need to
+        support older versions of Python, you can use an identity function:
+
+        .. code-block:: python
+
+            def identity(x):
+                return x
+
+
+            @identity(example(...).xfail())
+            def test(x):
+                pass
+
         """
         check_type(bool, condition, "condition")
         check_type(str, reason, "reason")
@@ -469,11 +501,17 @@ def execute_explicit_examples(state, wrapped_test, arguments, kwargs, original_s
                         execute_example()
                     else:
                         # @example(...).xfail(...)
+
                         try:
                             execute_example()
                         except failure_exceptions_to_catch() as err:
                             if not isinstance(err, example.raises):
                                 raise
+                            # Save a string form of this example; we'll warn if it's
+                            # ever generated by the strategy (which can't be xfailed)
+                            state.xfail_example_reprs.add(
+                                repr_call(state.test, arguments, example_kwargs)
+                            )
                         except example.raises as err:
                             # We'd usually check this as early as possible, but it's
                             # possible for failure_exceptions_to_catch() to grow when
@@ -679,6 +717,7 @@ def __init__(self, test_runner, stuff, test, settings, random, wrapped_test):
 
         self.is_find = getattr(wrapped_test, "_hypothesis_internal_is_find", False)
         self.wrapped_test = wrapped_test
+        self.xfail_example_reprs = set()
 
         self.test = test
 
@@ -761,6 +800,15 @@ def run(data):
                         if expected_failure is not None:
                             nonlocal text_repr
                             text_repr = repr_call(test, args, kwargs)
+                            if text_repr in self.xfail_example_reprs:
+                                warnings.warn(
+                                    f"We generated {text_repr}, which seems identical "
+                                    "to one of your `@example(...).xfail()` cases.  "
+                                    "Revise the strategy to avoid this overlap?",
+                                    HypothesisWarning,
+                                    # Checked in test_generating_xfailed_examples_warns!
+                                    stacklevel=6,
+                                )
 
                         if print_example or current_verbosity() >= Verbosity.verbose:
                             printer = RepresentationPrinter(context=context)


Test patch:
diff --git a/hypothesis-python/tests/cover/test_example.py b/hypothesis-python/tests/cover/test_example.py
--- a/hypothesis-python/tests/cover/test_example.py
+++ b/hypothesis-python/tests/cover/test_example.py
@@ -18,6 +18,7 @@
 from hypothesis import example, find, given, strategies as st
 from hypothesis.errors import (
     HypothesisException,
+    HypothesisWarning,
     InvalidArgument,
     NonInteractiveExampleWarning,
     Unsatisfiable,
@@ -194,3 +195,22 @@ def test_error_on_unexpected_pass_single_elem_tuple(x):
 @given(st.none())
 def test_error_on_unexpected_pass_multi(x):
     pass
+
+
+def test_generating_xfailed_examples_warns():
+    @given(st.integers())
+    @example(1)
+    @identity(example(0).xfail(raises=ZeroDivisionError))
+    def foo(x):
+        assert 1 / x
+
+    with pytest.warns(
+        HypothesisWarning,
+        match=r"Revise the strategy to avoid this overlap",
+    ) as wrec:
+        with pytest.raises(ZeroDivisionError):
+            foo()
+
+        warning_locations = sorted(w.filename for w in wrec.list)
+        # See the reference in core.py to this test
+        assert __file__ in warning_locations, "probable stacklevel bug"