Repository: HypothesisWorks/hypothesis (Python)

Issue:
example.xfail() defaults don't match pytest's defaults, no option to set strict parameter
Caveat: I'm new to both `pytest` and `hypothesis`. As such, it's possible that this is the expected behaviour, but it is simply unintuitive to me. So far my searches in issues and documentation have not yielded any meaningful results, so I'm hoping to be illuminated if this is indeed the expected behaviour. Any pointers to the relevant sections in the documentation are much appreciated.

Consider the following test suite, `test_xfail.py`:

```python
import pytest
from hypothesis import given, strategies as st, example

@given(x=st.integers())
@example(x=0).xfail()
def test_zero_hy(x):
    assert x != 0

@given(x=st.integers())
@example(x=0).xfail()
def test_one_hy(x):
    assert True

@pytest.mark.xfail
@given(x=st.integers())
@example(x=0)
def test_zero_pt(x):
    assert x != 0

@pytest.mark.xfail
@given(x=st.integers())
@example(x=0)
def test_one_pt(x):
    assert True
```

with the following versions installed:
```
platform linux -- Python 3.9.18, pytest-7.4.2, pluggy-1.3.0
hypothesis profile 'default-with-verbose-verbosity' -> verbosity=Verbosity.verbose, database=DirectoryBasedExampleDatabase('/home/nfraser/workspace/hyp_xfail_test/.hypothesis/examples')
rootdir: /home/nfraser/workspace/hyp_xfail_test
plugins: hypothesis-6.84.3
```

Note, I also tried python versions `3.10.12` & `3.11.5` with the same results.

Running the following (with nothing else in the directory except the above file):

```bash
rm -rf .hypothesis/ .pytest_cache/ __pycache__/ && pytest | tail -n 4
```

Note, I removed all other folders to prevent what appears to be another issue from occurring (I'll save that for a separate issue).

```bash
=========================== short test summary info ============================
FAILED test_xfail.py::test_zero_hy - assert 0 != 0
FAILED test_xfail.py::test_one_hy - AssertionError: Expected an exception fro...
=================== 2 failed, 1 xfailed, 1 xpassed in 0.19s ====================
```

I would've expected the results to be `2 xfailed, 2 xpassed`. As far as I can tell, the unexpected pass from `test_one_hy` results in a fail (as if `strict=True`, [ref](https://docs.pytest.org/en/7.4.x/reference/reference.html#pytest.mark.xfail)), but I'm not sure why the expected fail, fails. Finally, it would be nice to support the `strict` argument for example.xfail for `test_one_hy`.

Note, for brevity I've not provided a full log, but I can if that would be useful.


Discussion: Some extra information:

If I add the ` --hypothesis-verbosity=verbose` to my command above, explicitly:

```bash
rm -rf .hypothesis/ .pytest_cache/ __pycache__/ && pytest --hypothesis-verbosity=verbose
```

I see the following output related to `test_zero_hy`:

```
-----------------------Hypothesis--------------------------------
Trying explicit example: test_zero_hy(
    x=0,
)
Trying example: test_zero_hy(
    x=0,
)
x = 0

    @given(x=st.integers())
    @example(x=0).xfail()
    def test_zero_hy(x):
>       assert x != 0
E       assert 0 != 0

test_xfail.py:8: AssertionError

Trying example: test_zero_hy(
    x=55,
)
Trying example: test_zero_hy(
    x=5548580897629562706,
)
Trying example: test_zero_hy(
    x=16532,
)
Trying example: test_zero_hy(
    x=20680,
)
Trying example: test_zero_hy(
    x=30,
)
Trying example: test_zero_hy(
    x=-31766,
)
Trying example: test_zero_hy(
    x=0,
)
x = 0

    @given(x=st.integers())
    @example(x=0).xfail()
    def test_zero_hy(x):
>       assert x != 0
E       assert 0 != 0

test_xfail.py:8: AssertionError

Trying example: test_zero_hy(
    x=70,
)
Trying example: test_zero_hy(
    x=786818658,
)
Trying example: test_zero_hy(
    x=0,
)
x = 0

    @given(x=st.integers())
    @example(x=0).xfail()
    def test_zero_hy(x):
>       assert x != 0
E       assert 0 != 0

test_xfail.py:8: AssertionError
```

Is it possible that the `x=0` example is being generated from my strategy for `x` _as well_ as being set explicitly by the example decorator? Or to put another way, in order to get the expected behaviour, do I need to ensure that my strategies within `given` *never* produce the expected failing example?
More follow-up, modifying `test_zero_hy` as follows:

```python
@given(x=st.integers(min_value=1))
@example(x=0).xfail()
def test_zero_hy(x):
    assert x != 0
```

Results in this test being reported as passed. Explicitly, making the modification above to the original test results in the following:

```bash
rm -rf .hypothesis/ .pytest_cache/ __pycache__/ && pytest | tail -n -3
```

produces:

```
=========================== short test summary info ============================
FAILED test_xfail.py::test_one_hy - AssertionError: Expected an exception fro...
============== 1 failed, 1 passed, 1 xfailed, 1 xpassed in 0.21s ===============
```

Which still differs from what I would expect (`2 xfailed, 2 xpassed`), so I'm interested to understand if this is the expected behaviour or not.

\* apologies, I accidentally clicked close on this issue - I'm still interested in understanding the expected behaviour here.
> Note, I removed all other folders to prevent what appears to be another issue from occurring (I'll save that for a separate issue).

BTW, I believe this was a artefact of the following behaviour as well.

> Is it possible that the `x=0` example is being generated from my strategy for `x` as well as being set explicitly by the example decorator?



Fix (patch):
diff --git a/hypothesis-python/RELEASE.rst b/hypothesis-python/RELEASE.rst
new file mode 100644
index 0000000000..e187f5954f
--- /dev/null
+++ b/hypothesis-python/RELEASE.rst
@@ -0,0 +1,6 @@
+RELEASE_TYPE: patch
+
+This patch improves the documentation of :obj:`@example(...).xfail() <hypothesis.example.xfail>`
+by adding a note about :pep:`614`, similar to :obj:`@example(...).via() <hypothesis.example.via>`,
+and adds a warning when a strategy generates a test case which seems identical to
+one provided by an xfailed example.
diff --git a/hypothesis-python/docs/changes.rst b/hypothesis-python/docs/changes.rst
index 2f28b625b4..08c237f8cd 100644
--- a/hypothesis-python/docs/changes.rst
+++ b/hypothesis-python/docs/changes.rst
@@ -25,7 +25,7 @@ Hypothesis 6.x
 -------------------
 
 This release enables the :obj:`~hypothesis.Phase.explain` :ref:`phase <phases>`
-by default.  We hope it helps you to understand _why_ your failing tests have
+by default.  We hope it helps you to understand *why* your failing tests have
 failed!
 
 .. _v6.85.1:
diff --git a/hypothesis-python/src/hypothesis/core.py b/hypothesis-python/src/hypothesis/core.py
index b31101fea1..0b106b4239 100644
--- a/hypothesis-python/src/hypothesis/core.py
+++ b/hypothesis-python/src/hypothesis/core.py
@@ -173,7 +173,8 @@ def xfail(
             Type[BaseException], Tuple[Type[BaseException], ...]
         ] = BaseException,
     ) -> "example":
-        """Mark this example as an expected failure, like pytest.mark.xfail().
+        """Mark this example as an expected failure, similarly to
+        :obj:`pytest.mark.xfail(strict=True) <pytest.mark.xfail>`.
 
         Expected-failing examples allow you to check that your test does fail on
         some examples, and therefore build confidence that *passing* tests are
@@ -188,6 +189,37 @@ def xfail(
             @example(...).xfail(condition=sys.platform != "linux", raises=OSError)
             def test(x):
                 pass
+
+        .. note::
+
+            Expected-failing examples are handled separately from those generated
+            by strategies, so you should usually ensure that there is no overlap.
+
+            .. code-block:: python
+
+                @example(x=1, y=0).xfail(raises=ZeroDivisionError)
+                @given(x=st.just(1), y=st.integers())  # Missing `.filter(bool)`!
+                def test_fraction(x, y):
+                    # This test will try the explicit example and see it fail as
+                    # expected, then go on to generate more examples from the
+                    # strategy.  If we happen to generate y=0, the test will fail
+                    # because only the explicit example is treated as xfailing.
+                    x / y
+
+        Note that this "method chaining" syntax requires Python 3.9 or later, for
+        :pep:`614` relaxing grammar restrictions on decorators.  If you need to
+        support older versions of Python, you can use an identity function:
+
+        .. code-block:: python
+
+            def identity(x):
+                return x
+
+
+            @identity(example(...).xfail())
+            def test(x):
+                pass
+
         """
         check_type(bool, condition, "condition")
         check_type(str, reason, "reason")
@@ -469,11 +501,17 @@ def execute_explicit_examples(state, wrapped_test, arguments, kwargs, original_s
                         execute_example()
                     else:
                         # @example(...).xfail(...)
+
                         try:
                             execute_example()
                         except failure_exceptions_to_catch() as err:
                             if not isinstance(err, example.raises):
                                 raise
+                            # Save a string form of this example; we'll warn if it's
+                            # ever generated by the strategy (which can't be xfailed)
+                            state.xfail_example_reprs.add(
+                                repr_call(state.test, arguments, example_kwargs)
+                            )
                         except example.raises as err:
                             # We'd usually check this as early as possible, but it's
                             # possible for failure_exceptions_to_catch() to grow when
@@ -679,6 +717,7 @@ def __init__(self, test_runner, stuff, test, settings, random, wrapped_test):
 
         self.is_find = getattr(wrapped_test, "_hypothesis_internal_is_find", False)
         self.wrapped_test = wrapped_test
+        self.xfail_example_reprs = set()
 
         self.test = test
 
@@ -761,6 +800,15 @@ def run(data):
                         if expected_failure is not None:
                             nonlocal text_repr
                             text_repr = repr_call(test, args, kwargs)
+                            if text_repr in self.xfail_example_reprs:
+                                warnings.warn(
+                                    f"We generated {text_repr}, which seems identical "
+                                    "to one of your `@example(...).xfail()` cases.  "
+                                    "Revise the strategy to avoid this overlap?",
+                                    HypothesisWarning,
+                                    # Checked in test_generating_xfailed_examples_warns!
+                                    stacklevel=6,
+                                )
 
                         if print_example or current_verbosity() >= Verbosity.verbose:
                             printer = RepresentationPrinter(context=context)