Repository: HypothesisWorks/hypothesis

Issue:
Stop silently dropping examples when the `@example()` decorator is applied to itself
Suppose you wanted to write a function which added explicit `@hypothesis.example()`s to a function for the cross-product of a few sequences.  A correct implementation would look something like:

```python
import itertools
from hypothesis import example

def cross_product_examples(**kwargs):
    def inner(f):
        params, values = zip(*kwargs.items())
        for value_set in itertools.product(*values):
            kw = dict(zip(params, value_set))
            f = example(**kw)(f)
    return inner
```

But if you forget the inner function, you'd end up with something like this:

```python
def buggy_cross_product_examples(**kwargs):
    # This function is broken in a nastily subtle way: everything will appear
    # to work, *except* that only the last-defined example will ever be run!
    params, values = zip(*kwargs.items())

    decorator = lambda f: f
    for value_set in itertools.product(*values):
        kw = dict(zip(params, value_set))
        # Here's the bug: we're applying the @example() decorator to *another decorator*,
        # not to the test function.  This silently discards everything from previous loop
        # iterations, and so we'll only ever run the last-defined example.
        decorator = example(**kw)(decorator)

    return decorator
```

To fix this, we should modify the definition of `@example()` (below), so that the `hypothesis_explicit_examples` list is defined outside of `accept`, and added as an attribute on *both* `test` and `accept`.

https://github.com/HypothesisWorks/hypothesis/blob/7f1fa0baf279e4db09c2a3d6c9e89867ab04ee85/hypothesis-python/src/hypothesis/core.py#L136-L151

For a complete pull request ("PR"), you'll also need to add a test which checks that examples are not lost when you chain together e.g. `example("outer")(example("inner"))(f)`, as well as writing a `RELEASE.rst` and adding your name to the `AUTHORS.rst` list.


Discussion:

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/core.py b/hypothesis-python/src/hypothesis/core.py
--- a/hypothesis-python/src/hypothesis/core.py
+++ b/hypothesis-python/src/hypothesis/core.py
@@ -142,12 +142,15 @@ def example(*args: Any, **kwargs: Any) -> Callable[[TestFunc], TestFunc]:
     if not (args or kwargs):
         raise InvalidArgument("An example must provide at least one argument")
 
+    hypothesis_explicit_examples: List[Example] = []
+
     def accept(test):
         if not hasattr(test, "hypothesis_explicit_examples"):
-            test.hypothesis_explicit_examples = []
+            test.hypothesis_explicit_examples = hypothesis_explicit_examples
         test.hypothesis_explicit_examples.append(Example(tuple(args), kwargs))
         return test
 
+    accept.hypothesis_explicit_examples = hypothesis_explicit_examples  # type: ignore
     return accept
 
 


Test patch:
diff --git a/hypothesis-python/tests/cover/test_explicit_examples.py b/hypothesis-python/tests/cover/test_explicit_examples.py
--- a/hypothesis-python/tests/cover/test_explicit_examples.py
+++ b/hypothesis-python/tests/cover/test_explicit_examples.py
@@ -282,3 +282,11 @@ def t(s):
         assert isinstance(err.__cause__, AssertionError)
     else:
         raise NotImplementedError("should be unreachable")
+
+
+def test_stop_silently_dropping_examples_when_decorator_is_applied_to_itself():
+    def f():
+        pass
+
+    test = example("outer")(example("inner"))(f)
+    assert len(test.hypothesis_explicit_examples) == 2