Repository: HypothesisWorks/hypothesis (2021-03-04 ~ 2021-04-06)

problem_statement: Improve handling of self-referential strategies
As mentioned in #2783, the following strategy is not explicitly forbidden but fails with an `AssertionError`:

```python
from hypothesis import strategies as st

SELF_REF = st.recursive(
    st.deferred(lambda: SELF_REF | st.booleans()),
    lambda s: st.lists(s, min_size=1)
)
```

There is an alternative strategy that produces the same data, but doesn't fail the same way:

```python
SELF_REF = st.recursive(
    st.booleans(),
    lambda s: st.lists(s, min_size=1)
)
```

I am not sure if all self-referential strategies can be rewritten like this, but probably we can either explicitly forbid such strategies or revisit our drawing approach for them.

Here is my reasoning from #2783 

> As far as I see, the cap is needed to prevent the drawing from this strategy & generating a certain maximum amount of leaves. However, assuming a single thread (more on the multi-threaded behavior in the next section) and such a self-referential strategy, I am not sure if capping is needed as it is - we can just apply it once on the first capped usage and make all subsequent calls no-op (e.g., just yield without modifying marked). Then we still have the marker set only once on the very first RecursiveStrategy.do_draw call, and it will be monotonically decreasing. Therefore, we'll have the max size properly maintained, and there will be no oversized subtrees because, at some point, LimitReached will occur.


date: 2021-03-04
patch: diff --git a/hypothesis-python/src/hypothesis/strategies/_internal/recursive.py b/hypothesis-python/src/hypothesis/strategies/_internal/recursive.py
index 6cbb71175..093f8a441 100644
--- a/hypothesis-python/src/hypothesis/strategies/_internal/recursive.py
+++ b/hypothesis-python/src/hypothesis/strategies/_internal/recursive.py
@@ -66,13 +66,13 @@ class LimitedStrategy(SearchStrategy):
 
     @contextmanager
     def capped(self, max_templates):
-        assert not self.currently_capped
         try:
+            was_capped = self.currently_capped
             self.currently_capped = True
             self.marker = max_templates
             yield
         finally:
-            self.currently_capped = False
+            self.currently_capped = was_capped
 
 
 class RecursiveStrategy(SearchStrategy):

test_patch: diff --git a/hypothesis-python/tests/nocover/test_recursive.py b/hypothesis-python/tests/nocover/test_recursive.py
index c4fce937b..67b07983c 100644
--- a/hypothesis-python/tests/nocover/test_recursive.py
+++ b/hypothesis-python/tests/nocover/test_recursive.py
@@ -156,3 +156,15 @@ def test_drawing_from_recursive_strategy_is_thread_safe():
         thread.join()
 
     assert not errors
+
+
+SELF_REF = st.recursive(
+    st.deferred(lambda: st.booleans() | SELF_REF),
+    lambda s: st.lists(s, min_size=1),
+)
+
+
+@given(SELF_REF)
+def test_self_ref_regression(_):
+    # See https://github.com/HypothesisWorks/hypothesis/issues/2794
+    pass


----

problem_statement: Show falsifying keyword arguments in the same order as the arguments
Falsifying examples are shown with positional arguments converted to keyword arguments. However, these are sorted in alphabetical order. This can be confusing. It would be better if the arguments were shown in the original order.

For example:

```py
from hypothesis import given
from hypothesis.strategies import integers

@given(integers(), integers(), integers())
def test(b, a, c):
    assert a == b == c
```

gives

```
Falsifying example: test(
    a=0, b=0, c=1,
)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 5, in test
  File "/Users/aaronmeurer/anaconda3/envs/ndindex/lib/python3.8/site-packages/hypothesis/core.py", line 1171, in wrapped_test
    raise the_error_hypothesis_found
  File "<stdin>", line 6, in test
AssertionError
```

I dug into it. Hypothesis internally converts the entire argument spec into a single strategy, and converts positional arguments into keyword arguments where possible.  

The issue seems to be that fixed_dictionaries does not maintain key order for normal dictionaries. I'm not sure why this is the case, since dictionaries are supposed to be ordered since Python 3.6. Supposing that isn't going to change, however, I think this can be fixed by explicitly passing an OrderedDict 

If this weren't the case, I believe this would just work as expected, except there is also an oddity where the keyword arguments are actually reversed for some reason [here](https://github.com/HypothesisWorks/hypothesis/blob/3d5761ed9254b39260f39cf52bad1293ccf733ab/hypothesis-python/src/hypothesis/core.py#L975). I'm not entirely clear about that logic, but I guess it is supposed to handle the case where one argument to the `zip` is longer than the other (if so, it can easily be fixed by using an additional reversed on the zip). 

In short, this patch fixes the issue

```patch
diff --git a/hypothesis-python/src/hypothesis/core.py b/hypothesis-python/src/hypothesis/core.py
index 6089b3efb..606c5d1e2 100644
--- a/hypothesis-python/src/hypothesis/core.py
+++ b/hypothesis-python/src/hypothesis/core.py
@@ -27,7 +27,7 @@ import traceback
 import types
 import warnings
 import zlib
-from collections import defaultdict
+from collections import defaultdict, OrderedDict
 from inspect import getfullargspec
 from io import StringIO
 from random import Random
@@ -436,7 +436,7 @@ def process_arguments_to_given(wrapped_test, arguments, kwargs, given_kwargs, ar
     search_strategy = TupleStrategy(
         (
             st.just(arguments),
-            st.fixed_dictionaries(given_kwargs).map(lambda args: dict(args, **kwargs)),
+            st.fixed_dictionaries(OrderedDict(given_kwargs)).map(lambda args: dict(args, **kwargs)),
         )
     )
 
@@ -971,9 +971,9 @@ def given(
         # positional arguments into keyword arguments for simplicity.
         if given_arguments:
             assert not given_kwargs
-            for name, strategy in zip(
+            for name, strategy in reversed(list(zip(
                 reversed(original_argspec.args), reversed(given_arguments)
-            ):
+            ))):
                 given_kwargs[name] = strategy
         # These have been converted, so delete them to prevent accidental use.
         del given_arguments
```

Although (to me at least), making fixed_dictionaries not sort normal dicts would be preferable. 
date: 2021-03-28
patch: diff --git a/hypothesis-python/src/hypothesis/core.py b/hypothesis-python/src/hypothesis/core.py
index 6089b3efb..a9af27aa8 100644
--- a/hypothesis-python/src/hypothesis/core.py
+++ b/hypothesis-python/src/hypothesis/core.py
@@ -971,10 +971,8 @@ def given(
         # positional arguments into keyword arguments for simplicity.
         if given_arguments:
             assert not given_kwargs
-            for name, strategy in zip(
-                reversed(original_argspec.args), reversed(given_arguments)
-            ):
-                given_kwargs[name] = strategy
+            nargs = len(given_arguments)
+            given_kwargs.update(zip(original_argspec.args[-nargs:], given_arguments))
         # These have been converted, so delete them to prevent accidental use.
         del given_arguments
 
diff --git a/hypothesis-python/src/hypothesis/strategies/_internal/collections.py b/hypothesis-python/src/hypothesis/strategies/_internal/collections.py
index fc2f5f6bf..b72c4cd05 100644
--- a/hypothesis-python/src/hypothesis/strategies/_internal/collections.py
+++ b/hypothesis-python/src/hypothesis/strategies/_internal/collections.py
@@ -13,8 +13,6 @@
 #
 # END HEADER
 
-from collections import OrderedDict
-
 from hypothesis.errors import InvalidArgument
 from hypothesis.internal.conjecture import utils as cu
 from hypothesis.internal.conjecture.junkdrawer import LazySequenceCopy
@@ -229,14 +227,7 @@ class FixedKeysDictStrategy(MappedSearchStrategy):
 
     def __init__(self, strategy_dict):
         self.dict_type = type(strategy_dict)
-
-        if isinstance(strategy_dict, OrderedDict):
-            self.keys = tuple(strategy_dict.keys())
-        else:
-            try:
-                self.keys = tuple(sorted(strategy_dict.keys()))
-            except TypeError:
-                self.keys = tuple(sorted(strategy_dict.keys(), key=repr))
+        self.keys = tuple(strategy_dict.keys())
         super().__init__(strategy=TupleStrategy(strategy_dict[k] for k in self.keys))
 
     def calc_is_empty(self, recur):
@@ -262,14 +253,6 @@ class FixedAndOptionalKeysDictStrategy(SearchStrategy):
         self.fixed = FixedKeysDictStrategy(strategy_dict)
         self.optional = optional
 
-        if isinstance(self.optional, OrderedDict):
-            self.optional_keys = tuple(self.optional.keys())
-        else:
-            try:
-                self.optional_keys = tuple(sorted(self.optional.keys()))
-            except TypeError:
-                self.optional_keys = tuple(sorted(self.optional.keys(), key=repr))
-
     def calc_is_empty(self, recur):
         return recur(self.fixed)
 
@@ -278,7 +261,7 @@ class FixedAndOptionalKeysDictStrategy(SearchStrategy):
 
     def do_draw(self, data):
         result = data.draw(self.fixed)
-        remaining = [k for k in self.optional_keys if not self.optional[k].is_empty]
+        remaining = [k for k, v in self.optional.items() if not v.is_empty]
         should_draw = cu.many(
             data, min_size=0, max_size=len(remaining), average_size=len(remaining) / 2
         )

test_patch: diff --git a/hypothesis-python/tests/nocover/test_testdecorators.py b/hypothesis-python/tests/nocover/test_testdecorators.py
index 3c3c83908..896c6536d 100644
--- a/hypothesis-python/tests/nocover/test_testdecorators.py
+++ b/hypothesis-python/tests/nocover/test_testdecorators.py
@@ -13,9 +13,8 @@
 #
 # END HEADER
 
-from hypothesis import HealthCheck, given, reject, settings
+from hypothesis import HealthCheck, given, reject, settings, strategies as st
 from hypothesis.errors import InvalidArgument, Unsatisfiable
-from hypothesis.strategies import integers
 
 from tests.common.utils import raises
 
@@ -23,7 +22,7 @@ from tests.common.utils import raises
 def test_contains_the_test_function_name_in_the_exception_string():
     look_for_one = settings(max_examples=1, suppress_health_check=HealthCheck.all())
 
-    @given(integers())
+    @given(st.integers())
     @look_for_one
     def this_has_a_totally_unique_name(x):
         reject()
@@ -33,7 +32,7 @@ def test_contains_the_test_function_name_in_the_exception_string():
     assert this_has_a_totally_unique_name.__name__ in e.value.args[0]
 
     class Foo:
-        @given(integers())
+        @given(st.integers())
         @look_for_one
         def this_has_a_unique_name_and_lives_on_a_class(self, x):
             reject()
@@ -47,7 +46,7 @@ def test_signature_mismatch_error_message():
     # Regression test for issue #1978
 
     @settings(max_examples=2)
-    @given(x=integers())
+    @given(x=st.integers())
     def bad_test():
         pass
 
@@ -58,3 +57,9 @@ def test_signature_mismatch_error_message():
             str(e) == "bad_test() got an unexpected keyword argument 'x', "
             "from `x=integers()` in @given"
         )
+
+
+@given(data=st.data(), keys=st.lists(st.integers(), unique=True))
+def test_fixed_dict_preserves_iteration_order(data, keys):
+    d = data.draw(st.fixed_dictionaries({k: st.none() for k in keys}))
+    assert all(a == b for a, b in zip(keys, d)), f"keys={keys}, d.keys()={d.keys()}"


----

problem_statement: Add helpful error message for `st.sampled_from(EmptyEnumWithAnnotations)`
Following the discussion at #2693, the documented method of sampling from an Enum does not appear to work (or the documentation could use clarification and/or an example). The approach I tried is:

```python
from enum import Enum
from hypothesis import given, strategies as st

class MyEnum(Enum):
    a: 'a'
    b: 'b'

@given(st.sampled_from(MyEnum))
def test_example(instance):
    assert True
```

This raises `hypothesis.errors.InvalidArgument: Cannot sample from a length-zero sequence.` Additionally, `sampled_from(list(MyEnum))` raises the same issue. Hypothesis version is 6.8.5, python is 3.9.2 


date: 2021-04-06
patch: diff --git a/hypothesis-python/src/hypothesis/strategies/_internal/core.py b/hypothesis-python/src/hypothesis/strategies/_internal/core.py
index 3001d27b9..186be894b 100644
--- a/hypothesis-python/src/hypothesis/strategies/_internal/core.py
+++ b/hypothesis-python/src/hypothesis/strategies/_internal/core.py
@@ -158,6 +158,17 @@ def sampled_from(elements):
     """
     values = check_sample(elements, "sampled_from")
     if not values:
+        if (
+            isinstance(elements, type)
+            and issubclass(elements, enum.Enum)
+            and vars(elements).get("__annotations__")
+        ):
+            # See https://github.com/HypothesisWorks/hypothesis/issues/2923
+            raise InvalidArgument(
+                f"Cannot sample from {elements.__module__}.{elements.__name__} "
+                "because it contains no elements.  It does however have annotations, "
+                "so maybe you tried to write an enum as if it was a dataclass?"
+            )
         raise InvalidArgument("Cannot sample from a length-zero sequence.")
     if len(values) == 1:
         return just(values[0])
@@ -385,13 +396,12 @@ def fixed_dictionaries(
     """Generates a dictionary of the same type as mapping with a fixed set of
     keys mapping to strategies. ``mapping`` must be a dict subclass.
 
-    Generated values have all keys present in mapping, with the
-    corresponding values drawn from mapping[key]. If mapping is an
-    instance of OrderedDict the keys will also be in the same order,
-    otherwise the order is arbitrary.
+    Generated values have all keys present in mapping, in iteration order,
+    with the corresponding values drawn from mapping[key].
 
     If ``optional`` is passed, the generated value *may or may not* contain each
     key from ``optional`` and a value drawn from the corresponding strategy.
+    Generated values may contain optional keys in an arbitrary order.
 
     Examples from this strategy shrink by shrinking each individual value in
     the generated dictionary, and omitting optional key-value pairs.

test_patch: diff --git a/hypothesis-python/tests/cover/test_sampled_from.py b/hypothesis-python/tests/cover/test_sampled_from.py
index babcf8de7..c8ef6d5df 100644
--- a/hypothesis-python/tests/cover/test_sampled_from.py
+++ b/hypothesis-python/tests/cover/test_sampled_from.py
@@ -186,3 +186,12 @@ def test_mutability_2(data):
     assert data.draw(s) != 2
     x.append(2)
     assert data.draw(s) != 2
+
+
+class AnnotationsInsteadOfElements(enum.Enum):
+    a: "a"
+
+
+def test_suggests_elements_instead_of_annotations():
+    with pytest.raises(InvalidArgument, match="Cannot sample.*annotations.*dataclass"):
+        st.sampled_from(AnnotationsInsteadOfElements).example()