Repository: HypothesisWorks/hypothesis

Issue:
from_type() does not resolve forward references outside of function annotations
The sample code:
```python
import typing as T

import hypothesis as H
import hypothesis.strategies as S

class Tree(T.NamedTuple):
    val: int
    l: T.Optional["Tree"]
    r: T.Optional["Tree"]
    def size(self):
        return 1 + (self.l.size() if self.l else 0) + (self.r.size() if self.r else 0)

@H.given(t = S.infer)
def test_tree(t: Tree):
    if t.size() > 3:
        assert False
        H.note(t)
    else:
        assert True
```

(Part of) the stack trace:
```
platform darwin -- Python 3.6.1, pytest-3.2.1, py-1.4.34, pluggy-0.4.0
rootdir: /Users/desmond/Documents/code/code-python/Try1/tests, inifile:
plugins: xonsh-0.5.12, hypothesis-3.38.9
collected 4 items
foo_test.py ...F
foo_test.py:36 (test_tree)
args = (<class 'foo_test.Tree'>,), kwargs = {}, kwargs_cache_key = set()
cache_key = (<function from_type at 0x112216a60>, ((<class 'type'>, <class 'foo_test.Tree'>),), frozenset())
result = builds(Tree, l=builds(lazy_error), r=builds(lazy_error), val=integers())

    @proxies(fn)
    def cached_strategy(*args, **kwargs):
        kwargs_cache_key = set()
        try:
            for k, v in kwargs.items():
                kwargs_cache_key.add((k, convert_value(v)))
        except TypeError:
            return fn(*args, **kwargs)
        cache_key = (
            fn,
            tuple(map(convert_value, args)), frozenset(kwargs_cache_key))
        try:
>           return STRATEGY_CACHE[cache_key]

/Users/desmond/anaconda3/lib/python3.6/site-packages/hypothesis/strategies.py:107: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = {(<function booleans at 0x112763e18>, (), frozenset()): booleans(), (<function fractions at 0x112216d90>, ((<class 'No...(tuples(), fixed_dictionaries({})).map(lambda value: target(*value[0], **value[1])),
 'val': WideRangeIntStrategy()}))}
key = (<function from_type at 0x112216a60>, ((<class 'type'>, <class 'foo_test.Tree'>),), frozenset())

    def __getitem__(self, key):
>       i = self.keys_to_indices[key]
E       KeyError: (<function from_type at 0x112216a60>, ((<class 'type'>, <class 'foo_test.Tree'>),), frozenset())

```


Discussion:
Thanks for the report - I'll need to poke around a bit to decide on the best fix, but this is a great bug report
 and will make that much easier 😄 
> I'll need to poke around a bit to decide on the best fix

FWIW, I'd guess that wrapping `from_type` output in a `deferred` strategy is probably the way to go. This sort of free recursion is kinda what they're for.
Yes, probably a couple of `deferred` needed. I might do it by hand like this (though I think they came out lopsided when I tried this):
```
GenTree = S.builds(Tree,
                   S.integers(),
                   S.one_of(S.none(), S.deferred(lambda: GenTree)),
                   S.one_of(S.none(), S.deferred(lambda: GenTree)))
```

(Side comment: I wonder why type annotations use strings for recursive types or forward references, instead of thunks / lambdas).

Related question: I was originally going to jump into stateful testing to try out trees, but got confused by some things in the examples described, so I'll ask here. Feel free to remove / move if appropriate:

- (Caveat: my understanding only) The example of *BalancedTrees* has no mutable state or side effects other than the Bundles, and those exist to help create a reasonable set of immutable trees using pure functions, to then feed to the `balance_tree` pure function and then to the `check_balanced` invariant. Is that the intent of the example? Why not just assert a boolean version of `check_balanced` at the end of `balance_tree`? Does the instantiation of these Bundles need to be user-visible? This may be partly a question about example selection.

- Why does *BalancedTrees* need to be a class? Would a module provide a lighter touch, much like pyunit / unittest vs. pytest?

Thanks for sharing a great library.

> Side comment: I wonder why type annotations use strings for recursive types or forward references, instead of thunks / lambdas.

I'll leave the others, but this is because type annotations are meant to work for static analysis and thus can't require evaluating code.
We actually have two problems here: 

1. Forward references are not currently handled, and are resolved for function annotations but not always for types (as in this case)
2. I'm pretty sure recursion in the lookup is also broken, but I'll have to fix (1) before diagnosing and handling recursion

Resolving patch:
diff --git a/src/hypothesis/searchstrategy/types.py b/src/hypothesis/searchstrategy/types.py
--- a/src/hypothesis/searchstrategy/types.py
+++ b/src/hypothesis/searchstrategy/types.py
@@ -26,12 +26,14 @@
 import collections
 
 import hypothesis.strategies as st
-from hypothesis.errors import ResolutionFailed
+from hypothesis.errors import InvalidArgument, ResolutionFailed
 from hypothesis.internal.compat import text_type, integer_types
 
 
 def type_sorting_key(t):
     """Minimise to None, then non-container types, then container types."""
+    if not isinstance(t, type):
+        raise InvalidArgument('thing=%s must be a type' % (t,))
     if t is None or t is type(None):  # noqa: E721
         return -1
     return issubclass(t, collections.abc.Container)
@@ -194,12 +196,20 @@ def really_inner(thing):
     def resolve_Type(thing):
         if thing.__args__ is None:
             return st.just(type)
-        inner = thing.__args__[0]
-        if getattr(inner, '__origin__', None) is typing.Union:
-            return st.sampled_from(inner.__args__)
-        elif hasattr(inner, '__union_params__'):  # pragma: no cover
-            return st.sampled_from(inner.__union_params__)
-        return st.just(inner)
+        args = (thing.__args__[0],)
+        if getattr(args[0], '__origin__', None) is typing.Union:
+            args = args[0].__args__
+        elif hasattr(args[0], '__union_params__'):  # pragma: no cover
+            args = args[0].__union_params__
+        if isinstance(typing._ForwardRef, type):  # pragma: no cover
+            # Duplicate check from from_type here - only paying when needed.
+            for a in args:
+                if type(a) == typing._ForwardRef:
+                    raise ResolutionFailed(
+                        'thing=%s cannot be resolved.  Upgrading to '
+                        'python>=3.6 may fix this problem via improvements '
+                        'to the typing module.' % (thing,))
+        return st.sampled_from(sorted(args, key=type_sorting_key))
 
     @register(typing.List, st.builds(list))
     def resolve_List(thing):
diff --git a/src/hypothesis/strategies.py b/src/hypothesis/strategies.py
--- a/src/hypothesis/strategies.py
+++ b/src/hypothesis/strategies.py
@@ -972,30 +972,16 @@ def builds(target, *args, **kwargs):
     )
 
 
-def delay_error(func):
-    """A decorator to make exceptions lazy but success immediate.
-
-    We want from_type to resolve to a strategy immediately if possible,
-    for a useful repr and interactive use, but delay errors until a
-    value would be drawn to localise them to a particular test.
-
-    """
+def _defer_from_type(func):
+    """Decorator to make from_type lazy to support recursive definitions."""
     @proxies(func)
     def inner(*args, **kwargs):
-        try:
-            return func(*args, **kwargs)
-        except Exception as e:
-            error = e
-
-            def lazy_error():
-                raise error
-
-            return builds(lazy_error)
+        return deferred(lambda: func(*args, **kwargs))
     return inner
 
 
 @cacheable
-@delay_error
+@_defer_from_type
 def from_type(thing):
     """Looks up the appropriate search strategy for the given type.
 
@@ -1024,12 +1010,12 @@ def from_type(thing):
 
     """
     from hypothesis.searchstrategy import types
-    if not isinstance(thing, type):
-        try:
+    try:
+        import typing
+        if not isinstance(thing, type):
             # At runtime, `typing.NewType` returns an identity function rather
             # than an actual type, but we can check that for a possible match
             # and then read the magic attribute to unwrap it.
-            import typing
             if all([
                 hasattr(thing, '__supertype__'), hasattr(typing, 'NewType'),
                 isfunction(thing), getattr(thing, '__module__', 0) == 'typing'
@@ -1040,8 +1026,16 @@ def from_type(thing):
             if getattr(thing, '__origin__', None) is typing.Union:
                 args = sorted(thing.__args__, key=types.type_sorting_key)
                 return one_of([from_type(t) for t in args])
-        except ImportError:  # pragma: no cover
-            pass
+        # We can't resolve forward references, and under Python 3.5 (only)
+        # a forward reference is an instance of type.  Hence, explicit check:
+        elif type(thing) == typing._ForwardRef:  # pragma: no cover
+            raise ResolutionFailed(
+                'thing=%s cannot be resolved.  Upgrading to python>=3.6 may '
+                'fix this problem via improvements to the typing module.'
+                % (thing,))
+    except ImportError:  # pragma: no cover
+        pass
+    if not isinstance(thing, type):
         raise InvalidArgument('thing=%s must be a type' % (thing,))
     # Now that we know `thing` is a type, the first step is to check for an
     # explicitly registered strategy.  This is the best (and hopefully most


Test patch:
diff --git a/tests/py3/test_lookup.py b/tests/py3/test_lookup.py
--- a/tests/py3/test_lookup.py
+++ b/tests/py3/test_lookup.py
@@ -365,3 +365,34 @@ def test_required_args(target, args, kwargs):
     # Mostly checking that `self` (and only self) is correctly excluded
     st.builds(target, *map(st.just, args),
               **{k: st.just(v) for k, v in kwargs.items()}).example()
+
+
+@pytest.mark.parametrize('thing', [
+    typing.Optional, typing.List, getattr(typing, 'Type', typing.Set)
+])  # check Type if it's available, otherwise Set is redundant but harmless
+def test_cannot_resolve_bare_forward_reference(thing):
+    with pytest.raises(InvalidArgument):
+        t = thing['int']
+        if type(getattr(t, '__args__', [None])[0]) != typing._ForwardRef:
+            assert sys.version_info[:2] == (3, 5)
+            pytest.xfail('python 3.5 typing module is really weird')
+        st.from_type(t).example()
+
+
+class Tree:
+    def __init__(self,
+                 left: typing.Optional['Tree'],
+                 right: typing.Optional['Tree']):
+        self.left = left
+        self.right = right
+
+    def __repr__(self):
+        return 'Tree({}, {})'.format(self.left, self.right)
+
+
+def test_resolving_recursive_type():
+    try:
+        assert isinstance(st.builds(Tree).example(), Tree)
+    except ResolutionFailed:
+        assert sys.version_info[:2] == (3, 5)
+        pytest.xfail('python 3.5 typing module may not resolve annotations')