Repository: HypothesisWorks/hypothesis

Issue:
from_type() does not resolve forward references outside of function annotations
The sample code:
```python
import typing as T

import hypothesis as H
import hypothesis.strategies as S

class Tree(T.NamedTuple):
    val: int
    l: T.Optional["Tree"]
    r: T.Optional["Tree"]
    def size(self):
        return 1 + (self.l.size() if self.l else 0) + (self.r.size() if self.r else 0)

@H.given(t = S.infer)
def test_tree(t: Tree):
    if t.size() > 3:
        assert False
        H.note(t)
    else:
        assert True
```

(Part of) the stack trace:
```
platform darwin -- Python 3.6.1, pytest-3.2.1, py-1.4.34, pluggy-0.4.0
rootdir: /Users/desmond/Documents/code/code-python/Try1/tests, inifile:
plugins: xonsh-0.5.12, hypothesis-3.38.9
collected 4 items
foo_test.py ...F
foo_test.py:36 (test_tree)
args = (<class 'foo_test.Tree'>,), kwargs = {}, kwargs_cache_key = set()
cache_key = (<function from_type at 0x112216a60>, ((<class 'type'>, <class 'foo_test.Tree'>),), frozenset())
result = builds(Tree, l=builds(lazy_error), r=builds(lazy_error), val=integers())

    @proxies(fn)
    def cached_strategy(*args, **kwargs):
        kwargs_cache_key = set()
        try:
            for k, v in kwargs.items():
                kwargs_cache_key.add((k, convert_value(v)))
        except TypeError:
            return fn(*args, **kwargs)
        cache_key = (
            fn,
            tuple(map(convert_value, args)), frozenset(kwargs_cache_key))
        try:
>           return STRATEGY_CACHE[cache_key]

/Users/desmond/anaconda3/lib/python3.6/site-packages/hypothesis/strategies.py:107: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = {(<function booleans at 0x112763e18>, (), frozenset()): booleans(), (<function fractions at 0x112216d90>, ((<class 'No...(tuples(), fixed_dictionaries({})).map(lambda value: target(*value[0], **value[1])),
 'val': WideRangeIntStrategy()}))}
key = (<function from_type at 0x112216a60>, ((<class 'type'>, <class 'foo_test.Tree'>),), frozenset())

    def __getitem__(self, key):
>       i = self.keys_to_indices[key]
E       KeyError: (<function from_type at 0x112216a60>, ((<class 'type'>, <class 'foo_test.Tree'>),), frozenset())

```


Discussion:
Thanks for the report - I'll need to poke around a bit to decide on the best fix, but this is a great bug report
 and will make that much easier 😄 
> I'll need to poke around a bit to decide on the best fix

FWIW, I'd guess that wrapping `from_type` output in a `deferred` strategy is probably the way to go. This sort of free recursion is kinda what they're for.
Yes, probably a couple of `deferred` needed. I might do it by hand like this (though I think they came out lopsided when I tried this):
```
GenTree = S.builds(Tree,
                   S.integers(),
                   S.one_of(S.none(), S.deferred(lambda: GenTree)),
                   S.one_of(S.none(), S.deferred(lambda: GenTree)))
```

(Side comment: I wonder why type annotations use strings for recursive types or forward references, instead of thunks / lambdas).

Related question: I was originally going to jump into stateful testing to try out trees, but got confused by some things in the examples described, so I'll ask here. Feel free to remove / move if appropriate:

- (Caveat: my understanding only) The example of *BalancedTrees* has no mutable state or side effects other than the Bundles, and those exist to help create a reasonable set of immutable trees using pure functions, to then feed to the `balance_tree` pure function and then to the `check_balanced` invariant. Is that the intent of the example? Why not just assert a boolean version of `check_balanced` at the end of `balance_tree`? Does the instantiation of these Bundles need to be user-visible? This may be partly a question about example selection.

- Why does *BalancedTrees* need to be a class? Would a module provide a lighter touch, much like pyunit / unittest vs. pytest?

Thanks for sharing a great library.

> Side comment: I wonder why type annotations use strings for recursive types or forward references, instead of thunks / lambdas.

I'll leave the others, but this is because type annotations are meant to work for static analysis and thus can't require evaluating code.
We actually have two problems here: 

1. Forward references are not currently handled, and are resolved for function annotations but not always for types (as in this case)
2. I'm pretty sure recursion in the lookup is also broken, but I'll have to fix (1) before diagnosing and handling recursion
@Zac-HD based on https://github.com/HypothesisWorks/hypothesis-python/pull/1026#discussion_r161369257, is this actually closed? It seems like the fix doesn't support NamedTuple (though based on discussion it seems like we might not be able to)
Unfortunately there is no way to resolve forward references in Python 3.6, if they are not function annotations.  The example above should now fail with an explicit `ResolutionFailed` though, and explain that forward references are not supported - that's the best I can do 😢
@Zac-HD In which case I think I'll reopen the issue. Even if we can't/won't fix this any time soon I think it's a legitimate thing to want supported. Hopefully in future it will become easier to do so (or maybe we'll figure out some clever trick).
Fair enough!  I consider that a feature request though, which might be possible on Python 3.7 😄

(retagged, retitled, unassigned myself)
Deferred annotations make this slightly less annoying, because an official way to resolve forward references has now been published: https://www.python.org/dev/peps/pep-0563/#resolving-type-hints-at-runtime

What to do when this fails at runtime - and especially how to make the error message useful - is left as an exercise for the implementer.

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/internal/compat.py b/hypothesis-python/src/hypothesis/internal/compat.py
--- a/hypothesis-python/src/hypothesis/internal/compat.py
+++ b/hypothesis-python/src/hypothesis/internal/compat.py
@@ -125,14 +125,25 @@ def get_type_hints(thing):
 else:
 
     def get_type_hints(thing):
-        if inspect.isclass(thing) and not hasattr(thing, "__signature__"):
-            if is_typed_named_tuple(thing):
-                # Special handling for typing.NamedTuple
-                return thing._field_types  # type: ignore
-            thing = thing.__init__  # type: ignore
+        """Like the typing version, but tries harder and never errors.
+
+        Tries harder: if the thing to inspect is a class but typing.get_type_hints
+        raises an error or returns no hints, then this function will try calling it
+        on the __init__ method.  This second step often helps with user-defined
+        classes on older versions of Python.
+
+        Never errors: instead of raising TypeError for uninspectable objects, or
+        NameError for unresolvable forward references, just return an empty dict.
+        """
         try:
-            return typing.get_type_hints(thing)
-        except TypeError:
+            hints = typing.get_type_hints(thing)
+        except (TypeError, NameError):
+            hints = {}
+        if hints or not inspect.isclass(thing):
+            return hints
+        try:
+            return typing.get_type_hints(thing.__init__)
+        except (TypeError, NameError, AttributeError):
             return {}
 
 


Test patch:
diff --git a/hypothesis-python/tests/conftest.py b/hypothesis-python/tests/conftest.py
--- a/hypothesis-python/tests/conftest.py
+++ b/hypothesis-python/tests/conftest.py
@@ -29,6 +29,8 @@
 # Skip collection of tests which require the Django test runner,
 # or that don't work on the current version of Python.
 collect_ignore_glob = ["django/*"]
+if sys.version_info < (3, 6):  # Remove after dropping Python 3.5
+    collect_ignore_glob.append("cover/*py36*")
 if sys.version_info < (3, 8):
     collect_ignore_glob.append("cover/*py38*")
 
diff --git a/hypothesis-python/tests/cover/test_lookup_py36.py b/hypothesis-python/tests/cover/test_lookup_py36.py
new file mode 100644
--- /dev/null
+++ b/hypothesis-python/tests/cover/test_lookup_py36.py
@@ -0,0 +1,33 @@
+# This file is part of Hypothesis, which may be found at
+# https://github.com/HypothesisWorks/hypothesis/
+#
+# Most of this work is copyright (C) 2013-2020 David R. MacIver
+# (david@drmaciver.com), but it contains contributions by others. See
+# CONTRIBUTING.rst for a full list of people who may hold copyright, and
+# consult the git log if you need to determine who owns an individual
+# contribution.
+#
+# This Source Code Form is subject to the terms of the Mozilla Public License,
+# v. 2.0. If a copy of the MPL was not distributed with this file, You can
+# obtain one at https://mozilla.org/MPL/2.0/.
+#
+# END HEADER
+
+import typing
+
+import pytest
+
+from hypothesis import given, strategies as st
+from hypothesis.internal.compat import PYPY
+
+
+class TreeForwardRefs(typing.NamedTuple):
+    val: int
+    l: typing.Optional["TreeForwardRefs"]
+    r: typing.Optional["TreeForwardRefs"]
+
+
+@pytest.mark.skipif(PYPY, reason="pypy36 does not resolve the forward refs")
+@given(st.builds(TreeForwardRefs))
+def test_resolves_forward_references_outside_annotations(t):
+    assert isinstance(t, TreeForwardRefs)
diff --git a/hypothesis-python/tests/cover/test_lookup_py38.py b/hypothesis-python/tests/cover/test_lookup_py38.py
--- a/hypothesis-python/tests/cover/test_lookup_py38.py
+++ b/hypothesis-python/tests/cover/test_lookup_py38.py
@@ -13,6 +13,7 @@
 #
 # END HEADER
 
+import dataclasses
 import typing
 
 import pytest
@@ -101,3 +102,14 @@ def test_layered_optional_key_is_optional():
     # Optional keys are not currently supported, as PEP-589 leaves no traces
     # at runtime.  See https://github.com/python/cpython/pull/17214
     find_any(from_type(C), lambda d: "b" not in d)
+
+
+@dataclasses.dataclass()
+class Node:
+    left: typing.Union["Node", int]
+    right: typing.Union["Node", int]
+
+
+@given(st.builds(Node))
+def test_can_resolve_recursive_dataclass(val):
+    assert isinstance(val, Node)