Repository: HypothesisWorks/hypothesis
Pull request: Warn if search space for inferred strategy is small (no variation)
Fixes https://github.com/HypothesisWorks/hypothesis/issues/3653.

Problem statement:
Warn (or fail) when inferred strategy is the trivial default object
Given the following two tests:

```python
from hypothesis import given
import numpy as np

@given(x=...)
def test_int(x: int):
    assert x==0

@given(x=...)
def test_int8(x: np.int8):
    assert x==0
```

The first one fails (as it should!), but the second succeeds. In this contrived example, it is harmless; but for the unsuspecting user, it may cause belief that a method is properly tested when in reality the input is always zero.

The inferred strategy for `int` is `st.integers()`. For `np.int8`, it is `st.builds(np.int8)`. I'm not asking for full support for `np.int8` and other arbitrary data types, but would it be possible to warn (or even fail) if the inferred strategy is to just return default-valued objects? That way, the user would be informed and could specify it properly when necessary.

Hints from issue discussion:
I think emitting a warning when `st.from_type(thing)` resolves to a non-registered zero-argument `st.builds(thing)` would be reasonable, even though that won't catch all cases where we only generate a single value (and in fact doesn't guarantee that it always generates a single value!).  Implementation of that will go here, warning if `kwargs` is empty:

https://github.com/HypothesisWorks/hypothesis/blob/661af850bbfcb091a820eb16c84f56048c8e21c8/hypothesis-python/src/hypothesis/strategies/_internal/core.py#L1222

We can _also_ add proper handling for these numpy classes without too much trouble, via the `from_dtype()` functionality.  IMO it makes sense to do both; the latter just after we handle `enum`s here:

https://github.com/HypothesisWorks/hypothesis/blob/661af850bbfcb091a820eb16c84f56048c8e21c8/hypothesis-python/src/hypothesis/strategies/_internal/core.py#L1185-L1188
Thanks @Zac-HD , that sounds very reasonable. I would say it is the simple cases that are most dangerous since there is less expectation that it would "just work" for more complex examples.

Btw: It looks like this line may also be affected?

https://github.com/HypothesisWorks/hypothesis/blob/661af850bbfcb091a820eb16c84f56048c8e21c8/hypothesis-python/src/hypothesis/strategies/_internal/core.py#L1212



I can create a PR if you want, if you also point me to a line showing how to emit a warning I think I have enough information
Yep, well spotted; we'd want a warning there too.

```python
warnings.warn(
    "Some long message here",
    HypothesisWarning,
)
```

should be all it takes; along with updating tests to expect the new warnings (instead of crashing on them).  We may also want to define a new `class SmallSearchSpaceWarning(HypothesisWarning):` (name suggestions welcome) in `hypothesis.errors`, so that people can set up custom filters to apply only to it.  

As for the warning message, I'd go with something like

> from_type(thing) resolved to builds(thing), because {we couldn't get the signature / it has no (non-varargs) arguments}.  Use `st.register_type_strategy()` to resolve to a strategy which can generate more than one value, or silence this warning.

using `hypothesis.internal.reflection.repr_call()` to show the from_type() and builds() reprs.

Commit message: Fixes more tests

Patch to review:
diff --git a/hypothesis-python/RELEASE.rst b/hypothesis-python/RELEASE.rst
new file mode 100644
index 0000000000..a589d77751
--- /dev/null
+++ b/hypothesis-python/RELEASE.rst
@@ -0,0 +1,5 @@
+RELEASE_TYPE: patch
+
+Warn in :func:`~hypothesis.strategies.from_type` if the inferred strategy
+has no variation (always returning default instances). Also ensures
+variation in numpy data types by inferring h.extras.numpy.from_dtype().
diff --git a/hypothesis-python/src/hypothesis/errors.py b/hypothesis-python/src/hypothesis/errors.py
index 024d9dfded..7b6de721fb 100644
--- a/hypothesis-python/src/hypothesis/errors.py
+++ b/hypothesis-python/src/hypothesis/errors.py
@@ -175,3 +175,8 @@ class RewindRecursive(Exception):
 
     def __init__(self, target):
         self.target = target
+
+
+class SmallSearchSpaceWarning(HypothesisWarning):
+    """Indicates that an inferred strategy does not span the search space
+    in a meaningful way, for example by only creating default instances."""
diff --git a/hypothesis-python/src/hypothesis/strategies/_internal/core.py b/hypothesis-python/src/hypothesis/strategies/_internal/core.py
index 539adbff36..360e5e2b65 100644
--- a/hypothesis-python/src/hypothesis/strategies/_internal/core.py
+++ b/hypothesis-python/src/hypothesis/strategies/_internal/core.py
@@ -16,6 +16,7 @@
 import string
 import sys
 import typing
+import warnings
 from decimal import Context, Decimal, localcontext
 from fractions import Fraction
 from functools import lru_cache, reduce
@@ -47,7 +48,12 @@
 
 from hypothesis._settings import note_deprecation
 from hypothesis.control import cleanup, current_build_context, note
-from hypothesis.errors import InvalidArgument, ResolutionFailed, RewindRecursive
+from hypothesis.errors import (
+    InvalidArgument,
+    ResolutionFailed,
+    RewindRecursive,
+    SmallSearchSpaceWarning,
+)
 from hypothesis.internal.cathetus import cathetus
 from hypothesis.internal.charmap import as_general_categories
 from hypothesis.internal.compat import (
@@ -71,6 +77,7 @@
     get_signature,
     is_first_param_referenced_in_function,
     nicerepr,
+    repr_call,
     required_args,
 )
 from hypothesis.internal.validation import (
@@ -988,7 +995,7 @@ def builds(
 
 @cacheable
 @defines_strategy(never_lazy=True)
-def from_type(thing: Type[Ex], *, force_defer: bool = False) -> SearchStrategy[Ex]:
+def from_type(thing: Type[Ex]) -> SearchStrategy[Ex]:
     """Looks up the appropriate search strategy for the given type.
 
     ``from_type`` is used internally to fill in missing arguments to
@@ -1040,11 +1047,13 @@ def everything_except(excluded_types):
     This is useful when writing tests which check that invalid input is
     rejected in a certain way.
     """
-    if not force_defer:
-        try:
-            return _from_type(thing, [])
-        except Exception:
-            pass
+    try:
+        return _from_type(thing, [])
+    except Exception:
+        return _from_type_deferred(thing)
+
+
+def _from_type_deferred(thing: Type[Ex]) -> SearchStrategy[Ex]:
     # This tricky little dance is because we want to show the repr of the actual
     # underlying strategy wherever possible, as a form of user education, but
     # would prefer to fall back to the default "from_type(...)" repr instead of
@@ -1079,13 +1088,13 @@ def as_strategy(strat_or_callable, thing, final=True):
             raise ResolutionFailed(f"Error: {thing!r} resolved to an empty strategy")
         return strategy
 
-    def defer_recursion(thing, producer):
+    def from_type_guarded(thing):
         """Returns the result of producer, or ... if recursion on thing is encountered"""
         if thing in recurse_guard:
             raise RewindRecursive(thing)
         recurse_guard.append(thing)
         try:
-            return producer()
+            return _from_type(thing, recurse_guard)
         except RewindRecursive as rr:
             if rr.target != thing:
                 raise
@@ -1158,9 +1167,9 @@ def defer_recursion(thing, producer):
                     raise InvalidArgument(
                         f"`{k}: {v.__name__}` is not a valid type annotation"
                     ) from None
-            anns[k] = defer_recursion(v, lambda: _from_type(v, recurse_guard))
+            anns[k] = from_type_guarded(v)
             if anns[k] is ...:
-                anns[k] = from_type(v, force_defer=True)
+                anns[k] = _from_type_deferred(v)
         if (
             (not anns)
             and thing.__annotations__
@@ -1204,6 +1213,13 @@ def defer_recursion(thing, producer):
     # may be able to fall back on type annotations.
     if issubclass(thing, enum.Enum):
         return sampled_from(thing)
+    # Handle numpy types through extras
+    if thing.__module__ == "numpy":
+        import numpy as np
+
+        from hypothesis.extra.numpy import from_dtype
+
+        return from_dtype(np.dtype(thing))
 
     # Finally, try to build an instance by calling the type object.  Unlike builds(),
     # this block *does* try to infer strategies for arguments with default values.
@@ -1227,19 +1243,27 @@ def defer_recursion(thing, producer):
             hints = get_type_hints(thing)
             params = get_signature(thing).parameters
         except Exception:
-            return builds(thing)
+            params = {}  # type: ignore
         kwargs = {}
         for k, p in params.items():
             if (
                 k in hints
                 and k != "return"
-                and p.default is not Parameter.empty
                 and p.kind in (Parameter.POSITIONAL_OR_KEYWORD, Parameter.KEYWORD_ONLY)
             ):
-                kwargs[k] = defer_recursion(
-                    hints[k],
-                    lambda: just(p.default) | _from_type(hints[k], recurse_guard),
-                )
+                kwargs[k] = from_type_guarded(hints[k])
+                if p.default is not Parameter.empty and kwargs[k] is not ...:
+                    kwargs[k] |= just(p.default)
+        if not kwargs:
+            from_type_repr = repr_call(from_type, (thing,), {})
+            builds_repr = repr_call(builds, (thing,), {})
+            warnings.warn(
+                f"{from_type_repr} resolved to {builds_repr}, because we could not "
+                "find any (non-varargs) arguments. Use st.register_type_strategy() "
+                "to resolve to a strategy which can generate more than one value, "
+                "or silence this warning.",
+                SmallSearchSpaceWarning,
+            )
         return builds(thing, **kwargs)
     # And if it's an abstract type, we'll resolve to a union of subclasses instead.
     subclasses = thing.__subclasses__()
diff --git a/hypothesis-python/tests/cover/test_annotations.py b/hypothesis-python/tests/cover/test_annotations.py
index d073c12645..56a9b611b2 100644
--- a/hypothesis-python/tests/cover/test_annotations.py
+++ b/hypothesis-python/tests/cover/test_annotations.py
@@ -14,6 +14,7 @@
 import pytest
 
 from hypothesis import given, strategies as st
+from hypothesis.errors import SmallSearchSpaceWarning
 from hypothesis.internal.reflection import (
     convert_positional_arguments,
     define_function_signature,
@@ -124,6 +125,8 @@ def test_attrs_inference_builds(c):
     pass
 
 
-@given(st.from_type(Inferrables))
-def test_attrs_inference_from_type(c):
-    pass
+with pytest.warns(SmallSearchSpaceWarning):
+
+    @given(st.from_type(Inferrables))
+    def test_attrs_inference_from_type(c):
+        pass
diff --git a/hypothesis-python/tests/cover/test_lookup.py b/hypothesis-python/tests/cover/test_lookup.py
index ee725f82f9..d8f4c206bc 100644
--- a/hypothesis-python/tests/cover/test_lookup.py
+++ b/hypothesis-python/tests/cover/test_lookup.py
@@ -116,6 +116,10 @@ class Elem:
     pass
 
 
+# To avoid SmallSearchSpaceWarning
+st.register_type_strategy(Elem, st.builds(Elem))
+
+
 @pytest.mark.parametrize(
     "typ,coll_type",
     [
@@ -152,6 +156,10 @@ class ElemValue:
     pass
 
 
+# To avoid SmallSearchSpaceWarning
+st.register_type_strategy(ElemValue, st.builds(ElemValue))
+
+
 @pytest.mark.parametrize(
     "typ,coll_type",
     [
@@ -702,6 +710,10 @@ def foo(self):
         pass
 
 
+# To avoid SmallSearchSpaceWarning
+st.register_type_strategy(ConcreteFoo, st.builds(ConcreteFoo))
+
+
 @given(st.from_type(AbstractFoo))
 def test_can_resolve_abstract_class(instance):
     assert isinstance(instance, ConcreteFoo)
@@ -962,14 +974,16 @@ def test_from_type_can_be_default_or_annotation():
 
 @pytest.mark.parametrize("t", BUILTIN_TYPES, ids=lambda t: t.__name__)
 def test_resolves_builtin_types(t):
-    v = st.from_type(t).example()
-    assert isinstance(v, t)
+    with pytest.warns():  # some types may warn SmallSearchSpaceWarning
+        v = st.from_type(t).example()
+        assert isinstance(v, t)
 
 
 @pytest.mark.parametrize("t", BUILTIN_TYPES, ids=lambda t: t.__name__)
 def test_resolves_forwardrefs_to_builtin_types(t):
-    v = st.from_type(typing.ForwardRef(t.__name__)).example()
-    assert isinstance(v, t)
+    with pytest.warns():  # some types may warn SmallSearchSpaceWarning
+        v = st.from_type(typing.ForwardRef(t.__name__)).example()
+        assert isinstance(v, t)
 
 
 @pytest.mark.parametrize("t", BUILTIN_TYPES, ids=lambda t: t.__name__)
diff --git a/hypothesis-python/tests/cover/test_lookup_py37.py b/hypothesis-python/tests/cover/test_lookup_py37.py
index 6a1955eb6b..45778d8b44 100644
--- a/hypothesis-python/tests/cover/test_lookup_py37.py
+++ b/hypothesis-python/tests/cover/test_lookup_py37.py
@@ -18,7 +18,7 @@
 
 import pytest
 
-from hypothesis import assume, given
+from hypothesis import assume, given, strategies as st
 
 # On Python 3.7 and 3.8, `from __future__ import annotations` means
 # that the syntax is supported; but the feature fails at runtime.  On Python
@@ -40,6 +40,11 @@ class Value:
     pass
 
 
+# To avoid SmallSearchSpaceWarning
+st.register_type_strategy(Elem, st.builds(Elem))
+st.register_type_strategy(Value, st.builds(Value))
+
+
 def check(t, ex):
     assert isinstance(ex, t)
     assert all(isinstance(e, Elem) for e in ex)
diff --git a/hypothesis-python/tests/cover/test_type_lookup.py b/hypothesis-python/tests/cover/test_type_lookup.py
index f4537e8843..e5fb1c594e 100644
--- a/hypothesis-python/tests/cover/test_type_lookup.py
+++ b/hypothesis-python/tests/cover/test_type_lookup.py
@@ -20,6 +20,7 @@
     HypothesisDeprecationWarning,
     InvalidArgument,
     ResolutionFailed,
+    SmallSearchSpaceWarning,
 )
 from hypothesis.internal.compat import get_type_hints
 from hypothesis.internal.reflection import get_pretty_function_description
@@ -204,7 +205,8 @@ def test_uninspectable_builds():
 
 def test_uninspectable_from_type():
     with pytest.raises(TypeError, match="object is not callable"):
-        st.from_type(BrokenClass).example()
+        with pytest.warns(SmallSearchSpaceWarning):
+            st.from_type(BrokenClass).example()
 
 
 def _check_instances(t):
diff --git a/hypothesis-python/tests/cover/test_type_lookup_forward_ref.py b/hypothesis-python/tests/cover/test_type_lookup_forward_ref.py
index 488023b6c7..e91ce9dff1 100644
--- a/hypothesis-python/tests/cover/test_type_lookup_forward_ref.py
+++ b/hypothesis-python/tests/cover/test_type_lookup_forward_ref.py
@@ -27,7 +27,7 @@
 import pytest
 
 from hypothesis import given, strategies as st
-from hypothesis.errors import ResolutionFailed
+from hypothesis.errors import ResolutionFailed, SmallSearchSpaceWarning
 
 from tests.common import utils
 
@@ -91,10 +91,15 @@ def missing_dot_access_fun(thing: _MissingDotAccess) -> int:
     return 1
 
 
-@given(st.builds(correct_dot_access_fun))
-def test_bound_correct_dot_access_forward_ref(built):
-    """Correct resolution of dot access types."""
-    assert isinstance(built, int)
+def test_bound_correct_dot_access_forward_ref():
+    with pytest.warns(SmallSearchSpaceWarning):
+
+        @given(st.builds(correct_dot_access_fun))
+        def test(built):
+            """Correct resolution of dot access types."""
+            assert isinstance(built, int)
+
+        test()
 
 
 @pytest.mark.parametrize("function", [wrong_dot_access_fun, missing_dot_access_fun])
diff --git a/hypothesis-python/tests/ghostwriter/recorded/add_custom_classes.txt b/hypothesis-python/tests/ghostwriter/recorded/add_custom_classes.txt
index e315456ec4..b13ae3f76a 100644
--- a/hypothesis-python/tests/ghostwriter/recorded/add_custom_classes.txt
+++ b/hypothesis-python/tests/ghostwriter/recorded/add_custom_classes.txt
@@ -7,7 +7,10 @@ from example_code.future_annotations import CustomClass
 from hypothesis import given, strategies as st
 
 
-@given(c1=st.builds(CustomClass), c2=st.one_of(st.none(), st.builds(CustomClass)))
+@given(
+    c1=st.builds(CustomClass, number=st.integers()),
+    c2=st.one_of(st.none(), st.builds(CustomClass, number=st.integers())),
+)
 def test_fuzz_add_custom_classes(
     c1: example_code.future_annotations.CustomClass,
     c2: typing.Union[example_code.future_annotations.CustomClass, None],
diff --git a/hypothesis-python/tests/ghostwriter/test_expected_output.py b/hypothesis-python/tests/ghostwriter/test_expected_output.py
index 60a3c4f4d9..9c259a97c8 100644
--- a/hypothesis-python/tests/ghostwriter/test_expected_output.py
+++ b/hypothesis-python/tests/ghostwriter/test_expected_output.py
@@ -32,6 +32,7 @@
 )
 
 import hypothesis
+from hypothesis.errors import SmallSearchSpaceWarning
 from hypothesis.extra import ghostwriter
 from hypothesis.utils.conventions import not_set
 
@@ -267,8 +268,11 @@ def test_ghostwriter_example_outputs(update_recorded_outputs, data):
 
 
 def test_ghostwriter_on_hypothesis(update_recorded_outputs):
-    actual = ghostwriter.magic(hypothesis).replace("Strategy[+Ex]", "Strategy")
-    expected = get_recorded("hypothesis_module_magic", actual * update_recorded_outputs)
-    if sys.version_info[:2] < (3, 10):
-        assert actual == expected
-    exec(expected, {"not_set": not_set})
+    with pytest.warns(SmallSearchSpaceWarning):
+        actual = ghostwriter.magic(hypothesis).replace("Strategy[+Ex]", "Strategy")
+        expected = get_recorded(
+            "hypothesis_module_magic", actual * update_recorded_outputs
+        )
+        if sys.version_info[:2] < (3, 10):
+            assert actual == expected
+        exec(expected, {"not_set": not_set})
diff --git a/hypothesis-python/tests/numpy/test_from_dtype.py b/hypothesis-python/tests/numpy/test_from_dtype.py
index ee74060c2f..d9571ff8c2 100644
--- a/hypothesis-python/tests/numpy/test_from_dtype.py
+++ b/hypothesis-python/tests/numpy/test_from_dtype.py
@@ -17,11 +17,12 @@
 from hypothesis.errors import InvalidArgument
 from hypothesis.extra import numpy as nps
 from hypothesis.internal.floats import width_smallest_normals
+from hypothesis.strategies import from_type
 from hypothesis.strategies._internal import SearchStrategy
 
 from tests.common.debug import assert_no_examples, find_any
 
-STANDARD_TYPES = [
+STANDARD_DTYPES = [
     np.dtype(t)
     for t in (
         "int8",
@@ -47,7 +48,7 @@
 ]
 for nonstandard_typecode in ["g", "G", "S1", "q", "Q"]:
     try:
-        STANDARD_TYPES.append(np.dtype(nonstandard_typecode))
+        STANDARD_DTYPES.append(np.dtype(nonstandard_typecode))
     except Exception:
         pass
 
@@ -57,7 +58,7 @@ def test_strategies_for_standard_dtypes_have_reusable_values(dtype):
     assert nps.from_dtype(dtype).has_reusable_values
 
 
-@pytest.mark.parametrize("t", STANDARD_TYPES)
+@pytest.mark.parametrize("t", STANDARD_DTYPES)
 def test_produces_instances(t):
     @given(nps.from_dtype(t))
     def test_is_t(x):
@@ -283,3 +284,15 @@ def condition(n):
         find_any(strat, condition)
     else:
         assert_no_examples(strat, condition)
+
+
+@pytest.mark.parametrize("dtype", STANDARD_DTYPES)
+def test_resolves_and_varies_numpy_dtype(dtype):
+    @given(x=from_type(dtype.type))
+    def test_is_resolved_and_varied(x):
+        assert isinstance(x, dtype.type)
+        # Filter out those equal to the default value, to check that we see
+        # non-default values as well
+        assume(x != type(x)())
+
+    test_is_resolved_and_varied()


Code review comments:
diff_hunk: @@ -1204,6 +1213,13 @@ def defer_recursion(thing, producer):
     # may be able to fall back on type annotations.
     if issubclass(thing, enum.Enum):
         return sampled_from(thing)
+    # Handle numpy types through extras
+    if thing.__module__ == "numpy":
+        import numpy as np
+
+        from hypothesis.extra.numpy import from_dtype
+
+        return from_dtype(np.dtype(thing))
path: hypothesis-python/src/hypothesis/strategies/_internal/core.py
text: @user1:
I think this is _too_ general; thing might be an array type for example. 

@author:
Good point. Maybe
```
dtype = np.dtype(thing)
if dtype.kind != "O":
    return from_dtype(dtype)
```
what do you think?

@author:
Actually `st.from_type(np.ndarray)` works fine as-is, I have no idea why :-)

@author:
Ah yes, found it:
https://github.com/HypothesisWorks/hypothesis/blob/8cce2dbf733bba78257b4d3fe3ac5c42b9a43820/hypothesis-python/src/hypothesis/strategies/_internal/types.py#L602

So it's a bit inconsistent right now wrt where/how the types are mapped. It could potentially be unified by deleting this block and adding
```
    if thing == np.dtype:
        return array_dtypes()
    if thing == np.ndarray:
        return arrays(scalar_dtypes(), array_shapes(max_dims=2))
```
to the new `extras.numpy.from_type_internal` method.