Repository: HypothesisWorks/hypothesis

Issue:
What does HealthCheck.data_too_large actually mean?
I am finding the docs around this one really confusing and ambiguous... i get what the others mean but `data_too_large` always has me a bit confused 😵‍💫.

> Check for when the typical size of the examples you are generating exceeds the maximum allowed size too often.

https://hypothesis.readthedocs.io/en/latest/healthchecks.html#hypothesis.HealthCheck.data_too_large

too large how? is it defined somewhere? i have looked in the settings and theres nothing about max example size, nor does it seem to be anything in the docs...

Is it about the range of values that can be generated or the actual memory size in bytes? if i have a hypothetical strategy that generates a billion `00` bytes or a billion `01` bytes thats going to result in only 2 possible examples, but a gig of memory footprint, will it trigger the error? (ignoring the fact there seems to be a threshold of 20 times to trigger an error).

the code causing the healtcheck to fail mentions overruns but i cant find any details on that in the docs either? is it a low level conjecture thing?

is it just to save the testsuite from chewing through ram or is there a deeper meaning?
docs: calrify the description of data_too_large health check
fixes: #3032 

updates the description of `HealthCheck.data_too_large` from:

> Check for when the typical size of the examples you are generating exceeds the maximum allowed size too often.

to:

> Check for when the typical size of the examples you are generating too often exceed the maximum amount of random data hypothesis is able to generate.

making it clearer that it's about running out of random data.


Discussion:
It's a low-level conjecture thing, basically "your strategy wanted more random bits than we had available", so *mostly* the range of values but in a way that depends on *how* they get generated.

We should improve the docs and the error message though, thanks for pointing that out!
I am glad to help, if we can find a helpful description I am happy to PR the docs.

so my hypothetical above choosing between two huge constants wouldn't trigger it as it's a single bit of randomness.

but if I tried to generate a billion booleans it would as (to overly simplify) the RNG would run out of randomness?
Correct - we cap at `8192` bytes internally to avoid very slow generation+shrinking performance.
Would something like this work?

> Check for when the typical size of the examples you are generating too often exceed the maximum amount of random data hypothesis is able to generate.
should I create a release file for this? its just a docs change so does it warrant a changelog entry / version bump?

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/_settings.py b/hypothesis-python/src/hypothesis/_settings.py
--- a/hypothesis-python/src/hypothesis/_settings.py
+++ b/hypothesis-python/src/hypothesis/_settings.py
@@ -461,8 +461,14 @@ def all(cls) -> List["HealthCheck"]:
         return list(HealthCheck)
 
     data_too_large = 1
-    """Check for when the typical size of the examples you are generating
-    exceeds the maximum allowed size too often."""
+    """Checks if too many examples are aborted for being too large.
+
+    This is measured by the number of random choices that Hypothesis makes
+    in order to generate something, not the size of the generated object.
+    For example, choosing a 100MB object from a predefined list would take
+    only a few bits, while generating 10KB of JSON from scratch might trigger
+    this health check.
+    """
 
     filter_too_much = 2
     """Check for when the test is filtering out too many examples, either
diff --git a/hypothesis-python/src/hypothesis/strategies/_internal/core.py b/hypothesis-python/src/hypothesis/strategies/_internal/core.py
--- a/hypothesis-python/src/hypothesis/strategies/_internal/core.py
+++ b/hypothesis-python/src/hypothesis/strategies/_internal/core.py
@@ -1711,7 +1711,7 @@ def register_type_strategy(
     from hypothesis.strategies._internal import types
 
     if not types.is_a_type(custom_type):
-        raise InvalidArgument("custom_type=%r must be a type")
+        raise InvalidArgument(f"custom_type={custom_type!r} must be a type")
     elif not (isinstance(strategy, SearchStrategy) or callable(strategy)):
         raise InvalidArgument(
             "strategy=%r must be a SearchStrategy, or a function that takes "
diff --git a/hypothesis-python/src/hypothesis/strategies/_internal/types.py b/hypothesis-python/src/hypothesis/strategies/_internal/types.py
--- a/hypothesis-python/src/hypothesis/strategies/_internal/types.py
+++ b/hypothesis-python/src/hypothesis/strategies/_internal/types.py
@@ -98,13 +98,17 @@ def try_issubclass(thing, superclass):
 
 
 def is_a_new_type(thing):
-    # At runtime, `typing.NewType` returns an identity function rather
-    # than an actual type, but we can check whether that thing matches.
-    return (
-        hasattr(thing, "__supertype__")
-        and getattr(thing, "__module__", None) in ("typing", "typing_extensions")
-        and inspect.isfunction(thing)
-    )
+    if sys.version_info[:2] < (3, 10):
+        # At runtime, `typing.NewType` returns an identity function rather
+        # than an actual type, but we can check whether that thing matches.
+        return (
+            hasattr(thing, "__supertype__")
+            and getattr(thing, "__module__", None) in ("typing", "typing_extensions")
+            and inspect.isfunction(thing)
+        )
+    # In 3.10 and later, NewType is actually a class - which simplifies things.
+    # See https://bugs.python.org/issue44353 for links to the various patches.
+    return isinstance(thing, typing.NewType)  # pragma: no cover  # on 3.8, anyway
 
 
 def is_a_type(thing):


Test patch: