Repository: HypothesisWorks/hypothesis

Issue:
Constrain inferred Django field strategies based on validators before filtering
Currently, we infer strategies for Django fields almost solely based on the field type, then filter that strategy by calling any attached validators.

Unfortunately, this can be highly inefficient (eg #1112; could not generate default User model).  Instead of inferring a strategy from the field type alone, we should also choose arguments to the strategy by inspecting any validators.  For some (eg RegexValidator) we might even choose a different strategy.


Discussion:
Notes dump:

- Django has [a bunch of default validators](https://github.com/django/django/blob/master/django/core/validators.py), which have been pretty stable for a long time.  We can therefore make the field strategies substantially more powerful by introspecting validators as well as the field type.
- For some (eg RegexValidator, EmailValidator), we'll choose a different base strategy
- For most others, we can choose better arguments to the strategy based on the validator - eg MinValueValidator, MaxLengthValidator, etc.
- For custom (and some builtin) validators, filtering will remain the only way.  We should keep it in all cases as a final check for correctness.
- We should resolve the strategy type and default kwargs for the given field type, then resolve kwargs from the validators (min_value, max_value, min_length, etc), possibly switch strategy type, and reconcile the arguments to give the actual strategy.
- Multiple RegexValidators are tricky: The current solution is to take the union of the *strategies* for each regex, which does works if one generates output that usually passes the others.  It's possible to compute the intersection of regular expressions (not PCRE) by treating them as finite automata using [greenery](https://github.com/qntm/greenery), but this is well out of scope.
- Documentation should suggest use of validators derived from Django's as an aid to Hypothesis as well as general correctness.
- We'll want to overhaul strategy registration to match this, too - allow strategy objects, but prefer functions for numeric and string types where we can do validator-based constraints.
  - Check that functions can only be registered for string or numeric fields (because we can't usefully inspect others)
  - Check that functions have the argument names we expect (eg `min_size`, `max_value`) so we can actually use our inspection results
Another note to self: after this is done, we could work with Simon to integrate [hypothesis-drf](https://gitlab.com/schmidt.simon/hypothesis-drf) and support the [Django REST Framework](http://www.django-rest-framework.org/).  

Technically possible before, but it would be much easier to support this *well* if we get a solid field-level inference + validation + registration system going first.  At least if we bear that in mind, and thus this note!
Based on chatting with a few Django people at the PyCon Australia sprints, validators are rarely used for model fields but almost always for forms (see our oldest open issue, #35).  We can add them in either order, but this is low-priority before a `forms()` function exists to use it.

It is almost certainly not worth the code complexity to support a public API for customisable validator inference; we should just support numeric and string validators and stop there.
#1743 made this quite a bit easier, by cleaning up [the `from_field` API](https://github.com/HypothesisWorks/hypothesis/blob/master/hypothesis-python/src/hypothesis/extra/django/_fields.py).

- In `_for_text`, get the Min and MaxLengthValidator (if any) and if relevant use the bounding values in place of the field-level min_size and max_size arguments.  This only applies to fields without a RegexValidator.
- For numeric types, write a registering function instead of writing static strategies into the lookup table.  I would use a helper function `_get_bounds_from_field(field, min_=None, max_=None)` to factor out all the logic about getting the validator(s) and comparing bounds into a single place.
@Zac-HD would love an update on this feature. Deciding on what testing strategy to use with a new django project and would like to try Hypothesis out. 
Definitely don't let this stop you!

This issue is a nice-to-have which would make automatically created strategies more efficient in some cases. We already infer strategies and they're guaranteed to be valid, plus you can supply an explicit strategy for any field you like if the automatic ones aren't quite what you want (e.g. only positive numbers in a field that allows negatives). 
Hi, I would like to start working on it :)

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/extra/django/_fields.py b/hypothesis-python/src/hypothesis/extra/django/_fields.py
--- a/hypothesis-python/src/hypothesis/extra/django/_fields.py
+++ b/hypothesis-python/src/hypothesis/extra/django/_fields.py
@@ -29,7 +29,7 @@
 from django.db import models as dm
 
 from hypothesis import strategies as st
-from hypothesis.errors import InvalidArgument
+from hypothesis.errors import InvalidArgument, ResolutionFailed
 from hypothesis.extra.pytz import timezones
 from hypothesis.internal.validation import check_type
 from hypothesis.provisional import urls
@@ -39,14 +39,31 @@
 F = TypeVar("F", bound=AnyField)
 
 
+def numeric_bounds_from_validators(
+    field, min_value=float("-inf"), max_value=float("inf")
+):
+    for v in field.validators:
+        if isinstance(v, django.core.validators.MinValueValidator):
+            min_value = max(min_value, v.limit_value)
+        elif isinstance(v, django.core.validators.MaxValueValidator):
+            max_value = min(max_value, v.limit_value)
+    return min_value, max_value
+
+
+def integers_for_field(min_value, max_value):
+    def inner(field):
+        return st.integers(*numeric_bounds_from_validators(field, min_value, max_value))
+
+    return inner
+
+
 # Mapping of field types, to strategy objects or functions of (type) -> strategy
 _global_field_lookup = {
-    dm.SmallIntegerField: st.integers(-32768, 32767),
-    dm.IntegerField: st.integers(-2147483648, 2147483647),
-    dm.BigIntegerField: st.integers(-9223372036854775808, 9223372036854775807),
-    dm.PositiveIntegerField: st.integers(0, 2147483647),
-    dm.PositiveSmallIntegerField: st.integers(0, 32767),
-    dm.BinaryField: st.binary(),
+    dm.SmallIntegerField: integers_for_field(-32768, 32767),
+    dm.IntegerField: integers_for_field(-2147483648, 2147483647),
+    dm.BigIntegerField: integers_for_field(-9223372036854775808, 9223372036854775807),
+    dm.PositiveIntegerField: integers_for_field(0, 2147483647),
+    dm.PositiveSmallIntegerField: integers_for_field(0, 32767),
     dm.BooleanField: st.booleans(),
     dm.DateField: st.dates(),
     dm.EmailField: emails(),
@@ -57,8 +74,10 @@
     df.DateField: st.dates(),
     df.DurationField: st.timedeltas(),
     df.EmailField: emails(),
-    df.FloatField: st.floats(allow_nan=False, allow_infinity=False),
-    df.IntegerField: st.integers(-2147483648, 2147483647),
+    df.FloatField: lambda field: st.floats(
+        *numeric_bounds_from_validators(field), allow_nan=False, allow_infinity=False
+    ),
+    df.IntegerField: integers_for_field(-2147483648, 2147483647),
     df.NullBooleanField: st.one_of(st.none(), st.booleans()),
     df.URLField: urls(),
     df.UUIDField: st.uuids(),
@@ -155,14 +174,38 @@ def _for_form_ip(field):
         return st.ip_addresses(v=4).map(str)
     if validate_ipv6_address in field.default_validators:
         return _ipv6_strings
-    raise InvalidArgument("No IP version validator on field=%r" % field)
+    raise ResolutionFailed("No IP version validator on field=%r" % field)
 
 
 @register_for(dm.DecimalField)
 @register_for(df.DecimalField)
 def _for_decimal(field):
+    min_value, max_value = numeric_bounds_from_validators(field)
     bound = Decimal(10 ** field.max_digits - 1) / (10 ** field.decimal_places)
-    return st.decimals(min_value=-bound, max_value=bound, places=field.decimal_places)
+    return st.decimals(
+        min_value=max(min_value, -bound),
+        max_value=min(max_value, bound),
+        places=field.decimal_places,
+    )
+
+
+def length_bounds_from_validators(field):
+    min_size = 1
+    max_size = field.max_length
+    for v in field.validators:
+        if isinstance(v, django.core.validators.MinLengthValidator):
+            min_size = max(min_size, v.limit_value)
+        elif isinstance(v, django.core.validators.MaxLengthValidator):
+            max_size = min(max_size or v.limit_value, v.limit_value)
+    return min_size, max_size
+
+
+@register_for(dm.BinaryField)
+def _for_binary(field):
+    min_size, max_size = length_bounds_from_validators(field)
+    if getattr(field, "blank", False) or not getattr(field, "required", True):
+        return st.just(b"") | st.binary(min_size=min_size, max_size=max_size)
+    return st.binary(min_size=min_size, max_size=max_size)
 
 
 @register_for(dm.CharField)
@@ -188,18 +231,16 @@ def _for_text(field):
         # compute intersections of the full Python regex language.
         return st.one_of(*[st.from_regex(r) for r in regexes])
     # If there are no (usable) regexes, we use a standard text strategy.
-    min_size = 1
-    if getattr(field, "blank", False) or not getattr(field, "required", True):
-        min_size = 0
+    min_size, max_size = length_bounds_from_validators(field)
     strategy = st.text(
         alphabet=st.characters(
             blacklist_characters="\x00", blacklist_categories=("Cs",)
         ),
         min_size=min_size,
-        max_size=field.max_length,
-    )
-    if getattr(field, "required", True):
-        strategy = strategy.filter(lambda s: s.strip())
+        max_size=max_size,
+    ).filter(lambda s: min_size <= len(s.strip()))
+    if getattr(field, "blank", False) or not getattr(field, "required", True):
+        return st.just("") | strategy
     return strategy
 
 
@@ -271,7 +312,7 @@ def from_field(field: F) -> st.SearchStrategy[Union[F, None]]:
         if type(field) not in _global_field_lookup:
             if getattr(field, "null", False):
                 return st.none()
-            raise InvalidArgument("Could not infer a strategy for %r", (field,))
+            raise ResolutionFailed("Could not infer a strategy for %r", (field,))
         strategy = _global_field_lookup[type(field)]  # type: ignore
         if not isinstance(strategy, st.SearchStrategy):
             strategy = strategy(field)


Test patch:
diff --git a/hypothesis-python/tests/django/toystore/forms.py b/hypothesis-python/tests/django/toystore/forms.py
--- a/hypothesis-python/tests/django/toystore/forms.py
+++ b/hypothesis-python/tests/django/toystore/forms.py
@@ -14,6 +14,12 @@
 # END HEADER
 
 from django import forms
+from django.core.validators import (
+    MaxLengthValidator,
+    MaxValueValidator,
+    MinLengthValidator,
+    MinValueValidator,
+)
 from django.forms import widgets
 
 from tests.django.toystore.models import (
@@ -104,6 +110,15 @@ class TemporalFieldForm(ReprForm):
     _split_date_time = forms.SplitDateTimeField()
 
 
+class WithValidatorsForm(ReprForm):
+    num_validators = [MinValueValidator(1), MaxValueValidator(5)]
+    _int_one_to_five = forms.IntegerField(validators=num_validators)
+    _decimal_one_to_five = forms.FloatField(validators=num_validators)
+    _float_one_to_five = forms.FloatField(validators=num_validators)
+    len_validators = [MinLengthValidator(5), MaxLengthValidator(10)]
+    _string_five_to_ten = forms.CharField(validators=len_validators)
+
+
 class EmailFieldForm(ReprForm):
     _email = forms.EmailField()
 
diff --git a/hypothesis-python/tests/django/toystore/test_given_forms.py b/hypothesis-python/tests/django/toystore/test_given_forms.py
--- a/hypothesis-python/tests/django/toystore/test_given_forms.py
+++ b/hypothesis-python/tests/django/toystore/test_given_forms.py
@@ -34,6 +34,7 @@
     TemporalFieldForm,
     URLFieldForm,
     UUIDFieldForm,
+    WithValidatorsForm,
 )
 
 register_field_strategy(
@@ -114,3 +115,10 @@ def test_excessive_values_in_multi_value_field(self, excessive_form):
     @given(from_form(ShortStringForm))
     def test_short_string_form(self, short_string_form):
         self.assertTrue(short_string_form.is_valid())
+
+    @given(from_form(WithValidatorsForm))
+    def test_tight_validators_form(self, x):
+        self.assertTrue(1 <= x.data["_int_one_to_five"] <= 5)
+        self.assertTrue(1 <= x.data["_decimal_one_to_five"] <= 5)
+        self.assertTrue(1 <= x.data["_float_one_to_five"] <= 5)
+        self.assertTrue(5 <= len(x.data["_string_five_to_ten"]) <= 10)