Repository: HypothesisWorks/hypothesis

Issue:
FailedHealthCheck with hypothesis.extra.django for django.contrib.auth.models.User model
I just use create the most simple strategy like this:
`USER_STRATEGY = models.models(User)`

Then I get a FailedHealthCheck:
`hypothesis.errors.FailedHealthCheck: It looks like your strategy is filtering out a lot of data. Health check found 50 filtered examples but only 3 good ones. This will make your tests much slower ...`

Any idea what I can do to get User objects for my tests?


Discussion:
I would guess you have some complex validation on one or more fields, and the default strategy is giving you data of the right type (eg `DateField`) that fails validation (eg 'only the first of the month').

You can override the strategy for each field of a model, with the example of customer age [here](https://hypothesis.readthedocs.io/en/latest/django.html) - but I'll admit the docs don't make it obvious!  (We'll therefore keep this issue open to improve them.)  For example:

    USER_STRATEGY = models.models(User, age=st.integers(0, 100))

Does that help?
Yeah I suspected that at first too, because we have a lot of ForeignKeys to User. But even in a fresh project the same happens.
My whole test code is this:

```python
from django.contrib.auth.models import User
from django.test import TestCase
from hypothesis import given
from hypothesis.extra.django import models as model_st

USER_STRATEGY = model_st.models(User)


class TestUser(TestCase):
    @given(USER_STRATEGY)
    def test_user(self, user):
        assert user.username
```

My django version is Django 2.0.2 with Hypothesis 3.45.0
You're using `django.test.TestCase` but you need to be using `hypothesis.extra.django.TestCase`. What's happening is that generated users are colliding with previous generated users because the database state isn't being reset at the start of each generation (though I'm a little surprised that this is how that manifests).

We should probably be better about signalling this problem.
Ooohh, yep, that would do it.  Related to #1071 and the (hopefully someday soon) solution in #1072 - in both cases we need to check the type of the class that our test method is defined on.
I'm going to close this as it's not actually a bug in Hypothesis, but I've open #1113 for communicating this problem more clearly.
That doesn't change it for me. 

```python
# Create your tests here.
import hypothesis.extra.django
from django.contrib.auth.models import User
from hypothesis import given
from hypothesis.extra.django import models as model_st

USER_STRATEGY = model_st.models(User)


class TestUser(hypothesis.extra.django.TestCase):
    @given(USER_STRATEGY)
    def test_user(self, user):
        assert user.username
```

still gives me 

`hypothesis.errors.FailedHealthCheck: It looks like your strategy is filtering out a lot of data. Health check found 50 filtered examples but only 5 good ones`

Interesting. You're absolutely right. I could have sworn we had tests that this worked with the standard auth User, but apparently we don't! Sorry for jumping to conclusions.
Yeah, regressions happen. No worries ;)
Hope you manage to find a solution and thanks for all the help!
Based on some investigation, I'm pretty sure the problem is that the user model has a very restrictive idea of what characters a username can contain, and we're trying to generate arbitrary unicode text for it.

We should figure out a proper fix, but the following will let you work around it until we do:

```python

import hypothesis.strategies as st
import string

USER_STRATEGY = model_st.models(
    User,
    username=st.text(
        alphabet=string.ascii_letters + string.digits + '@/./+/-/_'))
```
nitpick: @DRMacIver, I don't think you meant to include the slashes in the character list there :wink:
> (I know for a fact there are people using Hypothesis for Django, but if many of them are using its model introspection then I'm frankly astonished that we've never seen this before)

I thought this might be new in Django 2.0, but no - supported since 1.10 😞

> nitpick: @DRMacIver, I don't think you meant to include the slashes in the character list there 😉

That's straight outta the Django docs, but you're right that they shouldn't be in the string literal.  Instead, why not use [the regex that we're failing to validate against](https://github.com/django/django/blob/2014db50f4522243dba3190c640f64cf124d5b68/django/contrib/auth/validators.py)?  `username=st.from_regex(r'^[\w.@+-]+$')`

Which, uh, probably *is* the proper fix: for all string fields that use a [`RegexValidator`](https://docs.djangoproject.com/en/2.0/ref/validators/#regexvalidator), we should use `from_regex` instead of `text`.  Note that we will need to be careful about inverted matches, flags, and compiled expressions vs string patterns.
> nitpick: @DRMacIver, I don't think you meant to include the slashes in the character list there 😉

Uh, yeah, I just copied those out of the validator error message without really thinking about it.
> I thought this might be new in Django 2.0, but no - supported since 1.10 😞

I mean I do know that there are people using this stuff! I know there are people using Hypothesis + Django reasonably heavily, but I think the number of people using model introspection must be *way* lower than I thought if this is the first time we've seen this.
Notes dump:

- Django has [a bunch of default validators](https://github.com/django/django/blob/master/django/core/validators.py), which have been pretty stable for a long time.  We can therefore make the field strategies substantially more powerful by introspecting validators as well as the field type.
- For some (eg RegexValidator, EmailValidator), we'll choose a different base strategy
- For most others, we can choose better arguments to the strategy based on the validator - eg MinValueValidator, MaxLengthValidator, etc.
- For custom (and some builtin) validators, filtering will remain the only way.  We should keep it in all cases as a final check for correctness.
- We should resolve the strategy type and default kwargs for the given field type, then resolve kwargs from the validators (min_value, max_value, min_length, etc), possibly switch strategy type, and reconcile the arguments to give the actual strategy.  
- We should do some sanity-checks here too, eg for empty size ranges or multiple RegexValidators.  It's *technically possible* to compute the intersection of *n* regular expressions by analysis as finite automata, but that doesn't admit backreferences (ie PCRE) and is IMO out of scope for this.  Simpler lookahead-based combination of the patterns is equivalent to just filtering from a Hypothesis generation perspective, so that doesn't help.
- Documentation should be clearer on how to override the strategy for a particular field, but also suggest careful use of validators once they're supported.

This is a substantial new feature, but it would be really really nice to be able to generate instances of the default user model (!!), and the principle unlocks a massive improvement in specificity of inference and I assume real-world usefulness.  If anyone else is using it in the real world 😕 
I suggest we put together a minimal fix - just improving the docs and handling RegexValidator for CharField and TextField - and merge that, leaving a new issue (linked from said docs) with more detailed notes on the larger body of work for external or funded contributions.
> I suggest we put together a minimal fix - just improving the docs and handling RegexValidator for CharField and TextField - and merge that, leaving a new issue (linked from said docs) with more detailed notes on the larger body of work for external or funded contributions.

👍 with the possible caveat that it would be really nice to support the default User object and I bet we'll find some other problems when we try to do that. But I agree that just supporting the minimal amount to make that work is 100% the right starting point.

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -70,6 +70,7 @@
     'numpy': ('https://docs.scipy.org/doc/numpy/', None),
     'pandas': ('https://pandas.pydata.org/pandas-docs/stable/', None),
     'pytest': ('https://docs.pytest.org/en/stable/', None),
+    'django': ('https://django.readthedocs.io/en/stable/', None),
 }
 
 autodoc_mock_imports = ['numpy', 'pandas']
diff --git a/src/hypothesis/extra/django/models.py b/src/hypothesis/extra/django/models.py
--- a/src/hypothesis/extra/django/models.py
+++ b/src/hypothesis/extra/django/models.py
@@ -17,6 +17,7 @@
 
 from __future__ import division, print_function, absolute_import
 
+import re
 import string
 from decimal import Decimal
 from datetime import timedelta
@@ -24,6 +25,7 @@
 import django.db.models as dm
 from django.db import IntegrityError
 from django.conf import settings as django_settings
+from django.core import validators
 from django.core.exceptions import ValidationError
 
 import hypothesis.strategies as st
@@ -128,6 +130,24 @@ def _get_strategy_for_field(f):
             min_size=(None if f.blank else 1),
             max_size=f.max_length,
         )
+        # We can infer a vastly more precise strategy by considering the
+        # validators as well as the field type.  This is a minimal proof of
+        # concept, but we intend to leverage the idea much more heavily soon.
+        # See https://github.com/HypothesisWorks/hypothesis-python/issues/1116
+        re_validators = [
+            v for v in f.validators
+            if isinstance(v, validators.RegexValidator) and not v.inverse_match
+        ]
+        if re_validators:
+            regexes = [re.compile(v.regex, v.flags) if isinstance(v.regex, str)
+                       else v.regex for v in re_validators]
+            # This strategy generates according to one of the regexes, and
+            # filters using the others.  It can therefore learn to generate
+            # from the most restrictive and filter with permissive patterns.
+            # Not maximally efficient, but it makes pathological cases rarer.
+            # If you want a challenge: extend https://qntm.org/greenery to
+            # compute intersections of the full Python regex language.
+            strategy = st.one_of(*[st.from_regex(r) for r in regexes])
     elif type(f) == dm.DecimalField:
         bound = Decimal(10 ** f.max_digits - 1) / (10 ** f.decimal_places)
         strategy = st.decimals(min_value=-bound, max_value=bound,
@@ -141,12 +161,38 @@ def _get_strategy_for_field(f):
     return strategy
 
 
-def models(model, **extra):
-    """Return a strategy for instances of a model."""
-    result = {k: v for k, v in extra.items() if v is not default_value}
+def models(model, **field_strategies):
+    """Return a strategy for examples of ``model``.
+
+    .. warning::
+        Hypothesis creates saved models. This will run inside your testing
+        transaction when using the test runner, but if you use the dev console
+        this will leave debris in your database.
+
+    ``model`` must be an subclass of :class:`~django:django.db.models.Model`.
+    Strategies for fields may be passed as keyword arguments, for example
+    ``is_staff=st.just(False)``.
+
+    Hypothesis can often infer a strategy based the field type and validators
+    - for best results, make sure your validators are derived from Django's
+    and therefore have the known types and attributes.
+    Passing a keyword argument skips inference for that field; pass a strategy
+    or pass :const:`hypothesis.extra.django.models.default_value` to skip
+    inference for that field.
+
+    Foreign keys are not automatically derived. If they're nullable they will
+    default to always being null, otherwise you always have to specify them.
+    For example, examples of a Shop type with a foreign key to Company could
+    be generated with::
+
+      shop_strategy = models(Shop, company=models(Company))
+
+    """
+    result = {k: v for k, v in field_strategies.items()
+              if v is not default_value}
     missed = []
     for f in model._meta.concrete_fields:
-        if not (f.name in extra or isinstance(f, dm.AutoField)):
+        if not (f.name in field_strategies or isinstance(f, dm.AutoField)):
             result[f.name] = _get_strategy_for_field(f)
             if result[f.name].is_empty:
                 missed.append(f.name)


Test patch:
diff --git a/tests/django/toystore/test_given_models.py b/tests/django/toystore/test_given_models.py
--- a/tests/django/toystore/test_given_models.py
+++ b/tests/django/toystore/test_given_models.py
@@ -21,6 +21,7 @@
 from uuid import UUID
 
 from django.conf import settings as django_settings
+from django.contrib.auth.models import User
 
 from hypothesis import HealthCheck, given, assume, settings
 from hypothesis.errors import InvalidArgument
@@ -157,3 +158,10 @@ def test_constructs_valid_instance(self, instance):
                          instance.choice_field_grouped.lower())
         self.assertEqual(instance.even_number_field % 2, 0)
         self.assertTrue(instance.non_blank_text_field)
+
+
+class TestValidatorInference(TestCase):
+
+    @given(models(User))
+    def test_user_issue_1112_regression(self, user):
+        assert user.username