Repository: HypothesisWorks/hypothesis
Pull request: Minimum viable Django validator introspection
*Just* enough work towards #1116 that it closes #1112.

CC @DRMacIver, @MrGreenTea

Problem statement:
FailedHealthCheck with hypothesis.extra.django for django.contrib.auth.models.User model
I just use create the most simple strategy like this:
`USER_STRATEGY = models.models(User)`

Then I get a FailedHealthCheck:
`hypothesis.errors.FailedHealthCheck: It looks like your strategy is filtering out a lot of data. Health check found 50 filtered examples but only 3 good ones. This will make your tests much slower ...`

Any idea what I can do to get User objects for my tests?

Hints from issue discussion:
I would guess you have some complex validation on one or more fields, and the default strategy is giving you data of the right type (eg `DateField`) that fails validation (eg 'only the first of the month').

You can override the strategy for each field of a model, with the example of customer age [here](https://hypothesis.readthedocs.io/en/latest/django.html) - but I'll admit the docs don't make it obvious!  (We'll therefore keep this issue open to improve them.)  For example:

    USER_STRATEGY = models.models(User, age=st.integers(0, 100))

Does that help?
Yeah I suspected that at first too, because we have a lot of ForeignKeys to User. But even in a fresh project the same happens.
My whole test code is this:

```python
from django.contrib.auth.models import User
from django.test import TestCase
from hypothesis import given
from hypothesis.extra.django import models as model_st

USER_STRATEGY = model_st.models(User)


class TestUser(TestCase):
    @given(USER_STRATEGY)
    def test_user(self, user):
        assert user.username
```

My django version is Django 2.0.2 with Hypothesis 3.45.0
You're using `django.test.TestCase` but you need to be using `hypothesis.extra.django.TestCase`. What's happening is that generated users are colliding with previous generated users because the database state isn't being reset at the start of each generation (though I'm a little surprised that this is how that manifests).

We should probably be better about signalling this problem.
Ooohh, yep, that would do it.  Related to #1071 and the (hopefully someday soon) solution in #1072 - in both cases we need to check the type of the class that our test method is defined on.
I'm going to close this as it's not actually a bug in Hypothesis, but I've open #1113 for communicating this problem more clearly.
That doesn't change it for me. 

```python
# Create your tests here.
import hypothesis.extra.django
from django.contrib.auth.models import User
from hypothesis import given
from hypothesis.extra.django import models as model_st

USER_STRATEGY = model_st.models(User)


class TestUser(hypothesis.extra.django.TestCase):
    @given(USER_STRATEGY)
    def test_user(self, user):
        assert user.username
```

still gives me 

`hypothesis.errors.FailedHealthCheck: It looks like your strategy is filtering out a lot of data. Health check found 50 filtered examples but only 5 good ones`

Interesting. You're absolutely right. I could have sworn we had tests that this worked with the standard auth User, but apparently we don't! Sorry for jumping to conclusions.
Yeah, regressions happen. No worries ;)
Hope you manage to find a solution and thanks for all the help!
Based on some investigation, I'm pretty sure the problem is that the user model has a very restrictive idea of what characters a username can contain, and we're trying to generate arbitrary unicode text for it.

We should figure out a proper fix, but the following will let you work around it until we do:

```python

import hypothesis.strategies as st
import string

USER_STRATEGY = model_st.models(
    User,
    username=st.text(
        alphabet=string.ascii_letters + string.digits + '@/./+/-/_'))
```
nitpick: @DRMacIver, I don't think you meant to include the slashes in the character list there :wink:

Commit message: Add regression test for default User model

Currently fails because we use a generic text strategy instead of
generating from the regex.

Patch to review:
diff --git a/tests/django/toystore/test_given_models.py b/tests/django/toystore/test_given_models.py
index 2f1657dc47..3af9387f63 100644
--- a/tests/django/toystore/test_given_models.py
+++ b/tests/django/toystore/test_given_models.py
@@ -21,6 +21,7 @@
 from uuid import UUID
 
 from django.conf import settings as django_settings
+from django.contrib.auth.models import User
 
 from hypothesis import HealthCheck, given, assume, settings
 from hypothesis.errors import InvalidArgument
@@ -157,3 +158,10 @@ def test_constructs_valid_instance(self, instance):
                          instance.choice_field_grouped.lower())
         self.assertEqual(instance.even_number_field % 2, 0)
         self.assertTrue(instance.non_blank_text_field)
+
+
+class TestValidatorInference(TestCase):
+
+    @given(models(User))
+    def test_user_issue_1112_regression(self, user):
+        assert user.username


Code review comments:
diff_hunk: @@ -141,12 +154,38 @@ def _get_strategy_for_field(f):
     return strategy
 
 
-def models(model, **extra):
-    """Return a strategy for instances of a model."""
-    result = {k: v for k, v in extra.items() if v is not default_value}
+def models(model, **field_strategies):
+    """Return a strategy for examples of ``model``.
+
+    .. warning::
+        Hypothesis creates saved models. This will run inside your testing
+        transaction when using the test runner, but if you use the dev console
+        this will leave debris in your database.
+
+    ``model`` must be an instance of :class:`~django:django.db.models.Model`.
path: src/hypothesis/extra/django/models.py
text: @user1:
must be a *subclass* of Model, not an instance.

diff_hunk: @@ -128,6 +130,17 @@ def _get_strategy_for_field(f):
             min_size=(None if f.blank else 1),
             max_size=f.max_length,
         )
+        # We can infer a vastly more precise strategy by considering the
+        # validators as well as the field type.  This is a minimal proof of
+        # concept, but we intend to leverage the idea much more heavily soon.
+        # See https://github.com/HypothesisWorks/hypothesis-python/issues/1116
+        for v in f.validators:
+            if isinstance(v, validators.RegexValidator) and \
+                    not v.inverse_match:
+                if isinstance(v.regex, str):
path: src/hypothesis/extra/django/models.py
text: @user1:
Currently if there are multiple `RegexValidator`s this picks the last one in the list. I don't have a problem with that, but it would be nice to add a comment indicating that this is an edge case that may not work that well.

If you'd rather deal with it than comment, one option in this case would be to use `one_of(from_regex(a), from_regex(b), ...)` on those. It'll still be an exact match because we filter by the validators later, but will deal with the case where the results of one regex are much more likely to match than the results of the other. Up to you if you care enough to want to deal with that.

@author:
I'll use `one_of` tactic, it sounds good enough to me.

The "let's just be silly" alternative is to compute the intersection of multiple regex by interpreting them as finite automata.  This has actually been implemented in Python - https://qntm.org/greenery - but of course it only works for truly regular expressions (eg. no forward- or back-references) and covering the remaining edge cases would be far more work than I'm interested in.

@user1:
I've actually got a largish amount of prototype-quality code that does just this sort of thing (e.g https://travis-ci.org/DRMacIver/FALBS).

I would say that I am like 80% confident that with judicious use of laziness and regular expression derivatives I could make an automaton based solution that incorporates both `from_regex` and intersection given, say, two solid weeks of work.

There are number of amusing obstacles to overcome (like the fact that Python regex are actually context-free languages, so the resulting decision problem of deciding whether the intersection of two "regex" has a match is undecidable. But that's OK, we can work around that pretty easily because we only have to be better than generate + filter we don't have to be perfect), but I've got a pretty good idea of how they can all be overcome.

The biggest obstacle however is that I don't actually care enough to want to spend two solid weeks of work on this. :-p

It might be worth revisiting if and when I've added a bunch more internal machinery for this though, which I'd say currently has some chance of happening given my current research direction.