Repository: HypothesisWorks/hypothesis

Issue:
Use distinct DB keys for parametrized Hypothesis tests
#1995 fixes an issue with database collision, but there's a deeper issue in that these tests should have been using distinct keys in the first place.

So I propose that we should firm up the design of  `_hypothesis_internal_add_digest` (currently only used for `run_state_machine_as_test`, so we need to decide on e.g. how multiple applications work), then use it for `@pytest.mark.parametrize` too on at least a best-effort basis.
Use distinct DB keys for parametrized Hypothesis tests
#1995 fixes an issue with database collision, but there's a deeper issue in that these tests should have been using distinct keys in the first place.

So I propose that we should firm up the design of  `_hypothesis_internal_add_digest` (currently only used for `run_state_machine_as_test`, so we need to decide on e.g. how multiple applications work), then use it for `@pytest.mark.parametrize` too on at least a best-effort basis.


Discussion:
>  on at least a best-effort basis.

Best-effort might be quite bad. `pytest.mark.parametrize` can offer very general arguments that we might struggle to hash in a meaningful way.

I wonder if there's an easy way to get an incrementing sequence that corresponds to the parameter order? We can't just use the number of times the test has been run this process (which was my initial thought) for a wide variety of reasons (xdist, filtering of tests run, etc).

Another possibility is that rather than using distinct keys we could change the eviction behaviour to be in some way more relaxed when parametrizing.
Is there any reasonable way to get a suitable test name/identifier directly from pytest?

(If our DB key is based on the test name that xdist sees, then we should sidestep any xdist conflicts.)
I was thinking of adding the pytest item id to the internal hash, which should deal with both issues. That's a unique string that identifies the test, so xdist and hashing both work. 

Design note: we should *only* add the item ID to the hash for `pytest.mark.parametrize`d tests, so that calling nonparametrized tests directly uses the same database entry as invoking them via pytest.
Ah, yes, that's a good solution.
I would be happy to take a look at this in the spirit of Hacktoberfest. I still need to dig through the code a bit to get a feel for what change should go where. I'll get back here once I have an idea for an approach, I hope that will be somewhere later this week.
Beyond that I can't say anything on timescales, since I also have thesis to finish :sob: 
I just did a `git grep _hypothesis_internal_add_digest` to see the current uses - there's only one in the internals of stateful testing, and since there's no way for that to be pytest-parametrized you can just reuse the same logic 😄 

More specifically, go to the following branch:

https://github.com/HypothesisWorks/hypothesis/blob/f8dff6b13c2c6c85d8637497d44bd61f864e49d6/hypothesis-python/src/hypothesis/extra/pytestplugin.py#L181-L182

and add a check:

```python
if is_hypothesis_test(item.obj):
    item.add_marker("hypothesis")
    if we_can_tell_this_is_parametrized(obj):  # This is the tricky part!
        item.obj.hypothesis.inner_test._hypothesis_internal_add_digest = \
            item.nodeid.encode("utf-8")
```

And then write a test under `hypothesis-python/tests/pytest/` that checks we end up with more than one entry from a `@parametrized` test with an `InMemoryDatabase`.
Sounds good. I'll try to find some time soon :)
>  on at least a best-effort basis.

Best-effort might be quite bad. `pytest.mark.parametrize` can offer very general arguments that we might struggle to hash in a meaningful way.

I wonder if there's an easy way to get an incrementing sequence that corresponds to the parameter order? We can't just use the number of times the test has been run this process (which was my initial thought) for a wide variety of reasons (xdist, filtering of tests run, etc).

Another possibility is that rather than using distinct keys we could change the eviction behaviour to be in some way more relaxed when parametrizing.
Is there any reasonable way to get a suitable test name/identifier directly from pytest?

(If our DB key is based on the test name that xdist sees, then we should sidestep any xdist conflicts.)
I was thinking of adding the pytest item id to the internal hash, which should deal with both issues. That's a unique string that identifies the test, so xdist and hashing both work. 

Design note: we should *only* add the item ID to the hash for `pytest.mark.parametrize`d tests, so that calling nonparametrized tests directly uses the same database entry as invoking them via pytest.
Ah, yes, that's a good solution.
I would be happy to take a look at this in the spirit of Hacktoberfest. I still need to dig through the code a bit to get a feel for what change should go where. I'll get back here once I have an idea for an approach, I hope that will be somewhere later this week.
Beyond that I can't say anything on timescales, since I also have thesis to finish :sob: 
I just did a `git grep _hypothesis_internal_add_digest` to see the current uses - there's only one in the internals of stateful testing, and since there's no way for that to be pytest-parametrized you can just reuse the same logic 😄 

More specifically, go to the following branch:

https://github.com/HypothesisWorks/hypothesis/blob/f8dff6b13c2c6c85d8637497d44bd61f864e49d6/hypothesis-python/src/hypothesis/extra/pytestplugin.py#L181-L182

and add a check:

```python
if is_hypothesis_test(item.obj):
    item.add_marker("hypothesis")
    if we_can_tell_this_is_parametrized(obj):  # This is the tricky part!
        item.obj.hypothesis.inner_test._hypothesis_internal_add_digest = \
            item.nodeid.encode("utf-8")
```

And then write a test under `hypothesis-python/tests/pytest/` that checks we end up with more than one entry from a `@parametrized` test with an `InMemoryDatabase`.
Sounds good. I'll try to find some time soon :)

Resolving patch:
diff --git a/hypothesis-python/setup.py b/hypothesis-python/setup.py
--- a/hypothesis-python/setup.py
+++ b/hypothesis-python/setup.py
@@ -58,7 +58,7 @@ def local_file(name):
     "lark": ["lark-parser>=0.6.5"],
     "numpy": ["numpy>=1.9.0"],
     "pandas": ["pandas>=0.19"],
-    "pytest": ["pytest>=3.0"],
+    "pytest": ["pytest>=4.3"],
     "dpcontracts": ["dpcontracts>=0.4"],
     # We only support Django versions with upstream support - see
     # https://www.djangoproject.com/download/#supported-versions
diff --git a/hypothesis-python/src/hypothesis/extra/pytestplugin.py b/hypothesis-python/src/hypothesis/extra/pytestplugin.py
--- a/hypothesis-python/src/hypothesis/extra/pytestplugin.py
+++ b/hypothesis-python/src/hypothesis/extra/pytestplugin.py
@@ -115,6 +115,12 @@ def pytest_runtest_call(item):
     if not (hasattr(item, "obj") and is_hypothesis_test(item.obj)):
         yield
     else:
+        if item.get_closest_marker("parametrize") is not None:
+            # Give every parametrized test invocation a unique database key
+            item.obj.hypothesis.inner_test._hypothesis_internal_add_digest = item.nodeid.encode(
+                "utf-8"
+            )
+
         store = StoringReporter(item.config)
 
         def note_statistics(stats):
diff --git a/tooling/src/hypothesistooling/__main__.py b/tooling/src/hypothesistooling/__main__.py
--- a/tooling/src/hypothesistooling/__main__.py
+++ b/tooling/src/hypothesistooling/__main__.py
@@ -460,7 +460,7 @@ def standard_tox_task(name):
 
 
 standard_tox_task("nose")
-standard_tox_task("pytest30")
+standard_tox_task("pytest43")
 
 for n in [20, 21, 22, 111]:
     standard_tox_task("django%d" % (n,))


Test patch:
diff --git a/hypothesis-python/tests/pytest/test_parametrized_db_keys.py b/hypothesis-python/tests/pytest/test_parametrized_db_keys.py
new file mode 100644
--- /dev/null
+++ b/hypothesis-python/tests/pytest/test_parametrized_db_keys.py
@@ -0,0 +1,44 @@
+# coding=utf-8
+#
+# This file is part of Hypothesis, which may be found at
+# https://github.com/HypothesisWorks/hypothesis/
+#
+# Most of this work is copyright (C) 2013-2019 David R. MacIver
+# (david@drmaciver.com), but it contains contributions by others. See
+# CONTRIBUTING.rst for a full list of people who may hold copyright, and
+# consult the git log if you need to determine who owns an individual
+# contribution.
+#
+# This Source Code Form is subject to the terms of the Mozilla Public License,
+# v. 2.0. If a copy of the MPL was not distributed with this file, You can
+# obtain one at https://mozilla.org/MPL/2.0/.
+#
+# END HEADER
+
+from __future__ import absolute_import, division, print_function
+
+DB_KEY_TESTCASE = """
+from hypothesis import settings, given
+from hypothesis.database import InMemoryExampleDatabase
+from hypothesis.strategies import booleans
+import pytest
+
+DB = InMemoryExampleDatabase()
+
+
+@settings(database=DB)
+@given(booleans())
+@pytest.mark.parametrize("hi", (1, 2, 3))
+@pytest.mark.xfail()
+def test_dummy_for_parametrized_db_keys(hi, i):
+    assert Fail  # Test *must* fail for it to end up the database anyway
+
+
+def test_DB_keys_for_parametrized_test():
+    assert len(DB.data) == 3
+"""
+
+
+def test_db_keys_for_parametrized_tests_are_unique(testdir):
+    script = testdir.makepyfile(DB_KEY_TESTCASE)
+    testdir.runpytest(script).assert_outcomes(xfailed=3, passed=1)