Repository: HypothesisWorks/hypothesis

Issue:
given's interaction with mock.patch decorator is broken in almost every way
@waynew is trying to use unitest.mock's patch decorator together with Hypothesis's given and py.test's fixtures. He's not having a very good time of it, which is unsurprising because this combination almost entirely doesn't work.

The first way it goes wrong:

You can mostly use mock.patch with given if you don't care about py.test fixtures. For example, the following will work:

``` python
import thing
import pytest
import hypothesis.strategies as st
from hypothesis import given
from unittest import mock

@mock.patch('thing.fnord')
@given(thing=st.text())
def test_a_thing(fnord, thing):
    pass
```

But if you try to add an extra named argument then everything will go wrong: Hypothesis will expose something with just _args and *_kwargs because that's what mock.patch gives it, so py.test won't be able to find the fixture.

It's unclear that there is a correct thing Hypothesis could do at this point. mock.patch _is_ using functools.wrap, so in Python 3 we could propagate the **wrapped** attribute, but AFAIK there's no equivalent in Python 2. Additionally, note that the signature of the wrapped function is not correct, because the first arguments are filled in manually. Historically, propagating wrapped has broken py.test (https://github.com/pytest-dev/pytest/issues/677), but this may be affected differently now?

Anyway: Can of worms, unclear there's a correct thing to do here.

The second way it goes wrong is more amusing and is _probably_ fixable (update: was fixed in #779), or at least can be mitigated:

``` python
import thing
import pytest
import hypothesis.strategies as st
from hypothesis import given
from unittest import mock

@given(thing=st.text())
@mock.patch('thing.fnord')
def test_a_thing(fnord, thing):
    assert False
```

This test passes.

Why does this test pass? It passes because Hypothesis can't currently tell the difference between being called as a bound method and just having an argument explicitly passed to it as the first positional argument. This means that it thinks that the mock is self and begins the shenanigans for [the executor API](http://hypothesis.readthedocs.org/en/latest/details.html#custom-function-execution). It looks for a method called execute_example, finds it, and then calls it with the test function.

It then of course immediately returns, never executing the test.

Possible ways to work around:
- Have given return a descriptor, so that we can actually detect if it's been bound as a method.
- Explicitly check for being passed a mock instance as self (I don't want to do this one, but it would work)
- Tell people not to do this, or to use autospec when using patch.

Seed not always printed on failure
I'm a bit confused on when the seed is printed and when it is not. I've seen it printed at some moments, but at other times it is unexpectedly missing.

Looking at the code, it seems that the printing happens here:

https://github.com/HypothesisWorks/hypothesis/blob/c877d1d7690c73a793bbaa02c1d9dd62e99aa9d0/hypothesis-python/src/hypothesis/core.py#L1049-L1059

In a case where the seed is not printed, debug printing shows that `state.failed_normally` is True, which suppressed the seed printing.

Digging more, I found that commit 6aadcc02694dfea2aca008e751f367956c5cf64a introduces this `failed_normally` check. Apparently the ide is that for "normal" failures, a reproduction blob is printed for each individual failure, which (I guess) removes the need for printing the seed. However, blobs are not printed in all cases, which (at first sight) leaves you with nothing.

In my case, the blob isn't printed because `can_use_repr` is true, which I guess means that I could just paste the failing test description and put it into an @example decorator (this does not seem [documented](https://hypothesis.readthedocs.io/en/latest/reproducing.html)). Also, I guess the failure database should make sure that a failure is retried on the next run, making it less essential to keep the seed.

Still, I was a bit confused by not (always) seeing the seed after failures. Would it not be more consistent to just always print it (unless it was explicitly set), reverting part of the above linked commit? The only downside of this that I can imagine is that people will assume the seed is the only way to reproduce, and not realize there is a database and you can also use @example...


Discussion:
Note: Let me know if you really really care about this bug, because the combination of "good workarounds", "all the fixes are awful and/or partial" and "I don't actually like mocking" means that unless this is actually bothering people I am very unmotivated to work on it.

Thanks for the details on this. Fixing this is not really important to me, I don't think the workarounds are burdensome at all. I've been using pytest and hypothesis for years and this has never been an issue before. I was just perplexed at the behavior but I understand the reasons it happens better now.
@DRMacIver I ran into this same issue (i.e. the second of the two reported problems) when injecting mocks using `pytest.fixture` instead of `mock.patch`.  Specifically, when mocks appear first in the arg list, they will be misidentified as `self`. If you have other non-mock args that appear in the arg list first, you'll not see the problem.  I independently verified the same behavior under the hood as the OP.  Though there are decent workarounds, I'm inclined to at least take a stab at a PR to fix this.  Having your tests return false positives, possibly MANY false positives that are hard to see is a serious and insidious issue IMO.

By hacking some code in `executors.py`, I was able to avoid the bug.  It's ghetto as all hell, literally checking the type string looking for mocks.  If you're OK with doing this sort of stuff, I'd be happy to roll up my sleeves and try making a few tests and fixing this.  I really like hypothesis, and I really like mocks and fixtures so I'm suitably motivated =)  If you have any suggestions on other code that might be relevant let me know.  

Thanks!
> If you're OK with doing this sort of stuff, I'd be happy to roll up my sleeves and try making a few tests and fixing this. 

Hypothesis is full of hacky workarounds for weird behaviour, so no objections on that front. :-) Very happy to accept a pull request for this.

The other thing I thought about but never got around to investigating as a way of detecting mocks is to check for the presence of an arbitrary attribute. e.g. if `hasattr(selfy, 'hypothesis_internal_this_attribute_should_not_exist')` is True then it's probably a mock, or something sufficiently mocklike that it will cause the same problems. Might be worth trying?

A thing worth noting BTW is that pytest fixtures *also* interact unexpectedly with Hypothesis (function level fixtures are once for the whole test, not once per example), so you may run into problems on that front too. Fixing this one is a rather larger task though (or requires work on pytest's side).
Alright @DRMacIver , so I made a pull request to address a portion of this issue: https://github.com/HypothesisWorks/hypothesis-python/pull/779.  As you mentioned, this is not really going to do much in relation to the other challenges re: fixtures, but it's an improvement nonetheless.  I wonder - should this bug (198) be broken down into smaller issues?  Seems kind of overwhelming in its current form due to its scope.
> In my case, the blob isn't printed because can_use_repr is true, which I guess means that I could just paste the failing test description and put it into an `@example` decorator (this does not seem documented). Also, I guess the failure database should make sure that a failure is retried on the next run, making it less essential to keep the seed.

We should document this - possibly adding a TLDR listing the ways to reproduce a failure and the order you should try them in - but I'm not personally inclined to start printing more details than are actually needed.

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/internal/compat.py b/hypothesis-python/src/hypothesis/internal/compat.py
--- a/hypothesis-python/src/hypothesis/internal/compat.py
+++ b/hypothesis-python/src/hypothesis/internal/compat.py
@@ -374,7 +374,7 @@ def update_code_location(code, newfile, newlineno):
 
     Why do we want to do this? It's for really shallow reasons involving
     hiding the hypothesis_temporary_module code from test runners like
-    py.test's verbose mode. This is a vastly disproportionate terrible
+    pytest's verbose mode. This is a vastly disproportionate terrible
     hack that I've done purely for vanity, and if you're reading this
     code you're probably here because it's broken something and now
     you're angry at me. Sorry.
diff --git a/hypothesis-python/src/hypothesis/strategies.py b/hypothesis-python/src/hypothesis/strategies.py
--- a/hypothesis-python/src/hypothesis/strategies.py
+++ b/hypothesis-python/src/hypothesis/strategies.py
@@ -617,12 +617,13 @@ def lists(
     Hypothesis 1.x mean we no longer needed this hint to generate useful data.
 
     If unique is True (or something that evaluates to True), we compare direct
-    object equality, as if unique_by was `lambda x: x`. This comparison only
+    object equality, as if unique_by was ``lambda x: x``. This comparison only
     works for hashable types.
 
     if unique_by is not None it must be a function returning a hashable type
     when given a value drawn from elements. The resulting list will satisfy the
-    condition that for i != j, unique_by(result[i]) != unique_by(result[j]).
+    condition that for ``i`` != ``j``, ``unique_by(result[i])`` !=
+    ``unique_by(result[j])``.
 
     Examples from this strategy shrink by trying to remove elements from the
     list, and by shrinking each individual element of the list.
@@ -1830,8 +1831,8 @@ def complex_numbers(min_magnitude=0, max_magnitude=None,
 
     If you need to generate complex numbers with particular real and
     imaginary parts or relationships between parts, consider using
-    `builds(complex, ...) <hypothesis.strategies.builds>` or
-    `@composite <hypothesis.strategies.composite>` respectively.
+    :func:`builds(complex, ...) <hypothesis.strategies.builds>` or
+    :func:`@composite <hypothesis.strategies.composite>` respectively.
     """
     check_valid_magnitude(min_magnitude, 'min_magnitude')
     check_valid_magnitude(max_magnitude, 'max_magnitude')
@@ -2163,12 +2164,11 @@ def deferred(definition):
 @defines_strategy_with_reusable_values
 def emails():
     """A strategy for generating email addresses as unicode strings. The
-    address format is specific in :rfc:`5322#section-3.4.1`. Values shrink
+    address format is specified in :rfc:`5322#section-3.4.1`. Values shrink
     towards shorter local-parts and host domains.
 
     This strategy is useful for generating "user data" for tests, as
-    mishandling of email addresses is a common source of bugs. Future
-    updates will generate more complicated addresses allowed by the RFC.
+    mishandling of email addresses is a common source of bugs.
     """
     from hypothesis.provisional import domains
     local_chars = string.ascii_letters + string.digits + "!#$%&'*+-/=^_`{|}~"


Test patch:
diff --git a/hypothesis-python/tests/README.rst b/hypothesis-python/tests/README.rst
--- a/hypothesis-python/tests/README.rst
+++ b/hypothesis-python/tests/README.rst
@@ -112,7 +112,7 @@ Still here?  Here's a note on what to expect in each directory.
     for inference from type hints.
 
 ``pytest/``
-    Hypothesis has excellent integration with ``py.test``, though we are careful
+    Hypothesis has excellent integration with ``pytest``, though we are careful
     to support other test runners including unittest and nose.  This is where we
     test that our pytest integration is working properly.
 
diff --git a/hypothesis-python/tests/pandas/test_data_frame.py b/hypothesis-python/tests/pandas/test_data_frame.py
--- a/hypothesis-python/tests/pandas/test_data_frame.py
+++ b/hypothesis-python/tests/pandas/test_data_frame.py
@@ -241,3 +241,11 @@ def test_uniqueness_does_not_affect_other_rows_2():
 def test_will_fill_missing_columns_in_tuple_row(df):
     for d in df['A']:
         assert d == 7
+
+
+@given(pdst.data_frames(
+    index=pdst.range_indexes(10, 10),
+    columns=[pdst.column(elements=st.integers(0, 9), fill=None, unique=True)],
+))
+def test_cen_generate_unique_columns(df):
+    assert set(df[0]) == set(range(10))
diff --git a/hypothesis-python/tests/py3/test_mock.py b/hypothesis-python/tests/py3/test_mock.py
new file mode 100644
--- /dev/null
+++ b/hypothesis-python/tests/py3/test_mock.py
@@ -0,0 +1,53 @@
+# coding=utf-8
+#
+# This file is part of Hypothesis, which may be found at
+# https://github.com/HypothesisWorks/hypothesis-python
+#
+# Most of this work is copyright (C) 2013-2018 David R. MacIver
+# (david@drmaciver.com), but it contains contributions by others. See
+# CONTRIBUTING.rst for a full list of people who may hold copyright, and
+# consult the git log if you need to determine who owns an individual
+# contribution.
+#
+# This Source Code Form is subject to the terms of the Mozilla Public License,
+# v. 2.0. If a copy of the MPL was not distributed with this file, You can
+# obtain one at http://mozilla.org/MPL/2.0/.
+#
+# END HEADER
+
+"""Checks that @given, @mock.patch, and pytest fixtures work as expected."""
+
+from __future__ import division, print_function, absolute_import
+
+import math
+from unittest import mock
+
+from _pytest.capture import CaptureFixture
+
+import hypothesis.strategies as st
+from hypothesis import given
+
+
+@given(thing=st.text())
+@mock.patch('math.atan')
+def test_can_mock_inside_given_without_fixture(atan, thing):
+    assert isinstance(atan, mock.MagicMock)
+    assert isinstance(math.atan, mock.MagicMock)
+
+
+@mock.patch('math.atan')
+@given(thing=st.text())
+def test_can_mock_outside_given_with_fixture(atan, capsys, thing):
+    assert isinstance(atan, mock.MagicMock)
+    assert isinstance(math.atan, mock.MagicMock)
+    assert isinstance(capsys, CaptureFixture)
+
+
+@given(thing=st.text())
+def test_can_mock_within_test_with_fixture(capsys, thing):
+    assert isinstance(capsys, CaptureFixture)
+    assert not isinstance(math.atan, mock.MagicMock)
+    with mock.patch('math.atan') as atan:
+        assert isinstance(atan, mock.MagicMock)
+        assert isinstance(math.atan, mock.MagicMock)
+    assert not isinstance(math.atan, mock.MagicMock)
diff --git a/hypothesis-python/tests/pytest/test_capture.py b/hypothesis-python/tests/pytest/test_capture.py
--- a/hypothesis-python/tests/pytest/test_capture.py
+++ b/hypothesis-python/tests/pytest/test_capture.py
@@ -69,7 +69,7 @@ def test_should_emit_unicode(t):
 @pytest.mark.xfail(
     WINDOWS,
     reason=(
-        "Encoding issues in running the subprocess, possibly py.test's fault"))
+        "Encoding issues in running the subprocess, possibly pytest's fault"))
 @pytest.mark.skipif(
     PY2, reason="Output streams don't have encodings in python 2")
 def test_output_emitting_unicode(testdir, monkeypatch):