Repository: HypothesisWorks/hypothesis

Issue:
@settings somehow conflates a broken test method with one that doesn't use @given
(In some situations?) `@settings` thinks that a method hasn't been decorated with `given`, when instead the issue is simply a broken method signature. See below, where you'll notice I simply forgot to add `x` and `y` arguments to the test method itself:

```
⊙  python -m pip list | rg hypothesis                                                                                                                                julian@Air ●
hypothesis                         4.23.4

~/Desktop
⊙  cat bar.py                                                                                                                                                        julian@Air ●
from unittest import TestCase
from hypothesis import given, settings, strategies


class TestStuff(TestCase):
    @settings(max_examples=1)
    @given(x=strategies.integers(), y=strategies.integers())
    def test_add(self):
        pass

~/Desktop
⊙  trial bar.py                                                                                                                                                      julian@Air ●
bar
  TestStuff
    test_add ...                                                        [ERROR]

===============================================================================
[ERROR]
Traceback (most recent call last):
  File "/usr/local/Cellar/pypy/7.1.0/libexec/lib-python/2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/Users/julian/.local/share/virtualenvs/dev/site-packages/hypothesis/core.py", line 230, in wrapped_test
    raise InvalidArgument(message)
  File "/Users/julian/.local/share/virtualenvs/dev/site-packages/hypothesis/_settings.py", line 255, in new_test
    "Using `@settings` on a test without `@given` is completely pointless."
hypothesis.errors.InvalidArgument: Using `@settings` on a test without `@given` is completely pointless.

bar.TestStuff.test_add
-------------------------------------------------------------------------------
Ran 1 tests in 0.229s

FAILED (errors=1)
```

After adding them, all's fine:

```
⊙  trial bar.py                                                                                                                                                      julian@Air ●
bar
  TestStuff
    test_add ...                                                           [OK]

-------------------------------------------------------------------------------
Ran 1 tests in 0.208s
```

Not sure I see how this is possible given https://github.com/HypothesisWorks/hypothesis/blob/master/hypothesis-python/src/hypothesis/_settings.py#L264 but maybe I'm missing something.


Discussion:
I think I see what's happening. We return a special fake test method when you get the signature of `@given` wrong, and that's not got `is_hypothesis_test` set to `True` on it, so `@settings` treats it as a non-`@given` method.

(Side note: I don't like that error message)
> (Side note: I don't like that error message)

Agreed :)
Hi,
If we assign `True` to `is_hypothesis_test` in that invalid method hypothesis will throw an exception that is (somewhat) more helpful.

However, adding a flag would be more explicit + it won't affect code reading the `is_hypothesis_test` flag.

Below is a link to illustrate what I mean.

[diff](https://github.com/gabrielBusta/hypothesis/compare/master...gabrielBusta:issue-1978-better-error-msg)

Below you will find the exception thrown with that change... it is still confusing, it almost seems like the message should read `@given got an unexpected keyword argument 'x'`.

```
test_add (exp.TestStuff) ... ERROR

======================================================================
ERROR: test_add (exp.TestStuff)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/gabriel/hypo-dev/hypothesis-python/src/hypothesis/core.py", line 231, in wrapped_test
    raise InvalidArgument(message)
hypothesis.errors.InvalidArgument: test_add() got an unexpected keyword argument 'x'

----------------------------------------------------------------------
Ran 1 test in 0.002s

FAILED (errors=1)
```
> Below you will find the exception thrown with that change... it is still confusing[...]

Actually, the message:

 `hypothesis.errors.InvalidArgument: test_add() got an unexpected keyword argument 'x'`

is appropriate here. `@given` is responsible for calling your function/method, so the error message here should be no different as if you had manually called `test_add(x=2)`.

> [...] it almost seems like the message should read `@given got an unexpected keyword argument x`

Reporting that `@given got an unexpected keyword argument 'x'` would imply that there is something fundamentally wrong with invoking `@given(x=st.integers())` under *any* circumstances.


Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/core.py b/hypothesis-python/src/hypothesis/core.py
--- a/hypothesis-python/src/hypothesis/core.py
+++ b/hypothesis-python/src/hypothesis/core.py
@@ -230,6 +230,7 @@ def invalid(message):
         def wrapped_test(*arguments, **kwargs):
             raise InvalidArgument(message)
 
+        wrapped_test.is_hypothesis_test = True
         return wrapped_test
 
     if not (generator_arguments or generator_kwargs):
@@ -244,12 +245,11 @@ def wrapped_test(*arguments, **kwargs):
         )
 
     if len(generator_arguments) > len(original_argspec.args):
+        args = tuple(generator_arguments)
         return invalid(
-            (
-                "Too many positional arguments for %s() (got %d but"
-                " expected at most %d"
-            )
-            % (name, len(generator_arguments), len(original_argspec.args))
+            "Too many positional arguments for %s() were passed to @given "
+            "- expected at most %d arguments, but got %d %r"
+            % (name, len(original_argspec.args), len(args), args)
         )
 
     if infer in generator_arguments:
@@ -266,16 +266,15 @@ def wrapped_test(*arguments, **kwargs):
         if k not in original_argspec.args + original_argspec.kwonlyargs
     ]
     if extra_kwargs and not original_argspec.varkw:
+        arg = extra_kwargs[0]
         return invalid(
-            "%s() got an unexpected keyword argument %r" % (name, extra_kwargs[0])
+            "%s() got an unexpected keyword argument %r, from `%s=%r` in @given"
+            % (name, arg, arg, generator_kwargs[arg])
         )
     for a in original_argspec.args:
         if isinstance(a, list):  # pragma: no cover
             return invalid(
-                (
-                    "Cannot decorate function %s() because it has "
-                    "destructuring arguments"
-                )
+                "Cannot decorate function %s() because it has destructuring arguments"
                 % (name,)
             )
     if original_argspec.defaults or original_argspec.kwonlydefaults:


Test patch:
diff --git a/hypothesis-python/tests/nocover/test_testdecorators.py b/hypothesis-python/tests/nocover/test_testdecorators.py
--- a/hypothesis-python/tests/nocover/test_testdecorators.py
+++ b/hypothesis-python/tests/nocover/test_testdecorators.py
@@ -18,7 +18,7 @@
 from __future__ import absolute_import, division, print_function
 
 from hypothesis import HealthCheck, given, reject, settings
-from hypothesis.errors import Unsatisfiable
+from hypothesis.errors import InvalidArgument, Unsatisfiable
 from hypothesis.strategies import integers
 from tests.common.utils import raises
 
@@ -44,3 +44,20 @@ def this_has_a_unique_name_and_lives_on_a_class(self, x):
     with raises(Unsatisfiable) as e:
         Foo().this_has_a_unique_name_and_lives_on_a_class()
     assert (Foo.this_has_a_unique_name_and_lives_on_a_class.__name__) in e.value.args[0]
+
+
+def test_signature_mismatch_error_message():
+    # Regression test for issue #1978
+
+    @settings(max_examples=2)
+    @given(x=integers())
+    def bad_test():
+        pass
+
+    try:
+        bad_test()
+    except InvalidArgument as e:
+        assert (
+            str(e) == "bad_test() got an unexpected keyword argument 'x', "
+            "from `x=integers()` in @given"
+        )