Repository: HypothesisWorks/hypothesis

Issue:
Missing argument validation for `rule()`
I was playing around with [stateful testing](https://hypothesis.readthedocs.io/en/latest/stateful.html) earlier today, and messed up a function call.  Here's the problem: this caused an internal error instead of a validation error with a nice message!

`hypothesis.stateful.rule()`, and `initialize()`, should use `check_type` to validate the types of all their arguments.  There are examples of this usage elsewhere in the module, though you'll need to loop over `kwargs.items()` to get the argument names.

Then add tests to `hypothesis-python/tests/cover/test_argument_validation.py`, a minor release by creating `hypothesis-python/RELEASE.rst` (check the other open PRs for examples), and you'll be good to go!  

You're welcome to check for other cases of missing validation, and we'd love fixes for them, but we would encourage that to happen in follow-up PRs.  Better to merge small changes regularly than have them build up and get stuck!


Discussion:
If no one minds, I think I'll take a look at this. :) 
@Zac-HD,
Hi, I'm new to hypothesis and type hinting in general, and I have a few questions:
> `hypothesis.stateful.rule()`, and `initialize()`, should use `check_type` to validate the types of all their arguments.

How do we know what types to compare against? Do we need to somehow extract the usage of kwargs from f?
>There are examples of this usage elsewhere in the module, though you'll need to loop over `kwargs.items()` to get the argument names.

In both cases (Settings, RuleBasedStateMachine), the check_type is against a locally created object. This is not the case with kwargs.
> Then add tests to `hypothesis-python/tests/cover/test_argument_validation.py`, a minor release by creating `hypothesis-python/RELEASE.rst` (check the other open PRs for examples), and you'll be good to go!

I couldn't find this in other PRs, do you have an example?

Thanks in advance

Hey @NNRepos - I hope you're enjoying Hypothesis :smile:  This isn't really type hinting, but explicit runtime checks.  We know the types because we only handle certain types - `target` must be a Bundle , `targets` must be a tuple of Bundles, and the value of every keyword argument must be a strategy.  Passing other types is already an *internal* error, but we want it to be an explict `InvalidArgument` error - does that make sense?

[Here's a good example of a pull request](https://github.com/HypothesisWorks/hypothesis/pull/2162/files) including contributors entry, release file, change, and tests!
Hey @fragmad, any progress?  Happy to give you some tips or help out if you're stuck :slightly_smiling_face: 
Hey @Zac-HD, 

I'm also new to hypothesis and would like to help contribute if this issue is still open. I was hoping you could clarify something for me.

It looks like both `hypothesis.stateful.rule()` and `initialize()` have the target, and targets argument validation done inside of `_convert_targets()`. Are you looking to separate some of that logic and perform validation separately from converting while also including validation of **kwargs?

Thanks!

Hey @benjpalmer - welcome!  We'll reserve this issue for you for a week; after that anyone else can open a PR 😄 

> It looks like both `rule()` and `initialize()` have the `target`, and `targets` argument validation done inside of `_convert_targets()`. Are you looking to separate some of that logic and perform validation separately from converting while also including validation of **kwargs?

The validation for `target` and `targets` is just fine - all we need to add is a loop that validates that each value in `kwargs` is a strategy, and if not also says which.  Something like

```python
for k, v in kwargs.items():
    check_type(SearchStrategy, v, k)
```

plus tests, a changelog, and your name in the authors list ✨ 

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/stateful.py b/hypothesis-python/src/hypothesis/stateful.py
--- a/hypothesis-python/src/hypothesis/stateful.py
+++ b/hypothesis-python/src/hypothesis/stateful.py
@@ -432,6 +432,8 @@ def rule(targets=(), target=None, **kwargs):
     provided.
     """
     converted_targets = _convert_targets(targets, target)
+    for k, v in kwargs.items():
+        check_type(SearchStrategy, v, k)
 
     def accept(f):
         existing_rule = getattr(f, RULE_MARKER, None)
@@ -467,6 +469,8 @@ def initialize(targets=(), target=None, **kwargs):
     arbitrary order.
     """
     converted_targets = _convert_targets(targets, target)
+    for k, v in kwargs.items():
+        check_type(SearchStrategy, v, k)
 
     def accept(f):
         existing_rule = getattr(f, RULE_MARKER, None)


Test patch:
diff --git a/hypothesis-python/tests/cover/test_stateful.py b/hypothesis-python/tests/cover/test_stateful.py
--- a/hypothesis-python/tests/cover/test_stateful.py
+++ b/hypothesis-python/tests/cover/test_stateful.py
@@ -718,6 +718,26 @@ def do_stuff(self):
     run_state_machine_as_test(Invariant)
 
 
+def test_invalid_rule_argument():
+    """Rule kwargs that are not a Strategy are expected to raise an InvalidArgument error."""
+    with pytest.raises(InvalidArgument):
+
+        class InvalidRuleMachine(RuleBasedStateMachine):
+            @rule(strategy=object())
+            def do_stuff(self):
+                pass
+
+
+def test_invalid_initialize_argument():
+    """Initialize kwargs that are not a Strategy are expected to raise an InvalidArgument error."""
+    with pytest.raises(InvalidArgument):
+
+        class InvalidInitialize(RuleBasedStateMachine):
+            @initialize(strategy=object())
+            def initialize(self):
+                pass
+
+
 def test_multiple_invariants():
     """If multiple invariants are present, they all get run."""