Repository: HypothesisWorks/hypothesis

Issue:
Add static typing to Hypothesis API
The new typing module in 3.5 and [PEP 0484](https://www.python.org/dev/peps/pep-0484/) add static type annotations to Python. It would be nice for the Hypothesis API to support this, as the public API is actually pretty static in nature. stub files offer a compatibility mode, so this can be done transparently without dropping support for Python 2 (which is sadly there for the long haul).

Steps this would require:
1. Making Strategy a generic type with a single parameter, Strategy[T] that specifies the type of value generated. Note that Strategy is covariant in its type parameter.
2. Providing static types for all of the public API in hypothesis and hypothesis.strategies.
3. Running a mypy checker on Hypothesis + its test suite as part of the build

Things to note:
1. `given` and `builds` cannot be precisely typed with this type system, but they can still have types that are better than nothing by constraining their *args and **kwargs to have a strategy type (this is actually quite useful because it will catch two common gotchas when using this).
2. Settings is going to be quite hard to type correctly and is probably not worth bothering with.
3. I am _not_ proposing to make Hypothesis internals typed. In particular templates for strategies will remain completely untyped. I've worked out what typing this accurately requires in the past and PEP 0484 can't begin to express it.



Discussion:
Note: On investigation, I don't think running mypy as part of the Hypothesis build is currently feasible, and I'm not very interested in adding this without a static checking layer as part of the build. I will leave this open, but this should be considered a long-term objective rather than something viable right now.

Another note: the comments above were written about Hypothesis v1.11, and Hypothesis 3.x is going to be much nicer to annotate.  MyPy has also improved substantially.

That said, we'd still have to use stub (.pyi) files, so I'm not volunteering to add type hints before we drop Python 2 at EOL in 2020.  My work on #643 also led to a terrible cynicism about the `typing` module, which is shockingly inconsistent between 3.5, 3.6, and even patch versions.
It seems that this issue so far has been planning to type-annotate as much as possible.  However, one of the main points of gradual typing is that you can add it to a project incrementally!

[This post about Zulip](https://blog.zulip.org/2016/10/13/static-types-in-python-oh-mypy/) has some good advice, which - summarised for Hypothesis - suggests:

- Writing a `make` task that runs the smallest possible set of checks, getting a clean run, adding it to `.travis.yml`.  This is 'glorified linter' territory, and shouldn't require code changes beyond (maybe) annotating empty global variables.
- Adding the  `--check-untyped-defs` option makes mypy try to infer variable types anywhere it can, without annotations.  With luck, 'independently useful linter' territory.  
- Then anyone can add annotations (in comments or stubs, for Py2 compatibility) to various parts of the codebase in future PRs.  This is where type hints become really useful and mypy does substantially more than any other Python linters.

@DRMacIver, if you think steps one and two are useful in isolation I'd be happy to make an attempt, and from there it's mostly an incremental job.

The remaining question is basically whether to use comments or stub files (until we drop Python2 support, at which point we can use inline annotations).  Comments are more local to the related code, but if we use stub files it would be possible to exclude them from the `has_source_changes` check and therefore avoid releases for hint-only PRs.
Stub files are probably preferable for now because mypy doesn't yet have an
method for pip-installed packages to provide their own annotations to
consumers - anything other than the local code itself needs to be in
typeshed to provide annotations to consumers, and that needs stubs. Also
hypothesis has a fairly stable interface so I think it wouldn't be quite so
bad to have header files. Additionally it lets you use the nicer python3
syntax instead of the python2 comments

On Wed, Sep 13, 2017 at 3:49 AM Zac Hatfield Dodds <notifications@github.com>
wrote:

> It seems that this issue so far has been planning to type-annotate as much
> as possible. However, one of the main points of gradual typing is that you
> can add it to a project incrementally!
>
> This post about Zulip
> <http://blog.zulip.org/2016/10/13/static-types-in-python-oh-mypy/> has
> some good advice, which - summarised for Hypothesis - suggests:
>
>    - Writing a make task that runs the smallest possible set of checks,
>    getting a clean run, adding it to .travis.yml. This is 'glorified
>    linter' territory, and shouldn't require code changes beyond (maybe)
>    annotating empty global variables.
>    - Adding the --check-untyped-defs option makes mypy try to infer
>    variable types anywhere it can, without annotations. With luck,
>    'independently useful linter' territory.
>    - Then anyone can add annotations (in comments or stubs, for Py2
>    compatibility) to various parts of the codebase in future PRs. This is
>    where type hints become really useful and mypy does substantially more than
>    any other Python linters.
>
> @DRMacIver <https://github.com/drmaciver>, if you think steps one and two
> are useful in isolation I'd be happy to make an attempt, and from there
> it's mostly an incremental job.
>
> The remaining question is basically whether to use comments or stub files
> (until we drop Python2 support, at which point we can use inline
> annotations). Comments are more local to the related code, but if we use
> stub files it would be possible to exclude them from the
> has_source_changes check and therefore avoid releases for hint-only PRs.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/HypothesisWorks/hypothesis-python/issues/200#issuecomment-329087300>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAA5NNDBN6dM7UZr1uaIS3ddZCTUUmKIks5sh4kDgaJpZM4GOkyp>
> .
>

Based on [the PEP](https://www.python.org/dev/peps/pep-0484/#storing-and-distributing-stub-files) and [typeshed docs](https://github.com/python/typeshed/blob/master/CONTRIBUTING.md#adding-a-new-library), I thought that either comments or stubs in the Hypothesis package (ie in local code, not typeshed) would work - assuming that people use the `--follow -imports` option, obviously.  I think this makes the stubs/comments distinction moot for downstream type-checking.
Currently either comments or stubs will work for local checking within the hypothesis project. It won't work for people pip installing hypothesis. 
See https://github.com/python/typing/issues/84
I've gotta be honest, if the process of doing this correctly requires submitting pull requests to typeshed, my inclination is not to bother and to wait for the ecosystem around this to mature into something actually useful.
I can't argue with that. The work involved is silly, though there's a pep in the progress of being drafted to address the issue.

I think it's fairly reasonable to also just have them locally inline (with comments) and wait for the ecosystem to allow pip-install to provide them, ignoring typeshed and downstream consumers.
> I think it's fairly reasonable to also just have them locally inline (with comments) and wait for the ecosystem to allow pip-install to provide them, ignoring typeshed and downstream consumers.

That's true. I'm definitely not against that solution, and if @Zac-HD (or anyone else!) wants to put in the work to start doing that incrementally as he proposed, I'd be perfectly happy with that.

I'm personally still a bit sceptical about the typing module, but I don't have any problem with us giving it a go.
Whew, that was a longish read.  My interpretation of the various issues and [PEP 561](https://www.python.org/dev/peps/pep-0561/) is that putting `foo.pyi` next to each `foo.py` will work for us now, and for downstream users once tooling improves.  Until then, distributing type hints is simply more trouble than it's worth.
[Tooling has improved!](https://mypy-lang.blogspot.com.au/2018/04/mypy-0590-released.html)  Mypy now supports distributing type hints as described in PEP561 ([docs](https://mypy.readthedocs.io/en/latest/installed_packages.html)).  Options for discussion:

1. provide stubs for downstream users, but don't run Mypy against Hypothesis
	- lightweight, but easy for implementation and stubs to get out of sync
2. annotate Hypothesis API but avoid typing internals or defining generics
	- could use stubs or type comments (or annotations in 2020 once 3.4 and 2.7 die)
	- get *some* consistency checks from running Mypy in CI
3. [start to] annotate everything
	- heavyweight; ugly if supporting 2.7 and 3.4
	- finds more bugs and design issues in Hypothesis
	- tried and rejected in #858 

My preference is for option two:
- aim for enough "gradual typing" to error if you pass the wrong type or number of arguments to public API or selected internals
- don't bother (eg) defining SearchStrategy as a generic type, or typing `@given` beyond "pass it strategies"
- ship as comments rather than stubs, to stay in sync (harder to ignore, for good or ill)
My preference is also for 2. Is there a reason you don't want SearchStrategy to be a generic type though? I feel like type checking the API will work much better if it is. 
Making `SearchStrategy` and `@given` [generic](https://mypy.readthedocs.io/en/latest/generics.html) would be nice, but may be an impractically large project and (without a clear mandate and excited users) I have other priorities.

I think I also have a higher opinion of 'shallow' typing than you!  Especially given the error messages you're likely to see from typical code...  For example, last time I tried Mypy on Hypothesis I found the dead `classpath` module, and this time I realised that `Verbosity` was trying to be an Enum (fixed in #1211).  

TLDR: do generics later.
> I think I also have a higher opinion of 'shallow' typing than you! 

Well it would be hard to have a *lower* one. ;-)

> TLDR: do generics later.

I don't think "do generics later" is actually a viable plan. The history of attempts at it is quite unpleasant. What happens if e.g. someone wants to use `SearchStrategy` in their own type signatures? (OK technically this isn't even public API, but it should probably be treated implicitly as such)
As a user I'd be pretty excited by generic SearchStrategy. In fact, that's the *only* thing I care about being typed. The rest is so trivial it hardly helps, but generic SearchStrategy, especially with `draw` and `flatmap` is huge.
- Works on Python2
- Works for Mypy
- Annotations available for runtime inspection (`builds`, Sphinx, etc)

Wouldn't it be nice if we could have all three of these?  Well, all we'd need to do is a spot of runtime introspection to translate either stubs or comments into an `__annotations__` attribute for each callable in the public API :smile: 

This idea brought to you by sphinx-doc/sphinx#2755 and sphinx-doc/sphinx#4824, and thinking "hey, that could actually work..." (and "what could possibly go wrong?").
OK, OK, if you want generics up front we can do that.  

(Fair warning: I will be *very unhappy* if we do [this](https://github.com/HypothesisWorks/hypothesis/pull/858#issuecomment-343676410) again, so please give frequent and detailed feedback)

> What happens if e.g. someone wants to use SearchStrategy in their own type signatures? (OK technically this isn't even public API, but it should probably be treated implicitly as such)

We could insert a parent class `Strategy` with the same structure for use in type annotations, which just raises 'use the strategy functions' errors if you try to implement anything based on it.
> (Fair warning: I will be very unhappy if we do this again, so please give frequent and detailed feedback)

I also want to avoid that! But I think the easiest way to make sure there is frequent and detailed feedback is to try to do the work in smaller chunks which can be shipped independently.

May I recommend an initial PR that does just enough work to make `SearchStrategy` generic and type check *something* in our build, but doesn't try to annotate the whole public API.

> We could insert a parent class Strategy with the same structure for use in type annotations, which just raises 'use the strategy functions' errors if you try to implement anything based on it.

Is this a solution to the lack of it being in the public API or to it being non-generic? If the former we should probably resolve it as part of the renaming thing (both #804  and #1193)
Fully on board with shipping in small pieces.  

How would you feel about a patch that just sets everything up for the real work in a future release?  ie sort out build, packaging, get a clean Mypy run in Travis, etc; I've got most of that in a branch ready to go - or at least review.
>  how would you feel about a patch that just sets everything up for the real work in a future release?

Fine by me. I don't really mind either way - I'm not at all attached to this feature, so I'm mostly interested in making sure this goes smoothly for everyone than I am in any particular timeline.

Resolving patch:
diff --git a/hypothesis-python/setup.py b/hypothesis-python/setup.py
--- a/hypothesis-python/setup.py
+++ b/hypothesis-python/setup.py
@@ -86,7 +86,7 @@ def local_file(name):
     author_email='david@drmaciver.com',
     packages=setuptools.find_packages(SOURCE),
     package_dir={'': SOURCE},
-    # package_data={'': ['py.typed']},  # un-comment to release type hints
+    package_data={'hypothesis': ['py.typed']},
     url=(
         'https://github.com/HypothesisWorks/hypothesis/'
         'tree/master/hypothesis-python'
diff --git a/hypothesis-python/src/hypothesis/strategies.py b/hypothesis-python/src/hypothesis/strategies.py
--- a/hypothesis-python/src/hypothesis/strategies.py
+++ b/hypothesis-python/src/hypothesis/strategies.py
@@ -80,14 +80,13 @@
     from random import Random  # noqa
     from typing import Any, Dict, Union, Sequence, Callable, Pattern  # noqa
     from typing import TypeVar, Tuple, List, Set, FrozenSet, overload  # noqa
-    from typing import Type, Mapping, Text, AnyStr, Optional  # noqa
+    from typing import Type, Text, AnyStr, Optional  # noqa
 
     from hypothesis.utils.conventions import InferType  # noqa
     from hypothesis.searchstrategy.strategies import T, Ex  # noqa
     K, V = TypeVar['K'], TypeVar['V']
     # See https://github.com/python/mypy/issues/3186 - numbers.Real is wrong!
     Real = Union[int, float, Fraction, Decimal]
-    ExtendFunc = Callable[[SearchStrategy[Union[T, Ex]]], SearchStrategy[T]]
 else:
     def overload(f):
         return f
@@ -260,10 +259,23 @@ def none():
     return just(None)
 
 
-def one_of(
-    *args  # type: Union[SearchStrategy[Ex], Sequence[SearchStrategy[Ex]]]
-):
-    # type: (...) -> SearchStrategy[Ex]
+@overload
+def one_of(args):
+    # type: (Sequence[SearchStrategy[Any]]) -> SearchStrategy[Any]
+    pass  # pragma: no cover
+
+
+@overload
+def one_of(*args):
+    # type: (SearchStrategy[Any]) -> SearchStrategy[Any]
+    pass  # pragma: no cover
+
+
+def one_of(*args):
+    # Mypy workaround alert:  Any is too loose above; the return paramater
+    # should be the union of the input parameters.  Unfortunately, Mypy <=0.600
+    # raises errors due to incompatible inputs instead.  See #1270 for links.
+    # v0.610 doesn't error; it gets inference wrong for 2+ arguments instead.
     """Return a strategy which generates values from any of the argument
     strategies.
 
@@ -704,7 +716,7 @@ def iterables(elements=None, min_size=None, average_size=None, max_size=None,
 
 @defines_strategy
 def fixed_dictionaries(
-    mapping  # type: Mapping[T, SearchStrategy[Ex]]
+    mapping  # type: Dict[T, SearchStrategy[Ex]]
 ):
     # type: (...) -> SearchStrategy[Dict[T, Ex]]
     """Generates a dictionary of the same type as mapping with a fixed set of
@@ -729,12 +741,14 @@ def fixed_dictionaries(
 def dictionaries(
     keys,  # type: SearchStrategy[Ex]
     values,  # type: SearchStrategy[T]
-    dict_class=dict,  # type: Type[Mapping]
+    dict_class=dict,  # type: type
     min_size=None,  # type: int
     average_size=None,  # type: int
     max_size=None,  # type: int
 ):
-    # type: (...) -> SearchStrategy[Mapping[Ex, T]]
+    # type: (...) -> SearchStrategy[Dict[Ex, T]]
+    # Describing the exact dict_class to Mypy drops the key and value types,
+    # so we report Dict[K, V] instead of Mapping[Any, Any] for now.  Sorry!
     """Generates dictionaries of type dict_class with keys drawn from the keys
     argument and values drawn from the values argument.
 
@@ -875,7 +889,7 @@ def characters(
 @cacheable
 @defines_strategy_with_reusable_values
 def text(
-    alphabet=None,  # type: Union[Text, SearchStrategy[Text]]
+    alphabet=None,  # type: Union[Sequence[Text], SearchStrategy[Text]]
     min_size=None,   # type: int
     average_size=None,   # type: int
     max_size=None  # type: int
@@ -1440,7 +1454,7 @@ def fraction_to_decimal(val):
 
 def recursive(
     base,  # type: SearchStrategy[Ex]
-    extend,  # type: ExtendFunc
+    extend,  # type: Callable[[SearchStrategy[Any]], SearchStrategy[T]]
     max_leaves=100,  # type: int
 ):
     # type: (...) -> SearchStrategy[Union[T, Ex]]
diff --git a/tooling/src/hypothesistooling/__main__.py b/tooling/src/hypothesistooling/__main__.py
--- a/tooling/src/hypothesistooling/__main__.py
+++ b/tooling/src/hypothesistooling/__main__.py
@@ -74,11 +74,6 @@ def lint():
     )
 
 
-@task(if_changed=hp.PYTHON_SRC)
-def check_type_hints():
-    pip_tool('mypy', hp.PYTHON_SRC)
-
-
 HEAD = tools.hash_for_name('HEAD')
 MASTER = tools.hash_for_name('origin/master')
 


Test patch:
diff --git a/hypothesis-python/tests/cover/test_regex.py b/hypothesis-python/tests/cover/test_regex.py
--- a/hypothesis-python/tests/cover/test_regex.py
+++ b/hypothesis-python/tests/cover/test_regex.py
@@ -131,10 +131,10 @@ def test_can_generate(pattern, encode):
 
 
 @pytest.mark.parametrize('pattern', [
-    re.compile(u'a', re.IGNORECASE),
-    u'(?i)a',
-    re.compile(u'[ab]', re.IGNORECASE),
-    u'(?i)[ab]',
+    re.compile(u'\\Aa\\Z', re.IGNORECASE),
+    u'(?i)\\Aa\\Z',
+    re.compile(u'\\A[ab]\\Z', re.IGNORECASE),
+    u'(?i)\\A[ab]\\Z',
 ])
 def test_literals_with_ignorecase(pattern):
     strategy = st.from_regex(pattern)
@@ -155,15 +155,19 @@ def test_not_literal_with_ignorecase(pattern):
 
 
 def test_any_doesnt_generate_newline():
-    assert_all_examples(st.from_regex(u'.'), lambda s: s != u'\n')
+    assert_all_examples(st.from_regex(u'\\A.\\Z'), lambda s: s != u'\n')
 
 
-@pytest.mark.parametrize('pattern', [re.compile(u'.', re.DOTALL), u'(?s).'])
+@pytest.mark.parametrize('pattern', [
+    re.compile(u'\\A.\\Z', re.DOTALL), u'(?s)\\A.\\Z'
+])
 def test_any_with_dotall_generate_newline(pattern):
     find_any(st.from_regex(pattern), lambda s: s == u'\n')
 
 
-@pytest.mark.parametrize('pattern', [re.compile(b'.', re.DOTALL), b'(?s).'])
+@pytest.mark.parametrize('pattern', [
+    re.compile(b'\\A.\\Z', re.DOTALL), b'(?s)\\A.\\Z'
+])
 def test_any_with_dotall_generate_newline_binary(pattern):
     find_any(st.from_regex(pattern), lambda s: s == b'\n')
 
@@ -214,7 +218,7 @@ def test_end_with_terminator_does_not_pad():
 
 
 def test_end():
-    strategy = st.from_regex(u'abc$')
+    strategy = st.from_regex(u'\\Aabc$')
 
     find_any(strategy, lambda s: s == u'abc')
     find_any(strategy, lambda s: s == u'abc\n')
@@ -303,7 +307,7 @@ def test_group_backref_may_not_be_present(s):
 @pytest.mark.skipif(sys.version_info[:2] < (3, 6),
                     reason='requires Python 3.6')
 def test_subpattern_flags():
-    strategy = st.from_regex(u'(?i)a(?-i:b)')
+    strategy = st.from_regex(u'(?i)\\Aa(?-i:b)\\Z')
 
     # "a" is case insensitive
     find_any(strategy, lambda s: s[0] == u'a')
diff --git a/whole-repo-tests/test_type_hints.py b/whole-repo-tests/test_type_hints.py
new file mode 100644
--- /dev/null
+++ b/whole-repo-tests/test_type_hints.py
@@ -0,0 +1,74 @@
+# coding=utf-8
+#
+# This file is part of Hypothesis, which may be found at
+# https://github.com/HypothesisWorks/hypothesis-python
+#
+# Most of this work is copyright (C) 2013-2018 David R. MacIver
+# (david@drmaciver.com), but it contains contributions by others. See
+# CONTRIBUTING.rst for a full list of people who may hold copyright, and
+# consult the git log if you need to determine who owns an individual
+# contribution.
+#
+# This Source Code Form is subject to the terms of the Mozilla Public License,
+# v. 2.0. If a copy of the MPL was not distributed with this file, You can
+# obtain one at http://mozilla.org/MPL/2.0/.
+#
+# END HEADER
+
+from __future__ import division, print_function, absolute_import
+
+import os
+import subprocess
+
+import pytest
+
+from hypothesistooling.scripts import pip_tool, tool_path
+from hypothesistooling.projects.hypothesispython import PYTHON_SRC
+
+
+def test_mypy_passes_on_hypothesis():
+    pip_tool('mypy', PYTHON_SRC)
+
+
+def get_mypy_analysed_type(fname, val):
+    out = subprocess.Popen(
+        [tool_path('mypy'), fname],
+        stdout=subprocess.PIPE, encoding='utf-8', universal_newlines=True,
+        # We set the MYPYPATH explicitly, because PEP561 discovery wasn't
+        # working in CI as of mypy==0.600 - hopefully a temporary workaround.
+        env=dict(os.environ, MYPYPATH=PYTHON_SRC),
+    ).stdout.read()
+    assert len(out.splitlines()) == 1
+    # See https://mypy.readthedocs.io/en/latest/common_issues.html#reveal-type
+    # The shell output for `reveal_type([1, 2, 3])` looks like a literal:
+    # file.py:2: error: Revealed type is 'builtins.list[builtins.int*]'
+    typ = out.split('error: Revealed type is ')[1].strip().strip("'")
+    qualname = 'hypothesis.searchstrategy.strategies.SearchStrategy'
+    assert typ.startswith(qualname)
+    return typ[len(qualname) + 1:-1].replace('builtins.', '').replace('*', '')
+
+
+@pytest.mark.parametrize('val,expect', [
+    ('integers()', 'int'),
+    ('text()', 'str'),
+    ('integers().map(str)', 'str'),
+    ('booleans().filter(bool)', 'bool'),
+    ('lists(none())', 'list[None]'),
+    ('dictionaries(integers(), datetimes())', 'dict[int, datetime.datetime]'),
+    # Ex`-1 stands for recursion in the whole type, i.e. Ex`0 == Union[...]
+    ('recursive(integers(), lists)', 'Union[list[Ex`-1], int]'),
+    # See https://github.com/python/mypy/issues/5269 - fix the hints on
+    # `one_of` and document the minimum Mypy version when the issue is fixed.
+    ('one_of(integers(), text())', 'Any'),
+])
+def test_revealed_types(tmpdir, val, expect):
+    """Check that Mypy picks up the expected `X` in SearchStrategy[`X`]."""
+    f = tmpdir.join(expect + '.py')
+    f.write(
+        'from hypothesis.strategies import *\n'
+        's = {}\n'
+        'reveal_type(s)\n'
+        .format(val)
+    )
+    got = get_mypy_analysed_type(str(f.realpath()), val)
+    assert got == expect