Repository: HypothesisWorks/hypothesis

Issue:
HealthCheck sensitivity is not appropriate in all contexts
When generating relatively large json objects (or even arrays of such objects), one will frequently run into either a `too_slow` or a `data_too_large` HealthCheck failure.

While it is possible to disable these checks, doing so foregoes any benefit that the HealthCheck is supposed to bring.

In many situations, it would be preferable to have an option to tweak the sensitivity level of health checks. Maybe I know roughly how much data I want to generate, and I am okay with CI taking a few extra seconds to run my tests.
Install of 6.47.2 fails with pipenv 
It appears that the following line in hypothesis-python/setup.py causes installation with pipenv (2022.6.7) to fail:

https://github.com/HypothesisWorks/hypothesis/blob/b73d6fd51d8dc31f23da1ad4bfdd9dc18401c3b8/hypothesis-python/setup.py#L105

The pipenv traceback is:

```
Traceback (most recent call last):
  File "/Users/andrewcordery/.local/pipx/venvs/pipenv/lib/python3.9/site-packages/pipenv/resolver.py", line 856, in <module>
    main()
  File "/Users/andrewcordery/.local/pipx/venvs/pipenv/lib/python3.9/site-packages/pipenv/resolver.py", line 842, in main
    _main(
  File "/Users/andrewcordery/.local/pipx/venvs/pipenv/lib/python3.9/site-packages/pipenv/resolver.py", line 822, in _main
    resolve_packages(
  File "/Users/andrewcordery/.local/pipx/venvs/pipenv/lib/python3.9/site-packages/pipenv/resolver.py", line 783, in resolve_packages
    results = clean_results(results, resolver, project, dev)
  File "/Users/andrewcordery/.local/pipx/venvs/pipenv/lib/python3.9/site-packages/pipenv/resolver.py", line 652, in clean_results
    entry_dict = translate_markers(entry.get_cleaned_dict(keep_outdated=False))
  File "/Users/andrewcordery/.local/pipx/venvs/pipenv/lib/python3.9/site-packages/pipenv/resolver.py", line 277, in get_cleaned_dict
    if self.original_markers and not self.markers:
  File "/Users/andrewcordery/.local/pipx/venvs/pipenv/lib/python3.9/site-packages/pipenv/resolver.py", line 634, in __getattribute__
    return super().__getattribute__(key)
  File "/Users/andrewcordery/.local/pipx/venvs/pipenv/lib/python3.9/site-packages/pipenv/resolver.py", line 239, in original_markers
    original_markers, lockfile_dict = self.get_markers_from_dict(self.lockfile_dict)
  File "/Users/andrewcordery/.local/pipx/venvs/pipenv/lib/python3.9/site-packages/pipenv/resolver.py", line 212, in get_markers_from_dict
    markers.add(normalize_marker_str(entry_dict["markers"]))
  File "/Users/andrewcordery/.local/pipx/venvs/pipenv/lib/python3.9/site-packages/pipenv/vendor/requirementslib/models/markers.py", line 692, in normalize_marker_str
    parts = cleanup_pyspecs(pyversion)
  File "/Users/andrewcordery/.local/pipx/venvs/pipenv/lib/python3.9/site-packages/pipenv/vendor/requirementslib/models/markers.py", line 286, in cleanup_pyspecs
    for op_and_version_type, versions in _group_by_op(tuple(specs)):
  File "/Users/andrewcordery/.local/pipx/venvs/pipenv/lib/python3.9/site-packages/pipenv/vendor/requirementslib/models/markers.py", line 200, in _group_by_op
    specs = [_get_specs(x) for x in list(specs)]
  File "/Users/andrewcordery/.local/pipx/venvs/pipenv/lib/python3.9/site-packages/pipenv/vendor/requirementslib/models/markers.py", line 200, in <listcomp>
    specs = [_get_specs(x) for x in list(specs)]
  File "/Users/andrewcordery/.local/pipx/venvs/pipenv/lib/python3.9/site-packages/pipenv/vendor/requirementslib/models/markers.py", line 193, in _get_specs
    result.append((spec.operator, _tuplize_version(spec.version)))
  File "/Users/andrewcordery/.local/pipx/venvs/pipenv/lib/python3.9/site-packages/pipenv/vendor/requirementslib/models/markers.py", line 124, in _tuplize_version
    return tuple(int(x) for x in filter(lambda i: i != "*", version.split(".")))
  File "/Users/andrewcordery/.local/pipx/venvs/pipenv/lib/python3.9/site-packages/pipenv/vendor/requirementslib/models/markers.py", line 124, in <genexpr>
    return tuple(int(x) for x in filter(lambda i: i != "*", version.split(".")))
ValueError: invalid literal for int() with base 10: '0b1'
✘ Locking Failed!
```
The relevant line in pipenv is attempting to turn all parts of the version into integers:  https://github.com/pypa/pipenv/blob/2482e2d3a49fec6830a80606c90c6be9d6bf7946/pipenv/vendor/requirementslib/models/markers.py#L124

I'm not sure the `python_version` marker is intended to support full version numbers including the beta suffix.  Looking at https://peps.python.org/pep-0508/ it seems to indicate you'd need to use `python_full_version` to do that, which is perhaps why pipenv is attempting to turn each part of the python_version version string into an integer.




Discussion:
The goal of the HealthCheck system is just to ensure that you know if you're hitting these unusually large or slow cases - if that's intentional (which in your case it is), disabling them is perfectly fine.  I don't think that making health checks configurable in general is valuable enough to justify the more complicated interface though.

We should probably [reframe `HealthCheck.too_slow` in terms of the `deadline` setting](https://github.com/HypothesisWorks/hypothesis/blob/ef463c1507eb9c4cc1e8387959b44ad8860775d8/hypothesis-python/src/hypothesis/internal/conjecture/engine.py#L387-L398), though 🤔 
I consider this a bug in pipenv, but unfortunately one that we might need to work around downstream too.
I think the linked code (`requirementslib`) is vendored, I [opened an issue there](https://github.com/sarugaku/requirementslib/issues/317), but maybe they'll point out the `python_version` vs `python_full_version` thing as well (though the breaking code seems to be a pretty general function).
The reason I mention that the `python_version` marker may not be intended to support full versions including beta is that [PEP508](https://peps.python.org/pep-0508/#environment-markers) describes `python_version` as being equivalent to `'.'.join(platform.python_version_tuple()[:2])`, so assuming that the 'b' beta indicator is always on the patch portion of the version string, it would never be included here.  
  
<img width="536" alt="image" src="https://user-images.githubusercontent.com/274325/173564105-ee3c4198-2c6d-4c7d-8df0-fad3c6b4ede3.png">

That said, looking at the requirementslib code, it appears that they do not need to be converting all parts of the version to an integer, only the first two.  Additionally, they do automatically correct 'python_version' to 'python_full_version' if it includes three version parts, but that only happens after the forced conversion to integers.

On a rather amusing note, at the moment you cannot `pipenv install -e .[dev,tests,typing]` to get a local dev setup of requirementslib going because..... requirementslib uses hypothesis for tests!   https://github.com/sarugaku/requirementslib/blob/main/setup.cfg#L67

To summarize:  requirementslib shouldn't crash on parsing `python_version<'3.11.0b1'"` but strictly speaking it probably shouldn't pay attention to the '.0b1' portion either when specified in `python_version`.


Resolving patch:
diff --git a/hypothesis-python/setup.py b/hypothesis-python/setup.py
--- a/hypothesis-python/setup.py
+++ b/hypothesis-python/setup.py
@@ -102,7 +102,7 @@ def local_file(name):
     extras_require=extras,
     install_requires=[
         "attrs>=19.2.0",
-        "exceptiongroup>=1.0.0rc8 ; python_version<'3.11.0b1'",
+        "exceptiongroup>=1.0.0rc8 ; python_version<'3.11'",
         "sortedcontainers>=2.1.0,<3.0.0",
     ],
     python_requires=">=3.7",
diff --git a/hypothesis-python/src/hypothesis/internal/conjecture/engine.py b/hypothesis-python/src/hypothesis/internal/conjecture/engine.py
--- a/hypothesis-python/src/hypothesis/internal/conjecture/engine.py
+++ b/hypothesis-python/src/hypothesis/internal/conjecture/engine.py
@@ -13,6 +13,7 @@
 import time
 from collections import defaultdict
 from contextlib import contextmanager
+from datetime import timedelta
 from enum import Enum
 from random import Random, getrandbits
 from weakref import WeakKeyDictionary
@@ -386,7 +387,10 @@ def record_for_health_check(self, data):
 
         draw_time = sum(state.draw_times)
 
-        if draw_time > 1.0:
+        # Allow at least the greater of one second or 5x the deadline.  If deadline
+        # is None, allow 30s - the user can disable the healthcheck too if desired.
+        draw_time_limit = 5 * (self.settings.deadline or timedelta(seconds=6))
+        if draw_time > max(1.0, draw_time_limit.total_seconds()):
             fail_health_check(
                 self.settings,
                 "Data generation is extremely slow: Only produced "


Test patch:
diff --git a/hypothesis-python/tests/common/utils.py b/hypothesis-python/tests/common/utils.py
--- a/hypothesis-python/tests/common/utils.py
+++ b/hypothesis-python/tests/common/utils.py
@@ -11,6 +11,7 @@
 import contextlib
 import sys
 from io import StringIO
+from types import SimpleNamespace
 
 from hypothesis import Phase, settings
 from hypothesis.errors import HypothesisDeprecationWarning
@@ -29,9 +30,11 @@
 
     @contextlib.contextmanager
     def raises(expected_exception, match=None):
+        err = SimpleNamespace(value=None)
         try:
-            yield
+            yield err
         except expected_exception as e:
+            err.value = e
             if match is not None:
                 import re
 
@@ -175,11 +178,14 @@ def assert_falsifying_output(
         if expected_exception is None:
             # Some tests want to check the output of non-failing runs.
             test()
+            msg = ""
         else:
-            with raises(expected_exception):
+            with raises(expected_exception) as exc_info:
                 test()
+            notes = "\n".join(getattr(exc_info.value, "__notes__", []))
+            msg = str(exc_info.value) + "\n" + notes
 
-    output = out.getvalue()
+    output = out.getvalue() + msg
     assert f"{example_type} example:" in output
     assert_output_contains_failure(output, test, **kwargs)
 
diff --git a/hypothesis-python/tests/cover/test_control.py b/hypothesis-python/tests/cover/test_control.py
--- a/hypothesis-python/tests/cover/test_control.py
+++ b/hypothesis-python/tests/cover/test_control.py
@@ -74,7 +74,7 @@ def test_does_not_suppress_exceptions():
 
 
 def test_suppresses_exceptions_in_teardown():
-    with pytest.raises(ValueError) as err:
+    with pytest.raises(ValueError) as exc_info:
         with bc():
 
             def foo():
@@ -83,12 +83,12 @@ def foo():
             cleanup(foo)
             raise AssertionError
 
-    assert isinstance(err.value, ValueError)
-    assert isinstance(err.value.__cause__, AssertionError)
+    assert isinstance(exc_info.value, ValueError)
+    assert isinstance(exc_info.value.__cause__, AssertionError)
 
 
 def test_runs_multiple_cleanup_with_teardown():
-    with pytest.raises(ExceptionGroup) as err:
+    with pytest.raises(ExceptionGroup) as exc_info:
         with bc():
 
             def foo():
@@ -101,9 +101,9 @@ def bar():
             cleanup(bar)
             raise AssertionError
 
-    assert isinstance(err.value, ExceptionGroup)
-    assert isinstance(err.value.__cause__, AssertionError)
-    assert {type(e) for e in err.value.exceptions} == {ValueError, TypeError}
+    assert isinstance(exc_info.value, ExceptionGroup)
+    assert isinstance(exc_info.value.__cause__, AssertionError)
+    assert {type(e) for e in exc_info.value.exceptions} == {ValueError, TypeError}
     assert _current_build_context.value is None
 
 
diff --git a/hypothesis-python/tests/cover/test_health_checks.py b/hypothesis-python/tests/cover/test_health_checks.py
--- a/hypothesis-python/tests/cover/test_health_checks.py
+++ b/hypothesis-python/tests/cover/test_health_checks.py
@@ -45,7 +45,7 @@ def test(x):
 
 
 def test_slow_generation_inline_fails_a_health_check():
-    @settings(HEALTH_CHECK_SETTINGS, deadline=None)
+    @HEALTH_CHECK_SETTINGS
     @given(st.data())
     def test(data):
         data.draw(st.integers().map(lambda x: time.sleep(0.2)))
diff --git a/hypothesis-python/tests/nocover/test_interesting_origin.py b/hypothesis-python/tests/nocover/test_interesting_origin.py
--- a/hypothesis-python/tests/nocover/test_interesting_origin.py
+++ b/hypothesis-python/tests/nocover/test_interesting_origin.py
@@ -58,9 +58,5 @@ def test_fn(x, y):
         # Indirection to fix https://github.com/HypothesisWorks/hypothesis/issues/2888
         return function(x, y)
 
-    try:
+    with pytest.raises(MultipleFailures):
         test_fn()
-    except MultipleFailures:
-        pass
-    else:
-        raise AssertionError("Expected MultipleFailures")
diff --git a/hypothesis-python/tests/nocover/test_targeting.py b/hypothesis-python/tests/nocover/test_targeting.py
--- a/hypothesis-python/tests/nocover/test_targeting.py
+++ b/hypothesis-python/tests/nocover/test_targeting.py
@@ -29,7 +29,7 @@ def test_threshold_problem(x):
 @pytest.mark.parametrize("multiple", [False, True])
 def test_reports_target_results(testdir, multiple):
     script = testdir.makepyfile(TESTSUITE.format("" if multiple else "# "))
-    result = testdir.runpytest(script)
+    result = testdir.runpytest(script, "--tb=native")
     out = "\n".join(result.stdout.lines)
     assert "Falsifying example" in out
     assert "x=101" in out