Repository: HypothesisWorks/hypothesis

Issue:
Set appropriate `stacklevel=` in all calls to `warnings.warn()`
I noticed [in this lovely video](https://youtu.be/mkgd9iOiICc?t=1039) that the warning for `st.integers().example()` pointed into the Hypothesis internals, rather than at the calling user code.  The solution is to set `stacklevel=` in:

- [x] `SearchStrategy.example()` in `strategies.py`
- [x] `warn_on_missing_dtypes()` in `array_api.py` - this might need to accept a new `stacklevel=` argument, since it's called from multiple different places.  [turns out not to be worth the trouble 😞]

Test plan: manually check that these warnings now point to user code - the problem isn't that we break such handling but rather that we hadn't added it in the first place.
Set appropriate `stacklevel=` in all calls to `warnings.warn()`
I noticed [in this lovely video](https://youtu.be/mkgd9iOiICc?t=1039) that the warning for `st.integers().example()` pointed into the Hypothesis internals, rather than at the calling user code.  The solution is to set `stacklevel=` in:

- [x] `SearchStrategy.example()` in `strategies.py`
- [x] `warn_on_missing_dtypes()` in `array_api.py` - this might need to accept a new `stacklevel=` argument, since it's called from multiple different places.  [turns out not to be worth the trouble 😞]

Test plan: manually check that these warnings now point to user code - the problem isn't that we break such handling but rather that we hadn't added it in the first place.


Discussion:
👋 I'm going to attempt this one.
I've created the following minimal, reproducible example for this.

<details><summary>summary of versions (click me)</summary>

```text
platform darwin -- Python 3.9.13, pytest-7.1.2, pluggy-1.0.0
rootdir: /Users/jlamb/repos/hypothesis, configfile: pytest.ini
plugins: xdist-2.5.0, forked-1.4.0, hypothesis-6.50.1
```

</details>

Given a file `test_warnings.py` with the following.

```python
from hypothesis import (
    example,
    given,
    strategies as st
)


def _bad_max(x, y):
    if x == 2:
        return -1
    else:
        return max(x, y)


@given(st.integers(), st.integers())
@example(2, st.integers().example())
def test_max_works(x, y):
    assert _bad_max(x, y) == max(x, y)
```

Running this with all of `pytest`'s defaults

```shell
pytest ./test_warnings.py
```

Returns a non-0 exit code as expected, and produces output that shows user code.

```text
================================================= FAILURES ==================================================
______________________________________________ test_max_works _______________________________________________
Traceback (most recent call last):
  File "/Users/jlamb/repos/hypothesis/test_warnings.py", line 16, in test_max_works
    @example(2, st.integers().example())
  File "/Users/jlamb/repos/hypothesis/hypothesis-python/src/hypothesis/core.py", line 1208, in wrapped_test
    raise the_error_hypothesis_found
  File "/Users/jlamb/repos/hypothesis/test_warnings.py", line 18, in test_max_works
    assert _bad_max(x, y) == max(x, y)
AssertionError: assert -1 == 3263640430859197895
 +  where -1 = _bad_max(2, 3263640430859197895)
 +  and   3263640430859197895 = max(2, 3263640430859197895)
------------------------------------------------ Hypothesis -------------------------------------------------
Falsifying explicit example: test_max_works(
    x=2, y=3263640430859197895,
)
```

But using warnings-as-errors, Hypothesis's warning is printed instead.

```shell
pytest -Werror ./test_warnings.py
```

Returns a non-0 exit code, and points to a line in the user's test file, but doesn't show the test name.

```text
================================================== ERRORS ===================================================
_____________________________________ ERROR collecting test_warnings.py _____________________________________
Traceback (most recent call last):
  File "/Users/jlamb/repos/hypothesis/test_warnings.py", line 16, in <module>
    @example(2, st.integers().example())
  File "/Users/jlamb/repos/hypothesis/hypothesis-python/src/hypothesis/strategies/_internal/strategies.py", line 285, in example
    warnings.warn(
hypothesis.errors.NonInteractiveExampleWarning: The `.example()` method is good for exploring strategies, but should only be used interactively.  We recommend using `@given` for tests - it performs better, saves and replays failures to avoid flakiness, and reports minimal examples. (strategy: integers())
========================================== short test summary info ==========================================
ERROR test_warnings.py - hypothesis.errors.NonInteractiveExampleWarning: The `.example()` method is good f...
```
Nice!  I think an even simpler repro for that warning would be:

```python
from hypothesis import strategies as st

def test():
    st.integers().example()
```

(in your MRE, the call to `.example()` happens at import time, when the arguments to the `@example()` decorator are evaluated)
Cool, cool ok.

So I think that both in my example (error at test collection time) and your example (error raised directly in the test codde), `hypothesis's` behavior is what I'd expect.

You get a line in the traceback pointing to the specific test file + line in that test file which caused the warning.

Running your example with `pytest -Werror test_warnings.py`, I see this:

```text
================================================= FAILURES ==================================================
___________________________________________________ test ____________________________________________________
Traceback (most recent call last):
  File "/Users/jlamb/repos/hypothesis/test_warnings.py", line 24, in test
    st.integers().example()
  File "/Users/jlamb/repos/hypothesis/hypothesis-python/src/hypothesis/strategies/_internal/strategies.py", line 286, in example
    warnings.warn(
hypothesis.errors.NonInteractiveExampleWarning: The `.example()` method is good for exploring strategies...
```

That seems pretty good to me. Can you help me understand what type of output you'd expect to see instead? I might be misunderstanding what "point to user code" means.
Ok we talked about this in person. The issue is not about the behavior with warnings-as-errors, but in other settings where `pytest` just collects warnings and prints at the end of results.

For example:

```python
"""test_warnings.py"""

from hypothesis import strategies as st


def test():
    st.integers().example()
```

And running like this

```shell
pytest -Wdefault ./test_warnings.py
```

You'll currently get output that points to the line in `hypothesis` where the warning was raised. That isn't enough information to understand what, in your own code, is leading to that warning.

<img width="775" alt="Screen Shot 2022-07-16 at 11 37 57 AM" src="https://user-images.githubusercontent.com/7608904/179364005-7bfd7215-3273-4858-a613-ad2fd081d822.png">

Instead, we'd like to have the line point to user code, e.g.

```text
  /Users/jlamb/repos/hypothesis/test_warnings:5: NonInteractiveExampleWarning: The `.example()` method is good
```
👋 I'm going to attempt this one.
I've created the following minimal, reproducible example for this.

<details><summary>summary of versions (click me)</summary>

```text
platform darwin -- Python 3.9.13, pytest-7.1.2, pluggy-1.0.0
rootdir: /Users/jlamb/repos/hypothesis, configfile: pytest.ini
plugins: xdist-2.5.0, forked-1.4.0, hypothesis-6.50.1
```

</details>

Given a file `test_warnings.py` with the following.

```python
from hypothesis import (
    example,
    given,
    strategies as st
)


def _bad_max(x, y):
    if x == 2:
        return -1
    else:
        return max(x, y)


@given(st.integers(), st.integers())
@example(2, st.integers().example())
def test_max_works(x, y):
    assert _bad_max(x, y) == max(x, y)
```

Running this with all of `pytest`'s defaults

```shell
pytest ./test_warnings.py
```

Returns a non-0 exit code as expected, and produces output that shows user code.

```text
================================================= FAILURES ==================================================
______________________________________________ test_max_works _______________________________________________
Traceback (most recent call last):
  File "/Users/jlamb/repos/hypothesis/test_warnings.py", line 16, in test_max_works
    @example(2, st.integers().example())
  File "/Users/jlamb/repos/hypothesis/hypothesis-python/src/hypothesis/core.py", line 1208, in wrapped_test
    raise the_error_hypothesis_found
  File "/Users/jlamb/repos/hypothesis/test_warnings.py", line 18, in test_max_works
    assert _bad_max(x, y) == max(x, y)
AssertionError: assert -1 == 3263640430859197895
 +  where -1 = _bad_max(2, 3263640430859197895)
 +  and   3263640430859197895 = max(2, 3263640430859197895)
------------------------------------------------ Hypothesis -------------------------------------------------
Falsifying explicit example: test_max_works(
    x=2, y=3263640430859197895,
)
```

But using warnings-as-errors, Hypothesis's warning is printed instead.

```shell
pytest -Werror ./test_warnings.py
```

Returns a non-0 exit code, and points to a line in the user's test file, but doesn't show the test name.

```text
================================================== ERRORS ===================================================
_____________________________________ ERROR collecting test_warnings.py _____________________________________
Traceback (most recent call last):
  File "/Users/jlamb/repos/hypothesis/test_warnings.py", line 16, in <module>
    @example(2, st.integers().example())
  File "/Users/jlamb/repos/hypothesis/hypothesis-python/src/hypothesis/strategies/_internal/strategies.py", line 285, in example
    warnings.warn(
hypothesis.errors.NonInteractiveExampleWarning: The `.example()` method is good for exploring strategies, but should only be used interactively.  We recommend using `@given` for tests - it performs better, saves and replays failures to avoid flakiness, and reports minimal examples. (strategy: integers())
========================================== short test summary info ==========================================
ERROR test_warnings.py - hypothesis.errors.NonInteractiveExampleWarning: The `.example()` method is good f...
```
Nice!  I think an even simpler repro for that warning would be:

```python
from hypothesis import strategies as st

def test():
    st.integers().example()
```

(in your MRE, the call to `.example()` happens at import time, when the arguments to the `@example()` decorator are evaluated)
Cool, cool ok.

So I think that both in my example (error at test collection time) and your example (error raised directly in the test codde), `hypothesis's` behavior is what I'd expect.

You get a line in the traceback pointing to the specific test file + line in that test file which caused the warning.

Running your example with `pytest -Werror test_warnings.py`, I see this:

```text
================================================= FAILURES ==================================================
___________________________________________________ test ____________________________________________________
Traceback (most recent call last):
  File "/Users/jlamb/repos/hypothesis/test_warnings.py", line 24, in test
    st.integers().example()
  File "/Users/jlamb/repos/hypothesis/hypothesis-python/src/hypothesis/strategies/_internal/strategies.py", line 286, in example
    warnings.warn(
hypothesis.errors.NonInteractiveExampleWarning: The `.example()` method is good for exploring strategies...
```

That seems pretty good to me. Can you help me understand what type of output you'd expect to see instead? I might be misunderstanding what "point to user code" means.
Ok we talked about this in person. The issue is not about the behavior with warnings-as-errors, but in other settings where `pytest` just collects warnings and prints at the end of results.

For example:

```python
"""test_warnings.py"""

from hypothesis import strategies as st


def test():
    st.integers().example()
```

And running like this

```shell
pytest -Wdefault ./test_warnings.py
```

You'll currently get output that points to the line in `hypothesis` where the warning was raised. That isn't enough information to understand what, in your own code, is leading to that warning.

<img width="775" alt="Screen Shot 2022-07-16 at 11 37 57 AM" src="https://user-images.githubusercontent.com/7608904/179364005-7bfd7215-3273-4858-a613-ad2fd081d822.png">

Instead, we'd like to have the line point to user code, e.g.

```text
  /Users/jlamb/repos/hypothesis/test_warnings:5: NonInteractiveExampleWarning: The `.example()` method is good
```

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/strategies/_internal/strategies.py b/hypothesis-python/src/hypothesis/strategies/_internal/strategies.py
--- a/hypothesis-python/src/hypothesis/strategies/_internal/strategies.py
+++ b/hypothesis-python/src/hypothesis/strategies/_internal/strategies.py
@@ -288,6 +288,7 @@ def example(self) -> Ex:
                 "it performs better, saves and replays failures to avoid flakiness, "
                 "and reports minimal examples. (strategy: %r)" % (self,),
                 NonInteractiveExampleWarning,
+                stacklevel=2,
             )
 
         context = _current_build_context.value


Test patch: