TST: assert_produces_warning works with filterwarnings
State: closed

Previously, assert_produces_warning did not play well with pytest's
filterwarnings.

```
===================================================================================== FAILURES ======================================================================================
____________________________________________________________________ test_assert_produces_warning_honors_filter _____________________________________________________________________

    @pytest.mark.filterwarnings('ignore:f1:FutureWarning')
    def test_assert_produces_warning_honors_filter():
        with tm.assert_produces_warning(RuntimeWarning):
>           f()

pandas/tests/util/test_assert_produces_warning.py:17:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <contextlib._GeneratorContextManager object at 0x11dd61128>, type = None, value = None, traceback = None

    def __exit__(self, type, value, traceback):
        if type is None:
            try:
>               next(self.gen)
E               AssertionError: Caused unexpected warning(s): [('FutureWarning', FutureWarning('f1'), '/Users/taugspurger/sandbox/pandas/pandas/tests/util/test_assert_produces_warni
ng.py', 10)].

/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py:119: AssertionError
============================================================================= 1 failed in 0.20 seconds ==============================================================================
```

Internally, `assert_produces_warning` sets a new `filter`, which
overrides any filters set by pytest. We allow for `filter_level=None` to
not set a filter.


This will be useful for the SparseDataFrame deprecation, where I'd like to just set a `pytest.mark.filterwarning` on the class, and not worry about catching individual warnings.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25721?src=pr&el=h1) Report
> Merging [#25721](https://codecov.io/gh/pandas-dev/pandas/pull/25721?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/79205ea8a6aac4c82a1572276ede7510f5a38e8e?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25721/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25721?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #25721      +/-   ##
==========================================
+ Coverage   91.24%   91.24%   +<.01%     
==========================================
  Files         172      172              
  Lines       52967    52968       +1     
==========================================
+ Hits        48331    48332       +1     
  Misses       4636     4636
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.82% <100%> (ø)` | :arrow_up: |
| #single | `41.75% <100%> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25721?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/25721/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `88.99% <100%> (+0.01%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25721?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25721?src=pr&el=footer). Last update [79205ea...f0ad708](https://codecov.io/gh/pandas-dev/pandas/pull/25721?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25721?src=pr&el=h1) Report
> Merging [#25721](https://codecov.io/gh/pandas-dev/pandas/pull/25721?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/79205ea8a6aac4c82a1572276ede7510f5a38e8e?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25721/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25721?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #25721      +/-   ##
==========================================
+ Coverage   91.24%   91.25%   +<.01%     
==========================================
  Files         172      172              
  Lines       52967    52973       +6     
==========================================
+ Hits        48331    48339       +8     
+ Misses       4636     4634       -2
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.82% <100%> (ø)` | :arrow_up: |
| #single | `41.74% <100%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25721?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/25721/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `89.18% <100%> (+0.19%)` | :arrow_up: |
| [pandas/core/indexes/range.py](https://codecov.io/gh/pandas-dev/pandas/pull/25721/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9yYW5nZS5weQ==) | `97.41% <0%> (+0.04%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25721?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25721?src=pr&el=footer). Last update [79205ea...3e01ab4](https://codecov.io/gh/pandas-dev/pandas/pull/25721?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Hmm this may require a slightly different approach.

As a test-written, I only care about the RuntimeWarning. But calling `f` may emit other warnings (from `f`,  or something it calls).

```python
def f():
    warnings.warn('f1', FutureWarning)
    warnings.warn('f2', RuntimeWarning)
```

So I think we need an option for whether "extra" warnings cause the test to fail (I really don't want to wrap every call to `SparseDataFrame` in an `assert_produces_warning` :)
> But calling f may emit other warnings

I think this is related to an issue that came up in `DatetimeArray._addsub_int` in cases where multiple warnings were expected.  The conclusion at the time was that `tm.assert_produces_warning` doesn't handle multiple expected warnings, so the workaround was to just suppress one and only test for the other.

Not a big deal, but any reason why this merits a new test file?
For the sparse array deprecation, I'd like to decorate the test class with
`@pytest.mark.filterwarnings(...)`
ignore warnings about the deprecation. But we'd like to assert that
expected warnings for other things still happen.

On Fri, Mar 15, 2019 at 10:25 AM jbrockmendel <notifications@github.com>
wrote:

> But calling f may emit other warnings
>
> I think this is related to an issue that came up in
> DatetimeArray._addsub_int in cases where multiple warnings were expected.
> The conclusion at the time was that tm.assert_produces_warning doesn't
> handle multiple expected warnings, so the workaround was to just suppress
> one and only test for the other.
>
> Not a big deal, but any reason why this merits a new test file?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/25721#issuecomment-473329315>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIhbVstqEzakbRSbPldBi3OAc1NM3ks5vW7tMgaJpZM4by-5g>
> .
>

> The conclusion at the time was that tm.assert_produces_warning doesn't handle multiple expected warnings

Right - it would be worthwhile though at some point to investigate adding such support.
thanks @TomAugspurger