MAINT: Complete Conversion to Pytest Idiom
State: closed

* Convert all test setup/teardown to `pytest` idiom
* Remove any classes that inherit from `unittest.TestCase`
* Update documentation that we are still using `tm.TestCase`

Closes #15990.

`pytest` idiom for test setup/teardown can be found <a href="https://docs.pytest.org/en/2.7.3/xunit_setup.html">here</a>.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
@jreback : This test failure <a href="https://travis-ci.org/pandas-dev/pandas/jobs/228021399">here</a> makes no sense to me at all.  I can't reproduce this when I walk through the test (though it replicates when I run the tests locally).  Any idea as to why this is happening?
```
/home/travis/build/pandas-dev/pandas/pandas/tests/frame/test_query_eval.py yield tests are deprecated, and scheduled to be removed in pytest 4.0
```
@jreback : I don't really follow.  What's the point of this comment?

that shouldn't happen, that means a fixture is being used incorrectly.
Yeah, there is some kind of cache poisoning somewhere, as when I check `IntervalDtype._cache` before the assertion, it already has `None` mapped to `interval64`.
hmm, that could be. so you can relax / eliminate that test then. prob has to do with ordering of the cache insertions.
But that cache insertion should never have happened though.  I'm surprised that this only surfaced now (don't really see how my change would have caused it).
Something goes wrong when you call `round_trip_pickle`.  You can replicate here:
~~~python
>>> from pandas.core.dtypes.dtypes import IntervalDtype
>>> import pandas.util.testing as tm
>>>
>>> dtype = IntervalDtype("int64")
>>> IntervalDtype._cache
{'int64': interval[int64]}
>>>
>>> none_dtype = IntervalDtype("interval")
>>> IntervalDtype._cache
{'int64': interval[int64], 'None': interval}
>>>
>>> result = tm.round_trip_pickle(dtype)
>>> IntervalDtype._cache  # what!?
{'int64': interval[int64], 'None': interval[int64]}
~~~
A closer look indicates that there is aliasing going on in the cache.  That's why the value mutates so mysteriously, which must happen when the pickled object is reloaded.  This makes sense given that we are <strike>deserializing</strike> unpickling, so we must have instantiated an empty `IntervalDtype` and then set the attributes for the instance.
@jreback : What was the rationale for having a `_cache` attribute (this type of construction seems particular to just these duck-typed `dtype`)?  Is it possible that we can just remove this caching given that this interferes with pickling?

The other option is that we add a `copy` parameter to the constructor, so that we can return copies and avoid this kind of aliasing.
i will fix 
the cache is important here
it shouldn't be serializing though 
Sorry, I should have said "unpickling," but okay, I will await a patch for the aliasing then.
FYI merging #16207 shortly. you can pick up ``pandas/tests/dtypes/test_dtypes.py`` from master as I fixed this for pytest.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=h1) Report
> Merging [#16201](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/ae70ecee9caa7173634756aa2daae41dde0fa16a?src=pr&el=desc) will **decrease** coverage by `0.63%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/16201/graphs/tree.svg?height=150&width=650&src=pr&token=eZ4WkYLtcO)](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #16201      +/-   ##
==========================================
- Coverage   90.86%   90.23%   -0.64%     
==========================================
  Files         162      162              
  Lines       50884    50883       -1     
==========================================
- Hits        46237    45914     -323     
- Misses       4647     4969     +322
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.01% <100%> (-0.64%)` | :arrow_down: |
| #single | `40.31% <100%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `78.86% <100%> (-0.21%)` | :arrow_down: |
| [pandas/io/formats/excel.py](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvZXhjZWwucHk=) | `73.15% <0%> (-23.49%)` | :arrow_down: |
| [pandas/io/excel.py](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=tree#diff-cGFuZGFzL2lvL2V4Y2VsLnB5) | `61.68% <0%> (-18.94%)` | :arrow_down: |
| [pandas/core/tools/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvdG9vbHMvZGF0ZXRpbWVzLnB5) | `66.94% <0%> (-18.62%)` | :arrow_down: |
| [pandas/core/config.py](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29uZmlnLnB5) | `69.72% <0%> (-18.37%)` | :arrow_down: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `95.75% <0%> (-0.46%)` | :arrow_down: |
| [pandas/compat/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9fX2luaXRfXy5weQ==) | `61.77% <0%> (-0.45%)` | :arrow_down: |
| [pandas/io/parsers.py](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=tree#diff-cGFuZGFzL2lvL3BhcnNlcnMucHk=) | `95.32% <0%> (-0.33%)` | :arrow_down: |
| [pandas/core/indexes/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZXMucHk=) | `95.23% <0%> (-0.2%)` | :arrow_down: |
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `94.8% <0%> (-0.19%)` | :arrow_down: |
| ... and [5 more](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=footer). Last update [ae70ece...51eab8c](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=h1) Report
> Merging [#16201](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/1002cc339d81591b280ec3ec3c8ac58ce157e3b6?src=pr&el=desc) will **decrease** coverage by `0.62%`.
> The diff coverage is `70%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/16201/graphs/tree.svg?src=pr&width=650&token=eZ4WkYLtcO&height=150)](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #16201      +/-   ##
==========================================
- Coverage   90.86%   90.24%   -0.63%     
==========================================
  Files         162      162              
  Lines       50887    50882       -5     
==========================================
- Hits        46240    45916     -324     
- Misses       4647     4966     +319
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.02% <70%> (-0.63%)` | :arrow_down: |
| #single | `40.31% <50%> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `78.97% <70%> (-0.1%)` | :arrow_down: |
| [pandas/io/formats/excel.py](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvZXhjZWwucHk=) | `73.15% <0%> (-23.49%)` | :arrow_down: |
| [pandas/io/excel.py](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=tree#diff-cGFuZGFzL2lvL2V4Y2VsLnB5) | `61.68% <0%> (-18.94%)` | :arrow_down: |
| [pandas/core/tools/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvdG9vbHMvZGF0ZXRpbWVzLnB5) | `66.94% <0%> (-18.62%)` | :arrow_down: |
| [pandas/core/config.py](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29uZmlnLnB5) | `69.72% <0%> (-18.37%)` | :arrow_down: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `95.75% <0%> (-0.46%)` | :arrow_down: |
| [pandas/compat/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9fX2luaXRfXy5weQ==) | `61.77% <0%> (-0.45%)` | :arrow_down: |
| [pandas/io/parsers.py](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=tree#diff-cGFuZGFzL2lvL3BhcnNlcnMucHk=) | `95.32% <0%> (-0.33%)` | :arrow_down: |
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `94.8% <0%> (-0.19%)` | :arrow_down: |
| ... and [5 more](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=footer). Last update [1002cc3...7bed86a](https://codecov.io/gh/pandas-dev/pandas/pull/16201?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@jreback : Circle is failing (see <a href="https://circleci.com/gh/pandas-dev/pandas/1650#tests/containers/2">here</a>), and I can't reproduce it.  It seems like there is some kind of weird `matplotlib` import failure that shouldn't be happening, but I have no idea why it is happening (can't reproduce locally with and without `matplotlib` installed).

One thought is that we can try using `pytest.importorskip` functionality, but that doesn't explain why Circle is failing in the first place when Travis and Appveyor are not.
not sure. you should try rolling back parts of the change until this work.
@jreback : I managed to patch the errors in Circle, so everything is green now.  PTAL.
@gfyoung I'll have a commit in a minute that fixes those yield warnings. Basically converting

```diff
-    def test_query_with_named_multiindex(self):
-        for parser, engine in product(['pandas'], ENGINES):
-            yield self.check_query_with_named_multiindex, parser, engine
-
-    def check_query_with_named_multiindex(self, parser, engine):
+    @pytest.mark.parametrize('parser', ['pandas'])
+    @pytest.mark.parametrize('engine', ENGINES)
+    def test_query_with_unnamed_multiindex(self, parser, engine):
 ```

In a bunch of places. Mind if I push that here?
@TomAugspurger : Certainly.  Perhaps wait until the tests pass though?
@gfyoung they all passed. Pushing that extra commit now.
@TomAugspurger : Awesome.  Nice that this PR has been able to pick several other things beyond its original purpose (cache bug, `pytest` warnings...) :smile: 
> in another PR can then remove tm.TestCase from the base class of test class where it is clearly not needed

The only thing that `tm.TestCase` contributes is raising on chained assignment (part of `setup_class`).  I presume that is to just discourage tests from being written like this.  Could we just remove that (and just make sure to be vigilant about it when writing tests)?
> The only thing that tm.TestCase contributes is raising on chained assignment (part of setup_class). I presume that is to just discourage tests from being written like this. Could we just remove that (and just make sure to be vigilant about it when writing tests)?

NO, this is the entire point of this. as I said you can remove from everywhere except pandas/tests/frame & pandas/tests/indexing
> and just make sure to be vigilant about it when writing tests

people are NEVER vigilant, assume that they aim to break every single rule :>
But the default for chained assignment is "warning" - would that not surface in the build logs if tests were incorrectly written with chained assignment?
ping when green. I have a PR after this one (moving stuff around).
> But the default for chained assignment is "warning" - would that not surface in the build logs if tests were incorrectly written with chained assignment?

no its not for testing, its raise.
> no its not for testing, its raise.

But that's because you set it that way when you set up each test class.  I meant in general, it's warning by default, hence my question.
@jreback : All green and ready to go.
> But that's because you set it that way when you set up each test class. I meant in general, it's warning by default, hence my question.

but that's exactly the point. if someone writes a *test* that uses chained indexing I WANT it to fail the tests. Sure we can look for it, but its best to automatically catch it.