TST: Use pytest
State: closed

- [x] closes https://github.com/pydata/pandas/issues/13097
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

Just a WIP for now, will need to iterate on the CI stuff to ensure that's all good.

What can I provide to make this easier to review? Refactoring tests is scary.
I'll give verbose outputs of nose on master and pytest on this branch.
junit.xml output too. Anything else?

Here are some notes I took along the way
### Kinds of Changes
- Nosetests Generators to fixtures

See TestPickle, TestMsgPack and [this](http://stackoverflow.com/questions/6689537/nose-test-generators-inside-class) SO post. We switched to using parametrized fixtures (yay) and dependency injection instead of setUp + a for loop inside the test.
- Different Discover Rules

See the sql tests. `py.test` discovers `_Test*` class (if they inherit from `TestCase`?), while unittest does not.
Solution: move TestCase inheritance down a level
- Assert Rewriting

See e.g. pandas/src/testing.py
- solution: rewrite `assert cond, msg` as `if not cond; raise AssertionError(msg)`
- looking into just excluding these files... They aren't actual tests, just test helpers.
- CI work:
  - updating nose to pytest


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration

Comments:
factoring asserts can be in a later PR

pytest will work with nose tools (though we mostly use our own tm.\* ones)

Thanks a lot @TomAugspurger ! 

One note on something above:

> py.test discovers _Test\* class (if they inherit from TestCase?), while unittest does not.

My understanding is that the only way to use pytest fixtures within a class is to _not_ inherit from `TestCase`. This can make it a bit more difficult to gradually transition, but you can use `autouse=True` on existing `setUp` methods, and the inheritance then isn't necessary.

And pytest discovers tests in `Test*` named classes, not `*Test`, unless they inherit from `TestCase`. 

Let me know if that's unclear.

> factoring asserts can be in a later PR

Agreed, the only asserts I've manually rewritten were in `src/testing.pyx` to get the tests to pass. I believe this is due to https://github.com/pytest-dev/pytest/issues/645, which has been fixed and will be included in pytest 3.0 which is coming out soonish. They were targeting Europython, but pushed it back.
I'll revert those changes and wait for pytest3 I think.

@MaximilianR thanks for the heads about `autouse=True`, I'll check that out. My initial goal was to have the test-suite runnable by either nose or pytest runners. Then followup PRs could make use of pytest features.
Doesn't look like I'll be able to achieve that though due to the nose generator tests.

pytest 3 is [out now](http://blog.pytest.org/2016/whats-new-in-pytest-30/). Do we want to make the switch just after 0.19 is released?

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/13856?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@3d6fcdc`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).


```diff
@@            Coverage Diff            @@
##             master   #13856   +/-   ##
=========================================
  Coverage          ?   87.88%           
=========================================
  Files             ?      142           
  Lines             ?    51106           
  Branches          ?        0           
=========================================
  Hits              ?    44913           
  Misses            ?     6193           
  Partials          ?        0
```


| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/13856?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/__init__.py](https://codecov.io/gh/pandas-dev/pandas/compare/3d6fcdcd356b2b1853346bc4e709baa3bf16ddad...1be6ea53ee5018f3ac1f954f8708c5945eb6f79e?src=pr&el=tree#diff-cGFuZGFzL19faW5pdF9fLnB5) | `91.42% <100%> (ø)` | |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/compare/3d6fcdcd356b2b1853346bc4e709baa3bf16ddad...1be6ea53ee5018f3ac1f954f8708c5945eb6f79e?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `82.79% <100%> (ø)` | |
| [pandas/util/_tester.py](https://codecov.io/gh/pandas-dev/pandas/compare/3d6fcdcd356b2b1853346bc4e709baa3bf16ddad...1be6ea53ee5018f3ac1f954f8708c5945eb6f79e?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvX3Rlc3Rlci5weQ==) | `46.15% <46.15%> (ø)` | |
| [pandas/conftest.py](https://codecov.io/gh/pandas-dev/pandas/compare/3d6fcdcd356b2b1853346bc4e709baa3bf16ddad...1be6ea53ee5018f3ac1f954f8708c5945eb6f79e?src=pr&el=tree#diff-cGFuZGFzL2NvbmZ0ZXN0LnB5) | `83.33% <83.33%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/13856?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/13856?src=pr&el=footer). Last update [3d6fcdc...59e2be9](https://codecov.io/gh/pandas-dev/pandas/compare/3d6fcdcd356b2b1853346bc4e709baa3bf16ddad...59e2be94b60ee13fee06bd84a44df0ac679ac0ba?src=pr&el=footer&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
@TomAugspurger is this working? (aside from the core dump in master). 

issue is `pd.test()` which is still using the older code as well

I think nose needs to be removed from the ci/requirements\* files as well

@jreback pretty close I think. The only ones I'm stumped on are the `read_stata` errors. You can see the failures in the last build [here](https://travis-ci.org/pandas-dev/pandas/jobs/174350589#L937), [This build](https://travis-ci.org/pandas-dev/pandas/jobs/173227412#L1080) from pre-segfaults has the tracebacks. I haven't been able to reproduce locally.

Other than that, I just have a bit of cleanup and verification that we didn't drop any tests. I'll do that tonight and on Sunday, hopefully get the merged soon. And the `pd.test` issue.

https://github.com/pandas-dev/pandas/pull/13856/commits/d653086948844c0eae6fad681b0696106756a48d fixed `pd.test()` to use pytest. Are we OK with adding that `pandas/api/test.py` file?

Without pytest:

``` python
>>> import pandas
>>> pandas.test()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/tom.augspurger/Envs/py3/lib/python3.5/site-packages/pandas/pandas/api/test.py", line 13, in test
    raise ImportError("Need pytest>=3.0 to run tests")
ImportError: Need pytest>=3.0 to run tests
```

With pytest (until I canceled it):

``` python
>>> import pandas
>>> pandas.test()
...............^C
```

I'd like to do the complete removal of nose (skips and generator-based tests for the most part) as a followup issue.

@bashtage, do you have any guesses on what's causing [this failure](https://travis-ci.org/pandas-dev/pandas/jobs/175935339#L1141) (scroll down a bit for the traceback)? The actual error comes from trying to lookup `StataReader.DTYPE_MAP[0]`, which throws a KeyError (that we catch and re-raise).  AFAICT, the `0` is coming from a null byte `ord('\x00') == 0`.

I suspect the problem is at a higher-level, since this passes with the nose runner, and it passes on other parts of our build matrix with pytest: https://travis-ci.org/pandas-dev/pandas/builds/175935337 Any hunches would be appreciated, but no worries if you're stumped.

I don't recall having any issues there, so nothing obvious. 

@TomAugspurger I have found the problem with `StataWriter`.  Date conversion is not robust to locale changes, and so when the file date is written, instead of `24 Jan 2017 12:10`, `24 ???? 2017 12:10` is being writen, where ???? is some unicode that I can't quite decipher.  Because this is inserting an extra byte, the rest of the header conversion is off. 

Will need to either use a custom date converter for the month or to temporarily set the locale when writing stata files. Former is probably easier although maybe uglier.
@bashtage thanks! One of the other failing tests is locale related as well. Haven't been able to determine if it's an actual bug in pandas, or just something wrong with how we're setting up the test that pytest doesn't like.
@TomAugspurger The reason why you found this is that you ran all tests with `zh_CN`.  In nose testing this config only runs `slow`, and none of `test_stata` are slow, and so this bug was allowed to persist.  I assume the other errors have occurred for the same reason. 
@bashtage hmm, https://travis-ci.org/pandas-dev/pandas/jobs/194558929

DOES run all tests with a locale (IT and only on 2.7)
@TomAugspurger if its easier (not sure), you can certainly make an additional build that runs py.test, of course then these changes need to be compat with ``nose`` . not sure if this possible / desirable. Then could merge and iterate rather than switch over all at once.
> @bashtage hmm, https://travis-ci.org/pandas-dev/pandas/jobs/194558929
> DOES run all tests with a locale (IT and only on 2.7)

But the short *month* for Italian is 3 non-unicode characters. For Chinese it is Unicode, which is why when writing the date as a string, Chinese requires an extra byte while Italian doesn't.  I suspect that Chinese is a somewhat harder language to pass w.r.t. locale issues than most western languages.

_edit_
so does this make sense to flip the locale for these tests? (e.g. not-slow tests with chinese, e.g. more unicode), and IT for slow?
I would think so.  Should be OK since the other locale testing is allowed failure. 
> In nose testing this config only runs slow,

Oh I guess I misunderstood our NOSE_ARGS. `slow` means *only* slow, not not all?
I can add another pytest option `--only-slow` to replicate that, if we want.
@bashtage 

> I would think so.  Should be OK since the other locale testing is allowed failure.

no ``allowed failure`` is simply to order the builds. these shouldn't actually fail (its just easier to have the 5 main builds run first and give feedback).
@TomAugspurger yes we have essentially split tests, e.g. slow/non-slow. I think replicating that is a good ideal.

in fact we can eventually define more markers to split things further at some point
> no allowed failure is simply to order the builds. these shouldn't actually fail (its just easier to have the 5 main builds run first and give feedback).

Then if you switched them master would be failing. 

It would probably make the most sense to run `zh_CN` with both `slow` and `not slow` configs so that all of pandas have to pass a more difficult locale. 
https://github.com/pandas-dev/pandas/pull/15211

though its on the 34_notslow

we try to cover everything.......but you know how that goes!
Have you tried using pytest-xdist on travis, 

```
pip install pytest-xdist 
pytest n auto pandas`
```

Seeing nice speedup on slow statsmodels runs (18-20 min down to 11-13).  Travis instances have 1.5 vCPUs.
> Have you tried using pytest-xdist on travis,

Will try that as soon as I get the basic version passing. Down to one failure on a single job: https://travis-ci.org/pandas-dev/pandas/jobs/197031785#L1445 a locale test on `27_nslow`

Any guess on why that is, before I dig into it? That job doesn't have any custom locale settings (and it passes fine locally).

Looking into the appveyor failures now.
@TomAugspurger you can simulate the locale locally easily

put in ``pandas/__init__.py``

```
import locale
locale.setlocale(locale.LC_ALL, 'zh_CN.UTF-8')
```

@TomAugspurger all passing! that's great! and of course love the ``pytest`` output.

I would remove ``-s`` from the invocation (only on travis), as we sometimes have weird things being printed, though not *too* much.

I'll have a look soon at the changes.

did you fix ``pd.test()``?
🎉 

> did you fix pd.test()?

I think so, but need to do a bit more testing here. I *think* that the appveyor testing proves that I have things setup correctly, since we `cd` to the root (outside of the pandas repo), and `pd.test()` is still able to find the `conftest.py` correctly, since we use the `--skip-slow --skip-network` options. AFAIK, we don't have to change the setup.py or anything, since it's just a regular python file inside the package.
some people use ``pytest --pyargs pandas`` and run from the outside (though I don't find that intuitve)
i have never have a problem putting conftest in the root directory - 

an entry point is for a separate program not sure that is needed here