DOC/CI: run doctests on travis
State: closed

Trying to see if we can check the doctests on travis. We were already running them, but at this point they were not failing travis if they failed.

Closes https://github.com/pandas-dev/pandas/issues/19934 
Closes https://github.com/pandas-dev/pandas/issues/3439

Labels:
name: Docs
name: CI
description: Continuous Integration

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/19952?src=pr&el=h1) Report
> Merging [#19952](https://codecov.io/gh/pandas-dev/pandas/pull/19952?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/b5d81cfe43eeccfc3641aa9578097f726da9ce9d?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/19952/graphs/tree.svg?token=eZ4WkYLtcO&src=pr&height=150&width=650)](https://codecov.io/gh/pandas-dev/pandas/pull/19952?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #19952   +/-   ##
=======================================
  Coverage   92.05%   92.05%           
=======================================
  Files         169      169           
  Lines       50709    50709           
=======================================
  Hits        46679    46679           
  Misses       4030     4030
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.46% <ø> (ø)` | :arrow_up: |
| #single | `42.25% <ø> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/19952?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/19952/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `96.44% <ø> (ø)` | :arrow_up: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/19952/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.24% <ø> (ø)` | :arrow_up: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/19952/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `85.85% <ø> (ø)` | :arrow_up: |
| [pandas/core/config.py](https://codecov.io/gh/pandas-dev/pandas/pull/19952/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29uZmlnLnB5) | `87.17% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/19952?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/19952?src=pr&el=footer). Last update [b5d81cf...25bc158](https://codecov.io/gh/pandas-dev/pandas/pull/19952?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

There were failures https://travis-ci.org/pandas-dev/pandas/jobs/347769808#L6516, but they build still passed.
Hmm, that is because the script is still called itself from another script that does `exit 0`. Added a commit to try to progagate this, but not sure that will work with this 'wait' stuff.

On the other hand, it is maybe better to move this to a separate script anyhow.
OK, this is working now, in the sense that travis is failing because of the failing doctests.

The question is now how to list the ones we want to test:

- manually list the files we want to test (and for some files, manually list the functions we want to test)
- or run all the doctests, and manually (for now) list the ones we want to skip

Checking how it would look like with the second way

Hmm, so running the doctests on travis gives segfaults for some of them (don't have this locally):

```
...
pandas/core/frame.py::pandas.core.frame.DataFrame.count PASSED           [  4%]
pandas/core/frame.py::pandas.core.frame.DataFrame.cov PASSED             [  4%]
pandas/core/frame.py::pandas.core.frame.DataFrame.cummax *** Error in `/home/travis/miniconda3/envs/pandas/bin/python': munmap_chunk(): invalid pointer: 0x00000000027217b0 ***
ci/doctests.sh: line 57:  5790 Aborted                 (core dumped) pytest --doctest-modules --ignore=pandas/tests -v pandas
```

Somebody any idea?
So this still has a segfault when running the doctests.

I cannot reproduce it locally with pytest (`pytest --doctest-modules --ignore=pandas/tests -v pandas/core/frame.py`). 
But I actually can reproduce it when calling it more manually

```
import pandas as pd
import numpy as np
import doctest
doctest.testmod(pd.core.frame, globs={'pd': pd, 'np': np})
```

@jreback @TomAugspurger @datapythonista @WillAyd any comments? 

This enable doctests on travis, and for now runs tests for frame/generic/series.py. 
I took the approach of skipping those docstrings that don't work yet. We can then do follow-up PRs gradually fixing the others + removing its skip, but this at least ensures the others keep working or newly added will work.


I think it'll be great to have this, but it'll take a while until we have all doctests passing (it's in my TODO list, together with fixing all warnings from sphinx).

I'm happy to merge this PR as it is, and keep adding docstrings and files as we fix the tests.
> Is there a doc about this in contributing? (certainly can do in later PRs), in particular formatting gotchas? Also list a couple of gold-star examples (in our current docs) that show good contstruction of these tests, so we can simply point to them and say, like this!

Yes, we have a very elaborate docstring guide: http://pandas.pydata.org/pandas-docs/stable/contributing_docstring.html#docstring, which also contains some tips to get the doctests passing

(it might need some better linking in the contributing docs though)