DOC/CI: Failing documentation build on warnings
State: closed

- [X] closes #24661, xref #24173
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs
name: CI
description: Continuous Integration

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26852?src=pr&el=h1) Report
> Merging [#26852](https://codecov.io/gh/pandas-dev/pandas/pull/26852?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/430f0fd04646a0ab44ccbfbad706dfe22e9fabfe?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26852/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26852?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26852      +/-   ##
==========================================
- Coverage   91.88%   91.87%   -0.01%     
==========================================
  Files         179      179              
  Lines       50696    50696              
==========================================
- Hits        46581    46576       -5     
- Misses       4115     4120       +5
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.46% <ø> (ø)` | :arrow_up: |
| #single | `41.09% <ø> (-0.08%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26852?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26852/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `88.88% <0%> (-11.12%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/26852/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.88% <0%> (-0.12%)` | :arrow_down: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/26852/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `90.84% <0%> (-0.11%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26852?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26852?src=pr&el=footer). Last update [430f0fd...f1fb939](https://codecov.io/gh/pandas-dev/pandas/pull/26852?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26852?src=pr&el=h1) Report
> Merging [#26852](https://codecov.io/gh/pandas-dev/pandas/pull/26852?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/adc656433ed959a9454ee6f90b5b6b1b9b9c3e42?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26852/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26852?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26852      +/-   ##
==========================================
- Coverage   91.87%   91.86%   -0.01%     
==========================================
  Files         180      180              
  Lines       50710    50712       +2     
==========================================
- Hits        46588    46585       -3     
- Misses       4122     4127       +5
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.45% <100%> (ø)` | :arrow_up: |
| #single | `41.1% <75%> (-0.08%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26852?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/dtypes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/26852/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2Jhc2UucHk=) | `100% <100%> (ø)` | :arrow_up: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/26852/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `96.71% <100%> (ø)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26852/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `88.88% <0%> (-11.12%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/26852/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.88% <0%> (-0.12%)` | :arrow_down: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/26852/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `90.84% <0%> (-0.11%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26852?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26852?src=pr&el=footer). Last update [adc6564...708dbb7](https://codecov.io/gh/pandas-dev/pandas/pull/26852?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

It's failing! So it is working :)
Yes, first test passed. Also, it's failing with return code 2, both sphinx and the ipython hack are failing (I test that well locally anyway, but good to see the CI red) :)

pushing some fixes soon, will get the remaining warnings fixed here, so when this is green, we're done :)
Not sure about the clipboard problem, it's not immediate to fix.

I'd merge this PR with the fix proposed in #26700. Or if anyone else wants to give a try to set up `xsel`...

CC: @jreback @jorisvandenbossche 
I am fine with that. We know now that clipboard doesn't work on Azure and so no clipboard tests are running there, so we can open an issue about that instead of keeping a failing example in the docs.
@jreback are you ok with fixing the clipboard problem in the docs by not running the code for now? i.e. adding the changes in #26700 to this PR?

This way we can start validating for warnings in the doc, before new are introduced.
i think we should get them running first
I created #26852/ for the issue.

Will close this then, but I don't think we're being smart here. By the time the issue is fixed we'll probably have introduced several new warnings as it happened last time we spent several days addressing them. Quite demotivating besides the extra work...
broken tests are much worse
we claim to do something but the the tests are not even being run properly

The tests not being run are not related to the documentation build. It's thanks to the docs that we now know they are not run. But now we know it, an open issue is just as good a reminder to fix it as a failing example in the docs.

Marc and I put a lot of effort in getting to this point of a warning-free doc build. The clipboard example is only one left. I think it is important to finally actually check the doc build in CI to prevent we constantly introduce new warnings.
I think we all agree that we need to fix the tests. But the tests will remain skipped whether we validate the warnings in the docs or not. I don't see the gain in blocking here. But anyway, it's just some doc warnings, not a big deal anyway.
Marc, don't dismiss your own work, a correct sphinx build *is* relevant.
@datapythonista i am actually ok merging this
but the next thing that needs to be fixed are these tests
they keep getting deferred 

having a healthy CI is he most important thing 
Thanks Jeff. Agree with prioritizing the CI, will do more tests in a separate PR, but I already spent few hours with the clipboard, and it's not trivial. It'll probably take a while, no idea what's the problem. That's why I think doesn't make sense to block this which is unrelated and I'd say urgent (really don't want to start getting new warnings added again after all the work).

This is passing the CI now, validating that no sphinx or ipython directive warnings are raised during the doc build.
@pandas-dev/pandas-core be aware that once this is merged, warnings in the documentation will leave the CI red.
Thanks for working on this Marc! It’s been quite an effort. 

> On Jun 16, 2019, at 17:00, Marc Garcia <notifications@github.com> wrote:
> 
> @pandas-dev/pandas-core be aware that once this is merged, warnings in the documentation will leave the CI red.
> 
> —
> You are receiving this because you are on a team that was mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.

Let's give this a go!
> @pandas-dev/pandas-core be aware that once this is merged, warnings in the documentation will leave the CI red.

It might also be advisable to be extra careful to merge master in PRs before merging the coming days, certainly for PRs that touch the documentation.
Thanks for the fixup Joris. This failed in master, looks like the scipy documentation is down (we fetch https://docs.scipy.org/doc/numpy/objects.inv for the cross references).

@rgommers not sure if you're aware?
>  looks like the scipy documentation is down 

Yes, noticing this locally as well.
Thanks for the ping. No I didn't notice that yet. Will follow up on that now.
scipy docs are now working, and the doc build is green again.