DOC/TST: Validate documentation pages
State: closed

- [x] closes #21520
- [ ] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Docs
name: CI
description: Continuous Integration

Comments:
Hello @FHaase! Thanks for updating the PR.

 - There are no PEP8 issues in the file [`doc/make.py`](https://github.com/FHaase/pandas/blob/93974fb7dce50a9bdc2233d544e17d00f45ed43d/doc/make.py) !

 - There are no PEP8 issues in the file [`scripts/tests/test_validate_documentation.py`](https://github.com/FHaase/pandas/blob/93974fb7dce50a9bdc2233d544e17d00f45ed43d/scripts/tests/test_validate_documentation.py) !

 - There are no PEP8 issues in the file [`scripts/validate_documentation.py`](https://github.com/FHaase/pandas/blob/93974fb7dce50a9bdc2233d544e17d00f45ed43d/scripts/validate_documentation.py) !



##### Comment last updated on December 10, 2018 at 23:59 Hours UTC
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24216?src=pr&el=h1) Report
> Merging [#24216](https://codecov.io/gh/pandas-dev/pandas/pull/24216?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/91802fb0accde031c3b6aca040a8b533a193fef6?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24216/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24216?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #24216   +/-   ##
=======================================
  Coverage   92.21%   92.21%           
=======================================
  Files         162      162           
  Lines       51763    51763           
=======================================
  Hits        47733    47733           
  Misses       4030     4030
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.61% <ø> (ø)` | :arrow_up: |
| #single | `43% <ø> (ø)` | :arrow_up: |


------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24216?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24216?src=pr&el=footer). Last update [91802fb...5e7bca4](https://codecov.io/gh/pandas-dev/pandas/pull/24216?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24216?src=pr&el=h1) Report
> Merging [#24216](https://codecov.io/gh/pandas-dev/pandas/pull/24216?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/91802fb0accde031c3b6aca040a8b533a193fef6?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24216/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24216?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #24216      +/-   ##
==========================================
+ Coverage   92.21%   92.21%   +<.01%     
==========================================
  Files         162      162              
  Lines       51763    51763              
==========================================
+ Hits        47733    47734       +1     
+ Misses       4030     4029       -1
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.61% <ø> (ø)` | :arrow_up: |
| #single | `43.01% <ø> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24216?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/24216/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `87.41% <0%> (-0.1%)` | :arrow_down: |
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/24216/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `96.65% <0%> (ø)` | :arrow_up: |
| [pandas/io/json/json.py](https://codecov.io/gh/pandas-dev/pandas/pull/24216/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2pzb24vanNvbi5weQ==) | `93.09% <0%> (+0.47%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24216?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24216?src=pr&el=footer). Last update [91802fb...f21a465](https://codecov.io/gh/pandas-dev/pandas/pull/24216?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

I've added the checks to `./ci/build_docs.sh` as the `doctrees` need to exist when running the script.
@datapythonista The ci integration is in `./ci/run_tests.sh` and generates errors in ['JOB="3.6, doc"'](https://travis-ci.org/pandas-dev/pandas/jobs/466799434)
Nice, lots of stuff to fix :)

I think you mention it somewhere, but I can't find it now. What's the reason to run in `run_tests.sh` instead of `code_checks.py`? can't be validate the `.rst` files directly?

And why adding it to `make.py`?
I opened #24281 to fix the tailing whitespace problems. We can create a separate issue to fix the lists, but I guess that will be harder to automate and we may need to do it manually.

Btw, you can see how lists are rendered with the new theme here: https://pandas-dev.github.io/pandas-sphinx-theme/pr-datapythonista_base/generated/pandas.DataFrame.head.html#pandas.DataFrame.head

I guess this won't fix the problem with the `Parameters` section. But you can check an example with 2 spaces, and one with 4, and see if they are rendered correctly in that url.
Adding it to `make.py` outputs the issues when running `make.py html --single <something>` at the end.
So if building one page those issues are easily seen. Running without `--single <something>` does **not** output something.
Basically I assumed that when building one page the next step would be to validate the issues anyway. I found this more convenient than having to type 2 commands and having to watch out that the page was build before by sphinx.

The last thing is also the reason why it could not be implemented in `./ci/code_checks.sh ` as that script is run on circle ci **without** generating the doc first. In order to check whether a bullet_list is within a block_quote, I load the `*.doctree` file in `./doc/build/doctrees` which exists only after the documentation was build.
The advantage is that all kinds of constraints to the doctree can be checked.

> I guess this won't fix the problem with the Parameters section. But you can check an example with 2 spaces, and one with 4, and see if they are rendered correctly in that url.

I don't really check the number of spaces in the file, but literally whether there is a `<blockquote>` generated around a `<ul>`.
BTW, as after the documentation build there are also all the docstrings generated for the classes functions, 
Those constraints get applied to those as well.

A possible solution for #22900 would be to run flake8-rst on the `docs/source/generated` files. I think that should be sufficient.
I see. My concern is in not making things too complex. There are lots of scripts in pandas, a lot of complexity in building the documentation... Linting the docs calling a script from `ci/code_checks.py` is very easy to understand and to keep under control. And it works in the same way as `validate_docstrings.py`, so when you know how one works, you know about the other.

Having something similar, but importing `scripts/validate_documentation.py` from `docs/make.py`, and having to call in the docs build instead of the `ci/code_checks.py`, make things more complex to learn, to remember and to maintain.

If it needs to run after (or during) the docs build. Would make sense to have this implemented as a sphinx extension? And I think it's generic enough to be independent of pandas and be used by other projects too. What do you think? Does it make sense?
I added validation for tailing whitespaces in all files (not only documentation) in #24286. I think that may simplify the work here, and it was trivial to validate with a grep there.
I don't think we'll merge this like it is. Not sure if that's too much complexity for the value it adds. But in any case, if we follow this approach, we should implement this as a sphinx extension, probably in a different package and install it via conda.