DOC: Use official numpydoc extension
State: closed

 - Replace custom numpydoc in `doc/sphinxext/numpydoc` with official
   numpydoc release
 - Remove `numpydoc_use_blockquotes` parameter

Labels:
name: Docs

Comments:
Hello @FHaase! Thanks for updating the PR.

 - There are no PEP8 issues in the file [`doc/source/conf.py`](https://github.com/FHaase/pandas/blob/afeb025ebbeb5e3d4545f18f4c4ade1a4f433302/doc/source/conf.py) !

 - There are no PEP8 issues in the file [`pandas/core/ops.py`](https://github.com/FHaase/pandas/blob/afeb025ebbeb5e3d4545f18f4c4ade1a4f433302/pandas/core/ops.py) !



##### Comment last updated on December 14, 2018 at 18:25 Hours UTC
@datapythonista is this what you meant in #23763 with:

> avoid that numpydoc parameters are rendered inside a \<blockquote\> tag, breaking the formatting


Before:
![numpydoc-before](https://user-images.githubusercontent.com/41163037/49467326-7fa5de80-f802-11e8-9c7f-f79c621330d5.png)

After:
![numpydoc-after](https://user-images.githubusercontent.com/41163037/49467325-7f0d4800-f802-11e8-8754-5b38bba84610.png)




Some xref issues for context https://github.com/pandas-dev/pandas/pull/11257, https://github.com/pandas-dev/pandas/pull/6003, https://github.com/numpy/numpydoc/pull/106#issuecomment-343220139.

@FHaase can you verify that those docstrings (e.g. Index.str) are rendering correctly?
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24098?src=pr&el=h1) Report
> Merging [#24098](https://codecov.io/gh/pandas-dev/pandas/pull/24098?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/1d3ed917638b9890e0410fa996a447c4a780429e?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24098/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24098?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #24098   +/-   ##
=======================================
  Coverage   42.52%   42.52%           
=======================================
  Files         161      161           
  Lines       51697    51697           
=======================================
  Hits        21982    21982           
  Misses      29715    29715
```

| Flag | Coverage Δ | |
|---|---|---|
| #single | `42.52% <ø> (ø)` | :arrow_up: |


------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24098?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24098?src=pr&el=footer). Last update [1d3ed91...b39f68e](https://codecov.io/gh/pandas-dev/pandas/pull/24098?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24098?src=pr&el=h1) Report
> Merging [#24098](https://codecov.io/gh/pandas-dev/pandas/pull/24098?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/040f06f731a09fc6e0663cada6697f6602b36f1d?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24098/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24098?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #24098   +/-   ##
=======================================
  Coverage   92.22%   92.22%           
=======================================
  Files         162      162           
  Lines       51828    51828           
=======================================
  Hits        47798    47798           
  Misses       4030     4030
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.62% <ø> (ø)` | :arrow_up: |
| #single | `43% <ø> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24098?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/ops.py](https://codecov.io/gh/pandas-dev/pandas/pull/24098/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvb3BzLnB5) | `94.26% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24098?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24098?src=pr&el=footer). Last update [040f06f...afeb025](https://codecov.io/gh/pandas-dev/pandas/pull/24098?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

`Index.str` renders identical with release version of [0.23.4](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Index.str.html)

We had an option to use a `member_list` over `param_list` to display the _Attribute section_ which made
the attribute section look like the _Methods section_ rather than the _Parameter section_.

The option is removed in favor of having this behavior always. So [pandas.Index](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Index.html) renders **unchanged**.
I don't understand this PR. Removing our copy of `numpydoc` and importing it as a dependency sounds good (not sure why was copied, I understand from the linked issues that it was because py3 compatibility?)

But what you're doing is have it as a dependency, and create an intermediate module to monkeypatch it? What is the advantage?
Regarding the blockquotes, the html wraps lists with a `<blockquote>`, because a list is indented, and indented code is used for a quote too. With the current style is not visible (I think we changed the css or something), but with the new theme I'm implementing the lists are rendered with a different font (because of the blockquote). See here:

![bloquote](https://user-images.githubusercontent.com/10058240/49487653-59e8fb80-f83b-11e8-8fb8-9dbe669cf19e.png)

> But what you're doing is have it as a dependency, and create an intermediate module to monkeypatch it? What is the advantage?

Moved the patching to `conf.py`. Didn't know that replacing a function has effect on a global level.
> Regarding the blockquotes, the html wraps lists with a \<blockquote\>, because a list is indented, and indented code is used for a quote too. With the current style is not visible (I think we changed the css or something), but with the new theme I'm implementing the lists are rendered with a different font (because of the blockquote).


@datapythonista I've compared the text in `io.rst` starting at line 1880 with the docutils example. When I indent by 2 spaces instead of 4 the \<blockquote\> doesn't get generated anymore. Can you confirm this?

```
If the JSON serializer cannot handle the container contents directly it will
fall back in the following manner:

* if the dtype is unsupported (e.g. ``np.complex``) then the ``default_handler``, if provided, will be called
  for each value, otherwise an exception is raised.

* if an object is unsupported it will attempt the following:


  * check if the object has defined a ``toDict`` method and call it.
    A ``toDict`` method should return a ``dict`` which will then be JSON serialized.

  * invoke the ``default_handler`` if one was provided.

  * convert the object to a ``dict`` by traversing its contents. However this will often fail
    with an ``OverflowError`` or give unexpected results.

```
I think the reStructuredText standard requires no indentation for first level lists, and 4 for sublists. We had some problems with the rendering in the past, you can take a look at #21520 and referenced issues, I don't remember the details.
![with-blockquote](https://user-images.githubusercontent.com/41163037/49591605-2d360080-f96f-11e8-9c79-297902507580.png)

```html
<ul>
<li><p class="first">if the dtype is unsupported (e.g. <code class="docutils literal notranslate"><span class="pre">np.complex</span></code>) then the <code class="docutils literal notranslate"><span class="pre">default_handler</span></code>, if provided, will be called
for each value, otherwise an exception is raised.</p>
</li>
<li><p class="first">if an object is unsupported it will attempt the following:</p>
<blockquote>
<div><ul class="simple">
<li>check if the object has defined a <code class="docutils literal notranslate"><span class="pre">toDict</span></code> method and call it.
A <code class="docutils literal notranslate"><span class="pre">toDict</span></code> method should return a <code class="docutils literal notranslate"><span class="pre">dict</span></code> which will then be JSON serialized.</li>
<li>invoke the <code class="docutils literal notranslate"><span class="pre">default_handler</span></code> if one was provided.</li>
<li>convert the object to a <code class="docutils literal notranslate"><span class="pre">dict</span></code> by traversing its contents. However this will often fail
with an <code class="docutils literal notranslate"><span class="pre">OverflowError</span></code> or give unexpected results.</li>
</ul>
</div></blockquote>
</li>
</ul>
```

vs

![without-blockquote](https://user-images.githubusercontent.com/41163037/49591625-3a52ef80-f96f-11e8-99da-a2f4423c6b07.png)
```html
<ul class="simple">
<li>if the dtype is unsupported (e.g. <code class="docutils literal notranslate"><span class="pre">np.complex</span></code>) then the <code class="docutils literal notranslate"><span class="pre">default_handler</span></code>, if provided, will be called
for each value, otherwise an exception is raised.</li>
<li>if an object is unsupported it will attempt the following:<ul>
<li>check if the object has defined a <code class="docutils literal notranslate"><span class="pre">toDict</span></code> method and call it.
A <code class="docutils literal notranslate"><span class="pre">toDict</span></code> method should return a <code class="docutils literal notranslate"><span class="pre">dict</span></code> which will then be JSON serialized.</li>
<li>invoke the <code class="docutils literal notranslate"><span class="pre">default_handler</span></code> if one was provided.</li>
<li>convert the object to a <code class="docutils literal notranslate"><span class="pre">dict</span></code> by traversing its contents. However this will often fail
with an <code class="docutils literal notranslate"><span class="pre">OverflowError</span></code> or give unexpected results.</li>
</ul>
</li>
</ul>
```
From [A ReStructuredText Primer](http://docutils.sourceforge.net/docs/user/rst/quickstart.html) at `bulleted list`

```
* a bullet point using "*"

  - a sub-list using "-"

    + yet another sub-list

  - another item
```

From [Sphinx Documentation](http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html)

```
* this is
* a list

  * with a nested list
  * and some subitems

* and here the parent list continues
```

Both use 2 spaces as indent.
See #21518 (the problem I fixed at that time was that some top level lists were indented). Not sure where I checked the standard, but for what I saw, while sphinx is flexible, the standard is 4 spaces, and other software may fail to render the lists with a different number.
You mentioned `pandoc` in that issue, looking at the documentation pandoc uses 2 spaces as well.

```
* fruits
  + apples
    - macintosh
    - red delicious
  + pears
  + peaches
* vegetables
  + broccoli
  + chard
```

I don't know from where you took 4 spaces as indentation for sublists, but docutils, sphinx, pandoc, ...
all seem to use 2.

pandoc uses 4 spaces for verbatim text:
> A block of text indented four spaces (or one tab) is treated as verbatim text: that is, special characters do not trigger special formatting, and all spaces and line breaks are preserved. For example, ...
 
cc @jorisvandenbossche.

I haven't had a chance to look at this yet. Will try to next week.
I am certainly fine with monkeypatching upstream numpydoc, instead of vendoring the full package as we do now. That makes it a bit more explicit which hacks we added / where the changes are compared to upstream.

I think in the past we had more hacks than just the attributes listing, but it might be that the others already have been solved in the meantime. 
I quickly looked at the build log, and we have a whole bunch of warnings related to numpydoc / autodoc, but we also have them on master, so this PR does not seem to add new warnings.
Reading through https://github.com/pandas-dev/pandas/pull/20383, it seems I have updated our vendored version to the upstream one earlier this year, with the attribute listing as the only 'hack' afterwards. And in that way removing the patches we had previously (https://github.com/pandas-dev/pandas/pull/11257)

There is a note there (https://github.com/pandas-dev/pandas/pull/20383#issue-175581885) about the `numpydoc_use_blockquotes` and `**kwargs` in the docstring. @FHaase this is still working now? (as I see you removed that setting; I don't really remember directly why it was needed at the time) 


> We had an option to use a member_list over param_list to display the Attribute section which made
the attribute section look like the Methods section rather than the Parameter section.
>
> The option is removed in favor of having this behavior always.

@FHaase yes, fine to remove that. I originally did it like that in the hope to push that change upstream (for which there is a PR: https://github.com/numpy/numpydoc/pull/161)
@jorisvandenbossche over to you; merge when satisfied
There are two pictures added above. With a before and after comparison. 
They feature a *args and **kwargs in parameters section. Which does look fine to me. 

The vendored version was 0.8.0.dev0 (something like that) now it's 0.8.0. I guess they fixed all those issues for the release version. 
Patched numpy/numpydoc#150 into `SphinxDocString`.
Running ` python make.py html --single api` doesn't generate any `WARNING: Inline strong start-string without end-string.`

Happened those issues were cut off after all the Warnings regarding the toctree were shown.
@FHaase you have one linting error: https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=5106


lgtm. can you merge master and ping on green. any issues closed by this?
Ping @jreback 
https://travis-ci.org/pandas-dev/pandas/jobs/468121566

this failed building docs, and *didn't* fail the jobs: @datapythonista 
I saw that couple of days ago working in some fixes to the doc build. We're using `os.system()` to call `sphinx-build`, and it returns the status code instead of raising an exception when the called command fails.

That will be fixed in the changes I'm doing, hopefully I'll be able to open a PR very soon, just need to fix some problems I'm getting with sphinx-autogen.
ok, but it appears our docs are completely broken now. so let's fix that before merging anything else.
I thought the failure was in this PR. Let me take a look then.
http://pandas-docs.github.io/pandas-docs-travis/
locally it fails because of the `enchant` (spellcheck) thing. Not sure if that's the only problem, as the error in the CI is different, but I'll get rid of that first, as it's something I wanted to do anyway (it can't be used, as the dependencies can't be installed)