DOC: Use nbsphinx
State: closed

Update header levels for nbsphinx

Link to nb, nicer default table

Closes #15539 

Labels:
name: Docs

Comments:
Here's a preview

![screen shot 2017-03-05 at 12 28 50 pm](https://cloud.githubusercontent.com/assets/1312546/23590087/573c8260-019f-11e7-982e-12caa84ae238.png)

The doc build may fail to convert the notebook if we don't have pandoc on the CI server. It's available through conda-forge if we need, but wanted to test without first.
Why does it need pandoc?

You also included two versions of the notebook, was that on purpose? 
I also don't see where you included it in any toctree.

But seems to look much nicer!
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15581?src=pr&el=h1) Report
> Merging [#15581](https://codecov.io/gh/pandas-dev/pandas/pull/15581?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/d60f490b4e764ab9af170adf4d88639014804fe6?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/15581/graphs/tree.svg?token=eZ4WkYLtcO&src=pr&width=650&height=150)](https://codecov.io/gh/pandas-dev/pandas/pull/15581?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #15581   +/-   ##
=======================================
  Coverage   90.99%   90.99%           
=======================================
  Files         145      145           
  Lines       49565    49565           
=======================================
  Hits        45101    45101           
  Misses       4464     4464
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.76% <ø> (ø)` | :arrow_up: |
| #single | `40.56% <ø> (ø)` | :arrow_up: |


------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15581?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15581?src=pr&el=footer). Last update [d60f490...a2f3056](https://codecov.io/gh/pandas-dev/pandas/pull/15581?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
> Why does it need pandoc?

Oh, I think it's the same as nbconvert, so nothing new. 

> You also included two versions of the notebook, was that on purpose?

Nope, fixed.

> I also don't see where you included it in any toctree.

We had the dummy `style.rst` file that just included the converted html in a raw block. I removed `style.rst` and renamed the notebook to `style.ipynb`, so it gets picked up by the `style` already in the `index.rst` toctree.
@TomAugspurger you need to add the install of ``nbsphinx`` in ``ci/requirements-3.5_DOC_BUILD.sh`` (you can add as same line as feather-format). That's the only place we explicitly install from conda-forge.
Fixed, thanks. Also added seaborn to the doc build (used in the notebook). I set the option to allow errors when executing the notebook, as that seems more consistent with the rest of the doc build.
@TomAugspurger doc build is failing on travis due to missing pandoc. So apparently this was previously not needed.
> @TomAugspurger doc build is failing on travis due to missing pandoc. So apparently this was previously not needed.

I'd rather not have pandoc as a required dependency to build the docs. I'll see what I can do to make this fail gracefully without pandoc installed.
Previously, we were using nbconvert to just convert it to html, which did not needed pandoc. But with nbsphinx I suppose this converts it to rst under the hood and therefore uses pandoc? (not sure if that is a correct assessment, the docs of nbsphinx are not very clear on the pandoc requirement in their installation notes) 
But, if that is correct, and if we want to include this in our dev doc build, then we just need to install pandoc? (you can add it to the conda-forge packages to install)

But indeed, it would be nice that this would not fail the doc build if you do that locally without having pandoc installed. EDIT: ah, this last thing was the point you were making I suppose :-)
this lgtm. @jorisvandenbossche had some comments though.
Oh, whoops, forgot I had already made a merge request. I'll let you know when this is ready for review.

Quick update though: I raised an issue on nbsphinx about gracefully failing if there's no pandoc. I think they're open, but 1.) they have a stalled PR removing the need for pandoc entirely, and 2.) it's a bit tricky since the errors are raised on each markdown cell (do you skip the cell, or the entire nb?)

So I think it's probably best we just workaround it for now by removing the ipynb file(s) before calling out to sphinx. Also, should we do something like my `maybe_exclude_notebooks` for the regular `.rst` files? It can be a bit strange for a new contributor to have a bunch of files removed when you tell them to use `--single=`.
Are people OK with using `requirements_all.txt` to contain all the optional runtime deps to use pandas, and `requirements_dev.txt` to contain all the optional deps to build docs and run tests?

Pip let's you source additional requirements files from another like so

```
# ci/requirements_dev.txt
# All additional dependencies required for running tests, building docs
# Main requirements are in ci/requirements_all.txt
-r requirements_all.txt
# Building
cython
# Testing
flake8
pytest
pytest-cov
pytest-xdist
# Doc building
sphinx
nbsphinx
nbconvert
ipython
seaborn
```

Does anyone know if this is possible with conda? If not I'll open an issue requesting it.
Reading a bit more, it seems like `requirements_dev.txt` was intended to be a minimal set of dependencies to develop pandas. I think it's a bit strange the doc dependencies (at least sphinx) aren't in there, but the test dependencies are.

One additional option is

- `ci/requirements_base.txt`: Required deps only (plus cython?)
- `ci/requirements_dev.txt`: Test and doc deps (and cython if it's not in base)
- `ci/requirements_all.txt`: all optional deps

And then the conda install instructions are

```bash
conda env create -n pandas_dev python=3 -f ci/requirements_base.txt
conda install -n pandas_dev --file=ci/requirements_all.txt
conda install -n pandas_dev --file=ci/requirements_dev.txt
[source] activate pandas_dev
```
> ci/requirements_dev.txt: Test and doc deps (and cython if it's not in base)

should also have a requirements_docs.txt I think (with only the doc-stuff).

Looks like conda is [close](https://github.com/conda/conda/pull/3969) to allowing `-r` syntax in a `requirements.txt`. I'm going to just add nbsphinx to `requirements_all.txt` for now, and open another issue for reorganizing the `requirements_*.txt` files.
@jorisvandenbossche 

> But with nbsphinx I suppose this converts it to rst under the hood and therefore uses pandoc? (not sure if that is a correct assessment, the docs of nbsphinx are not very clear on the pandoc requirement in their installation notes).

Yes, this is correct. At least until https://github.com/spatialaudio/nbsphinx/issues/36 is done (which is a biggie).

You are also right that the documentation doesn't mention that, but we had at least an issue for that: https://github.com/spatialaudio/nbsphinx/issues/72.
I just created a PR which hopefully closes this issue: https://github.com/spatialaudio/nbsphinx/pull/102.

Looking forward to Jupyter notebooks in the `pandas` docs!
@TomAugspurger how's this looking?
I think it's good now. My last change removed some warnings thrown by IPython / pygments about
not being able to find the ipython lexer.

```diff
+              'IPython.sphinxext.ipython_console_highlighting',  # lowercase didn't work
```

We already had `ipython.sphinxext.ipython_console_highlighting` in there (lowercase ipython), but that apparently wasn't working. I have both there now, but we can probably remove the lowercased version?
Ideally, if IPython is installed properly, you wouldn't need to mention `ipython_console_highlighting` at all. Sadly, there seems to be a bug in Anaconda: https://github.com/ContinuumIO/anaconda-issues/issues/1430 (see also https://github.com/spatialaudio/nbsphinx/issues/24). I don't know if this is still broken or not.

And yes, the module name is (unfortunately) `IPython`, AFAIK there is no module called `ipython` (but it *should* be named like this).
@TomAugspurger ok lmk. going to merge a bunch of things one after the other (because of annoying cache deleting once I merge the travis dedupe reversal).
can you rebase (happy to push this in as well if you are ready)
Rebased.

> What was the reason you needed this?

Otherwise we got a bunch of warnings about the IPython lever not being
installed.
Ideally, we wouldn't need it but there's an issue with how anaconda
installs the notebook [apparently](
https://github.com/spatialaudio/nbsphinx/issues/24).

On Wed, Mar 29, 2017 at 2:38 PM, Jeff Reback <notifications@github.com>
wrote:

> can you rebase (happy to push this in as well if you are ready)
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/15581#issuecomment-290201777>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIj1JBV4z31ZJdGmrnmk1rHtrhTRdks5rqrMxgaJpZM4MTfzT>
> .
>

@TomAugspurger merge? (minor comment on the installation of libs).
ready for merge?
Going to wait a day or two to see if https://github.com/pandas-dev/pandas/pull/14757 is updated, since there will be a notebook conflict.
sure
nice @TomAugspurger couple of comments on the built docs: http://pandas-docs.github.io/pandas-docs-travis/style.html

- maybe need some ``:okwarning:`` to eliminate some of the displayed warnings (or slight code changes)
- can you align the numbers in the tables (maybe make this the first table)
The [documentation built with nbsphinx](http://pandas-docs.github.io/pandas-docs-travis/style.html) uses a different font to the rest of the docs, i.e. a serif font.
Whoops, that's on me. Actually comes from my #15954 PR. I'll put up a fix. Thanks.