DOC: Splitting api.rst in several files
State: closed

- [X] closes #24451
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

@TomAugspurger do you know why moving the api `autosummary` to a directory, make the build not find the generated files? For what I understand, the `:toctree: generated/` is used for both, to know where to generate the API files, and later on to link them. But it looks like they are generated in `generated/`, but then I get warnings because sphinx is looking for them in `api/generated/`.

There was something a bit tricky in the `autosummary_generate` option that I removed in #24428 (it's not yet merged), but I don't see anything in the configuration that can be causing this behavior (and I've seen other projects using autosummaries in directories without problems, like https://github.com/bradgwest/esof523a3/tree/master/doc/reference).

Labels:
name: Docs

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24462?src=pr&el=h1) Report
> Merging [#24462](https://codecov.io/gh/pandas-dev/pandas/pull/24462?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/aa1549f7fd8ce81bf23296bf92fe9713aa8d6e20?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24462/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24462?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #24462      +/-   ##
==========================================
- Coverage    92.3%   92.29%   -0.01%     
==========================================
  Files         163      163              
  Lines       51969    51969              
==========================================
- Hits        47968    47967       -1     
- Misses       4001     4002       +1
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.7% <ø> (ø)` | :arrow_up: |
| #single | `43.01% <ø> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24462?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/24462/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `87.75% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24462?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24462?src=pr&el=footer). Last update [aa1549f...fc99580](https://codecov.io/gh/pandas-dev/pandas/pull/24462?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24462?src=pr&el=h1) Report
> Merging [#24462](https://codecov.io/gh/pandas-dev/pandas/pull/24462?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/d5e5bf761092c59eeb9b8750f05f2bc29fb45927?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24462/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24462?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #24462   +/-   ##
=======================================
  Coverage   92.31%   92.31%           
=======================================
  Files         166      166           
  Lines       52412    52412           
=======================================
  Hits        48382    48382           
  Misses       4030     4030
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.73% <ø> (ø)` | :arrow_up: |
| #single | `43.05% <ø> (ø)` | :arrow_up: |


------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24462?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24462?src=pr&el=footer). Last update [d5e5bf7...eba3985](https://codecov.io/gh/pandas-dev/pandas/pull/24462?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

I'm not sure what would be wrong with the autosummary. Your description sounds correct.
@TomAugspurger looks like the problem with the paths is indeed caused by setting the list of `.rst` in the top `doc/source` directory to `autosummary_generate`. That's addressed in #24428, which should be ready now. This will work once that one is merged.
This is looking good now, I think it can be merged. In this PR I just did the splitting without making any other change (more than standardizing blank lines, and the currentmodule directive).

A small change to the validation script was needed to keep it working, which is implemented here (at the very end of the changes).

In a follow up I'll propose changes to the titles of some pages, but I don't want to implement them here, as this is difficult to review, and I think it's better to not change any content here. 

There are some warnings, but none is introduced here, I'll keep taking care of them in separate PRs.

This is how the new API main page looks like:

![api_home](https://user-images.githubusercontent.com/10058240/50547864-b375f780-0c3a-11e9-934f-c4fa2c2c2773.png)

And this is a sample of one of the individual pages:

![api_io](https://user-images.githubusercontent.com/10058240/50547870-c5f03100-0c3a-11e9-911c-edbc8ae86eab.png)

CC: @TomAugspurger @jreback 
LGTM. Maybe wait till later today to merge in case other's have comments.

We might want to rethink some of the organization / section headings (e.g. "general functions" isn't especially informative) but those can be done as followups.
yeah this is ok. sure we need some work on the names / maybe some org of this, but a great first step.