DOC: Creating top-level user guide section, and moving pages inside
State: closed

- [X] xref #24499 
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This is part of #24499, will be moving the pages in different PRs, so it's easier to discuss.

Labels:
name: Docs

Comments:
Would it be possible to show a preview? (long time ago I eg just pushed the build docs to github repo to be able to show it : https://github.com/jorisvandenbossche/example-pandas-docs)
https://datapythonista.github.io/pandas-doc-preview/
should make a language section with the comparison ones (4), plus let’s remove rpy entirely
I agree on both things, both are proposed in #24499.

But I'd prefer to make the changes in different PRs, so the discussions can be focussed. So far I just created the PRs for the things I find more obvious. And I think grouping the comparison pages is quite obvious, but where to have it will require some discussion (I'd probably have it under the "Getting started" session, with all the new-comers documentation).
is it possible to make clicking on a top-level open AND close the section? (it just will open it now)
That's a good point. I think that should be implemented in the theme (as javascript). In the new theme we've been playing with, the navigation by default doesn't even have all the sections in the left sidebar, but just the current one (e.g. https://pandas-dev.github.io/pandas-sphinx-theme/pr-datapythonista_base/user_guide/io.html)

My conclusion after trying to customize those sidebars to do what we want with sphinx, is that we should not use sphinx (#23763). So, I don't think we'll be able to have a layout and navigation we're really happy with in the short term, but hopefully something significantly better than now.
@jorisvandenbossche do external links need to be updated for this one?
This PR (and the similar ones) break external links. I proposed to create redirect pages in #24705.

If we agree on that approach, I'd merge this PR and the rest, as they are quite likely to conflict soon, and I'll add all the redirects for the pages moved here when implementing the system.
Added the redirects in #24715
Would be nice to have this merged before merging other documentation changes. This will conflict with most changes to the docs.
does this need the re-directs?
They are implemented in #24715, I don't want to have both things in the same PR.

This should be merged first.
I am ok with this, @jorisvandenbossche @TomAugspurger 
Agreed that smaller PRs are preferable. I worry a bit about things like
this slipping through in the smaller PRs (e.g. I don't think this index
page will be touched again until
things are entirely done). In the meantime it'd be nice to not lose some of
the few benefits of the flat structure.

On Mon, Jan 14, 2019 at 8:10 AM Marc Garcia <notifications@github.com>
wrote:

> *@datapythonista* commented on this pull request.
> ------------------------------
>
> In doc/source/user_guide/index.rst
> <https://github.com/pandas-dev/pandas/pull/24677#discussion_r247504426>:
>
> > @@ -0,0 +1,30 @@
> +{{ header }}
> +
> +.. _user_guide:
> +
> +==========
> +User Guide
> +==========
> +
>
> I 100% agree, but it makes more sense to me to work on those in separate
> PRs after the whole restructure. There is already another PR for a top
> level "Getting started" section, that probably should be the linked for the
> "Users brand-new..." comment.
>
> I think moving all the pages and finding an agreement on all the
> structures will actually take quite a lot of time, and will require a
> decent amount of discussion. That's why I'm proposing smaller PRs, to be
> able to keep some progress. Also, I think that will let us see the progress
> at every step, and make us move in the right direction (kind of agile
> methodology).
>
> I mentioned it somewhere, but it was already work on trying to migrate all
> the documentation in a one-shot (see
> https://pandas-dev.github.io/pandas-sphinx-theme/pr-datapythonista_base/).
> After all the work on that, I think the approach will fail. If anyone else
> thinks a single PR or bigger PRs are better, I'm more than happy to leave
> this for someone else.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/24677#discussion_r247504426>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIn0J2-L3O3X5rs7nmREiQFEueUQkks5vDI_FgaJpZM4Z2lfX>
> .
>

On Mon, Jan 14, 2019 at 8:10 AM Marc Garcia <notifications@github.com>
wrote:

> *@datapythonista* commented on this pull request.
> ------------------------------
>
> In doc/source/user_guide/index.rst
> <https://github.com/pandas-dev/pandas/pull/24677#discussion_r247504426>:
>
> > @@ -0,0 +1,30 @@
> +{{ header }}
> +
> +.. _user_guide:
> +
> +==========
> +User Guide
> +==========
> +
>
> I 100% agree, but it makes more sense to me to work on those in separate
> PRs after the whole restructure. There is already another PR for a top
> level "Getting started" section, that probably should be the linked for the
> "Users brand-new..." comment.
>
> I think moving all the pages and finding an agreement on all the
> structures will actually take quite a lot of time, and will require a
> decent amount of discussion. That's why I'm proposing smaller PRs, to be
> able to keep some progress. Also, I think that will let us see the progress
> at every step, and make us move in the right direction (kind of agile
> methodology).
>
> I mentioned it somewhere, but it was already work on trying to migrate all
> the documentation in a one-shot (see
> https://pandas-dev.github.io/pandas-sphinx-theme/pr-datapythonista_base/).
> After all the work on that, I think the approach will fail. If anyone else
> thinks a single PR or bigger PRs are better, I'm more than happy to leave
> this for someone else.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/24677#discussion_r247504426>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIn0J2-L3O3X5rs7nmREiQFEueUQkks5vDI_FgaJpZM4Z2lfX>
> .
>

Sorry, I don't think I understand. What are the benefits of the flat structure that we'd loose?

Are you proposing not to make these changes, or just adding that explanation and links from the mentioned pages you'd be happy with this?
Sorry for being unclear. The benefit I forgot to state is "always being
able to see where you are in the (previously linear flow through the docs.
But I see in, e.g.,
https://datapythonista.github.io/pandas-doc-preview/user_guide/indexing.html#different-choices-for-indexing
that that's still the case, so
feel free to ignore me.

On Mon, Jan 14, 2019 at 8:29 AM Marc Garcia <notifications@github.com>
wrote:

> Sorry, I don't think I understand. What are the benefits of the flat
> structure that we'd loose?
>
> Are you proposing not to make these changes, or just adding that
> explanation and links from the mentioned pages you'd be happy with this?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/24677#issuecomment-454023475>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIiSgrFd4BRUAvjyDOjnTdkVPu4PQks5vDJRggaJpZM4Z2lfX>
> .
>

The navigation with the new template is different, and customization doesn't seem trivial (you saw #23763 with my experience trying). That part won't be easy.

But that will come later, so far, the PRs I'm opening don't change more than grouping some sections together, and everything else should be the same.
This looks good to me.  

I agree that it is good to do those splits in multiple PRs, but, I think it still can't hurt to already add the paragraph Tom is proposing in user_guide/index.rst (the part about explaining the scope of the User Guide, without any explicit links to the other parts, as those are being created in parallel)
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24677?src=pr&el=h1) Report
> Merging [#24677](https://codecov.io/gh/pandas-dev/pandas/pull/24677?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/597f9f31639eeb5724e49bec602e15b9bf8be092?src=pr&el=desc) will **increase** coverage by `49.49%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24677/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24677?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master   #24677       +/-   ##
===========================================
+ Coverage   42.88%   92.38%   +49.49%     
===========================================
  Files         166      166               
  Lines       52411    52310      -101     
===========================================
+ Hits        22478    48327    +25849     
+ Misses      29933     3983    -25950
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.8% <ø> (?)` | |
| #single | `43.06% <ø> (+0.17%)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24677?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/computation/pytables.py](https://codecov.io/gh/pandas-dev/pandas/pull/24677/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vcHl0YWJsZXMucHk=) | `92.37% <0%> (+0.32%)` | :arrow_up: |
| [pandas/io/pytables.py](https://codecov.io/gh/pandas-dev/pandas/pull/24677/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3B5dGFibGVzLnB5) | `92.31% <0%> (+0.92%)` | :arrow_up: |
| [pandas/util/\_test\_decorators.py](https://codecov.io/gh/pandas-dev/pandas/pull/24677/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvX3Rlc3RfZGVjb3JhdG9ycy5weQ==) | `90.54% <0%> (+4.05%)` | :arrow_up: |
| [pandas/compat/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/24677/diff?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9fX2luaXRfXy5weQ==) | `57.91% <0%> (+8.1%)` | :arrow_up: |
| [pandas/core/config\_init.py](https://codecov.io/gh/pandas-dev/pandas/pull/24677/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29uZmlnX2luaXQucHk=) | `99.24% <0%> (+9.84%)` | :arrow_up: |
| [pandas/core/api.py](https://codecov.io/gh/pandas-dev/pandas/pull/24677/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXBpLnB5) | `100% <0%> (+13.33%)` | :arrow_up: |
| [pandas/compat/numpy/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/24677/diff?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9udW1weS9fX2luaXRfXy5weQ==) | `92.85% <0%> (+14.28%)` | :arrow_up: |
| [pandas/core/computation/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/24677/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vY29tbW9uLnB5) | `85.71% <0%> (+14.28%)` | :arrow_up: |
| [pandas/core/indexes/api.py](https://codecov.io/gh/pandas-dev/pandas/pull/24677/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9hcGkucHk=) | `99% <0%> (+14.85%)` | :arrow_up: |
| [pandas/core/reshape/util.py](https://codecov.io/gh/pandas-dev/pandas/pull/24677/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS91dGlsLnB5) | `100% <0%> (+15%)` | :arrow_up: |
| ... and [123 more](https://codecov.io/gh/pandas-dev/pandas/pull/24677/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24677?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24677?src=pr&el=footer). Last update [597f9f3...175ef47](https://codecov.io/gh/pandas-dev/pandas/pull/24677?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24677?src=pr&el=h1) Report
> Merging [#24677](https://codecov.io/gh/pandas-dev/pandas/pull/24677?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/597f9f31639eeb5724e49bec602e15b9bf8be092?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24677/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24677?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #24677      +/-   ##
==========================================
- Coverage   92.39%   92.38%   -0.01%     
==========================================
  Files         166      166              
  Lines       52411    52411              
==========================================
- Hits        48423    48422       -1     
- Misses       3988     3989       +1
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.81% <ø> (ø)` | :arrow_up: |
| #single | `42.89% <ø> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24677?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/24677/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `88.04% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24677?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24677?src=pr&el=footer). Last update [597f9f3...97f427e](https://codecov.io/gh/pandas-dev/pandas/pull/24677?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Fixed the conflicts and added the text from Tom to the user guide index. This should be ready to be merged, and #24715 immediately after to create the redirects from the old urls to the new ones.
+1

________________________________
From: Marc Garcia <notifications@github.com>
Sent: Tuesday, January 22, 2019 6:32:11 PM
To: pandas-dev/pandas
Cc: Tom Augspurger; Mention
Subject: Re: [pandas-dev/pandas] DOC: Creating top-level user guide section, and moving pages inside (#24677)


Fixed the conflicts and added the text from Tom to the user guide index. This should be ready to be merged, and #24715<https://github.com/pandas-dev/pandas/pull/24715> immediately after to create the redirects from the old urls to the new ones.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://github.com/pandas-dev/pandas/pull/24677#issuecomment-456619579>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABQHIsgdSedgC_U_ljS2uRsnJrdXNLXOks5vF62LgaJpZM4Z2lfX>.

@datapythonista are you ready on https://github.com/pandas-dev/pandas/pull/24691 and https://github.com/pandas-dev/pandas/pull/24678 as well? I'd like to merge these all at the same time, and then https://github.com/pandas-dev/pandas/pull/24715.

https://github.com/pandas-dev/pandas/pull/24691/files has a merge conficlit.
My idea is to get this merge and the redirect system, and then add the redirects in those same PRs (can't create the redirects until the redirecting system is merged, and would prefer to not have the redirects of the 3 restructuring PRs in the same changes as the redirecting code).

I can have everything ready in minutes since this and the redirect system are merged if that makes sense.
Yeah by "same time" I meant same dayish :) Feel free to merge the three restructuring PRs when you're ready, and I'll take another look at the redirect PR, but I think that looked good as well.