DOC: Change release and whatsnew
State: closed

Some cleanup & changes to facilitate release automation

* We will include the correct (latest on master or maintenance branch)
  whatsnew directly in the index.rst toctree
* Contributors are included in the whatsnew for each version (automatically)
* Removed release.rst
* Added new releases.rst which has toctrees for each release
![screen shot 2018-06-22 at 3 10 23 pm](https://user-images.githubusercontent.com/1312546/41797116-6bfe43aa-762e-11e8-840f-9639faf14e8b.png)

Incidental changes

* Updated style.ipynb. Writing the jinja template was confusing sphinx. We
  included it in the git source now.
* Fixing some inconsitent header levels (will do more)
* Refactored announce.py to support auto-generated contributors

TODO:

- [x] Finish up the rest of the whatsnews

cc @jorisvandenbossche, @datapythonista

Labels:
name: Docs

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21599?src=pr&el=h1) Report
> Merging [#21599](https://codecov.io/gh/pandas-dev/pandas/pull/21599?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/fcb840349caa1a922b1d15e7a8e1ab05bfeed94e?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/21599/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/21599?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #21599   +/-   ##
=======================================
  Coverage   92.24%   92.24%           
=======================================
  Files         161      161           
  Lines       51340    51340           
=======================================
  Hits        47361    47361           
  Misses       3979     3979
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.64% <ø> (ø)` | :arrow_up: |
| #single | `42.34% <ø> (ø)` | :arrow_up: |


------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21599?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21599?src=pr&el=footer). Last update [fcb8403...bc5d49e](https://codecov.io/gh/pandas-dev/pandas/pull/21599?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Looks really good. There is just one thing that I'd consider changing. I don't think it's a good practice to import functions from `scripts/`. Whn in the future someone make changes to the scripts, I don't think we will feel that we need to keep compatibility on any function defined there, as they can be imported by other parts of the code like the doc.

What I'd do instead is creating somewhere a directory "dev utils", where we can not only have the code of `build_string` but also the generation of contributors (and anything that could potentially be reused by the docs, the release, and may be the web). And if possible, I'd also automatically generate there the list of releases (0.23.0, 0.23.1...) so `releases.rst` and anything that requires the list of releases can be generated with the info from there (`releases.rst` would be a template as `index.rst` in that case).
Hello @TomAugspurger! Thanks for updating the PR.









Cheers ! There are no PEP8 issues in this Pull Request. :beers: 

##### Comment last updated on June 26, 2018 at 12:00 Hours UTC
Sorry I didn't mean to push those changes. Not sure whether it's worth auto-discovering.
Cool! Didn't have time to look into detail yet (will do that tomorrow), but some very quick feedback: 

- The docs are not yet building on travis: ``ModuleNotFoundError: No module named 'announce'``
- ~~I get "/home/joris/scipy/pandas/doc/source/whatsnew/v0.20.3.txt: WARNING: document not readable. Ignored." for all whatsnew pages locally, but didn't further look into what is the cause (might be a problem on my side)~~
- Big +1 on having the separate whatsnew files to separate html pages (already wanted to propose that for some time). However, the only problem is that this will break quite some links?
- From a usability perspective of the "releases page", it would maybe be nice to include the highlights there, if this would be possible in some way.
- Generating all the `announce.build_string` takes a non-negligible amount of time (when building the full docs probably not significant, but for a single page it might get annoying). Do we want to keep the script to generate it, but then actually include the content in the actual files (and use the script to generate it on release time) instead of generating it dynamically? (or otherwise skip it in certain cases, eg when building a single page that is not the releases pages)

> Looks really good. There is just one thing that I'd consider changing. I don't think it's a good practice to import functions from scripts/

\+ 1. We can maybe also put things in or alongside doc/make.py ?
> alongside doc/make.py

Sure.

> this will break quite some links?

Which ones? Anything using the "canonical" format like `/version/x.y.z/` should be fine, since we aren't changing those.

> it would maybe be nice to include the highlights there, if this would be possible in some way.

Agreed. I think `highlights.rst` will be a stub or something that gets included in several places

> Do we want to keep the script to generate it, but then actually include the content in the actual files (and use the script to generate it on release time) instead of generating it dynamically?

I would slightly prefer to generate it dynamically. My goal is to remove the "final doc update" commit that's currently part of the release process. In theory, it should be possible to make this a Jinja2 macro that is called inside the actual whatsnew, rather than ahead of time.
I think it used to be to avoid warnings that they weren't included in a
toctree? Maybe we can just change them.

On Mon, Jun 25, 2018 at 3:51 PM, Joris Van den Bossche <
notifications@github.com> wrote:

> *@jorisvandenbossche* commented on this pull request.
> ------------------------------
>
> In doc/source/conf.py
> <https://github.com/pandas-dev/pandas/pull/21599#discussion_r197937968>:
>
> > @@ -120,7 +122,10 @@
>  templates_path = ['../_templates']
>
>  # The suffix of source filenames.
> -source_suffix = '.rst'
> +source_suffix = [
> +    '.rst',
> +    '.txt',
>
> Is there actually a reason they are not *.rst ?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/21599#discussion_r197937968>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIpGSK_j6L7FGlRHyK7AuMuGtSKBZks5uAU1pgaJpZM4U0XNK>
> .
>

> > this will break quite some links?

> Which ones? Anything using the "canonical" format like /version/x.y.z/ should be fine, since we aren't changing those.

Because it are now all separate pages, the url changes. For example, the first link in the highlights section of 0.23.0 points to http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#whatsnew-0230-enhancements-round-trippable-json. However, with this change the link will end in .../whatsnew/v0.23.0.html#whatsnew-0230-enhancements-round-trippable-json

Yeah, I'm removing all this. Didn't mean to push it.

I tried to write a new sphinx directive, so we can just do `..
contributors:: v0.23.1..v0.23.2`, but didn't get the rendering of the
inserted text to work yet.

On Tue, Jun 26, 2018 at 3:31 AM, Joris Van den Bossche <
notifications@github.com> wrote:

> *@jorisvandenbossche* commented on this pull request.
> ------------------------------
>
> In scripts/announce.py
> <https://github.com/pandas-dev/pandas/pull/21599#discussion_r198055743>:
>
> >      authors = get_authors(revision_range)
> -    heading = u"Contributors"
> -    print()
> -    print(heading)
> -    print(u"=" * len(heading))
> -    print(author_msg % len(authors))
> +    uline = '=' * len(heading)
>
> this header does not always match the header levels in the whatsnew file
> ------------------------------
>
> In doc/source/conf.py
> <https://github.com/pandas-dev/pandas/pull/21599#discussion_r198056099>:
>
> > +    'v0.16.2': announce.build_string('v0.16.1..v0.16.2'),
> +    'v0.17.0': announce.build_string('v0.16.2..v0.17.0'),
> +    'v0.17.1': announce.build_string('v0.17.0..v0.17.1'),
> +    'v0.18.0': announce.build_string('v0.10.0..v0.18.0'),
> +    'v0.18.1': announce.build_string('v0.10.0..v0.18.1'),
> +    'v0.19.0': announce.build_string('v0.10.0..v0.19.0'),
> +    'v0.19.1': announce.build_string('v0.10.0..v0.19.1'),
> +    'v0.19.2': announce.build_string('v0.10.0..v0.19.2'),
> +    'v0.20.0': announce.build_string('v0.10.0..v0.20.0'),
> +    'v0.20.1': announce.build_string('v0.10.0..v0.20.1'),
> +    'v0.20.2': announce.build_string('v0.10.0..v0.20.2'),
> +    'v0.20.3': announce.build_string('v0.10.0..v0.20.3'),
> +    'v0.21.0': announce.build_string('v0.10.0..v0.21.0'),
> +    'v0.21.1': announce.build_string('v0.10.0..v0.21.1'),
> +    'v0.22.0': announce.build_string('v0.10.0..v0.22.0'),
> +    'v0.22.0': announce.build_string('v0.22.0..v0.23.0'),
>
> I suppose you know you still have some typos here (a lot of "v0.10.0"
> above, and this line is duplicated)
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/21599#pullrequestreview-131913526>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIkE93cGThlHgr7y1seHnce7EFJKDks5uAfFPgaJpZM4U0XNK>
> .
>

Ok, cleaned up my mess from yesterday. Replaced with a custom directive so we write

```
.. contributors:: v0.23.0..v0.23.1
```

rather than building them up ahead of time.

I haven't updated all the old files yet.
Will think about the links as well.

Writing the directive took a long time (I didn't realize I had to build the list of elements myself. I was hoping I could just pass it a string and have sphinx parse it), so I'm not sure I'll have time to finish this for 0.23.2.
> so I'm not sure I'll have time to finish this for 0.23.2.

no problem, we can still go the normal way for that
> > > this will break quite some 

> > Which ones? Anything using the "canonical" format like /version/x.y.z/ should be fine, since we aren't changing those.

> Because it are now all separate pages, the url changes.

Hmm I'm not sure I see what will break.

For internal links, we're fine, since we use `:ref:`. The ones in the highlights will be fine.

For external links, we're hopefully using the canonical link with `/version/x.y.z/` wherever it matters (e.g. announce). Places that don't use the canonical link will break, but I think that's acceptable.
And it looks like sharing the highlights between the release notes, announcement, and website will be difficult.
> For external links, we're hopefully using the canonical link with /version/x.y.z/ wherever it matters (e.g. announce). Places that don't use the canonical link will break, but I think that's acceptable.

Did you see my comment above (https://github.com/pandas-dev/pandas/pull/21599#issuecomment-400223120)? 
It is not about the "base url" (/version/0.23.0/ vs /stable/ or so), but about the part that comes after it (now "whatsnew.html#internal-link-label", with this PR "whatsnew/v0.23.2.html#internal-link-label", even if we keep all "internal-link-label" the same, the actual url will change)

Anyhow, if we want to do a bit bigger docs reorganisation in the future as well, we will have to live with such breakages in any case ..

can you rebase or close
Updated.

@jorisvandenbossche I think I'm ok with breaking links like `http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#whatsnew-0190-enhancements-other`. I assume (hope) that people have the expectation that `/stable` is a snapshot, and the structure could change. Links like `http://pandas.pydata.org/pandas-docs/version/0.23.0/whatsnew.html#whatsnew-0190-enhancements-other` will continue to work.
> I assume (hope) that people have the expectation that /stable is a snapshot, and the structure could change.

I am not sure that assumption is correct :-) (I think people often point to stable (or at least, I also do that), to make sure to always point to the latest version)

But that said, that doesn't mean we shouldn't do it. As I said above, if we want to do bigger reorganisation of the docs, we will have such issues anyway. 

I could look into autogenerating some redirects (but not right now).

Didn't look at the latest updates yet.
@TomAugspurger can you rebase
I was thinking a bit... for links like `http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#whatsnew-0190-enhancements-other` I *think* we're completely out of luck for redirecting. nginx (our web server) doesn't see anchor tags (everything after the `#`) so we wouldn't be able to rewrite the url.

I'm going to rebase and I think we should merge. Every release we delay this is another releases's worth of URLs we'll break in the future if we ever end up doing this :)
Since we renamed the whatsnew suffixes from .txt to .rst, they're getting linted now. I'm not planning to go through the old whatsnews to fix linting errors, but new ones will be linted (and 0.24.0 is clean right now).
> I think we're completely out of luck for redirecting. nginx (our web server) doesn't see anchor tags (everything after the #) so we wouldn't be able to rewrite the url.

So you mean we can't "catch" the fact that a user is visiting eg `http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#whatsnew-0190-enhancements-other` ?

Since the `whatsnew` page is still there, in principle we could add a bunch of generated tags there with a link for people to click on? 
Or with some javascript (https://stackoverflow.com/questions/1305211/javascript-to-redirect-from-anchor-to-a-separate-page) 

(but that does not need to hold up this PR)
> So you mean we can't "catch" the fact that a user is visiting eg http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#whatsnew-0190-enhancements-other ?

If we agree that "catch" means *before* the user actually gets there, then yes that's what I'm saying.

> Since the whatsnew page is still there, in principle we could add a bunch of generated tags there with a link for people to click on?

Sounds pretty ugly :)

A javascript-based solution seems like the best bet.
> If we agree that "catch" means before the user actually gets there, then yes that's what I'm saying.

I am a complete web noob, so not sure what I meant :-)
@datapythonista it'd be nice if warnings like https://travis-ci.org/pandas-dev/pandas/jobs/454525382#L2853

```
doc/source/whatsnew/v0.24.0.rst:- Bug in :func:`pandas.concat` when concatenating a multicolumn DataFrame with tz-aware data against a DataFrame with a different number of columns (:issue`22796`)
```

printed the source line number. Is that possible? Do we have an issue for that?
"Check for backticks incorrectly rendering because of missing spaces" is actually also a strange description of the problem in this case?
I think that one is a bug in the linter (good it caught it though). It's really checking for things like :class:\`DataFrame\`s.
I think we have +1s here, so merging once the doc build passes.
Updated the headers.