Repository: pypa/pip

Issue:
Changelog/release notes sections do not have stable anchors
**Description**
The section headers in the changelog/release notes do not have proper stable anchor IDs. Instead, they have generic numbered IDs like `id123`. This makes it impossible to link to a specific section of the changelog in a way that will continue working in the future: for example the section for version 20.0 has anchor ID [`id6`](https://pip.pypa.io/en/stable/news/#id6) in the current stable version of the docs, and anchor ID [`id35`](https://pip.pypa.io/en/latest/news/#id35) in the current latest version. These IDs seem to be assigned sequentially from the start of the file, so they are guaranteed to change when a new version section is added.

**Expected behavior**
Each version section (and preferrably also each subsection) should have a unique anchor ID that does not change when a new pip release is added to the changelog.

(I'm violating the issue template a bit here - this is a documentation issue, so pip version, OS, etc. don't really apply.)


Discussion:
The only project that I know does anything to solve this issue is virtualenv: https://virtualenv.pypa.io/en/latest/changelog.html#features-20-0-21

I am not particularly a fan of solving this issue like that tho. :)
I think we need to clarify what the requirement is in this issue, there is some more discussion over in #8620
Sorry I was a little slow to re-read the initial report:

> Each version section (and preferrably also each subsection) should have a unique anchor ID that does not change when a new pip release is added to the changelog.

Each version section is very achievable and there are lots of examples to follow. Each sub section is more difficult. I suggest splitting this proposed requirement into two and quickly implementing "each version section".
Thank you for looking into this!

My main request is being able to link to a specific version in the release notes in a way that doesn't break when new versions are added. That way when I'm in an issue/review discussion talking about a change in pip's behavior, I can provide a link to the relevant section of the release notes and say "here's the pip release where the change happened".

The ability to also link to a specific subsection would be "nice to have", but I would also be happy if only the version sections have unique anchors and the subsections don't. In most cases the list of changes for one version is short enough that you can find a specific change even if you don't know what subsection it's under.
To keep the discussion easy to follow, I suggest commenting about the requirements and general approaches here and only talk about implementation-specific details in the PRs.

Personally I'd favor seeing this incrementally improved, i.e. merging GH-8620 almost as-is and perform the subsection work in a follow-up.

@maxwell-k, IIUC, by

> I am not particularly a fan of solving this issue like [that](https://virtualenv.pypa.io/en/latest/changelog.html) tho. :)

@pradyunsg probably mean he doesn't like the way e.g.

> Features - 20.0.21

looks.  As suggested in the mentioned PR, we can instead label each subsection using something like
```rst
.. _features-{(versiondata.version.replace('.', '-')}}:
```

Regardless of which path we take, we'll still need `towncrier>=19.9.0rc1` for this task, preferably by nudging its maintainers to prepare for a stable release if possible.
Just saw this, it seems to affect more than just the changelog section - same in the user guide 
https://pip.pypa.io/en/latest/user_guide/#using-pip-from-your-program
used to anchor but now it's https://pip.pypa.io/en/latest/user_guide/#id9 which won't be stable in the likely event that a section gets added above.
@wimglenn The problem to this issue is that the changelog page is machine-generated. The section you are referring is written by a human, and likely broken by a botched merge conflict resolution. Feel free to submit a pull request to amend it. Specifically, [this line](https://github.com/pypa/pip/blob/2e4d748/docs/html/user_guide.rst#L791) should be moved to right above the relevent secion.
So the issue is that Sphinx can not slugify headers with numbers like `19.2 (2019-07-22)` to turn them into html link anchors.

Can anyone compile a minimal example of this misbehaviour to report upstream?
@abitrolly that's because it's a heading without "letters". Any page with only digit+symbols heading would have this issue. Here's an example page:

```
Hi
==

1.2.0
-----

1.1.0
-----

1.0.0
-----
```
@pradyunsg yep, that's exactly what I've meant. So is there a minimal example to reproduce?
@abitrolly I'm not sure what you're asking for. The reproducer is any page with such headings, being built by sphinx.

If you're reporting upstream,. please make sure that it's not a duplicate of some issue already on the repository, and pointing the maintainer to pip's Changelog page should be sufficient to demonstrate this. Otherwise, my previous comment contains a minimal rst that you can use in any sphinx documentation to test the behaviour under question here.
@pradyunsg sorry, I've meant Short, Self Contained, Correct (Compilable), Example - http://sscce.org/

I can just chime in into https://github.com/sphinx-doc/sphinx/issues and say "Hey, you know `pip` has this problem - its Changelog ToC is autogenerated and slugs in html anchors are changing each time a new version is released". I haven't done that yet, because it is unlikely to be treated as good issue report.
@abitrolly In [this comment](https://github.com/pypa/pip/issues/8152#issuecomment-762833074) @pradyunsg provided a very short reST document that exhibits the issue if you use Sphinx to render it to HTML. You can't really make it any more short, self-contained and compilable than that 🙂

Also, as I understand it, Sphinx uses [docutils](https://sourceforge.net/projects/docutils/) to perform the conversion from reST to HTML. So if you want to report this upstream, you should probably do so on the docutils [issue tracker](https://sourceforge.net/p/docutils/bugs/) or [mailing list](https://sourceforge.net/p/docutils/mailman/), rather than the Sphinx issue tracker.
@abitrolly if I had the time to write down step by step instructions, I would have filed the issue upstream myself. :)

As mentioned, the rst is available above, and you can use sphinx-quickstart to get a sphinx documentation set.

^^^ reported the bug with the SSCCE. This is the most I can do.
I wonder if it is possible to resolve this in Towncrier by adding an explicit ref derivitive to each release section.

Resolving patch:
diff --git a/docs/pip_sphinxext.py b/docs/pip_sphinxext.py
--- a/docs/pip_sphinxext.py
+++ b/docs/pip_sphinxext.py
@@ -1,12 +1,13 @@
 """pip sphinx extensions"""
 
 import optparse
+import pathlib
 import re
 import sys
 from textwrap import dedent
 from typing import Iterable, List, Optional
 
-from docutils import nodes
+from docutils import nodes, statemachine
 from docutils.parsers import rst
 from docutils.statemachine import StringList, ViewList
 from sphinx.application import Sphinx
@@ -16,6 +17,63 @@
 from pip._internal.req.req_file import SUPPORTED_OPTIONS
 
 
+class PipNewsInclude(rst.Directive):
+    required_arguments = 1
+
+    def _is_version_section_title_underline(self, prev, curr):
+        """Find a ==== line that marks the version section title."""
+        if prev is None:
+            return False
+        if re.match(r"^=+$", curr) is None:
+            return False
+        if len(curr) < len(prev):
+            return False
+        return True
+
+    def _iter_lines_with_refs(self, lines):
+        """Transform the input lines to add a ref before each section title.
+
+        This is done by looking one line ahead and locate a title's underline,
+        and add a ref before the title text.
+
+        Dots in the version is converted into dash, and a ``v`` is prefixed.
+        This makes Sphinx use them as HTML ``id`` verbatim without generating
+        auto numbering (which would make the the anchors unstable).
+        """
+        prev = None
+        for line in lines:
+            # Transform the previous line to include an explicit ref.
+            if self._is_version_section_title_underline(prev, line):
+                vref = prev.split(None, 1)[0].replace(".", "-")
+                yield f".. _`v{vref}`:"
+                yield ""  # Empty line between ref and the title.
+            if prev is not None:
+                yield prev
+            prev = line
+        if prev is not None:
+            yield prev
+
+    def run(self):
+        source = self.state_machine.input_lines.source(
+            self.lineno - self.state_machine.input_offset - 1,
+        )
+        path = (
+            pathlib.Path(source)
+            .resolve()
+            .parent
+            .joinpath(self.arguments[0])
+            .resolve()
+        )
+        include_lines = statemachine.string2lines(
+            path.read_text(encoding="utf-8"),
+            self.state.document.settings.tab_width,
+            convert_whitespace=True,
+        )
+        include_lines = list(self._iter_lines_with_refs(include_lines))
+        self.state_machine.insert_input(include_lines, str(path))
+        return []
+
+
 class PipCommandUsage(rst.Directive):
     required_arguments = 1
     optional_arguments = 3
@@ -245,4 +303,5 @@ def setup(app: Sphinx) -> None:
     app.add_directive(
         "pip-requirements-file-options-ref-list", PipReqFileOptionsReference
     )
+    app.add_directive('pip-news-include', PipNewsInclude)
     app.add_directive("pip-cli", PipCLIDirective)


Test patch: