Repository: sphinx-doc/sphinx

Issue:
toc_object_entries uses language (Python) scoping instead of documentation scoping
### Describe the bug

The entries added by ``toc_object_entries`` for Python (and other domains?) seems to use the scoping of the language to nest the toc entries, instead of how they were declared in the document. E.g., if a nested class is declared after the class, then its entry is nested inside the class instead of appended.

### How to Reproduce

E.g., with
```rst
.. py:currentmodule:: myMod
.. py:class:: MyClass

	.. py:method:: f()

.. py:class:: MyClass.MyNestedClass
```
I get the toc
```
- MyClass
  - MyClass.f()
  - MyClass.MyNestedClass
```
instead of
```
- MyClass
  - MyClass.f()
- MyClass.MyNestedClass
```

### Environment Information

```text
Tested with current master.
```


### Sphinx extensions

_No response_

### Additional context

Could the check at https://github.com/sphinx-doc/sphinx/blob/88a54d8a1394b09c7425d7a5bfebae55ab615d32/sphinx/environment/collectors/toctree.py#L146 be changed to check if the parent in the language is also the parent in the current toctree generation?

Discussion:
chrisjsewell: > Are there any tests for this? None seems to break with the change.

Hey @jakobandersen cheers, but yes I feel we should look to add a MWE test that breaks on the current behaviour and is fixed by this change

Patch:
diff --git a/CHANGES.rst b/CHANGES.rst
index ac359fc8c..d75bde8eb 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -26,6 +26,8 @@ Deprecated
 Features added
 --------------
 
+.. rst-class:: compact
+
 * Add optional ``description`` argument to
   :meth:`~sphinx.application.Sphinx.add_config_value`.
   Patch by Chris Sewell.
@@ -69,7 +71,6 @@ Features added
     parses the provided text into inline elements and text nodes.
 
   Patch by Adam Turner.
-
 * #12258: Support ``typing_extensions.Unpack``
   Patch by Bénédikt Tran and Adam Turner.
 * #12524: Add a ``class`` option to the :rst:dir:`toctree` directive.
@@ -98,6 +99,9 @@ Features added
 * #12508: LaTeX: Revamped styling of all admonitions, with addition of a
   title row with icon.
   Patch by Jean-François B.
+* #11773: Display :py:class:`~typing.Annotated` annotations
+  with their metadata in the Python domain.
+  Patch by Adam Turner and David Stansby.
 
 Bugs fixed
 ----------
@@ -145,6 +149,11 @@ Bugs fixed
   Patch by James Addison and Will Lachance.
 * #9634: Do not add a fallback language by stripping the country code.
   Patch by Alvin Wong.
+* #12352: Add domain objects to the table of contents
+  in the same order as defined in the document.
+  Previously, each domain used language-specific nesting rules,
+  which removed control from document authors.
+  Patch by Jakob Lykke Andersen and Adam Turner.
 
 Testing
 -------
diff --git a/sphinx/environment/collectors/toctree.py b/sphinx/environment/collectors/toctree.py
index 6ea148c63..dcf4ee178 100644
--- a/sphinx/environment/collectors/toctree.py
+++ b/sphinx/environment/collectors/toctree.py
@@ -68,8 +68,6 @@ def build_toc(
         ) -> nodes.bullet_list | None:
             # list of table of contents entries
             entries: list[Element] = []
-            # cache of parents -> list item
-            memo_parents: dict[tuple[str, ...], nodes.list_item] = {}
             for sectionnode in node:
                 # find all toctree nodes in this section and add them
                 # to the toc (just copying the toctree node which is then
@@ -103,6 +101,8 @@ def build_toc(
                         entries.append(onlynode)
                 # check within the section for other node types
                 elif isinstance(sectionnode, nodes.Element):
+                    # cache of parent node -> list item
+                    memo_parents: dict[nodes.Element, nodes.list_item] = {}
                     toctreenode: nodes.Node
                     for toctreenode in sectionnode.findall():
                         if isinstance(toctreenode, nodes.section):
@@ -114,6 +114,10 @@ def build_toc(
                             note_toctree(app.env, docname, toctreenode)
                         # add object signatures within a section to the ToC
                         elif isinstance(toctreenode, addnodes.desc):
+                            # The desc has one or more nested desc_signature,
+                            # and then a desc_content, which again may have desc nodes.
+                            # Thus, desc is the one we can bubble up to through parents.
+                            entry: nodes.list_item | None = None
                             for sig_node in toctreenode:
                                 if not isinstance(sig_node, addnodes.desc_signature):
                                     continue
@@ -136,22 +140,28 @@ def build_toc(
                                 para = addnodes.compact_paragraph('', '', reference,
                                                                   skip_section_number=True)
                                 entry = nodes.list_item('', para)
-                                *parents, _ = sig_node['_toc_parts']
-                                parents = tuple(parents)
 
-                                # Cache parents tuple
-                                memo_parents[sig_node['_toc_parts']] = entry
-
-                                # Nest children within parents
-                                if parents and parents in memo_parents:
-                                    root_entry = memo_parents[parents]
+                                # Find parent node
+                                parent = sig_node.parent
+                                while parent not in memo_parents and parent != sectionnode:
+                                    parent = parent.parent
+                                # Note, it may both be the limit and in memo_parents,
+                                # prefer memo_parents, so we get the nesting.
+                                if parent in memo_parents:
+                                    root_entry = memo_parents[parent]
                                     if isinstance(root_entry[-1], nodes.bullet_list):
                                         root_entry[-1].append(entry)
                                     else:
                                         root_entry.append(nodes.bullet_list('', entry))
-                                    continue
-
-                                entries.append(entry)
+                                else:
+                                    assert parent == sectionnode
+                                    entries.append(entry)
+
+                            # Save the latest desc_signature as the one we put sub entries in.
+                            # If there are multiple signatures, then the latest is used.
+                            if entry is not None:
+                                # are there any desc nodes without desc_signature nodes?
+                                memo_parents[toctreenode] = entry
 
             if entries:
                 return nodes.bullet_list('', *entries)
diff --git a/sphinx/util/typing.py b/sphinx/util/typing.py
index f19e054b3..28ca86490 100644
--- a/sphinx/util/typing.py
+++ b/sphinx/util/typing.py
@@ -261,6 +261,14 @@ def restify(cls: Any, mode: _RestifyMode = 'fully-qualified-except-typing') -> s
             # evaluated before determining whether *cls* is a mocked object
             # or not; instead of two try-except blocks, we keep it here.
             return f':py:class:`{module_prefix}{_INVALID_BUILTIN_CLASSES[cls]}`'
+        elif _is_annotated_form(cls):
+            args = restify(cls.__args__[0], mode)
+            meta = ', '.join(map(repr, cls.__metadata__))
+            if sys.version_info[:2] <= (3, 11):
+                # Hardcoded to fix errors on Python 3.11 and earlier.
+                return fr':py:class:`~typing.Annotated`\ [{args}, {meta}]'
+            return (f':py:class:`{module_prefix}{cls.__module__}.{cls.__name__}`'
+                    fr'\ [{args}, {meta}]')
         elif inspect.isNewType(cls):
             if sys.version_info[:2] >= (3, 10):
                 # newtypes have correct module info since Python 3.10+
@@ -497,7 +505,14 @@ def stringify_annotation(
                              for a in annotation_args)
             return f'{module_prefix}Literal[{args}]'
         elif _is_annotated_form(annotation):  # for py39+
-            return stringify_annotation(annotation_args[0], mode)
+            args = stringify_annotation(annotation_args[0], mode)
+            meta = ', '.join(map(repr, annotation.__metadata__))
+            if sys.version_info[:2] <= (3, 11):
+                if mode == 'fully-qualified-except-typing':
+                    return f'Annotated[{args}, {meta}]'
+                module_prefix = module_prefix.replace('builtins', 'typing')
+                return f'{module_prefix}Annotated[{args}, {meta}]'
+            return f'{module_prefix}Annotated[{args}, {meta}]'
         elif all(is_system_TypeVar(a) for a in annotation_args):
             # Suppress arguments if all system defined TypeVars (ex. Dict[KT, VT])
             return module_prefix + qualname


Test patch:
diff --git a/tests/roots/test-toctree-domain-objects/document_scoping.rst b/tests/roots/test-toctree-domain-objects/document_scoping.rst
new file mode 100644
index 000000000..49aba9e4b
--- /dev/null
+++ b/tests/roots/test-toctree-domain-objects/document_scoping.rst
@@ -0,0 +1,23 @@
+Level 1
+=======
+
+.. py:class:: ClassLevel1a
+              ClassLevel1b
+
+   .. py:method:: f()
+
+.. py:method:: ClassLevel1a.g()
+
+.. py:method:: ClassLevel1b.g()
+
+Level 2
+-------
+
+.. py:class:: ClassLevel2a
+              ClassLevel2b
+
+   .. py:method:: f()
+
+.. py:method:: ClassLevel2a.g()
+
+.. py:method:: ClassLevel2b.g()
diff --git a/tests/roots/test-toctree-domain-objects/index.rst b/tests/roots/test-toctree-domain-objects/index.rst
index 77ee01009..5f041725a 100644
--- a/tests/roots/test-toctree-domain-objects/index.rst
+++ b/tests/roots/test-toctree-domain-objects/index.rst
@@ -4,3 +4,4 @@
    :name: mastertoc
 
    domains
+   document_scoping
diff --git a/tests/test_environment/test_environment_toctree.py b/tests/test_environment/test_environment_toctree.py
index 175c6abe4..6979a129e 100644
--- a/tests/test_environment/test_environment_toctree.py
+++ b/tests/test_environment/test_environment_toctree.py
@@ -132,7 +132,7 @@ def test_domain_objects(app):
 
     assert app.env.toc_num_entries['index'] == 0
     assert app.env.toc_num_entries['domains'] == 9
-    assert app.env.toctree_includes['index'] == ['domains']
+    assert app.env.toctree_includes['index'] == ['domains', 'document_scoping']
     assert 'index' in app.env.files_to_rebuild['domains']
     assert app.env.glob_toctrees == set()
     assert app.env.numbered_toctrees == {'index'}
@@ -161,6 +161,41 @@ def test_domain_objects(app):
                 [list_item, ([compact_paragraph, reference, literal, "HelloWorldPrinter.print()"])])
 
 
+@pytest.mark.sphinx('dummy', testroot='toctree-domain-objects')
+def test_domain_objects_document_scoping(app):
+    app.build()
+
+    # tocs
+    toctree = app.env.tocs['document_scoping']
+    assert_node(
+        toctree,
+        [bullet_list, list_item, (
+            compact_paragraph,  # [0][0]
+            [bullet_list, (  # [0][1]
+                [list_item, compact_paragraph, reference, literal, 'ClassLevel1a'],  # [0][1][0]
+                [list_item, (  # [0][1][1]
+                    [compact_paragraph, reference, literal, 'ClassLevel1b'],  # [0][1][1][0]
+                    [bullet_list, list_item, compact_paragraph, reference, literal, 'ClassLevel1b.f()'],  # [0][1][1][1][0]
+                )],
+                [list_item, compact_paragraph, reference, literal, 'ClassLevel1a.g()'],  # [0][1][2]
+                [list_item, compact_paragraph, reference, literal, 'ClassLevel1b.g()'],  # [0][1][3]
+                [list_item, (  # [0][1][4]
+                    [compact_paragraph, reference, 'Level 2'],  # [0][1][4][0]
+                    [bullet_list, (  # [0][1][4][1]
+                        [list_item, compact_paragraph, reference, literal, 'ClassLevel2a'],  # [0][1][4][1][0]
+                        [list_item, (  # [0][1][4][1][1]
+                            [compact_paragraph, reference, literal, 'ClassLevel2b'],  # [0][1][4][1][1][0]
+                            [bullet_list, list_item, compact_paragraph, reference, literal, 'ClassLevel2b.f()'],  # [0][1][4][1][1][1][0]
+                        )],
+                        [list_item, compact_paragraph, reference, literal, 'ClassLevel2a.g()'],  # [0][1][4][1][2]
+                        [list_item, compact_paragraph, reference, literal, 'ClassLevel2b.g()'],  # [0][1][4][1][3]
+                    )],
+                )],
+            )],
+        )],
+    )
+
+
 @pytest.mark.sphinx('xml', testroot='toctree')
 @pytest.mark.test_params(shared_result='test_environment_toctree_basic')
 def test_document_toc(app):
diff --git a/tests/test_util/test_util_typing.py b/tests/test_util/test_util_typing.py
index 2873bb1a1..044b45990 100644
--- a/tests/test_util/test_util_typing.py
+++ b/tests/test_util/test_util_typing.py
@@ -1,5 +1,6 @@
 """Tests util.typing functions."""
 
+import dataclasses
 import sys
 import typing as t
 from collections import abc
@@ -73,6 +74,11 @@ class BrokenType:
     __args__ = int
 
 
+@dataclasses.dataclass(frozen=True)
+class Gt:
+    gt: float
+
+
 def test_restify():
     assert restify(int) == ":py:class:`int`"
     assert restify(int, "smart") == ":py:class:`int`"
@@ -187,10 +193,11 @@ def test_restify_type_hints_containers():
                                            "[:py:obj:`None`]")
 
 
-@pytest.mark.xfail(sys.version_info[:2] <= (3, 11), reason='Needs fixing.')
 def test_restify_Annotated():
-    assert restify(Annotated[str, "foo", "bar"]) == ':py:class:`~typing.Annotated`\\ [:py:class:`str`]'
-    assert restify(Annotated[str, "foo", "bar"], 'smart') == ':py:class:`~typing.Annotated`\\ [:py:class:`str`]'
+    assert restify(Annotated[str, "foo", "bar"]) == ":py:class:`~typing.Annotated`\\ [:py:class:`str`, 'foo', 'bar']"
+    assert restify(Annotated[str, "foo", "bar"], 'smart') == ":py:class:`~typing.Annotated`\\ [:py:class:`str`, 'foo', 'bar']"
+    assert restify(Annotated[float, Gt(-10.0)]) == ':py:class:`~typing.Annotated`\\ [:py:class:`float`, Gt(gt=-10.0)]'
+    assert restify(Annotated[float, Gt(-10.0)], 'smart') == ':py:class:`~typing.Annotated`\\ [:py:class:`float`, Gt(gt=-10.0)]'
 
 
 def test_restify_type_hints_Callable():
@@ -499,9 +506,12 @@ def test_stringify_type_hints_pep_585():
     assert stringify_annotation(tuple[List[dict[int, str]], str, ...], "smart") == "tuple[~typing.List[dict[int, str]], str, ...]"
 
 
+@pytest.mark.xfail(sys.version_info[:2] <= (3, 9), reason='Needs fixing.')
 def test_stringify_Annotated():
-    assert stringify_annotation(Annotated[str, "foo", "bar"], 'fully-qualified-except-typing') == "str"
-    assert stringify_annotation(Annotated[str, "foo", "bar"], "smart") == "str"
+    assert stringify_annotation(Annotated[str, "foo", "bar"], 'fully-qualified-except-typing') == "Annotated[str, 'foo', 'bar']"
+    assert stringify_annotation(Annotated[str, "foo", "bar"], 'smart') == "~typing.Annotated[str, 'foo', 'bar']"
+    assert stringify_annotation(Annotated[float, Gt(-10.0)], 'fully-qualified-except-typing') == "Annotated[float, Gt(gt=-10.0)]"
+    assert stringify_annotation(Annotated[float, Gt(-10.0)], 'smart') == "~typing.Annotated[float, Gt(gt=-10.0)]"
 
 
 def test_stringify_Unpack():
@@ -662,7 +672,6 @@ def test_stringify_type_hints_alias():
 
 
 def test_stringify_type_Literal():
-    from typing import Literal  # type: ignore[attr-defined]
     assert stringify_annotation(Literal[1, "2", "\r"], 'fully-qualified-except-typing') == "Literal[1, '2', '\\r']"
     assert stringify_annotation(Literal[1, "2", "\r"], "fully-qualified") == "typing.Literal[1, '2', '\\r']"
     assert stringify_annotation(Literal[1, "2", "\r"], "smart") == "~typing.Literal[1, '2', '\\r']"
@@ -704,8 +713,6 @@ def test_stringify_mock():
 
 
 def test_stringify_type_ForwardRef():
-    from typing import ForwardRef  # type: ignore[attr-defined]
-
     assert stringify_annotation(ForwardRef("MyInt")) == "MyInt"
     assert stringify_annotation(ForwardRef("MyInt"), 'smart') == "MyInt"