Repository: sphinx-doc/sphinx

Issue:
Sphinx 7.4: autodoc: ``typing.Annotated`` failure with pydantic
I think I found a regression about this. We're using the `Annotated` class + pydantic to validate types of a user-defined config in conf.py...

```py
CSSClassType = Annotated[str, pydantic.AfterValidator(nodes.make_id)]

class CustomAdmonitionConfig(pydantic.BaseModel):
    title: Annotated[Optional[str], pydantic.Field(validate_default=True)] = None
    classes: List[CSSClassType] = []
```
But autodoc-ing `CustomAdmonitionConfig`'s `title` and `classes` attributes results in errors trying to reference the metadata:
> /opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/sphinx_immaterial/custom_admonitions.py:docstring of sphinx_immaterial.custom_admonitions.CustomAdmonitionConfig.title:1: WARNING: py:obj reference target not found: typing.Annotated[str | None, FieldInfo(annotation=NoneType, required=True, validate_default=True)]


> /opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/sphinx_immaterial/custom_admonitions.py:docstring of sphinx_immaterial.custom_admonitions.CustomAdmonitionConfig.classes:1: WARNING: py:obj reference target not found: typing.List[~typing.Annotated[str, AfterValidator(func=<function make_id at 0x7fd2a7b5cae0>)]]

I think lesson here is that not all metadata is compatible with autodoc. Is there a way to disable it?

_Originally posted by @2bndy5 in https://github.com/sphinx-doc/sphinx/issues/11785#issuecomment-2230045019_
            

Discussion:
2bndy5: Using the zip from #12601 thread, I now get the following nitpicky warnings:
```
path/to/issue#12601/annotated.py:docstring of annotated.CustomAdmonitionConfig:1: WARNING: py:class reference target not found: pydantic.functional_validators.AfterValidator
path/to/issue#12601/annotated.py:docstring of annotated.CustomAdmonitionConfig:1: WARNING: py:class reference target not found: docutils.nodes.make_id
path/to/issue#12601/annotated.py:docstring of annotated.CustomAdmonitionConfig.classes:1: WARNING: py:class reference target not found: pydantic.functional_validators.AfterValidator
path/to/issue#12601/annotated.py:docstring of annotated.CustomAdmonitionConfig.classes:1: WARNING: py:class reference target not found: docutils.nodes.make_id
path/to/issue#12601/annotated.py:docstring of annotated.CustomAdmonitionConfig.title:1: WARNING: py:class reference target not found: FieldInfo
path/to/issue#12601/annotated.py:docstring of annotated.CustomAdmonitionConfig.title:1: WARNING: py:class reference target not found: NoneType
```
after adding to conf.py:
```py
extensions = [
    "sphinx.ext.autodoc",
    "sphinx.ext.intersphinx",
]
autodoc_mock_imports = [
    "docutils",
    "pydantic",
]
intersphinx_mapping = {
    "python": ("https://docs.python.org/3", None),
}
```
I get the following nitpicky warnings:
```
path/to/issue#12601/annotated.py:docstring of annotated.CustomAdmonitionConfig.classes:1: WARNING: py:class reference target not found: pydantic.AfterValidator
path/to/issue#12601/annotated.py:docstring of annotated.CustomAdmonitionConfig.title:1: WARNING: py:class reference target not found: pydantic.Field
```
with the following HTML output:
![image](https://github.com/user-attachments/assets/dd7a73b7-b0c4-409c-a3ee-a1eb4721c753)
Overall, I think this is a considerable improvement. Thanks!

Patch:
diff --git a/CHANGES.rst b/CHANGES.rst
index c53bb8f8b..66150d262 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -11,6 +11,10 @@ Bugs fixed
 * #12601, #12625: Support callable objects in :py:class:`~typing.Annotated` type
   metadata in the Python domain.
   Patch by Adam Turner.
+* #12601, #12622: Resolve :py:class:`~typing.Annotated` warnings with
+  ``sphinx.ext.autodoc``,
+  especially when using :mod:`dataclasses` as type metadata.
+  Patch by Adam Turner.
 
 Release 7.4.6 (released Jul 18, 2024)
 =====================================
diff --git a/sphinx/ext/autodoc/__init__.py b/sphinx/ext/autodoc/__init__.py
index b3fb3e0c4..41e128e07 100644
--- a/sphinx/ext/autodoc/__init__.py
+++ b/sphinx/ext/autodoc/__init__.py
@@ -2008,7 +2008,8 @@ def import_object(self, raiseerror: bool = False) -> bool:
                 with mock(self.config.autodoc_mock_imports):
                     parent = import_module(self.modname, self.config.autodoc_warningiserror)
                     annotations = get_type_hints(parent, None,
-                                                 self.config.autodoc_type_aliases)
+                                                 self.config.autodoc_type_aliases,
+                                                 include_extras=True)
                     if self.objpath[-1] in annotations:
                         self.object = UNINITIALIZED_ATTR
                         self.parent = parent
@@ -2097,7 +2098,8 @@ def add_directive_header(self, sig: str) -> None:
             if self.config.autodoc_typehints != 'none':
                 # obtain annotation for this data
                 annotations = get_type_hints(self.parent, None,
-                                             self.config.autodoc_type_aliases)
+                                             self.config.autodoc_type_aliases,
+                                             include_extras=True)
                 if self.objpath[-1] in annotations:
                     if self.config.autodoc_typehints_format == "short":
                         objrepr = stringify_annotation(annotations.get(self.objpath[-1]),
@@ -2541,7 +2543,8 @@ class Foo:
 
     def is_uninitialized_instance_attribute(self, parent: Any) -> bool:
         """Check the subject is an annotation only attribute."""
-        annotations = get_type_hints(parent, None, self.config.autodoc_type_aliases)
+        annotations = get_type_hints(parent, None, self.config.autodoc_type_aliases,
+                                     include_extras=True)
         return self.objpath[-1] in annotations
 
     def import_object(self, raiseerror: bool = False) -> bool:
@@ -2673,7 +2676,8 @@ def add_directive_header(self, sig: str) -> None:
             if self.config.autodoc_typehints != 'none':
                 # obtain type annotation for this attribute
                 annotations = get_type_hints(self.parent, None,
-                                             self.config.autodoc_type_aliases)
+                                             self.config.autodoc_type_aliases,
+                                             include_extras=True)
                 if self.objpath[-1] in annotations:
                     if self.config.autodoc_typehints_format == "short":
                         objrepr = stringify_annotation(annotations.get(self.objpath[-1]),
diff --git a/sphinx/util/inspect.py b/sphinx/util/inspect.py
index 04595fd8c..28bba0c30 100644
--- a/sphinx/util/inspect.py
+++ b/sphinx/util/inspect.py
@@ -652,7 +652,7 @@ def signature(
     try:
         # Resolve annotations using ``get_type_hints()`` and type_aliases.
         localns = TypeAliasNamespace(type_aliases)
-        annotations = typing.get_type_hints(subject, None, localns)
+        annotations = typing.get_type_hints(subject, None, localns, include_extras=True)
         for i, param in enumerate(parameters):
             if param.name in annotations:
                 annotation = annotations[param.name]
diff --git a/sphinx/util/typing.py b/sphinx/util/typing.py
index a295c0605..a5d241705 100644
--- a/sphinx/util/typing.py
+++ b/sphinx/util/typing.py
@@ -2,6 +2,7 @@
 
 from __future__ import annotations
 
+import dataclasses
 import sys
 import types
 import typing
@@ -157,6 +158,7 @@ def get_type_hints(
     obj: Any,
     globalns: dict[str, Any] | None = None,
     localns: dict[str, Any] | None = None,
+    include_extras: bool = False,
 ) -> dict[str, Any]:
     """Return a dictionary containing type hints for a function, method, module or class
     object.
@@ -167,7 +169,7 @@ def get_type_hints(
     from sphinx.util.inspect import safe_getattr  # lazy loading
 
     try:
-        return typing.get_type_hints(obj, globalns, localns)
+        return typing.get_type_hints(obj, globalns, localns, include_extras=include_extras)
     except NameError:
         # Failed to evaluate ForwardRef (maybe TYPE_CHECKING)
         return safe_getattr(obj, '__annotations__', {})
@@ -267,7 +269,20 @@ def restify(cls: Any, mode: _RestifyMode = 'fully-qualified-except-typing') -> s
             return f':py:class:`{module_prefix}{_INVALID_BUILTIN_CLASSES[cls]}`'
         elif _is_annotated_form(cls):
             args = restify(cls.__args__[0], mode)
-            meta = ', '.join(map(repr, cls.__metadata__))
+            meta_args = []
+            for m in cls.__metadata__:
+                if isinstance(m, type):
+                    meta_args.append(restify(m, mode))
+                elif dataclasses.is_dataclass(m):
+                    # use restify for the repr of field values rather than repr
+                    d_fields = ', '.join([
+                        fr"{f.name}=\ {restify(getattr(m, f.name), mode)}"
+                        for f in dataclasses.fields(m) if f.repr
+                    ])
+                    meta_args.append(fr'{restify(type(m), mode)}\ ({d_fields})')
+                else:
+                    meta_args.append(repr(m))
+            meta = ', '.join(meta_args)
             if sys.version_info[:2] <= (3, 11):
                 # Hardcoded to fix errors on Python 3.11 and earlier.
                 return fr':py:class:`~typing.Annotated`\ [{args}, {meta}]'
@@ -510,7 +525,25 @@ def stringify_annotation(
             return f'{module_prefix}Literal[{args}]'
         elif _is_annotated_form(annotation):  # for py39+
             args = stringify_annotation(annotation_args[0], mode)
-            meta = ', '.join(map(repr, annotation.__metadata__))
+            meta_args = []
+            for m in annotation.__metadata__:
+                if isinstance(m, type):
+                    meta_args.append(stringify_annotation(m, mode))
+                elif dataclasses.is_dataclass(m):
+                    # use stringify_annotation for the repr of field values rather than repr
+                    d_fields = ', '.join([
+                        f"{f.name}={stringify_annotation(getattr(m, f.name), mode)}"
+                        for f in dataclasses.fields(m) if f.repr
+                    ])
+                    meta_args.append(f'{stringify_annotation(type(m), mode)}({d_fields})')
+                else:
+                    meta_args.append(repr(m))
+            meta = ', '.join(meta_args)
+            if sys.version_info[:2] <= (3, 9):
+                if mode == 'smart':
+                    return f'~typing.Annotated[{args}, {meta}]'
+                if mode == 'fully-qualified':
+                    return f'typing.Annotated[{args}, {meta}]'
             if sys.version_info[:2] <= (3, 11):
                 if mode == 'fully-qualified-except-typing':
                     return f'Annotated[{args}, {meta}]'


Test patch:
diff --git a/tests/roots/test-ext-autodoc/target/annotated.py b/tests/roots/test-ext-autodoc/target/annotated.py
index 5b87518f9..7adc3e0f1 100644
--- a/tests/roots/test-ext-autodoc/target/annotated.py
+++ b/tests/roots/test-ext-autodoc/target/annotated.py
@@ -1,8 +1,42 @@
-from __future__ import annotations
+# from __future__ import annotations
 
+import dataclasses
+import types
 from typing import Annotated
 
 
+@dataclasses.dataclass(frozen=True)
+class FuncValidator:
+    func: types.FunctionType
+
+
+@dataclasses.dataclass(frozen=True)
+class MaxLen:
+    max_length: int
+    whitelisted_words: list[str]
+
+
+def validate(value: str) -> str:
+    return value
+
+
+#: Type alias for a validated string.
+ValidatedString = Annotated[str, FuncValidator(validate)]
+
+
 def hello(name: Annotated[str, "attribute"]) -> None:
     """docstring"""
     pass
+
+
+class AnnotatedAttributes:
+    """docstring"""
+
+    #: Docstring about the ``name`` attribute.
+    name: Annotated[str, "attribute"]
+
+    #: Docstring about the ``max_len`` attribute.
+    max_len: list[Annotated[str, MaxLen(10, ['word_one', 'word_two'])]]
+
+    #: Docstring about the ``validated`` attribute.
+    validated: ValidatedString
diff --git a/tests/test_extensions/test_ext_autodoc.py b/tests/test_extensions/test_ext_autodoc.py
index 53289f32c..e10850bac 100644
--- a/tests/test_extensions/test_ext_autodoc.py
+++ b/tests/test_extensions/test_ext_autodoc.py
@@ -2321,18 +2321,62 @@ def test_autodoc_TypeVar(app):
 
 @pytest.mark.sphinx('html', testroot='ext-autodoc')
 def test_autodoc_Annotated(app):
-    options = {"members": None}
+    options = {'members': None, 'member-order': 'bysource'}
     actual = do_autodoc(app, 'module', 'target.annotated', options)
     assert list(actual) == [
         '',
         '.. py:module:: target.annotated',
         '',
         '',
-        '.. py:function:: hello(name: str) -> None',
+        '.. py:class:: FuncValidator(func: function)',
+        '   :module: target.annotated',
+        '',
+        '',
+        '.. py:class:: MaxLen(max_length: int, whitelisted_words: list[str])',
+        '   :module: target.annotated',
+        '',
+        '',
+        '.. py:data:: ValidatedString',
+        '   :module: target.annotated',
+        '',
+        '   Type alias for a validated string.',
+        '',
+        '   alias of :py:class:`~typing.Annotated`\\ [:py:class:`str`, '
+        ':py:class:`~target.annotated.FuncValidator`\\ (func=\\ :py:class:`~target.annotated.validate`)]',
+        '',
+        '',
+        ".. py:function:: hello(name: ~typing.Annotated[str, 'attribute']) -> None",
+        '   :module: target.annotated',
+        '',
+        '   docstring',
+        '',
+        '',
+        '.. py:class:: AnnotatedAttributes()',
         '   :module: target.annotated',
         '',
         '   docstring',
         '',
+        '',
+        '   .. py:attribute:: AnnotatedAttributes.name',
+        '      :module: target.annotated',
+        "      :type: ~typing.Annotated[str, 'attribute']",
+        '',
+        '      Docstring about the ``name`` attribute.',
+        '',
+        '',
+        '   .. py:attribute:: AnnotatedAttributes.max_len',
+        '      :module: target.annotated',
+        "      :type: list[~typing.Annotated[str, ~target.annotated.MaxLen(max_length=10, whitelisted_words=['word_one', 'word_two'])]]",
+        '',
+        '      Docstring about the ``max_len`` attribute.',
+        '',
+        '',
+        '   .. py:attribute:: AnnotatedAttributes.validated',
+        '      :module: target.annotated',
+        '      :type: ~typing.Annotated[str, ~target.annotated.FuncValidator(func=~target.annotated.validate)]',
+        '',
+        '      Docstring about the ``validated`` attribute.',
+        '',
     ]
 
 
diff --git a/tests/test_util/test_util_typing.py b/tests/test_util/test_util_typing.py
index d00d69fb0..956cffe9d 100644
--- a/tests/test_util/test_util_typing.py
+++ b/tests/test_util/test_util_typing.py
@@ -196,8 +196,8 @@ def test_restify_type_hints_containers():
 def test_restify_Annotated():
     assert restify(Annotated[str, "foo", "bar"]) == ":py:class:`~typing.Annotated`\\ [:py:class:`str`, 'foo', 'bar']"
     assert restify(Annotated[str, "foo", "bar"], 'smart') == ":py:class:`~typing.Annotated`\\ [:py:class:`str`, 'foo', 'bar']"
-    assert restify(Annotated[float, Gt(-10.0)]) == ':py:class:`~typing.Annotated`\\ [:py:class:`float`, Gt(gt=-10.0)]'
-    assert restify(Annotated[float, Gt(-10.0)], 'smart') == ':py:class:`~typing.Annotated`\\ [:py:class:`float`, Gt(gt=-10.0)]'
+    assert restify(Annotated[float, Gt(-10.0)]) == ':py:class:`~typing.Annotated`\\ [:py:class:`float`, :py:class:`tests.test_util.test_util_typing.Gt`\\ (gt=\\ -10.0)]'
+    assert restify(Annotated[float, Gt(-10.0)], 'smart') == ':py:class:`~typing.Annotated`\\ [:py:class:`float`, :py:class:`~tests.test_util.test_util_typing.Gt`\\ (gt=\\ -10.0)]'
 
 
 def test_restify_type_hints_Callable():
@@ -521,12 +521,11 @@ def test_stringify_type_hints_pep_585():
     assert stringify_annotation(tuple[List[dict[int, str]], str, ...], "smart") == "tuple[~typing.List[dict[int, str]], str, ...]"
 
 
-@pytest.mark.xfail(sys.version_info[:2] <= (3, 9), reason='Needs fixing.')
 def test_stringify_Annotated():
     assert stringify_annotation(Annotated[str, "foo", "bar"], 'fully-qualified-except-typing') == "Annotated[str, 'foo', 'bar']"
     assert stringify_annotation(Annotated[str, "foo", "bar"], 'smart') == "~typing.Annotated[str, 'foo', 'bar']"
-    assert stringify_annotation(Annotated[float, Gt(-10.0)], 'fully-qualified-except-typing') == "Annotated[float, Gt(gt=-10.0)]"
-    assert stringify_annotation(Annotated[float, Gt(-10.0)], 'smart') == "~typing.Annotated[float, Gt(gt=-10.0)]"
+    assert stringify_annotation(Annotated[float, Gt(-10.0)], 'fully-qualified-except-typing') == "Annotated[float, tests.test_util.test_util_typing.Gt(gt=-10.0)]"
+    assert stringify_annotation(Annotated[float, Gt(-10.0)], 'smart') == "~typing.Annotated[float, ~tests.test_util.test_util_typing.Gt(gt=-10.0)]"
 
 
 def test_stringify_Unpack():