Repository: sphinx-doc/sphinx

Issue:
Sphinx 7.4: autodoc: ``typing.Annotated`` failure with pydantic
I think I found a regression about this. We're using the `Annotated` class + pydantic to validate types of a user-defined config in conf.py...

```py
CSSClassType = Annotated[str, pydantic.AfterValidator(nodes.make_id)]

class CustomAdmonitionConfig(pydantic.BaseModel):
    title: Annotated[Optional[str], pydantic.Field(validate_default=True)] = None
    classes: List[CSSClassType] = []
```
But autodoc-ing `CustomAdmonitionConfig`'s `title` and `classes` attributes results in errors trying to reference the metadata:
> /opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/sphinx_immaterial/custom_admonitions.py:docstring of sphinx_immaterial.custom_admonitions.CustomAdmonitionConfig.title:1: WARNING: py:obj reference target not found: typing.Annotated[str | None, FieldInfo(annotation=NoneType, required=True, validate_default=True)]


> /opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/sphinx_immaterial/custom_admonitions.py:docstring of sphinx_immaterial.custom_admonitions.CustomAdmonitionConfig.classes:1: WARNING: py:obj reference target not found: typing.List[~typing.Annotated[str, AfterValidator(func=<function make_id at 0x7fd2a7b5cae0>)]]

I think lesson here is that not all metadata is compatible with autodoc. Is there a way to disable it?

_Originally posted by @2bndy5 in https://github.com/sphinx-doc/sphinx/issues/11785#issuecomment-2230045019_
            


Discussion:
I'm not sure there's anything autodoc or intersphinx can do about this. Pydantic doesn't use sphinx to generate the their docs (sadly).

Honestly, this "feature" should be hidden behind a config option until if/when it is stable to use in autodoc.
Hi @2bndy5 are you able to help with a reproducer? I currently have the below, which doesn't generate the cited warnings/errors:

**`conf.py`**
```python
extensions = ['sphinx.ext.autodoc']
nitpicky = True
```

**`index.rst`**
```rst
.. automodule:: bug
   :members:
```

**`bug.py`**
```python
from typing import List, Optional, Annotated

from docutils import nodes
import pydantic

CSSClassType = Annotated[str, pydantic.AfterValidator(nodes.make_id)]


class CustomAdmonitionConfig(pydantic.BaseModel):
    title: Annotated[Optional[str], pydantic.Field(validate_default=True)] = None
    classes: List[CSSClassType] = []
```

**Output:**
```ps1con
PS> python -m sphinx build -M html . _build -T
<...output elided...>
...\bug.py:docstring of bug.CustomAdmonitionConfig.model_computed_fields:1: WARNING: py:class reference target not found: ComputedFieldInfo
...\bug.py:docstring of bug.CustomAdmonitionConfig.model_config:1: WARNING: py:class reference target not found: ConfigDict
...\bug.py:docstring of bug.CustomAdmonitionConfig.model_fields:1: WARNING: py:class reference target not found: FieldInfo
<...output elided...>
```


A
Example was taken straight from sphinx-immaterial docs. Maybe the theme does something extra to return types... I'll look into it.
I haven't narrowed it down yet, but I was able to suppress the warnings using `autoattribute`'s `:annotation:` option:
```rst
   .. autoclass:: sphinx_immaterial.custom_admonitions.CustomAdmonitionConfig
      :exclude-members: __new__, __init__

      .. autoattribute:: name
      .. autoattribute:: title
         :annotation: : str | None
      .. autoattribute:: icon
      .. autoattribute:: color
      .. autoattribute:: classes
         :annotation: : List[str]
      .. autoattribute:: override
```
<details><summary>Some theme-specific details.</summary>

So, the sphinx-immaterial theme (in `sphinx_immaterial.apidoc.python.attribute_style`) monkey-patches the `handle_signature()` function of `sphinx.domains.python.PyAttribute` (and `PyProperty`) to

1. add a punctuation to the signature
   - `sphinx.addnodes.desc_sig_punctuation("", " : ")` for a type
   - `sphinx.addnodes.desc_sig_punctuation("", " = ")` for a value
2. convert the signature's type into xrefs using a monkey-patched `sphinx.domains.python._parse_annotation()`

I'll discuss a solution downstream. It doesn't seem like the problem is specific to our monkey patches (when comparing the original code with the patched code).

</details>

@AA-Turner The example you posted is sufficient to reproduce, but we build our docs with both `--nitpicky --fail-on-warning` options enabled.

I can't just ignore a nitpicky warning that shows a function's memory offset (the `__str__()` output of `docutils.nodes.make_id()`) because the offset is different on every build, That is unless the `nitpicky_ignore` field supports a glob or regex pattern of which I'm unaware.
> That is unless the nitpicky_ignore field supports a glob or regex pattern of which I'm unaware.

Today is your lucky day: https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-nitpick_ignore_regex

A
I would still be interested in a reproducer that has new failures on 7.4 compared to 7.3 so that I can release a fix, if you have one.

A
I ended up factoring out our use of the `Annotated` class as an attribute type (see https://github.com/jbms/sphinx-immaterial/pull/370/commits/a1b0bfd29dfc98965c6394ed7eef46eff81f1c3c), at least for the documented API. Pydantic is refreshingly flexible. It just took a little more code to achieve the same behavior.

> Today is your lucky day

I'm glad to see I have multiple options to recommend others that may come across this (or a similar) problem. 👍🏼

> I would still be interested in a reproducer

You're example does reproduce it but it needs the `--fail-on-warning` option added to `sphinx-build` command. Just to be proper, I packaged it up into a zip file (with a requirements.txt file) and added to `:members:` for conciseness.

Steps: to reproduce
1. extract the zip: [issue#12601.zip](https://github.com/user-attachments/files/16279197/issue.12601.zip)
2. navigate to "issue#12601" directory
3. ```
   pip install -r requirements.txt
   ```
4. ```
   sphinx-build . ./_build -W --keep-going
   ```

> release a fix, if you have one.

I don't have a fix other than ways to avoid the warning. I just don't know the domain API well enough. Downstream, we were considering a theme-specific config option to help filter the transformations of type annotations (see [our docs](https://jbms.github.io/sphinx-immaterial/apidoc/python/index.html) for options already implemented), but nitpicky warnings are often ignored in the wild.
Ideally for sphinx maybe, but is it really pydantic's responsibility to provide representation of function pointers specified from user-space?

I don't think falling back to `__repr__()` is the best last resort, especially if Python's default implementation does not include the info that `ast.parse()` is looking for. It my understanding that the `ast` module's source is beyond the scope of Sphinx (as with the `inspect` module).

---
There is a similar problem with using an instantiated object as a parameter default value.
```py
# in defaults.py
class A:
    """Some class A doc"""
    pass

def some_fn(param: A = A()):
    """Some function doc"""
    pass
``` 

![image](https://github.com/user-attachments/assets/b50d3c07-e633-4701-82a4-16171c570ae9)

We've come across yet another seemingly similar problem in https://github.com/jbms/sphinx-immaterial/issues/140
https://github.com/AA-Turner/sphinx/tree/fix-annotated-autodoc has my working progress.

There are two parts to fix; autodoc's generation of reStructuredText source, and then the Python domain's parsing and resolution of those targets. I have the second fixed, but not the first.

A



Patch:
diff --git a/CHANGES.rst b/CHANGES.rst
index c53bb8f8b0f..66150d2624a 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -11,6 +11,10 @@ Bugs fixed
 * #12601, #12625: Support callable objects in :py:class:`~typing.Annotated` type
   metadata in the Python domain.
   Patch by Adam Turner.
+* #12601, #12622: Resolve :py:class:`~typing.Annotated` warnings with
+  ``sphinx.ext.autodoc``,
+  especially when using :mod:`dataclasses` as type metadata.
+  Patch by Adam Turner.
 
 Release 7.4.6 (released Jul 18, 2024)
 =====================================
diff --git a/sphinx/ext/autodoc/__init__.py b/sphinx/ext/autodoc/__init__.py
index b3fb3e0c41f..41e128e0732 100644
--- a/sphinx/ext/autodoc/__init__.py
+++ b/sphinx/ext/autodoc/__init__.py
@@ -2008,7 +2008,8 @@ def import_object(self, raiseerror: bool = False) -> bool:
                 with mock(self.config.autodoc_mock_imports):
                     parent = import_module(self.modname, self.config.autodoc_warningiserror)
                     annotations = get_type_hints(parent, None,
-                                                 self.config.autodoc_type_aliases)
+                                                 self.config.autodoc_type_aliases,
+                                                 include_extras=True)
                     if self.objpath[-1] in annotations:
                         self.object = UNINITIALIZED_ATTR
                         self.parent = parent
@@ -2097,7 +2098,8 @@ def add_directive_header(self, sig: str) -> None:
             if self.config.autodoc_typehints != 'none':
                 # obtain annotation for this data
                 annotations = get_type_hints(self.parent, None,
-                                             self.config.autodoc_type_aliases)
+                                             self.config.autodoc_type_aliases,
+                                             include_extras=True)
                 if self.objpath[-1] in annotations:
                     if self.config.autodoc_typehints_format == "short":
                         objrepr = stringify_annotation(annotations.get(self.objpath[-1]),
@@ -2541,7 +2543,8 @@ class Foo:
 
     def is_uninitialized_instance_attribute(self, parent: Any) -> bool:
         """Check the subject is an annotation only attribute."""
-        annotations = get_type_hints(parent, None, self.config.autodoc_type_aliases)
+        annotations = get_type_hints(parent, None, self.config.autodoc_type_aliases,
+                                     include_extras=True)
         return self.objpath[-1] in annotations
 
     def import_object(self, raiseerror: bool = False) -> bool:
@@ -2673,7 +2676,8 @@ def add_directive_header(self, sig: str) -> None:
             if self.config.autodoc_typehints != 'none':
                 # obtain type annotation for this attribute
                 annotations = get_type_hints(self.parent, None,
-                                             self.config.autodoc_type_aliases)
+                                             self.config.autodoc_type_aliases,
+                                             include_extras=True)
                 if self.objpath[-1] in annotations:
                     if self.config.autodoc_typehints_format == "short":
                         objrepr = stringify_annotation(annotations.get(self.objpath[-1]),
diff --git a/sphinx/util/inspect.py b/sphinx/util/inspect.py
index 04595fd8c58..28bba0c3032 100644
--- a/sphinx/util/inspect.py
+++ b/sphinx/util/inspect.py
@@ -652,7 +652,7 @@ def signature(
     try:
         # Resolve annotations using ``get_type_hints()`` and type_aliases.
         localns = TypeAliasNamespace(type_aliases)
-        annotations = typing.get_type_hints(subject, None, localns)
+        annotations = typing.get_type_hints(subject, None, localns, include_extras=True)
         for i, param in enumerate(parameters):
             if param.name in annotations:
                 annotation = annotations[param.name]
diff --git a/sphinx/util/typing.py b/sphinx/util/typing.py
index a295c0605e6..a5d24170576 100644
--- a/sphinx/util/typing.py
+++ b/sphinx/util/typing.py
@@ -2,6 +2,7 @@
 
 from __future__ import annotations
 
+import dataclasses
 import sys
 import types
 import typing
@@ -157,6 +158,7 @@ def get_type_hints(
     obj: Any,
     globalns: dict[str, Any] | None = None,
     localns: dict[str, Any] | None = None,
+    include_extras: bool = False,
 ) -> dict[str, Any]:
     """Return a dictionary containing type hints for a function, method, module or class
     object.
@@ -167,7 +169,7 @@ def get_type_hints(
     from sphinx.util.inspect import safe_getattr  # lazy loading
 
     try:
-        return typing.get_type_hints(obj, globalns, localns)
+        return typing.get_type_hints(obj, globalns, localns, include_extras=include_extras)
     except NameError:
         # Failed to evaluate ForwardRef (maybe TYPE_CHECKING)
         return safe_getattr(obj, '__annotations__', {})
@@ -267,7 +269,20 @@ def restify(cls: Any, mode: _RestifyMode = 'fully-qualified-except-typing') -> s
             return f':py:class:`{module_prefix}{_INVALID_BUILTIN_CLASSES[cls]}`'
         elif _is_annotated_form(cls):
             args = restify(cls.__args__[0], mode)
-            meta = ', '.join(map(repr, cls.__metadata__))
+            meta_args = []
+            for m in cls.__metadata__:
+                if isinstance(m, type):
+                    meta_args.append(restify(m, mode))
+                elif dataclasses.is_dataclass(m):
+                    # use restify for the repr of field values rather than repr
+                    d_fields = ', '.join([
+                        fr"{f.name}=\ {restify(getattr(m, f.name), mode)}"
+                        for f in dataclasses.fields(m) if f.repr
+                    ])
+                    meta_args.append(fr'{restify(type(m), mode)}\ ({d_fields})')
+                else:
+                    meta_args.append(repr(m))
+            meta = ', '.join(meta_args)
             if sys.version_info[:2] <= (3, 11):
                 # Hardcoded to fix errors on Python 3.11 and earlier.
                 return fr':py:class:`~typing.Annotated`\ [{args}, {meta}]'
@@ -510,7 +525,25 @@ def stringify_annotation(
             return f'{module_prefix}Literal[{args}]'
         elif _is_annotated_form(annotation):  # for py39+
             args = stringify_annotation(annotation_args[0], mode)
-            meta = ', '.join(map(repr, annotation.__metadata__))
+            meta_args = []
+            for m in annotation.__metadata__:
+                if isinstance(m, type):
+                    meta_args.append(stringify_annotation(m, mode))
+                elif dataclasses.is_dataclass(m):
+                    # use stringify_annotation for the repr of field values rather than repr
+                    d_fields = ', '.join([
+                        f"{f.name}={stringify_annotation(getattr(m, f.name), mode)}"
+                        for f in dataclasses.fields(m) if f.repr
+                    ])
+                    meta_args.append(f'{stringify_annotation(type(m), mode)}({d_fields})')
+                else:
+                    meta_args.append(repr(m))
+            meta = ', '.join(meta_args)
+            if sys.version_info[:2] <= (3, 9):
+                if mode == 'smart':
+                    return f'~typing.Annotated[{args}, {meta}]'
+                if mode == 'fully-qualified':
+                    return f'typing.Annotated[{args}, {meta}]'
             if sys.version_info[:2] <= (3, 11):
                 if mode == 'fully-qualified-except-typing':
                     return f'Annotated[{args}, {meta}]'


Test patch:
diff --git a/tests/roots/test-ext-autodoc/target/annotated.py b/tests/roots/test-ext-autodoc/target/annotated.py
index 5b87518f968..7adc3e0f152 100644
--- a/tests/roots/test-ext-autodoc/target/annotated.py
+++ b/tests/roots/test-ext-autodoc/target/annotated.py
@@ -1,8 +1,42 @@
-from __future__ import annotations
+# from __future__ import annotations
 
+import dataclasses
+import types
 from typing import Annotated
 
 
+@dataclasses.dataclass(frozen=True)
+class FuncValidator:
+    func: types.FunctionType
+
+
+@dataclasses.dataclass(frozen=True)
+class MaxLen:
+    max_length: int
+    whitelisted_words: list[str]
+
+
+def validate(value: str) -> str:
+    return value
+
+
+#: Type alias for a validated string.
+ValidatedString = Annotated[str, FuncValidator(validate)]
+
+
 def hello(name: Annotated[str, "attribute"]) -> None:
     """docstring"""
     pass
+
+
+class AnnotatedAttributes:
+    """docstring"""
+
+    #: Docstring about the ``name`` attribute.
+    name: Annotated[str, "attribute"]
+
+    #: Docstring about the ``max_len`` attribute.
+    max_len: list[Annotated[str, MaxLen(10, ['word_one', 'word_two'])]]
+
+    #: Docstring about the ``validated`` attribute.
+    validated: ValidatedString
diff --git a/tests/test_extensions/test_ext_autodoc.py b/tests/test_extensions/test_ext_autodoc.py
index 53289f32c1a..e10850bacd7 100644
--- a/tests/test_extensions/test_ext_autodoc.py
+++ b/tests/test_extensions/test_ext_autodoc.py
@@ -2321,18 +2321,62 @@ def test_autodoc_TypeVar(app):
 
 @pytest.mark.sphinx('html', testroot='ext-autodoc')
 def test_autodoc_Annotated(app):
-    options = {"members": None}
+    options = {'members': None, 'member-order': 'bysource'}
     actual = do_autodoc(app, 'module', 'target.annotated', options)
     assert list(actual) == [
         '',
         '.. py:module:: target.annotated',
         '',
         '',
-        '.. py:function:: hello(name: str) -> None',
+        '.. py:class:: FuncValidator(func: function)',
+        '   :module: target.annotated',
+        '',
+        '',
+        '.. py:class:: MaxLen(max_length: int, whitelisted_words: list[str])',
+        '   :module: target.annotated',
+        '',
+        '',
+        '.. py:data:: ValidatedString',
+        '   :module: target.annotated',
+        '',
+        '   Type alias for a validated string.',
+        '',
+        '   alias of :py:class:`~typing.Annotated`\\ [:py:class:`str`, '
+        ':py:class:`~target.annotated.FuncValidator`\\ (func=\\ :py:class:`~target.annotated.validate`)]',
+        '',
+        '',
+        ".. py:function:: hello(name: ~typing.Annotated[str, 'attribute']) -> None",
+        '   :module: target.annotated',
+        '',
+        '   docstring',
+        '',
+        '',
+        '.. py:class:: AnnotatedAttributes()',
         '   :module: target.annotated',
         '',
         '   docstring',
         '',
+        '',
+        '   .. py:attribute:: AnnotatedAttributes.name',
+        '      :module: target.annotated',
+        "      :type: ~typing.Annotated[str, 'attribute']",
+        '',
+        '      Docstring about the ``name`` attribute.',
+        '',
+        '',
+        '   .. py:attribute:: AnnotatedAttributes.max_len',
+        '      :module: target.annotated',
+        "      :type: list[~typing.Annotated[str, ~target.annotated.MaxLen(max_length=10, whitelisted_words=['word_one', 'word_two'])]]",
+        '',
+        '      Docstring about the ``max_len`` attribute.',
+        '',
+        '',
+        '   .. py:attribute:: AnnotatedAttributes.validated',
+        '      :module: target.annotated',
+        '      :type: ~typing.Annotated[str, ~target.annotated.FuncValidator(func=~target.annotated.validate)]',
+        '',
+        '      Docstring about the ``validated`` attribute.',
+        '',
     ]
 
 
diff --git a/tests/test_util/test_util_typing.py b/tests/test_util/test_util_typing.py
index d00d69fb04f..956cffe9dec 100644
--- a/tests/test_util/test_util_typing.py
+++ b/tests/test_util/test_util_typing.py
@@ -196,8 +196,8 @@ def test_restify_type_hints_containers():
 def test_restify_Annotated():
     assert restify(Annotated[str, "foo", "bar"]) == ":py:class:`~typing.Annotated`\\ [:py:class:`str`, 'foo', 'bar']"
     assert restify(Annotated[str, "foo", "bar"], 'smart') == ":py:class:`~typing.Annotated`\\ [:py:class:`str`, 'foo', 'bar']"
-    assert restify(Annotated[float, Gt(-10.0)]) == ':py:class:`~typing.Annotated`\\ [:py:class:`float`, Gt(gt=-10.0)]'
-    assert restify(Annotated[float, Gt(-10.0)], 'smart') == ':py:class:`~typing.Annotated`\\ [:py:class:`float`, Gt(gt=-10.0)]'
+    assert restify(Annotated[float, Gt(-10.0)]) == ':py:class:`~typing.Annotated`\\ [:py:class:`float`, :py:class:`tests.test_util.test_util_typing.Gt`\\ (gt=\\ -10.0)]'
+    assert restify(Annotated[float, Gt(-10.0)], 'smart') == ':py:class:`~typing.Annotated`\\ [:py:class:`float`, :py:class:`~tests.test_util.test_util_typing.Gt`\\ (gt=\\ -10.0)]'
 
 
 def test_restify_type_hints_Callable():
@@ -521,12 +521,11 @@ def test_stringify_type_hints_pep_585():
     assert stringify_annotation(tuple[List[dict[int, str]], str, ...], "smart") == "tuple[~typing.List[dict[int, str]], str, ...]"
 
 
-@pytest.mark.xfail(sys.version_info[:2] <= (3, 9), reason='Needs fixing.')
 def test_stringify_Annotated():
     assert stringify_annotation(Annotated[str, "foo", "bar"], 'fully-qualified-except-typing') == "Annotated[str, 'foo', 'bar']"
     assert stringify_annotation(Annotated[str, "foo", "bar"], 'smart') == "~typing.Annotated[str, 'foo', 'bar']"
-    assert stringify_annotation(Annotated[float, Gt(-10.0)], 'fully-qualified-except-typing') == "Annotated[float, Gt(gt=-10.0)]"
-    assert stringify_annotation(Annotated[float, Gt(-10.0)], 'smart') == "~typing.Annotated[float, Gt(gt=-10.0)]"
+    assert stringify_annotation(Annotated[float, Gt(-10.0)], 'fully-qualified-except-typing') == "Annotated[float, tests.test_util.test_util_typing.Gt(gt=-10.0)]"
+    assert stringify_annotation(Annotated[float, Gt(-10.0)], 'smart') == "~typing.Annotated[float, ~tests.test_util.test_util_typing.Gt(gt=-10.0)]"
 
 
 def test_stringify_Unpack():