Repository: pydantic/pydantic
Pull request: Fix various issues with dataclasses and `use_attribute_docstrings`
<!-- Thank you for your contribution! -->
<!-- Unless your change is trivial, please create an issue to discuss the change before creating a PR -->

## Change Summary

Fixes https://github.com/pydantic/pydantic/issues/11243.

Edit post 2.11 release: fixes https://github.com/pydantic/pydantic/issues/11133.

Best reviewed commit per commit.

<!-- Please give a short summary of the changes. -->

## Related issue number

<!-- WARNING: please use "fix #123" style references so the issue is closed when this PR is merged. -->

## Checklist

* [ ] The pull request title is a good summary of the changes - it will be used in the changelog
* [ ] Unit tests for the changes exist
* [ ] Tests pass on CI
* [ ] Documentation reflects the changes where applicable
* [ ] My PR is ready to review, **please add a comment including the phrase "please review" to assign reviewers**


Problem statement:
BeforeValidator `func` called twice when input field value is json object in `pydantic.dataclass`
### Initial Checks

- [X] I confirm that I'm using Pydantic V2

### Description

func `_json_to_dict` called twice when pydantic.dataclass A init, first param in is json str, second is dict. But it called right when use BaseModel AA.
I added a conditional `if isinstance(data, dict): return data` to avoid exception. Or it report excpetion `TypeError: the JSON object must be str, bytes or bytearray, not dict`.
My local pydantic version is 2.5.3 I tried to upgrade to 2.9.2 but not work.

### Example Code

```Python
# coding: utf-8

import json
import typing as t

from pydantic import ConfigDict, Field, BeforeValidator, BaseModel
from pydantic.dataclasses import dataclass


def _json_to_dict(data: str) -> dict:

    print("_json_to_dict", type(data), data)
    if not data:
        return {}
    # fixme: avoid exception
    if isinstance(data, dict):
        return data
    return json.loads(data)


def _to_str(data: t.Any) -> str:

    print("_to_str", type(data), data)
    if isinstance(data, str):
        return data
    if not data:
        return ""
    return str(data)


_CONFIG = ConfigDict(
    extra="allow",
    validate_assignment=True,
    arbitrary_types_allowed=True,
    coerce_numbers_to_str=True)


@dataclass(config=_CONFIG)
class B:

    a: t.Annotated[str, BeforeValidator(_to_str)] = "4"


@dataclass(config=_CONFIG)
class A:

    b: t.Annotated[B, BeforeValidator(_json_to_dict)] = Field(default_factory=B)


class AA(BaseModel):

    b: t.Annotated[B, BeforeValidator(_json_to_dict)] = Field(default_factory=B)


if __name__ == '__main__':

    data = {"b": json.dumps({"a": None, "b": 200})}
    a = A(**data)  # _json_to_dict <class 'str'> {"a": null, "b": 200}
                   # _json_to_dict <class 'dict'> {'a': None, 'b': 200}
    print(a)
    aa = AA(**data)  # _json_to_dict <class 'str'> {"a": null, "b": 200}
    print(aa)

```


### Python, Pydantic & OS Version

```Text
pydantic version: 2.9.2
        pydantic-core version: 2.23.4
          pydantic-core build: profile=release pgo=false
                 install path: /opt/miniconda3/envs/test39/lib/python3.9/site-packages/pydantic
               python version: 3.9.20 (main, Oct  3 2024, 02:27:54)  [Clang 14.0.6 ]
                     platform: macOS-10.15.7-x86_64-i386-64bit
             related packages: typing_extensions-4.12.2
                       commit: unknown
```


Automatic field description from docstrings for Pydantic dataclasses
### Initial Checks

- [X] I confirm that I'm using Pydantic V2

### Description

Unlike `BaseModel` and `TypedDict`, description of fields from Pydantic dataclasses is NOT automatically populated from docstrings (despite using `ConfigDict(use_attribute_docstrings=True)`)

### Example Code

```Python
from typing_extensions import TypedDict

from pydantic import BaseModel, TypeAdapter, ConfigDict, with_config
from pydantic.dataclasses import dataclass

PYDANTIC_CFG = ConfigDict(use_attribute_docstrings=True)


class NativeModel(BaseModel):
    model_config = PYDANTIC_CFG

    x: int
    """my docstring"""


@with_config(PYDANTIC_CFG)
class TypedDictModel(TypedDict):
    x: int
    """my docstring"""


@dataclass(config=PYDANTIC_CFG)
class DataclassModel:
    x: int
    """my docstring"""


print(NativeModel.model_json_schema())
# {'properties': {'x': {'description': 'my docstring', 'title': 'X', 'type': 'integer'}}, 'required': ['x'], 'title': 'NativeModel', 'type': 'object'}

print(TypeAdapter(TypedDictModel).json_schema())
# {'properties': {'x': {'description': 'my docstring', 'title': 'X', 'type': 'integer'}}, 'required': ['x'], 'title': 'TypedDictModel', 'type': 'object'}

print(TypeAdapter(DataclassModel).json_schema())
# {'properties': {'x': {'title': 'X', 'type': 'integer'}}, 'required': ['x'], 'title': 'DataclassModel', 'type': 'object'}
```


### Python, Pydantic & OS Version

```Text
pydantic version: 2.10.4
        pydantic-core version: 2.27.2
          pydantic-core build: profile=release pgo=false
               python version: 3.11.9 (main, May  2 2024, 10:11:35) [GCC 12.2.0]
                     platform: Linux-6.1.0-28-amd64-x86_64-with-glibc2.36
             related packages: typing_extensions-4.12.2
```

Hints from issue discussion:
Are you running this in an interactive terminal? If so, this won't work as described in [the documentation](https://docs.pydantic.dev/latest/api/config/#pydantic.config.ConfigDict.use_attribute_docstrings):

> This requires the source code of the class to be available at runtime.
> Are you running this in an interactive terminal? If so, this won't work as described in [the documentation](https://docs.pydantic.dev/latest/api/config/#pydantic.config.ConfigDict.use_attribute_docstrings):
> 
> > This requires the source code of the class to be available at runtime.

No no this is run as a regular Python program: models based on `pydantic.BaseModel` and `typing.TypedDict` are just fine, only models based on `pydantic.dataclasses.dataclass` do not have the correct behavior :(
Ah, yes, I can reproduce.

Commit message: Support `use_attribute_docstrings` with stdlib dataclasses

Patch to review:
diff --git a/pydantic/_internal/_fields.py b/pydantic/_internal/_fields.py
index 0bbbdd33c2d..7cf61461d4f 100644
--- a/pydantic/_internal/_fields.py
+++ b/pydantic/_internal/_fields.py
@@ -64,12 +64,11 @@ def __init__(self, metadata: Any):
     return _PydanticGeneralMetadata  # type: ignore
 
 
-def _update_fields_from_docstrings(cls: type[Any], fields: dict[str, FieldInfo], config_wrapper: ConfigWrapper) -> None:
-    if config_wrapper.use_attribute_docstrings:
-        fields_docs = extract_docstrings_from_cls(cls)
-        for ann_name, field_info in fields.items():
-            if field_info.description is None and ann_name in fields_docs:
-                field_info.description = fields_docs[ann_name]
+def _update_fields_from_docstrings(cls: type[Any], fields: dict[str, FieldInfo], use_inspect: bool = False) -> None:
+    fields_docs = extract_docstrings_from_cls(cls, use_inspect=use_inspect)
+    for ann_name, field_info in fields.items():
+        if field_info.description is None and ann_name in fields_docs:
+            field_info.description = fields_docs[ann_name]
 
 
 def collect_model_fields(  # noqa: C901
@@ -255,7 +254,8 @@ def collect_model_fields(  # noqa: C901
         for field in fields.values():
             field.apply_typevars_map(typevars_map)
 
-    _update_fields_from_docstrings(cls, fields, config_wrapper)
+    if config_wrapper.use_attribute_docstrings:
+        _update_fields_from_docstrings(cls, fields)
     return fields, class_vars
 
 
@@ -372,8 +372,16 @@ def collect_dataclass_fields(
             # Can't we juste use `_generics.replace_types`?
             field.apply_typevars_map(typevars_map)
 
-    if config_wrapper is not None:
-        _update_fields_from_docstrings(cls, fields, config_wrapper)
+    if config_wrapper is not None and config_wrapper.use_attribute_docstrings:
+        _update_fields_from_docstrings(
+            cls,
+            fields,
+            # This is an attribute that is only set when building a Pydantic dataclass,
+            # and we need to rely on the `inspect` module to get the source code otherwise.
+            # Note that it would be great to have an explicit `__is_pydantic_dataclass__`
+            # attribute instead.
+            use_inspect=not hasattr(cls, '__pydantic_complete__'),
+        )
 
     return fields
 
diff --git a/pydantic/_internal/_generate_schema.py b/pydantic/_internal/_generate_schema.py
index 710f08ec081..1aa77e0cd11 100644
--- a/pydantic/_internal/_generate_schema.py
+++ b/pydantic/_internal/_generate_schema.py
@@ -13,7 +13,7 @@
 import typing
 import warnings
 from contextlib import contextmanager
-from copy import copy, deepcopy
+from copy import copy
 from decimal import Decimal
 from enum import Enum
 from fractions import Fraction
@@ -1830,7 +1830,11 @@ def _dataclass_schema(
 
             with self._ns_resolver.push(dataclass), self._config_wrapper_stack.push(config):
                 if is_pydantic_dataclass(dataclass):
-                    fields = deepcopy(dataclass.__pydantic_fields__)
+                    # Copy the field info instances to avoid mutating the `FieldInfo` instances
+                    # of the generic dataclass generic origin (e.g. `apply_typevars_map` below).
+                    # Note that we don't apply `deepcopy` on `__pydantic_fields__`, we don't want
+                    # to copy the `FieldInfo` attributes:
+                    fields = {f_name: copy(field_info) for f_name, field_info in dataclass.__pydantic_fields__.items()}
                     if typevars_map:
                         for field in fields.values():
                             field.apply_typevars_map(typevars_map, *self._types_namespace)
@@ -1838,6 +1842,7 @@ def _dataclass_schema(
                     fields = collect_dataclass_fields(
                         dataclass,
                         typevars_map=typevars_map,
+                        config_wrapper=self._config_wrapper,
                     )
 
                 if self._config_wrapper.extra == 'allow':
diff --git a/pydantic/config.py b/pydantic/config.py
index 18f494c7dff..9de2b2fab5e 100644
--- a/pydantic/config.py
+++ b/pydantic/config.py
@@ -1006,10 +1006,12 @@ class Model(BaseModel):
     ```
     This requires the source code of the class to be available at runtime.
 
-    !!! warning "Usage with `TypedDict`"
-        Due to current limitations, attribute docstrings detection may not work as expected when using `TypedDict`
-        (in particular when multiple `TypedDict` classes have the same name in the same source file). The behavior
-        can be different depending on the Python version used.
+    !!! warning "Usage with `TypedDict` and stdlib dataclasses"
+        Due to current limitations, attribute docstrings detection may not work as expected when using
+        [`TypedDict`][typing.TypedDict] and stdlib dataclasses, in particular when:
+
+        - inheritance is being used.
+        - multiple classes have the same name in the same source file.
     '''
 
     cache_strings: bool | Literal['all', 'keys', 'none']
diff --git a/pydantic/dataclasses.py b/pydantic/dataclasses.py
index e249e5ae704..a9bc7ba027d 100644
--- a/pydantic/dataclasses.py
+++ b/pydantic/dataclasses.py
@@ -362,6 +362,6 @@ def is_pydantic_dataclass(class_: type[Any], /) -> TypeGuard[type[PydanticDatacl
         `True` if the class is a pydantic dataclass, `False` otherwise.
     """
     try:
-        return '__pydantic_validator__' in class_.__dict__ and dataclasses.is_dataclass(class_)
+        return '__pydantic_fields__' in class_.__dict__ and dataclasses.is_dataclass(class_)
     except AttributeError:
         return False
diff --git a/tests/test_docs_extraction.py b/tests/test_docs_extraction.py
index fd8616176be..a769e5f3d68 100644
--- a/tests/test_docs_extraction.py
+++ b/tests/test_docs_extraction.py
@@ -1,9 +1,11 @@
 import textwrap
+from dataclasses import dataclass
 from typing import Generic, TypeVar
 
+import pytest
 from typing_extensions import Annotated, TypedDict
 
-from pydantic import BaseModel, ConfigDict, Field, TypeAdapter, create_model
+from pydantic import BaseModel, ConfigDict, Field, TypeAdapter, create_model, with_config
 from pydantic.dataclasses import dataclass as pydantic_dataclass
 
 T = TypeVar('T')
@@ -253,6 +255,55 @@ def dummy_method(self) -> None:
     assert MyModel.__pydantic_fields__['g'].description == 'G docs'
     assert MyModel.__pydantic_fields__['i'].description == 'Real description'
 
+    # https://github.com/pydantic/pydantic/issues/11243:
+    # Even though the `FieldInfo` instances had the correct description set,
+    # a bug in the core schema generation logic omitted them:
+    assert TypeAdapter(MyModel).json_schema()['properties']['a']['description'] == 'A docs'
+
+
+def test_stdlib_docs_extraction():
+    @dataclass
+    @with_config({'use_attribute_docstrings': True})
+    class MyModel:
+        a: int
+        """A docs"""
+
+    ta = TypeAdapter(MyModel)
+
+    assert ta.json_schema() == {
+        'properties': {'a': {'title': 'A', 'type': 'integer', 'description': 'A docs'}},
+        'required': ['a'],
+        'title': 'MyModel',
+        'type': 'object',
+    }
+
+
+@pytest.mark.xfail(reason='Current implementation does not take inheritance into account.')
+def test_stdlib_docs_extraction_inheritance():
+    @dataclass
+    @with_config({'use_attribute_docstrings': True})
+    class Base:
+        a: int
+        """A docs"""
+
+    @dataclass
+    @with_config({'use_attribute_docstrings': True})
+    class MyStdlibDataclass(Base):
+        b: int
+        """B docs"""
+
+    ta = TypeAdapter(MyStdlibDataclass)
+
+    assert ta.json_schema() == {
+        'properties': {
+            'a': {'title': 'A', 'type': 'integer', 'description': 'A docs'},
+            'b': {'title': 'B', 'type': 'integer', 'description': 'B docs'},
+        },
+        'required': ['a', 'b'],
+        'title': 'MyStdlibDataclass',
+        'type': 'object',
+    }
+
 
 def test_typeddict():
     class MyModel(TypedDict):


Code review comments:
diff_hunk: @@ -362,6 +366,6 @@ def is_pydantic_dataclass(class_: type[Any], /) -> TypeGuard[type[PydanticDatacl
         `True` if the class is a pydantic dataclass, `False` otherwise.
     """
     try:
-        return '__pydantic_validator__' in class_.__dict__ and dataclasses.is_dataclass(class_)
path: pydantic/dataclasses.py
text: @user1:
Can't we just check `__is_pydantic_dataclass__` here instead?

@author:
Yes forgot about it