Repository: pydantic/pydantic
Pull request: Remove generics cache workaround

This results in parameterized classes being different type objects, although they represent the same type.

A new refactor (https://github.com/pydantic/pydantic/pull/11388) was introduced since then and fixes the root issue.

<!-- Thank you for your contribution! -->
<!-- Unless your change is trivial, please create an issue to discuss the change before creating a PR -->

Fixes https://github.com/pydantic/pydantic/issues/11747.

## Checklist

* [ ] The pull request title is a good summary of the changes - it will be used in the changelog
* [ ] Unit tests for the changes exist
* [ ] Tests pass on CI
* [ ] Documentation reflects the changes where applicable
* [ ] My PR is ready to review, **please add a comment including the phrase "please review" to assign reviewers**


Problem statement:
UserWarning: Pydantic serializer warnings >= v2.11.0
### Initial Checks

- [x] I confirm that I'm using Pydantic V2

### Description

For all pydantic versions >= 2.11.0 I receive the following warning for the code posted below.
- For pydantic 2.10.6 the warning doesn't occur.
- The serializing seems to works correctly.
- The warning itself is confusing, as it says it expects `AStageSeries[float]` and also tells us that it got `input_type=AStageSeries[float]`

```
UserWarning: Pydantic serializer warnings:
  PydanticSerializationUnexpectedValue(Expected `AStageSeries[float]` - serialized value may not be as expected [input_value=AStageSeries[float](stage....0, 2.1], a2=[3.0, 3.1]), input_type=AStageSeries[float]])
  PydanticSerializationUnexpectedValue(Expected `BStageSeries[float]` - serialized value may not be as expected [input_value=AStageSeries[float](stage....0, 2.1], a2=[3.0, 3.1]), input_type=AStageSeries[float]])
  return self.__pydantic_serializer__.to_json(
```


### Example Code

```Python
from enum import Enum
from typing import Literal, TypeVar, Generic, Annotated
from pydantic import BaseModel, Field

T = TypeVar("T", bound=float | str)

class StageData(BaseModel, Generic[T]):
    stage_no: int = Field(...)
    base: T = Field(...)

class AStageData(StageData[T], BaseModel, Generic[T]):
    type: Literal["A"] = Field(
        ...
    )
    a1: T = Field(...)
    a2: T = Field(...)

class BStageData(StageData[T], BaseModel, Generic[T]):
    type: Literal["B"] = Field(
        ...
    )
    b1: T = Field(...)
    b2: T = Field(...)

type StageDataBase[T] = Annotated[
    AStageData[T] | BStageData[T], Field(discriminator="type")
]


class StageSeries(BaseModel, Generic[T]):
    stage_no: int = Field(...)
    base: list[T] = Field(
        []
    )

    def append_value(self, stage_data: StageDataBase[T]) -> None:
        self.base.append(stage_data.base)

class AStageSeries(StageSeries[T], Generic[T]):
    type: Literal["A"] = Field(
        ...
    )
    a1: list[T] = Field(
        []
    )
    a2: list[T] = Field(
        []
    )

    def append_value(self, stage_data: AStageData[T]) -> None:
        super().append_value(stage_data)
        self.a1.append(stage_data.a1)
        self.a2.append(stage_data.a2)

class BStageSeries(StageSeries[T], Generic[T]):
    type: Literal["B"] = Field(
        ...
    )
    b1: list[T] = Field(
        []
    )
    b2: list[T] = Field(
        []
    )

    def append_value(self, stage_data: BStageData[T]) -> None:
        super().append_value(stage_data)
        self.b1.append(stage_data.b1)
        self.b2.append(stage_data.b2)

type StageSeriesBase[T] = Annotated[
    AStageSeries[T] | BStageSeries[T], Field(discriminator="type")
]

class Series(BaseModel, Generic[T]):
    stages: list[StageSeriesBase[T]] = Field(
        []
    )

    def append_value(self, stages: list[StageDataBase[T]]) -> None:
        for stage_data in stages:
            stage = self.get_stage(stage_data)
            stage.append_value(stage_data)

    def get_stage(self, stage_data: StageDataBase[T]) -> StageSeriesBase[T]:
        # this throws "Parameterized generics cannot be used with class or instance checks"
        # but this is the only possible way to instantiate the correct generic type class
        # google didn't provide any meaningful info to that mypy error either
        try:
            stage = next(s for s in self.stages if s.stage_no == stage_data.stage_no)
        except StopIteration:
            if isinstance(stage_data, AStageData[float]):
                stage = AStageSeries[float](
                    type="A", stage_no=stage_data.stage_no
                )
            elif isinstance(stage_data, AStageData[str]):
                stage = AStageSeries[str](
                    type="A", stage_no=stage_data.stage_no
                )
            elif isinstance(trend_stage, BStageData[float]):
                stage = BStageSeries[float](
                    type="B", stage_no=stage_data.stage_no
                )
            elif isinstance(trend_stage, BStageData[std]):
                stage = BlStageSeries[str](
                    type="B", stage_no=stage_data.stage_no
                )
            else:
                raise ValueError(f"Unsupported type: {type(stage_data)}") from None

            self.stages.append(stage)
        return stage

g = Series[float]()
g.append_value(
    stages = [
        AStageData[float](
            type="A",
            stage_no=1,
            base=1,
            a1=2,
            a2=3,
        )
    ]
)
g.append_value(
    stages = [
        AStageData[float](
            type="A",
            stage_no=1,
            base=1.1,
            a1=2.1,
            a2=3.1,
        )
    ]
)

g.model_dump_json()
```

### Python, Pydantic & OS Version

```Text
             pydantic version: 2.11.3
        pydantic-core version: 2.33.1
          pydantic-core build: profile=release pgo=false
                 install path: C:\Users\DEKOEHLS\AppData\Local\miniconda3\envs\jupyter-313\Lib\site-packages\pydantic
               python version: 3.13.3 | packaged by conda-forge | (main, Apr 10 2025, 22:12:50) [MSC v.1943 64 bit (AMD64)]
                     platform: Windows-10-10.0.19045-SP0
             related packages: typing_extensions-4.13.2
                       commit: unknown
```

Hints from issue discussion:
Bisected to https://github.com/pydantic/pydantic/pull/10704.

Pydantic has a different way of representing parameterized generic classes. Normally, parameterized user-defined generic classes aren't proper types, but rather what we call generic alias instances:

```python
class MyClass[T]: pass

MyClassInt = MyClass[int]
type(MyClassInt)  # typing._GenericAlias

isinstance(MyClass[int](), MyClass[int])  # TypeError
```

On the other hand, Pydantic creates a new class object when you parameterize your generic class. To avoid using too much memory, these parameterized classes are cached, so that we avoid recreating a new one:

```python
class MyClass[T](BaseModel):
    pass

M1 = MyClass[int]
M2 = MyClass[int]

M1 is M2  # True
```

However, the caching logic was changed in https://github.com/pydantic/pydantic/pull/10704 to fix an issue with recursive generics. As a result, in some circumstances, the parameterized classes may be different objects, even if the arguments are the same.

This what happens in your case, and because Pydantic checks if the serialized values are proper instances of the type, you get the serialization warning because the type of the instance of `g.stages[0]` is different from the actual annotation (even though they are supposed to be the same).

I have a couple patches that could fix this, so I need to decide which one's best.


Note that if you want to avoid relying on the `isinstance()` checks in `get_stage()` (which we recommend not doing), you can use the following implementation:

```python
    def get_stage(self, stage_data: StageDataBase[T]) -> StageSeriesBase[T]:
        # this throws "Parameterized generics cannot be used with class or instance checks"
        # but this is the only possible way to instantiate the correct generic type class
        # google didn't provide any meaningful info to that mypy error either
        try:
            stage = next(s for s in self.stages if s.stage_no == stage_data.stage_no)
        except StopIteration:
            if stage_data.__pydantic_generic_metadata__['origin'] is AStageData:
                if stage_data.__pydantic_generic_metadata__['args'] == (float,):
                    stage = AStageSeries[float](
                        type="A", stage_no=stage_data.stage_no
                    )
                elif stage_data.__pydantic_generic_metadata__['args'] == (str,):
                    stage = AStageSeries[str](
                        type="A", stage_no=stage_data.stage_no
                    )
            elif stage_data.__pydantic_generic_metadata__['origin'] is BStageData:
                if stage_data.__pydantic_generic_metadata__['args'] == (float,):
                    stage = BStageSeries[float](
                        type="B", stage_no=stage_data.stage_no
                    )
                elif stage_data.__pydantic_generic_metadata__['args'] == (str,):
                    stage = BStageSeries[str](
                        type="B", stage_no=stage_data.stage_no
                    )
            else:
                raise ValueError(f"Unsupported type: {type(stage_data)}") from None
```
Thank you, I adapted my code to your suggested approach in the second answer. I still receive mypy errors though: `Incompatible types in assignment (expression has type "AStageSeries[float]", variable has type "AStageSeries[T] | BStageSeries[T]")Mypy[assignment](https://mypy.readthedocs.io/en/latest/_refs.html#code-assignment)`

Regarding your first answer: I don't fully understand. Does it mean there would also be a better way for me to implement the UseCase?

Commit message: Remove generics cache workaround

This results in parameterized classes being different type objects,
although they represent the same type.

A new refactor (https://github.com/pydantic/pydantic/pull/11388)
was introduced since then and fixes the root issue.

Patch to review:
diff --git a/pydantic/main.py b/pydantic/main.py
index ba9324bad2a..820f34f406d 100644
--- a/pydantic/main.py
+++ b/pydantic/main.py
@@ -908,12 +908,7 @@ def __class_getitem__(
 
                 submodel = _generics.create_generic_submodel(model_name, origin, args, params)
 
-                # Cache the generated model *only* if not in the process of parametrizing
-                # another model. In some valid scenarios, we miss the opportunity to cache
-                # it but in some cases this results in `PydanticRecursiveRef` instances left
-                # on `FieldInfo` annotations:
-                if len(_generics.recursively_defined_type_refs()) == 1:
-                    _generics.set_cached_generic_type(cls, typevar_values, submodel, origin, args)
+                _generics.set_cached_generic_type(cls, typevar_values, submodel, origin, args)
 
         return submodel
 
diff --git a/tests/test_forward_ref.py b/tests/test_forward_ref.py
index 01d81f95d8a..1187d642b77 100644
--- a/tests/test_forward_ref.py
+++ b/tests/test_forward_ref.py
@@ -718,15 +718,9 @@ def test_recursive_models_union(create_module):
     # This test should pass because PydanticRecursiveRef.__or__ is implemented,
     # not because `eval_type_backport` magically makes `|` work,
     # since it's installed for tests but otherwise optional.
-    # When generic models are involved in recursive models, parametrizing a model
-    # can result in a `PydanticRecursiveRef` instance. This isn't ideal, as in the
-    # example below, this results in the `FieldInfo.annotation` attribute being changed,
-    # e.g. for `bar` to something like `PydanticRecursiveRef(...) | None`.
-    # We currently have a workaround (avoid caching parametrized models where this bad
-    # annotation mutation can happen).
     sys.modules['eval_type_backport'] = None  # type: ignore
     try:
-        create_module(
+        module = create_module(
             # language=Python
             """
 from __future__ import annotations
@@ -742,14 +736,20 @@ class Foo(BaseModel):
 
 class Bar(BaseModel, Generic[T]):
     foo: Foo
+
+Foo.model_rebuild()
     """
         )
     finally:
         del sys.modules['eval_type_backport']
 
+    assert module.Foo.model_fields['bar'].annotation == typing.Optional[module.Bar[str]]
+    assert module.Foo.model_fields['bar2'].annotation == typing.Union[int, module.Bar[float]]
+    assert module.Bar.model_fields['foo'].annotation == module.Foo
+
 
 def test_recursive_models_union_backport(create_module):
-    create_module(
+    module = create_module(
         # language=Python
         """
 from __future__ import annotations
@@ -768,9 +768,15 @@ class Foo(BaseModel):
 
 class Bar(BaseModel, Generic[T]):
     foo: Foo
+
+Foo.model_rebuild()
 """
     )
 
+    assert module.Foo.model_fields['bar'].annotation == typing.Optional[module.Bar[str]]
+    assert module.Foo.model_fields['bar2'].annotation == typing.Union[int, str, module.Bar[float]]
+    assert module.Bar.model_fields['foo'].annotation == module.Foo
+
 
 def test_force_rebuild():
     class Foobar(BaseModel):
diff --git a/tests/test_generics.py b/tests/test_generics.py
index 5d015a9e598..c1cab439f8f 100644
--- a/tests/test_generics.py
+++ b/tests/test_generics.py
@@ -27,6 +27,7 @@
     Never,
     NotRequired,
     ParamSpec,
+    TypeAliasType,
     TypedDict,
     TypeVarTuple,
     Unpack,
@@ -519,6 +520,37 @@ class M(BaseModel):
     del generics
 
 
+def test_generics_reused() -> None:
+    """https://github.com/pydantic/pydantic/issues/11747
+
+    To fix an issue with recursive generics, we introduced a change in 2.11 that would
+    skip caching the parameterized model under specific circumstances. The following setup
+    is an example of where this happens. As a result, we end up with two different `A[float]`
+    classes, although they are the same in practice.
+    When serializing, we check that the value instances are matching the type, but we end up
+    with warnings as `isinstance(value, A[float])` fails.
+    The fix was reverted as a refactor (https://github.com/pydantic/pydantic/pull/11388) fixed
+    the underlying issue.
+    """
+
+    T = TypeVar('T')
+
+    class A(BaseModel, Generic[T]):
+        pass
+
+    class B(BaseModel, Generic[T]):
+        pass
+
+    AorB = TypeAliasType('AorB', Union[A[T], B[T]], type_params=(T,))
+
+    class Main(BaseModel, Generic[T]):
+        ls: list[AorB[T]] = []
+
+    m = Main[int]()
+    m.ls.append(A[int]())
+    m.model_dump_json(warnings='error')
+
+
 def test_generic_config():
     data_type = TypeVar('data_type')
 
@@ -1445,7 +1477,9 @@ class InnerModel(OuterModel[T], Generic[T]):
 
     with pytest.raises(ValidationError):
         InnerModel[int](a=['s', {'a': 'wrong'}])
+
     assert InnerModel[int](a=['s', {'a': 1}]).a[1].a == 1
+    assert InnerModel[int].model_fields['a'].annotation == Optional[list[Union[ReferencedModel[int], str]]]
 
 
 def test_deep_generic_with_multiple_typevars():


Code review comments:
diff_hunk: @@ -519,6 +520,37 @@ class M(BaseModel):
     del generics
 
 
+def test_generics_reused() -> None:
+    """https://github.com/pydantic/pydantic/issues/11747
+
+    To fix an issue with recursive generics, we introduced a change in 2.11 that would
+    skip caching the parameterized model under specific circumstances. The following setup
+    is an example of where this happens. As a result, we end up with two different `A[float]`
+    classes, although they are the same in practice.
+    When serializing, we check that the value instances are matching the type, but we end up
+    with warnings as `isinstance(value, A[float])` fails.
path: tests/test_generics.py
text: @user1:
`int` not `float`? Also I think the tense might be past tense...

```suggestion
    is an example of where this would happen. As a result, we ended up with two different `A[int]`
    classes, although they were the same in practice.
    When serializing, we check that the value instances are matching the type, but we ended up
    with warnings as `isinstance(value, A[int])` fails.
```