Repository: pydantic/pydantic

Issue:
`KeyError` in GenericModel.__parameterized_bases__.build_base_model
# Bug

Output of `python -c "import pydantic.utils; print(pydantic.utils.version_info())"`:
```
             pydantic version: 1.9.1
            pydantic compiled: True
                 install path: /home/andi/virtualenvs/aimaas/lib/python3.9/site-packages/pydantic
               python version: 3.9.12 (main, Apr 18 2022, 22:40:46)  [GCC 9.4.0]
                     platform: Linux-5.4.0-113-generic-x86_64-with-glibc2.31
     optional deps. installed: ['dotenv', 'typing-extensions']
```
I'm afraid reproduction of this _issue_ is a bit tricky as I was not able to reproduce it in a production scenario nor with a small code snippet. So far, I only noticed this in the test suite of this [project](https://github.com/SUSE/aimaas), which contains multiple sub-modules for tests; but only tests in two **can** fail, i.e. when the entire testsuite runs, they fail; if I explicitly only run tests from these modules, they pass. 

This behavior probably also indicates a problem within the testsuite, which I intend to look for, but the problem does not occur with version 1.9.0 of pydantic. So, I figured it's worth sharing.

In the project, we dynamically define API routes for FastAPI for dynamically generated models (i.e. model definitions are stored in DB), i.e. the models and routes get generated/updated at start-up or when something changes in the DB. The callback function used for this looks something like this:

```python
def route_get_entities(router: APIRouter, schema: Schema):
    # this is the callback
    factory = EntityModelFactory()
    filter_model = _filters_request_model(schema=schema)

    @router.get(
        f'/{schema.slug}',
        response_model=Page[factory(schema=schema, variant=ModelVariant.GET)],
        response_model_exclude_unset=True
    )
    def get_entities(
        filters: filter_model = Depends(),
        order_by: str = Query('name', description=''),
        ascending: bool = Query(True, description=''),
        db: Session = Depends(get_db),
        params: Params = Depends()
    ):
        # the actual API route
        ...
```
The exception raised in the testsuite looks like this:
```
_____________________________________ ERROR at setup of TestRouteGetEntityChanges.test_raise_on_change_doesnt_exist ______________________________________

dbsession = <sqlalchemy.orm.session.Session object at 0x7fca28575580>

    @pytest.fixture
    def client(dbsession):
>       app = create_app(session=dbsession)

backend/tests/conftest.py:338: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
backend/__init__.py:53: in create_app
    load_dynamic_routes(db=session, app=app)
backend/__init__.py:40: in load_dynamic_routes
    create_dynamic_router(schema=schema, app=app)
backend/dynamic_routes.py:341: in create_dynamic_router
    route_get_entities(router=router, schema=schema)
backend/dynamic_routes.py:124: in route_get_entities
    response_model=Page[factory(schema=schema, variant=ModelVariant.GET)],
../../virtualenvs/aimaas/lib/python3.9/site-packages/pydantic/generics.py:98: in __class_getitem__
    __base__=(cls,) + tuple(cls.__parameterized_bases__(typevars_map)),
../../virtualenvs/aimaas/lib/python3.9/site-packages/pydantic/generics.py:224: in __parameterized_bases__
    yield from build_base_model(base_model, typevars_map)
../../virtualenvs/aimaas/lib/python3.9/site-packages/pydantic/generics.py:190: in build_base_model
    parameterized_base = base_model.__class_getitem__(base_parameters)
../../virtualenvs/aimaas/lib/python3.9/site-packages/pydantic/generics.py:98: in __class_getitem__
    __base__=(cls,) + tuple(cls.__parameterized_bases__(typevars_map)),
../../virtualenvs/aimaas/lib/python3.9/site-packages/pydantic/generics.py:224: in __parameterized_bases__
    yield from build_base_model(base_model, typevars_map)
../../virtualenvs/aimaas/lib/python3.9/site-packages/pydantic/generics.py:185: in build_base_model
    base_parameters = tuple([mapped_types[param] for param in base_model.__parameters__])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

.0 = <tuple_iterator object at 0x7fca2344e370>

>   base_parameters = tuple([mapped_types[param] for param in base_model.__parameters__])
E   KeyError: ~T

../../virtualenvs/aimaas/lib/python3.9/site-packages/pydantic/generics.py:185: KeyError
```
I checked the contents of `mapped_types` and `base_model.__parameters__` in the line where the exception gets raised:
```
>>> mapped_types
{~T: <class 'pydantic.main.PersonGet'>}
>>> a = list(mapped_types.keys())[0]
>>> a
~T
>>> b = base_model.__parameters__
>>> b[0]
~T
>>> a is b[0]
False
>>> type(a)
<class 'typing.TypeVar'>
>>> type(b[0])
<class 'typing.TypeVar'>
```
It looks like the `TypeVar`s, which are expected to be identical, no longer are. :thinking: 


Comments:
I was not able to create a short example for reproducibility, but I managed to identify the changes that "broke" our testsuite: #4083

If I revert both changes to [pydantic/generics.py](https://github.com/samuelcolvin/pydantic/blob/master/pydantic/generics.py), the errors in our testsuite disappear.

That might explain, why it's not easy to create a simple snippet for reproducibility, right? One needs to manage to exceed the limit...
Sorry for slow reply, yes makes sense.

I don't want to change the limits, but we should fail in a more transparent way.
I also added some debug messages in `pydantic/generics.py` to better understand what's happening.

I noticed for my case, that the caches `_generic_types_cache` and `_assigned_parameters` grow with each unittest, when the FastAPI app is re-created (in a fixture explicitly via `create_model`). It looks to me, like the retrieval from these caches is broken for dynamic models and/or `TypeVar`s:

The testsuite fixture defines a limited number of distinct models (including static and dynamic ones,  in total **23**), but whenever the application is rebuilt for the next test case, the models get rebuilt and added to the cache as the keys do not match. To be exact, `_generic_types_cache` very quickly grows to the specific limit, whereas `_assigned_parameters` goes up only to about 600.

To the best of my knowledge, this _cache problem_ (maybe the reason for the leak mentioned in #3829?) was also true for v1.9.0, but for some reasonit did not cause a `KeyError`.

> I don't want to change the limits

Yes, if my understanding of the observed problem is correct, changing the limit makes no difference. 

> we should fail in a more transparent way.

Given the specific error I observed, I'm wondering if it would make more sense to allow a _margin of error_ when comparing two different `TypeVars` that have the same name?

But that would probably only count as a workaround.

Maybe I should raise an issue with the developers of fastapi_pagination to not use `TypeVar`s or raise an issue with the developers of Python so that the comparison of `TypeVar`s changes:
```
>>> import typing
>>> t = typing.TypeVar("T")
>>> s = typing.TypeVar("T")
>>> t == s
False
>>> t is s
False
```


Patch:
diff --git a/pydantic/generics.py b/pydantic/generics.py
--- a/pydantic/generics.py
+++ b/pydantic/generics.py
@@ -17,6 +17,7 @@
     Union,
     cast,
 )
+from weakref import WeakKeyDictionary, WeakValueDictionary
 
 from typing_extensions import Annotated
 
@@ -25,7 +26,7 @@
 from .main import BaseModel, create_model
 from .types import JsonWrapper
 from .typing import display_as_type, get_all_type_hints, get_args, get_origin, typing_base
-from .utils import LimitedDict, all_identical, lenient_issubclass
+from .utils import all_identical, lenient_issubclass
 
 if sys.version_info >= (3, 10):
     from typing import _UnionGenericAlias
@@ -33,15 +34,28 @@
 GenericModelT = TypeVar('GenericModelT', bound='GenericModel')
 TypeVarType = Any  # since mypy doesn't allow the use of TypeVar as a type
 
+CacheKey = Tuple[Type[Any], Any, Tuple[Any, ...]]
 Parametrization = Mapping[TypeVarType, Type[Any]]
 
-_generic_types_cache: LimitedDict[Tuple[Type[Any], Union[Any, Tuple[Any, ...]]], Type[BaseModel]] = LimitedDict()
+# weak dictionaries allow the dynamically created parametrized versions of generic models to get collected
+# once they are no longer referenced by the caller.
+if sys.version_info >= (3, 9):  # Typing for weak dictionaries available at 3.9
+    GenericTypesCache = WeakValueDictionary[CacheKey, Type[BaseModel]]
+    AssignedParameters = WeakKeyDictionary[Type[BaseModel], Parametrization]
+else:
+    GenericTypesCache = WeakValueDictionary
+    AssignedParameters = WeakKeyDictionary
+
+# _generic_types_cache is a Mapping from __class_getitem__ arguments to the parametrized version of generic models.
+# This ensures multiple calls of e.g. A[B] return always the same class.
+_generic_types_cache = GenericTypesCache()
+
 # _assigned_parameters is a Mapping from parametrized version of generic models to assigned types of parametrizations
 # as captured during construction of the class (not instances).
 # E.g., for generic model `Model[A, B]`, when parametrized model `Model[int, str]` is created,
 # `Model[int, str]`: {A: int, B: str}` will be stored in `_assigned_parameters`.
 # (This information is only otherwise available after creation from the class name string).
-_assigned_parameters: LimitedDict[Type[Any], Parametrization] = LimitedDict()
+_assigned_parameters = AssignedParameters()
 
 
 class GenericModel(BaseModel):
@@ -67,7 +81,7 @@ def __class_getitem__(cls: Type[GenericModelT], params: Union[Type[Any], Tuple[T
 
         """
 
-        def _cache_key(_params: Any) -> Tuple[Type[GenericModelT], Any, Tuple[Any, ...]]:
+        def _cache_key(_params: Any) -> CacheKey:
             args = get_args(_params)
             # python returns a list for Callables, which is not hashable
             if len(args) == 2 and isinstance(args[0], list):
diff --git a/pydantic/utils.py b/pydantic/utils.py
--- a/pydantic/utils.py
+++ b/pydantic/utils.py
@@ -17,7 +17,6 @@
     Iterator,
     List,
     Mapping,
-    MutableMapping,
     NoReturn,
     Optional,
     Set,
@@ -80,7 +79,6 @@
     'get_unique_discriminator_alias',
     'get_discriminator_alias_and_values',
     'DUNDER_ATTRIBUTES',
-    'LimitedDict',
 )
 
 ROOT_KEY = '__root__'
@@ -803,39 +801,3 @@ def _get_union_alias_and_all_values(
     # unzip: [('alias_a',('v1', 'v2)), ('alias_b', ('v3',))] => [('alias_a', 'alias_b'), (('v1', 'v2'), ('v3',))]
     all_aliases, all_values = zip(*zipped_aliases_values)
     return get_unique_discriminator_alias(all_aliases, discriminator_key), all_values
-
-
-KT = TypeVar('KT')
-VT = TypeVar('VT')
-if TYPE_CHECKING:
-    # Annoying inheriting from `MutableMapping` and `dict` breaks cython, hence this work around
-    class LimitedDict(dict, MutableMapping[KT, VT]):  # type: ignore[type-arg]
-        def __init__(self, size_limit: int = 1000):
-            ...
-
-else:
-
-    class LimitedDict(dict):
-        """
-        Limit the size/length of a dict used for caching to avoid unlimited increase in memory usage.
-
-        Since the dict is ordered, and we always remove elements from the beginning, this is effectively a FIFO cache.
-
-        Annoying inheriting from `MutableMapping` breaks cython.
-        """
-
-        def __init__(self, size_limit: int = 1000):
-            self.size_limit = size_limit
-            super().__init__()
-
-        def __setitem__(self, __key: Any, __value: Any) -> None:
-            super().__setitem__(__key, __value)
-            if len(self) > self.size_limit:
-                excess = len(self) - self.size_limit + self.size_limit // 10
-                to_remove = list(self.keys())[:excess]
-                for key in to_remove:
-                    del self[key]
-
-        def __class_getitem__(cls, *args: Any) -> Any:
-            # to avoid errors with 3.7
-            pass


Test patch:
diff --git a/tests/requirements-testing.txt b/tests/requirements-testing.txt
--- a/tests/requirements-testing.txt
+++ b/tests/requirements-testing.txt
@@ -3,9 +3,9 @@ hypothesis==6.54.4
 # pin importlib-metadata as upper versions need typing-extensions to work if on Python < 3.8
 importlib-metadata==3.1.0;python_version<"3.8"
 mypy==0.971
-pytest==7.1.2
-pytest-cov==3.0.0
-pytest-mock==3.8.2
-pytest-sugar==0.9.5
+pytest==7.2.1
+pytest-cov==4.0.0
+pytest-mock==3.10.0
+pytest-sugar==0.9.6
 # pin typing-extensions to minimum requirement - see #4885
 typing-extensions==4.2.0
diff --git a/tests/test_edge_cases.py b/tests/test_edge_cases.py
--- a/tests/test_edge_cases.py
+++ b/tests/test_edge_cases.py
@@ -1894,7 +1894,7 @@ def get_double_a(self) -> float:
     model = Model(a=10.2)
     assert model.a == 10.2
     assert model.b == 10
-    return model.get_double_a() == 20.2
+    assert model.get_double_a() == 20.2
 
 
 def test_resolve_annotations_module_missing(tmp_path):
diff --git a/tests/test_generics.py b/tests/test_generics.py
--- a/tests/test_generics.py
+++ b/tests/test_generics.py
@@ -1,3 +1,5 @@
+import gc
+import itertools
 import json
 import sys
 from enum import Enum
@@ -6,12 +8,14 @@
     Callable,
     ClassVar,
     Dict,
+    FrozenSet,
     Generic,
     Iterable,
     List,
     Mapping,
     Optional,
     Sequence,
+    Set,
     Tuple,
     Type,
     TypeVar,
@@ -21,8 +25,19 @@
 import pytest
 from typing_extensions import Annotated, Literal
 
-from pydantic import BaseModel, Field, Json, ValidationError, root_validator, validator
-from pydantic.generics import GenericModel, _generic_types_cache, iter_contained_typevars, replace_types
+from pydantic import BaseModel, Field, Json, ValidationError, create_model, root_validator, validator
+from pydantic.generics import (
+    GenericModel,
+    _assigned_parameters,
+    _generic_types_cache,
+    iter_contained_typevars,
+    replace_types,
+)
+
+
+@pytest.fixture(autouse=True)
+def clean_cache():
+    gc.collect()  # cleans up _generic_types_cache for checking item counts in the cache
 
 
 def test_generic_name():
@@ -229,10 +244,13 @@ def test_cover_cache():
     class Model(GenericModel, Generic[T]):
         x: T
 
-    Model[int]  # adds both with-tuple and without-tuple version to cache
+    models = []  # keep references to models to get cache size
+
+    models.append(Model[int])  # adds both with-tuple and without-tuple version to cache
     assert len(_generic_types_cache) == cache_size + 2
-    Model[int]  # uses the cache
+    models.append(Model[int])  # uses the cache
     assert len(_generic_types_cache) == cache_size + 2
+    del models
 
 
 def test_cache_keys_are_hashable():
@@ -246,19 +264,66 @@ class MyGenericModel(GenericModel, Generic[T]):
     # Callable's first params get converted to a list, which is not hashable.
     # Make sure we can handle that special case
     Simple = MyGenericModel[Callable[[int], str]]
+    models = []  # keep references to models to get cache size
+    models.append(Simple)
     assert len(_generic_types_cache) == cache_size + 2
     # Nested Callables
-    MyGenericModel[Callable[[C], Iterable[str]]]
+    models.append(MyGenericModel[Callable[[C], Iterable[str]]])
     assert len(_generic_types_cache) == cache_size + 4
-    MyGenericModel[Callable[[Simple], Iterable[int]]]
+    models.append(MyGenericModel[Callable[[Simple], Iterable[int]]])
     assert len(_generic_types_cache) == cache_size + 6
-    MyGenericModel[Callable[[MyGenericModel[C]], Iterable[int]]]
+    models.append(MyGenericModel[Callable[[MyGenericModel[C]], Iterable[int]]])
     assert len(_generic_types_cache) == cache_size + 10
 
     class Model(BaseModel):
         x: MyGenericModel[Callable[[C], Iterable[str]]] = Field(...)
 
+    models.append(Model)
     assert len(_generic_types_cache) == cache_size + 10
+    del models
+
+
+def test_caches_get_cleaned_up():
+    types_cache_size = len(_generic_types_cache)
+    params_cache_size = len(_assigned_parameters)
+    T = TypeVar('T')
+
+    class MyGenericModel(GenericModel, Generic[T]):
+        x: T
+
+    Model = MyGenericModel[int]
+    assert len(_generic_types_cache) == types_cache_size + 2
+    assert len(_assigned_parameters) == params_cache_size + 1
+    del Model
+    gc.collect()
+    assert len(_generic_types_cache) == types_cache_size
+    assert len(_assigned_parameters) == params_cache_size
+
+
+def test_generics_work_with_many_parametrized_base_models():
+    cache_size = len(_generic_types_cache)
+    params_size = len(_assigned_parameters)
+    count_create_models = 1000
+    T = TypeVar('T')
+    C = TypeVar('C')
+
+    class A(GenericModel, Generic[T, C]):
+        x: T
+        y: C
+
+    class B(A[int, C], GenericModel, Generic[C]):
+        pass
+
+    models = [create_model(f'M{i}') for i in range(count_create_models)]
+    generics = []
+    for m in models:
+        Working = B[m]
+        generics.append(Working)
+
+    assert len(_generic_types_cache) == cache_size + count_create_models * 5 + 1
+    assert len(_assigned_parameters) == params_size + count_create_models * 3 + 1
+    del models
+    del generics
 
 
 def test_generic_config():
@@ -1379,3 +1444,57 @@ class Payload(BaseModel):
         'properties': {'payload_field': {'title': 'Payload Field', 'type': 'string'}},
         'required': ['payload_field'],
     }
+
+
+def memray_limit_memory(limit):
+    if '--memray' in sys.argv:
+        return pytest.mark.limit_memory(limit)
+    else:
+        return pytest.mark.skip(reason='memray not enabled')
+
+
+@memray_limit_memory('100 MB')
+def test_generics_memory_use():
+    """See:
+    - https://github.com/pydantic/pydantic/issues/3829
+    - https://github.com/pydantic/pydantic/pull/4083
+    - https://github.com/pydantic/pydantic/pull/5052
+    """
+
+    T = TypeVar('T')
+    U = TypeVar('U')
+    V = TypeVar('V')
+
+    class MyModel(GenericModel, Generic[T, U, V]):
+        message: Json[T]
+        field: Dict[U, V]
+
+    class Outer(GenericModel, Generic[T]):
+        inner: T
+
+    types = [
+        int,
+        str,
+        float,
+        bool,
+        bytes,
+    ]
+
+    containers = [
+        List,
+        Tuple,
+        Set,
+        FrozenSet,
+    ]
+
+    all = [*types, *[container[tp] for container in containers for tp in types]]
+
+    total = list(itertools.product(all, all, all))
+
+    for t1, t2, t3 in total:
+
+        class Foo(MyModel[t1, t2, t3]):
+            pass
+
+        class _(Outer[Foo]):
+            pass
diff --git a/tests/test_utils.py b/tests/test_utils.py
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -27,7 +27,6 @@
 from pydantic.utils import (
     BUILTIN_COLLECTIONS,
     ClassAttribute,
-    LimitedDict,
     ValueItems,
     all_identical,
     deep_update,
@@ -549,43 +548,3 @@ def test_on_lower_camel_one_length():
 
 def test_on_lower_camel_many_length():
     assert to_lower_camel('i_like_turtles') == 'iLikeTurtles'
-
-
-def test_limited_dict():
-    d = LimitedDict(10)
-    d[1] = '1'
-    d[2] = '2'
-    assert list(d.items()) == [(1, '1'), (2, '2')]
-    for no in '34567890':
-        d[int(no)] = no
-    assert list(d.items()) == [
-        (1, '1'),
-        (2, '2'),
-        (3, '3'),
-        (4, '4'),
-        (5, '5'),
-        (6, '6'),
-        (7, '7'),
-        (8, '8'),
-        (9, '9'),
-        (0, '0'),
-    ]
-    d[11] = '11'
-
-    # reduce size to 9 after setting 11
-    assert len(d) == 9
-    assert list(d.items()) == [
-        (3, '3'),
-        (4, '4'),
-        (5, '5'),
-        (6, '6'),
-        (7, '7'),
-        (8, '8'),
-        (9, '9'),
-        (0, '0'),
-        (11, '11'),
-    ]
-    d[12] = '12'
-    assert len(d) == 10
-    d[13] = '13'
-    assert len(d) == 9