Repository: pydantic/pydantic
Pull request: Add benchmarks for schema generation with recursive models
<!-- Thank you for your contribution! -->
<!-- Unless your change is trivial, please create an issue to discuss the change before creating a PR -->

## Change Summary

<!-- Please give a short summary of the changes. -->

A suite of benchmarks for schema generation was added when recursive models were present.

Some of the cases considered:
- Simple self-referencing models (with and without generics)
- Complex recursive models (with and without generics)
- Recursive models with discriminated unions (`BaseModel`, data classes, `TypedDict`)
- Deeply nested recursive models

This should be a good starting point. I'm open to edits, suggestions, and other use cases.

## Related issue number

<!-- WARNING: please use "fix #123" style references so the issue is closed when this PR is merged. -->
fix #9711 

## Checklist

* [ ] The pull request title is a good summary of the changes - it will be used in the changelog
* [ ] Unit tests for the changes exist
* [ ] Tests pass on CI
* [ ] Documentation reflects the changes where applicable
* [ ] My PR is ready to review, **please add a comment including the phrase "please review" to assign reviewers**


Problem statement:
Expand breadth of benchmarks
We have a robust benchmark test suite in `pydantic-core`, but are lacking similar coverage in `pydantic`.

Specifically, I'm interested in expanding this test suite in relation to our core schema build times, as we're looking to improve performance there in the short term. Secondarily, I'm interested in better benchmarking for our json schema generation. Benchmarks for validation and serialization are great, but we've already got pretty good coverage in `pydantic-core` for those, so I'm less concerned about such benchmarks.

This is a great first issue - you can help write tests that test a variety of use cases for `pydantic`, without having to delve into the internals of `pydantic` code to fix bugs or add new features.

Here are some cases for which I think we should expand our suite of benchmarks (assume these are for core schema generation, but you can write related tests for any of the above mentioned categories). As in, you could read the first bullet as "we need to write some tests showcasing how long schema building takes for models with custom validators".

- [x] custom validators (field and model, also before, wrap, after, and plain)
- [x] tagged and untagged unions
- [x] stdlib types
- [x] second class Pydantic types like `SecretStr` and `PositiveInt`
- [x] custom serializers (field and model, also before, wrap, after, and plain)
- [x] recursive model definitions -- need much more robust testing when defs / refs are involved
- [ ] third party, custom types

I wouldn't say these are necessary to close this issue, but one thing for more involved contributors to look at would be benchmarking specific parts of our core schema generation process in `pydantic`, such as:

- [ ] annotation application
- [ ] discriminator application
- [ ] schema cleaning (including defs / refs simplification)
- [ ] schema walking
- [ ] field acquisition / setup in `ModelMetaClass.__new__`

Hints from issue discussion:
Do you have a plan on what general areas we want to focus on increasing benchmark coverage? (i.e. serialization, deserialization, import, validation, etc)
Any of these would be great. I'd say the most pressing category is model schema generation, but we'd more than welcome additional benchmarks for cases of validation and serialization varying in complexity. I'd say that JSON schema generation is lower priority, in general.
This probably needs more specific acceptance criteria - when are we done with this?
Hello! @sydney-runkle, I created PR #10240 to handle this issue.

My idea was to create three separate test files to handle benchmarks for each of the following categories:

- Schema generation
- validations
- Serializations.

I hope this meets the issue's requirements. I am open to any corrections or suggestions you might have.

I created tests with an increasing degree of complexity. I got inspiration from the pydantic-core library.


@AdolfoVillalobos,

Amazing work on this! I'll dig into that odd test failure. This is a great extension of our test suite!!
I've added this to our current sprint - I'd like to add a few more benchmarks to round out this issue, but the work done above is a great start. @Viicos mentioned he might add some new ones related to defs / refs / schema cleaning as well.
Alright, I've designed some more helpful criteria here re how we can close this issue. I'll note (slightly correcting my previous statement), I'm less concerned about benchmarking validation / serialization, and more concerned about benchmarking core schema generation, and secondarily, json schema generation.

I'm going to update the description of this issue so that these criteria are obvious from the get go.
> I've added this to our current sprint - I'd like to add a few more benchmarks to round out this issue, but the work done above is a great start. @Viicos mentioned he might add some new ones related to defs / refs / schema cleaning as well.

Yei! That was the idea. To have a first stab at the problem. 

I have more clarity on the scope now, so I'll submit a new PR concerning core schema generation in some of the cases you highlighted (other than what @Viicos  is working on). 

Thanks for the guidance!
I added PR #10249  just for custom field/model validators to ensure I understood the requirements correctly. Any suggestions/improvements are welcome ;)
I saw #10247 . Nice!

Commit message: fix: use Dict instead of dict

Patch to review:
diff --git a/tests/benchmarks/test_model_schema_generation.py b/tests/benchmarks/test_model_schema_generation.py
index 5dc52b37936..359baa820b6 100644
--- a/tests/benchmarks/test_model_schema_generation.py
+++ b/tests/benchmarks/test_model_schema_generation.py
@@ -12,7 +12,7 @@
 )
 
 import pytest
-from typing_extensions import Annotated, Self
+from typing_extensions import Annotated, Self, TypedDict
 
 from pydantic import (
     AfterValidator,
@@ -307,3 +307,115 @@ class PydanticTypeModel(DeferredModel):
         field: field_type
 
     benchmark(rebuild_model, PydanticTypeModel)
+
+
+@pytest.mark.benchmark(group='model_schema_generation')
+def test_simple_recursive_model_schema_generation(benchmark):
+    class Foo(DeferredModel):
+        a: int = 123
+        sibling: 'Foo' = None
+
+    benchmark(rebuild_model, Foo)
+
+
+@pytest.mark.benchmark(group='model_schema_generation')
+def test_generic_recursive_model_schema_generation(benchmark):
+    T = TypeVar('T')
+
+    class GenericFoo(DeferredModel, Generic[T]):
+        value: T
+        sibling: Optional['GenericFoo[T]'] = None
+
+    benchmark(rebuild_model, GenericFoo[int])
+
+
+@pytest.mark.benchmark(group='model_schema_generation')
+def test_nested_recursive_model_schema_generation(benchmark):
+    class Node(DeferredModel):
+        value: int
+        left: Optional['Node'] = None
+        right: Optional['Node'] = None
+
+    class Tree(DeferredModel):
+        root: Node
+        metadata: Dict[str, 'Tree'] = Field(default_factory=dict)
+
+    benchmark(rebuild_model, Tree)
+
+
+@pytest.mark.benchmark(group='model_schema_generation')
+def test_nested_recursive_generic_model_schema_generation(benchmark):
+    T = TypeVar('T')
+
+    class GenericNode(DeferredModel, Generic[T]):
+        value: T
+        left: Optional['GenericNode[T]'] = None
+        right: Optional['GenericNode[T]'] = None
+
+    class GenericTree(DeferredModel, Generic[T]):
+        root: GenericNode[T]
+        metadata: Dict[str, 'GenericTree[T]'] = Field(default_factory=dict)
+
+    benchmark(rebuild_model, GenericTree[int])
+
+
+@pytest.mark.benchmark(group='model_schema_generation')
+def test_recursive_discriminated_union_with_base_model(benchmark) -> None:
+    class Foo(DeferredModel):
+        type: Literal['foo']
+        x: 'Foobar'
+
+    class Bar(DeferredModel):
+        type: Literal['bar']
+
+    class Foobar(DeferredModel):
+        value: Union[Foo, Bar] = Field(discriminator='type')
+
+    benchmark(rebuild_model, Foobar)
+
+
+@pytest.mark.benchmark(group='model_schema_generation')
+def test_recursive_discriminated_union_with_typed_dict(benchmark) -> None:
+    class Foo(TypedDict):
+        type: Literal['foo']
+        x: 'Foobar'
+
+    class Bar(TypedDict):
+        type: Literal['bar']
+
+    class Foobar(DeferredModel):
+        value: Union[Foo, Bar] = Field(discriminator='type')
+
+    benchmark(rebuild_model, Foobar)
+
+
+@pytest.mark.benchmark(group='model_schema_generation')
+def test_recursive_discriminated_union_with_pydantic_dataclass(benchmark) -> None:
+    @dataclass
+    class Foo:
+        type: Literal['foo']
+        x: 'Foobar'
+
+    @dataclass
+    class Bar:
+        type: Literal['bar']
+
+    @dataclass(frozen=True, kw_only=True, config={'defer_build': True})
+    class Foobar:
+        value: Union[Foo, Bar] = Field(discriminator='type')
+
+    benchmark(lambda: rebuild_dataclass(Foobar, force=True, _types_namespace={}))
+
+
+@pytest.mark.benchmark(group='model_schema_generation')
+def test_deeply_nested_recursive_model_schema_generation(benchmark):
+    class A(DeferredModel):
+        b: 'B'
+
+    class B(DeferredModel):
+        c: 'C'
+
+    class C(DeferredModel):
+        a: Optional['A']
+
+    benchmark(rebuild_model, C)


Code review comments:
diff_hunk: @@ -307,3 +307,115 @@ class PydanticTypeModel(DeferredModel):
         field: field_type
 
     benchmark(rebuild_model, PydanticTypeModel)
+
+
+@pytest.mark.benchmark(group='model_schema_generation')
+def test_simple_recursive_model_schema_generation(benchmark):
+    class Foo(DeferredModel):
+        a: int = 123
+        sibling: 'Foo' = None
+
+    benchmark(rebuild_model, Foo)
+
+
+@pytest.mark.benchmark(group='model_schema_generation')
+def test_generic_recursive_model_schema_generation(benchmark):
+    T = TypeVar('T')
+
+    class GenericFoo(DeferredModel, Generic[T]):
+        value: T
+        sibling: Optional['GenericFoo[T]'] = None
+
+    benchmark(rebuild_model, GenericFoo[int])
+
+
+@pytest.mark.benchmark(group='model_schema_generation')
+def test_nested_recursive_model_schema_generation(benchmark):
+    class Node(DeferredModel):
+        value: int
+        left: Optional['Node'] = None
+        right: Optional['Node'] = None
+
+    class Tree(DeferredModel):
+        root: Node
+        metadata: Dict[str, 'Tree'] = Field(default_factory=dict)
+
+    benchmark(rebuild_model, Tree)
+
+
+@pytest.mark.benchmark(group='model_schema_generation')
+def test_nested_recursive_generic_model_schema_generation(benchmark):
+    T = TypeVar('T')
+
+    class GenericNode(DeferredModel, Generic[T]):
+        value: T
+        left: Optional['GenericNode[T]'] = None
+        right: Optional['GenericNode[T]'] = None
+
+    class GenericTree(DeferredModel, Generic[T]):
+        root: GenericNode[T]
+        metadata: Dict[str, 'GenericTree[T]'] = Field(default_factory=dict)
+
+    benchmark(rebuild_model, GenericTree[int])
+
+
+@pytest.mark.benchmark(group='model_schema_generation')
+def test_recursive_discriminated_union_with_base_model(benchmark) -> None:
+    class Foo(DeferredModel):
+        type: Literal['foo']
+        x: 'Foobar'
+
+    class Bar(DeferredModel):
+        type: Literal['bar']
+
+    class Foobar(DeferredModel):
+        value: Union[Foo, Bar] = Field(discriminator='type')
+
+    benchmark(rebuild_model, Foobar)
+
+
+@pytest.mark.benchmark(group='model_schema_generation')
+def test_recursive_discriminated_union_with_typed_dict(benchmark) -> None:
+    class Foo(TypedDict):
+        type: Literal['foo']
+        x: 'Foobar'
+
+    class Bar(TypedDict):
+        type: Literal['bar']
+
+    class Foobar(DeferredModel):
+        value: Union[Foo, Bar] = Field(discriminator='type')
+
+    benchmark(rebuild_model, Foobar)
+
+
+@pytest.mark.benchmark(group='model_schema_generation')
+def test_recursive_discriminated_union_with_pydantic_dataclass(benchmark) -> None:
+    @dataclass
+    class Foo:
+        type: Literal['foo']
+        x: 'Foobar'
+
+    @dataclass
+    class Bar:
+        type: Literal['bar']
+
+    @dataclass(frozen=True, kw_only=True, config={'defer_build': True})
+    class Foobar:
+        value: Union[Foo, Bar] = Field(discriminator='type')
+
+    benchmark(lambda: rebuild_dataclass(Foobar, force=True, _types_namespace={}))
path: tests/benchmarks/test_model_schema_generation.py
text: @user1:
I think we can remove these ones, I don't expect any schema generation changes with Pydantic models

@author:
That makes sense. Thanks for the feedback!