Repository: samuelcolvin/pydantic
Pull request #2375 (2021-02-17 12:30:32)

Advanced use of generic fields in schema generation
<!-- Thank you for your contribution! -->
<!-- Unless your change is trivial, please create an issue to discuss the change before creating a PR -->
<!-- See https://pydantic-docs.helpmanual.io/contributing/ for help on Contributing -->

## Change Summary

Continuing #2262, I started to develop the use of generics as field types in models.
My initial implementation of this functionality was rather naive. I made a revision.
In particular, I changed how `modify_schema` is called for generics and how the nested type schema is generated.

For example, you are using a generic type `MyGeneric[MyCustomType, str]`.
You expect the data to be passed to validation as a Tuple, and a schema must be defined for each nested type and for the entire field.

The sequence of forming the scheme should be, as it seems to me, like this:
1. Collect all nested type schemas into one `allOf` schema.
2. Call `modify_schema` of the generic `MyGeneric`, to where we pass the schema collected at the previous stage. There it can be processed.
3. Update the schema described in the model itself with the data collected in the previous stage.

Moreover, there are two options for nested types - if one type is specified or if several are specified.

For one type, the schema will look like this:
`MyGeneric[MyCustomType]` -> `{"allOf": [{"$ref": "#/definitions/MyCustomType"}]}`

For two nested types like this:
`MyGeneric[MyCustomType, str]` -> `{"allOf": [{"type": "array", "items": [{"$ref": "#/definitions/CustomType"}, {"type": "string "}]}]}`

## Related issue number

<!-- Are there any issues opened that will be resolved by merging this change? -->

## Checklist

* [x] Unit tests for the changes exist
* [x] Tests pass on CI and coverage remains at 100%
* [x] Documentation reflects the changes where applicable
* [x] `changes/<pull request or issue id>-<github username>.md` file added describing change
  (see [changes/README.md](https://github.com/samuelcolvin/pydantic/blob/master/changes/README.md) for details)


codecov[bot]: # [Codecov](https://codecov.io/gh/samuelcolvin/pydantic/pull/2375?src=pr&el=h1) Report
> Merging [#2375](https://codecov.io/gh/samuelcolvin/pydantic/pull/2375?src=pr&el=desc) (2eaed2f) into [master](https://codecov.io/gh/samuelcolvin/pydantic/commit/cc3010c80d5c635680ca1cad35d4d5a9bf3e219b?el=desc) (cc3010c) will **decrease** coverage by `4.53%`.
> The diff coverage is `50.00%`.


```diff
@@            Coverage Diff             @@
##           master    #2375      +/-   ##
==========================================
- Coverage   99.97%   95.44%   -4.54%     
==========================================
  Files          23       23              
  Lines        4496     4500       +4     
  Branches      909      911       +2     
==========================================
- Hits         4495     4295     -200     
- Misses          1      178     +177     
- Partials        0       27      +27     
```


| [Impacted Files](https://codecov.io/gh/samuelcolvin/pydantic/pull/2375?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pydantic/schema.py](https://codecov.io/gh/samuelcolvin/pydantic/pull/2375/diff?src=pr&el=tree#diff-cHlkYW50aWMvc2NoZW1hLnB5) | `98.09% <50.00%> (-1.91%)` | :arrow_down: |
| [pydantic/generics.py](https://codecov.io/gh/samuelcolvin/pydantic/pull/2375/diff?src=pr&el=tree#diff-cHlkYW50aWMvZ2VuZXJpY3MucHk=) | `27.02% <0.00%> (-72.98%)` | :arrow_down: |
| [pydantic/annotated\_types.py](https://codecov.io/gh/samuelcolvin/pydantic/pull/2375/diff?src=pr&el=tree#diff-cHlkYW50aWMvYW5ub3RhdGVkX3R5cGVzLnB5) | `66.66% <0.00%> (-33.34%)` | :arrow_down: |
| [pydantic/typing.py](https://codecov.io/gh/samuelcolvin/pydantic/pull/2375/diff?src=pr&el=tree#diff-cHlkYW50aWMvdHlwaW5nLnB5) | `66.09% <0.00%> (-33.32%)` | :arrow_down: |
| [pydantic/version.py](https://codecov.io/gh/samuelcolvin/pydantic/pull/2375/diff?src=pr&el=tree#diff-cHlkYW50aWMvdmVyc2lvbi5weQ==) | `88.23% <0.00%> (-11.77%)` | :arrow_down: |
| [pydantic/decorator.py](https://codecov.io/gh/samuelcolvin/pydantic/pull/2375/diff?src=pr&el=tree#diff-cHlkYW50aWMvZGVjb3JhdG9yLnB5) | `88.43% <0.00%> (-11.57%)` | :arrow_down: |
| [pydantic/env\_settings.py](https://codecov.io/gh/samuelcolvin/pydantic/pull/2375/diff?src=pr&el=tree#diff-cHlkYW50aWMvZW52X3NldHRpbmdzLnB5) | `94.44% <0.00%> (-5.56%)` | :arrow_down: |
| [pydantic/fields.py](https://codecov.io/gh/samuelcolvin/pydantic/pull/2375/diff?src=pr&el=tree#diff-cHlkYW50aWMvZmllbGRzLnB5) | `97.07% <0.00%> (-2.93%)` | :arrow_down: |
| [pydantic/networks.py](https://codecov.io/gh/samuelcolvin/pydantic/pull/2375/diff?src=pr&el=tree#diff-cHlkYW50aWMvbmV0d29ya3MucHk=) | `98.52% <0.00%> (-1.48%)` | :arrow_down: |
| [pydantic/error\_wrappers.py](https://codecov.io/gh/samuelcolvin/pydantic/pull/2375/diff?src=pr&el=tree#diff-cHlkYW50aWMvZXJyb3Jfd3JhcHBlcnMucHk=) | `98.86% <0.00%> (-1.14%)` | :arrow_down: |
| ... and [4 more](https://codecov.io/gh/samuelcolvin/pydantic/pull/2375/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/samuelcolvin/pydantic/pull/2375?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/samuelcolvin/pydantic/pull/2375?src=pr&el=footer). Last update [cc3010c...2eaed2f](https://codecov.io/gh/samuelcolvin/pydantic/pull/2375?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

maximberg: @Mazyod What do you think? It may be useful to you.
Mazyod: Looks like a great addition! (although the details of schema generation as per openAPI standard is beyond me 🙇‍♂️)
samuelcolvin: conflicts.

But honestly, this is all beyond me. Can you give the simplest possible example of how this might be used IRL?
maximberg: @samuelcolvin, thanks for your comment.
First of all, we are not talking about data validation, but about the schema that describes our model. That is, we need to describe as accurately as possible what data must be transferred in order to successfully pass the validation. Validation for generics is a developer's job, as it seems to me.
In my project, I actively use schema to automatically generate documentation for the API, for example.
Here is a synthetic example, but similar to what I am working with in my project.
For example, the model has a parameter `user: User` which has a title. We pass a role and name as a tuple for validation. The role is the enum.
When forming the schema, we want to display that data have to be passed as a tuple. We will also describe our enum separately - as it has a title and description. And the generic has an example of use in the description of the scheme.
All this should be collected in the final scheme.
And now for some reason we need to teach model to validate both - one user and several users - in the same parameter. We will rewrite the validator in the generic `User`, description of it and also modify the schema modifier to show that an array of tuples can be passed too.
Hope my example explains the idea. If not - let me know, please.
maximberg: @samuelcolvin If you are going to issue new version, please take my PR. I'm living on my fork in my project waiting for it. :) Thanks!
samuelcolvin: I'm sorry this is still open and I've just got around to looking at it properly.

There are some trivial conflicts again, but I have some more fundamental questions here:
1. Does this change the current schema for tuples in any scenarios? If so it's a problem as the schema shouldn't change in minor versions
2. My small brain still doesn't understand exactly what case this is modifying for generics. Please could you create a self contained code example that shows what the current behaviour is and what the new behaviour is.

I'm worried this might need to wait for v2, unless either:
* this doesn't change current schema by default
* The current behaviour is obviously illogical or directly contradicts JSONSchema

https://m.xkcd.com/1172/
maximberg: Thanks for you comment.
1. My change will not affect the logic for converting tuples to JSONSchema. Moreover, I partly used tuple logic to convert generics to schema.
2. Below I have given an example in which I compared the previous version (which I made in #2262) and the new one that I propose in #2275.
3. New code for nested generic models from master changes my code logic. I need some time to merge.

```import pprint
from enum import IntEnum
from typing import TypeVar, Any, Generic, Optional

from pydantic import BaseModel, UUID4
from pydantic.fields import ModelField

T = TypeVar('T')
S = TypeVar('S')


class MyGeneric(Generic[T, S]):

    @classmethod
    def __modify_schema__(cls, field_schema):
        the_type = field_schema.pop('type', None)
        if not the_type:
            the_type = field_schema.pop('allOf', 'string')
        if isinstance(the_type, str):
            the_type = {'type': the_type}
        elif isinstance(the_type, list):
            the_type = the_type[0]
        field_schema.update(
            anyOf=[the_type, {'type': 'array', 'items': the_type}]
        )

    @classmethod
    def __get_validators__(cls):
        yield cls.validate

    @classmethod
    def validate(cls, v: Any, field: ModelField) -> 'MyGeneric':
        ...


class Status(IntEnum):
    new = 0
    active = 1
    closed = 2

    @classmethod
    def __modify_schema__(cls, field_schema):
        field_schema.update(dict(
            type='string',
            enum=[x.name for x in cls]
        ))


class MyModel(BaseModel):
    link_to_id: Optional[MyGeneric[UUID4, str]]
    status: MyGeneric[Status, str]


pprint.pp(MyModel.schema(), width=100)
```
#2262 produces:
```
{'title': 'MyModel',
 'type': 'object',
 'properties': {'link_to_id': {'title': 'Link To Id',
                               'anyOf': [{'type': 'string'},
                                         {'type': 'array', 'items': {'type': 'string'}}]},
                'status': {'title': 'Status',
                           'anyOf': [{'type': 'string'},
                                     {'type': 'array', 'items': {'type': 'string'}}]}},
 'required': ['status'],
 'definitions': {'Status': {'title': 'Status',
                            'description': 'An enumeration.',
                            'enum': ['new', 'active', 'closed'],
                            'type': 'string'}}}
 ```
 
 #2275 produces:
 ```
{'title': 'MyModel',
 'type': 'object',
 'properties': {'link_to_id': {'title': 'Link To Id',
                               'anyOf': [{'type': 'array',
                                          'items': [{'type': 'string', 'format': 'uuid4'},
                                                    {'type': 'string'}]},
                                         {'type': 'array',
                                          'items': {'type': 'array',
                                                    'items': [{'type': 'string', 'format': 'uuid4'},
                                                              {'type': 'string'}]}}]},
                'status': {'title': 'Status',
                           'anyOf': [{'type': 'array',
                                      'items': [{'$ref': '#/definitions/Status'},
                                                {'type': 'string'}]},
                                     {'type': 'array',
                                      'items': {'type': 'array',
                                                'items': [{'$ref': '#/definitions/Status'},
                                                          {'type': 'string'}]}}]}},
 'required': ['status'],
 'definitions': {'Status': {'title': 'Status',
                            'description': 'An enumeration.',
                            'enum': ['new', 'active', 'closed'],
                            'type': 'string'}}}
```
 

maximberg: @samuelcolvin Looks like I managed to merge.
maximberg: @samuelcolvin Should I do something else?

Patch:
diff --git a/changes/2375-maximberg.md b/changes/2375-maximberg.md
new file mode 100644
index 000000000..0ad29588f
--- /dev/null
+++ b/changes/2375-maximberg.md
@@ -0,0 +1,1 @@
+Support generating schema for `Generic` fields with subtypes. 
\ No newline at end of file
diff --git a/pydantic/schema.py b/pydantic/schema.py
index d640bc163..074240fbc 100644
--- a/pydantic/schema.py
+++ b/pydantic/schema.py
@@ -434,6 +434,8 @@ def field_type_schema(
     Take a single ``field`` and generate the schema for its type only, not including additional
     information as title, etc. Also return additional schema definitions, from sub-models.
     """
+    from .main import BaseModel  # noqa: F811
+
     definitions = {}
     nested_models: Set[str] = set()
     f_schema: Dict[str, Any]
@@ -473,7 +475,7 @@ def field_type_schema(
         elif items_schema:
             # The dict values are not simply Any, so they need a schema
             f_schema['additionalProperties'] = items_schema
-    elif field.shape == SHAPE_TUPLE:
+    elif field.shape == SHAPE_TUPLE or (field.shape == SHAPE_GENERIC and not issubclass(field.type_, BaseModel)):
         sub_schema = []
         sub_fields = cast(List[ModelField], field.sub_fields)
         for sf in sub_fields:
@@ -489,8 +491,14 @@ def field_type_schema(
             nested_models.update(sf_nested_models)
             sub_schema.append(sf_schema)
         if len(sub_schema) == 1:
-            sub_schema = sub_schema[0]  # type: ignore
-        f_schema = {'type': 'array', 'items': sub_schema}
+            if field.shape == SHAPE_GENERIC:
+                f_schema = sub_schema[0]
+            else:
+                f_schema = {'type': 'array', 'items': sub_schema[0]}
+        else:
+            f_schema = {'type': 'array', 'items': sub_schema}
+        if field.shape == SHAPE_GENERIC:
+            f_schema = {'allOf': [f_schema]}
     else:
         assert field.shape in {SHAPE_SINGLETON, SHAPE_GENERIC}, field.shape
         f_schema, f_definitions, f_nested_models = field_singleton_schema(


Test patch:
diff --git a/tests/test_schema.py b/tests/test_schema.py
index 43b6b69f0..9c620aa01 100644
--- a/tests/test_schema.py
+++ b/tests/test_schema.py
@@ -2266,7 +2266,7 @@ def test_schema_for_generic_field():
         'title': 'Model',
         'type': 'object',
         'properties': {
-            'data': {'title': 'Data', 'type': 'string'},
+            'data': {'allOf': [{'type': 'string'}], 'title': 'Data'},
             'data1': {
                 'title': 'Data1',
             },
@@ -2277,7 +2277,7 @@ def test_schema_for_generic_field():
     class GenModelModified(GenModel, Generic[T]):
         @classmethod
         def __modify_schema__(cls, field_schema):
-            field_schema.pop('type', None)
+            field_schema.pop('allOf', None)
             field_schema.update(anyOf=[{'type': 'string'}, {'type': 'array', 'items': {'type': 'string'}}])
 
     class ModelModified(BaseModel):
@@ -2317,6 +2317,111 @@ def test_namedtuple_default():
     }
 
 
+@pytest.mark.skipif(
+    sys.version_info < (3, 7), reason='schema generation for generic fields is not available in python < 3.7'
+)
+def test_advanced_generic_schema():
+    T = TypeVar('T')
+    K = TypeVar('K')
+
+    class Gen(Generic[T]):
+        def __init__(self, data: Any):
+            self.data = data
+
+        @classmethod
+        def __get_validators__(cls):
+            yield cls.validate
+
+        @classmethod
+        def validate(cls, v: Any):
+            return v
+
+        @classmethod
+        def __modify_schema__(cls, field_schema):
+            the_type = field_schema.pop('allOf', [{'type': 'string'}])[0]
+            field_schema.update(title='Gen title', anyOf=[the_type, {'type': 'array', 'items': the_type}])
+
+    class GenTwoParams(Generic[T, K]):
+        def __init__(self, x: str, y: Any):
+            self.x = x
+            self.y = y
+
+        @classmethod
+        def __get_validators__(cls):
+            yield cls.validate
+
+        @classmethod
+        def validate(cls, v: Any):
+            return cls(*v)
+
+        @classmethod
+        def __modify_schema__(cls, field_schema):
+            field_schema.update(examples='examples')
+
+    class CustomType(Enum):
+        A = 'a'
+        B = 'b'
+
+        @classmethod
+        def __modify_schema__(cls, field_schema):
+            field_schema.update(title='CustomType title', type='string')
+
+    class Model(BaseModel):
+        data0: Gen
+        data1: Gen[CustomType] = Field(title='Data1 title', description='Data 1 description')
+        data2: GenTwoParams[CustomType, UUID4] = Field(title='Data2 title', description='Data 2')
+        # check Tuple because changes in code touch that type
+        data3: Tuple
+        data4: Tuple[CustomType]
+        data5: Tuple[CustomType, str]
+
+    assert Model.schema() == {
+        'title': 'Model',
+        'type': 'object',
+        'properties': {
+            'data0': {
+                'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}],
+                'title': 'Gen title',
+            },
+            'data1': {
+                'title': 'Gen title',
+                'description': 'Data 1 description',
+                'anyOf': [
+                    {'$ref': '#/definitions/CustomType'},
+                    {'type': 'array', 'items': {'$ref': '#/definitions/CustomType'}},
+                ],
+            },
+            'data2': {
+                'allOf': [
+                    {
+                        'items': [{'$ref': '#/definitions/CustomType'}, {'format': 'uuid4', 'type': 'string'}],
+                        'type': 'array',
+                    }
+                ],
+                'title': 'Data2 title',
+                'description': 'Data 2',
+                'examples': 'examples',
+            },
+            'data3': {'title': 'Data3', 'type': 'array', 'items': {}},
+            'data4': {'title': 'Data4', 'type': 'array', 'items': {'$ref': '#/definitions/CustomType'}},
+            'data5': {
+                'title': 'Data5',
+                'type': 'array',
+                'items': [{'$ref': '#/definitions/CustomType'}, {'type': 'string'}],
+            },
+        },
+        'required': ['data0', 'data1', 'data2', 'data3', 'data4', 'data5'],
+        'definitions': {
+            'CustomType': {
+                'title': 'CustomType title',
+                'description': 'An enumeration.',
+                'enum': ['a', 'b'],
+                'type': 'string',
+            }
+        },
+    }
+
+
 @pytest.mark.skipif(
     sys.version_info < (3, 7), reason='schema generation for generic fields is not available in python < 3.7'
 )