Repository: pydantic/pydantic
Pull request: Use contentSchema keyword for JSON schema
Closes https://github.com/pydantic/pydantic/issues/6663

Selected Reviewer: @lig

Problem statement:
`Json[type]` schema generated now returns `string` rather than `type`
### Initial Checks

- [X] I confirm that I'm using Pydantic V2

### Description

With `Json[]` the schema type is now always `string`, rather than the type inside `Json[]`.

In `V1` the example would produce:

```JSON
{
   "title":"Foo",
   "type":"object",
   "properties":{
      "foo":{
         "title":"Foo",
         "type":"integer"
      }
   },
   "required":[
      "foo"
   ]
}
```


In `V2` it produces:
```JSON
{
   "properties":{
      "foo":{
         "format":"json-string",
         "title":"Foo",
         "type":"string"
      }
   },
   "required":[
      "foo"
   ],
   "title":"Foo",
   "type":"object"
}
```

I may be wrong, but given `"format": "json-string"` is added for clarity, there's no reason to change the type to `"string"` and this is a bug?

If this is intended and shouldn't be changed, is there any easy way to opt out without overriding `__get_pydantic_json_schema__` everywhere? The use case is most of our endpoints come in as json strings which are decoded, so this currently breaks a lot of the schema and validation externally.

### Example Code

```Python
from pydantic import BaseModel, Json


class Foo(BaseModel):
    foo: Json[int]


Foo.schema()
```


### Python, Pydantic & OS Version

```Text
pydantic version: 2.0.2
        pydantic-core version: 2.1.2 release build profile
                 install path: venv/lib/python3.11/site-packages/pydantic
               python version: 3.11.4 (main, Jun  7 2023, 12:45:48) [GCC 11.3.0]
                     platform: Linux-5.19.0-46-generic-x86_64-with-glibc2.35
     optional deps. installed: ['devtools', 'email-validator', 'typing-extensions']
```


Selected Assignee: @samuelcolvin

Hints from issue discussion:
Thanks @zakstucke for reporting this 🙏 

Isn't the `"type":"string"` correct? because the model accepts strings, not int.

```py
Foo(foo='1')  # Ok

Foo(foo=1)
"""
pydantic_core._pydantic_core.ValidationError: 1 validation error for Foo
foo
  JSON input should be string, bytes or bytearray [type=json_type, input_value=1, input_type=int]
    For further information visit https://errors.pydantic.dev/2.0.3/v/json_type
"""
```
@hramezani I would say no given a few reasons:
- `"format": "json-string"` is specifically stating that this input is expected to be coming in as a a jsonified version of the data type
- As a user of the schema, the user/external software can no longer validate the data *inside* the jsonified string, to make sure it's of the correct type pre-jsonification
- Example use case is a lot of my frontend's typing system has broken because of this, endpoint types have all converted to `string`, meaning the correct types are now seen as invalid.
Ok, I might be wrong.

Let's see what others think. @samuelcolvin @dmontagu 
This is intended: in V2, we now have the ability to create validation and serialization JSON schemas separately:
```python
from pydantic import BaseModel, Json


class Foo(BaseModel):
    foo: Json[int]


print(Foo.model_json_schema(mode='validation')['properties']['foo'])  # mode='validation' is the default
#> {'format': 'json-string', 'title': 'Foo', 'type': 'string'}

print(Foo.model_json_schema(mode='serialization')['properties']['foo'])  # mode='validation' is the default
#> {'title': 'Foo', 'type': 'integer'}
```

If you are using pydantic/calls to the `model_json_schema` method directly, the above should give you a way to get what you want; if you are using FastAPI or similar and it's not producing the right schema, let us know. (I don't believe FastAPI has added support for `mode='serialization'` schemas quite yet, but having some reported issues like this might increase the urgency on that.)
Also, I'll note that if you are using a web framework that is generating the schema for you and you want a short-term hacky fix to tide you over, you can do the following:
```python
import pydantic.json_schema
from pydantic.json_schema import GenerateJsonSchema, JsonSchemaValue
from pydantic_core import core_schema


class MyGenerateJsonSchema(GenerateJsonSchema):
    def json_schema(self, schema: core_schema.JsonSchema) -> JsonSchemaValue:
        # don't special-case validation
        # if self.mode == 'validation':
        #     return {'type': 'string', 'format': 'json-string'}
        if 'schema' in schema:
            return self.generate_inner(schema['schema'])
        else:
            # No wrapped schema, so return the same thing we would for an Any schema
            return self.generate_inner(core_schema.any_schema())


pydantic.json_schema.GenerateJsonSchema.json_schema = MyGenerateJsonSchema.json_schema

# ...

from pydantic import BaseModel, Json

class Foo(BaseModel):
    foo: Json[int]

print(Foo.model_json_schema(mode='validation')['properties']['foo'])
#> {'title': 'Foo', 'type': 'integer'}

print(Foo.model_json_schema(mode='serialization')['properties']['foo'])
#> {'title': 'Foo', 'type': 'integer'}
```

If you look at `GenerateJsonSchema.json_schema` you'll see the above override is very closely related to the parent method, it's just always using the `mode='serialization'` approach.
@dmontagu thanks, this workaround works very nicely to reproduce old behaviour. 

However, I'll admit I don't quite agree with your intended (once supported) usage of `serialization` instead of `validation`, in reality imo this doesn't make sense as the official workaround being to use `serialization`, because:
- it just is still validation, aka we want to know the data coming in, and the format it needs to be, not serialisation (how the data will come back out from a model). If my type is `Json[Foo]`, the endpoint isn't going to accept any old json input, the external party needs to know the exact structure of `Foo` as normal, since V2 this is all lost by default.
- Setting serialization could cause unintended consequences (think `Field(exclude=True)` etc). 

But I understand json here is a bit of a weird one, and both ways have downsides, I propose this as the new default `json_schema()`:

```Python
class MyGenerateJsonSchema(GenerateJsonSchema):
    def json_schema(self, schema: core_schema.JsonSchema) -> JsonSchemaValue:
        genned = self.generate_inner(schema["schema"]) if "schema" in schema else self.generate_inner(core_schema.any_schema())
        
        if self.mode == "validation":
            return {"type": "string", "format": "json-string", "contents": genned}
        
        return genned
```

Which would produce:
```JSON
{
   "contents":{
      "type":"integer"
   },
   "format":"json-string",
   "title":"Foo",
   "type":"string"
}
```

change the key as you wish, but this would prevent the default implementation just losing information at the very least, and could be used by downstream libraries/users if they needed it, making break from V1 much softer. 
Can PR it if you like, otherwise close this as the override of `.json_schema` works globally so I'm happy :+1: 
And for reference yes this is with `fastapi.openapi()`, that then gets passed into https://github.com/drwpow/openapi-typescript to produce typescript types from the schema. The hacky fix sorts it, but as said above I would personally use the hacky fix over the serialization method even if it was in some way supported in Fastapi, just seems too strange to use serialization.
> it just is still validation, aka we want to know the data coming in, and the format it needs to be, not serialisation (how the data will come back out from a model). If my type is Json[Foo], the endpoint isn't going to accept any old json input, the external party needs to know the exact structure of Foo as normal, since V2 this is all lost by default.

Unless I'm misunderstanding I guess this is the issue: there is no way to express "nested" json in json schema. You can either express it as a string, and loose information as you point out, or as the actual schema and have something technically incorrect since you're being sent a json string of that data, not the actual data.
@adriangb yeah pretty much. The hack https://github.com/pydantic/pydantic/issues/6663#issuecomment-1635934848 sorts it, but I still think the default should be changed pydantic side in one of these ways (in order of what I persoanlly think would be best):
- V1 behaviour but now the `"format":"json-string"` is added, it could arguably seen as correct and completely valid, no breaking change from V1, schema/swagger etc unchanged from V1.
- My suggestion in this [comment](https://github.com/pydantic/pydantic/issues/6663#issuecomment-1635992464) to just add the schema as a separate key, would still be breaking from V1, but less so than currently and probably as "correct" as it can be, no information is lost.
- Leave it as it is, but means lots of people will have to use this hack to get around it as jsonifying input params is quite a common use case, e.g. think when sending with `multipart/form-data`. The jsonifying/stringifying bit is usually quite obvious and probably happening automatically, what's actually important to get right is the structure of `Foo` inside `Json[Foo]`. 
But on a less concrete note, whenever I've found a need to use `Json[Foo]`, it's always because of an implementation level restriction that is forcing the usage of a json string, rather than the json being something that's actively chosen. In other words, In the schema and outside it, I've never cared / even known it gets to the endpoint as json, all that's been important is `Foo`, and why I feel most people would prefer old behaviour rather than this change.

Commit message: Use contentSchema keyword for JSON schema

Patch to review:
diff --git a/pydantic/json_schema.py b/pydantic/json_schema.py
index 2de8af4023e..955fff7f058 100644
--- a/pydantic/json_schema.py
+++ b/pydantic/json_schema.py
@@ -1609,13 +1609,13 @@ def json_schema(self, schema: core_schema.JsonSchema) -> JsonSchemaValue:
         Returns:
             The generated JSON schema.
         """
+        content_core_schema = schema['schema'] if 'schema' in schema else core_schema.any_schema()
+        content_json_schema = self.generate_inner(content_core_schema)
         if self.mode == 'validation':
-            return {'type': 'string', 'format': 'json-string'}
-        elif 'schema' in schema:
-            return self.generate_inner(schema['schema'])
+            return {'type': 'string', 'contentMediaType': 'application/json', 'contentSchema': content_json_schema}
         else:
-            # No wrapped schema, so return the same thing we would for an Any schema
-            return self.generate_inner(core_schema.any_schema())
+            # self.mode == 'serialization'
+            return content_json_schema
 
     def url_schema(self, schema: core_schema.UrlSchema) -> JsonSchemaValue:
         """Generates a JSON schema that matches a schema that defines a URL.
diff --git a/tests/test_generics.py b/tests/test_generics.py
index fbd30abbc23..44ea39212b7 100644
--- a/tests/test_generics.py
+++ b/tests/test_generics.py
@@ -2062,7 +2062,22 @@ class Payload(BaseModel):
 
     validation_schema = record.model_json_schema(mode='validation')
     assert validation_schema == {
-        'properties': {'message': {'format': 'json-string', 'title': 'Message', 'type': 'string'}},
+        '$defs': {
+            'Payload': {
+                'properties': {'payload_field': {'title': 'Payload Field', 'type': 'string'}},
+                'required': ['payload_field'],
+                'title': 'Payload',
+                'type': 'object',
+            }
+        },
+        'properties': {
+            'message': {
+                'contentMediaType': 'application/json',
+                'contentSchema': {'$ref': '#/$defs/Payload'},
+                'title': 'Message',
+                'type': 'string',
+            }
+        },
         'required': ['message'],
         'title': 'MessageWrapper[test_parse_generic_json.<locals>.Payload]',
         'type': 'object',
diff --git a/tests/test_json_schema.py b/tests/test_json_schema.py
index 8cb2865c093..bf32fd1d6d5 100644
--- a/tests/test_json_schema.py
+++ b/tests/test_json_schema.py
@@ -1078,14 +1078,19 @@ class Model(BaseModel):
         c: Json[Any]
 
     assert Model.model_json_schema() == {
-        'title': 'Model',
-        'type': 'object',
         'properties': {
-            'a': {'title': 'A', 'type': 'string', 'format': 'json-string'},
-            'b': {'title': 'B', 'type': 'string', 'format': 'json-string'},
-            'c': {'title': 'C', 'type': 'string', 'format': 'json-string'},
+            'a': {'contentMediaType': 'application/json', 'contentSchema': {}, 'title': 'A', 'type': 'string'},
+            'b': {
+                'contentMediaType': 'application/json',
+                'contentSchema': {'type': 'integer'},
+                'title': 'B',
+                'type': 'string',
+            },
+            'c': {'contentMediaType': 'application/json', 'contentSchema': {}, 'title': 'C', 'type': 'string'},
         },
         'required': ['a', 'b', 'c'],
+        'title': 'Model',
+        'type': 'object',
     }
     assert Model.model_json_schema(mode='serialization') == {
         'properties': {'a': {'title': 'A'}, 'b': {'title': 'B', 'type': 'integer'}, 'c': {'title': 'C'}},
@@ -4308,7 +4313,14 @@ class Outer(BaseModel):
     assert v_schema == {
         '$defs': {
             'Inner': {
-                'properties': {'x': {'format': 'json-string', 'title': 'X', 'type': 'string'}},
+                'properties': {
+                    'x': {
+                        'contentMediaType': 'application/json',
+                        'contentSchema': {'type': 'integer'},
+                        'title': 'X',
+                        'type': 'string',
+                    }
+                },
                 'required': ['x'],
                 'title': 'Inner',
                 'type': 'object',
@@ -4344,7 +4356,14 @@ class Outer(BaseModel):
     assert vs_schema == {
         '$defs': {
             'InnerInput': {
-                'properties': {'x': {'format': 'json-string', 'title': 'X', 'type': 'string'}},
+                'properties': {
+                    'x': {
+                        'contentMediaType': 'application/json',
+                        'contentSchema': {'type': 'integer'},
+                        'title': 'X',
+                        'type': 'string',
+                    }
+                },
                 'required': ['x'],
                 'title': 'Inner',
                 'type': 'object',
diff --git a/tests/test_types.py b/tests/test_types.py
index fd576135ffe..336f312f344 100644
--- a/tests/test_types.py
+++ b/tests/test_types.py
@@ -4542,7 +4542,12 @@ class Model(BaseModel):
             'my_none': {'type': 'null', 'title': 'My None'},
             'my_none_list': {'type': 'array', 'items': {'type': 'null'}, 'title': 'My None List'},
             'my_none_dict': {'type': 'object', 'additionalProperties': {'type': 'null'}, 'title': 'My None Dict'},
-            'my_json_none': {'type': 'string', 'format': 'json-string', 'title': 'My Json None'},
+            'my_json_none': {
+                'contentMediaType': 'application/json',
+                'contentSchema': {'type': 'null'},
+                'title': 'My Json None',
+                'type': 'string',
+            },
         },
         'required': ['my_none', 'my_none_list', 'my_none_dict', 'my_json_none'],
     }
@@ -4595,7 +4600,12 @@ class Model(BaseModel):
             'my_none': {'const': None, 'title': 'My None'},
             'my_none_list': {'type': 'array', 'items': {'const': None}, 'title': 'My None List'},
             'my_none_dict': {'type': 'object', 'additionalProperties': {'const': None}, 'title': 'My None Dict'},
-            'my_json_none': {'type': 'string', 'format': 'json-string', 'title': 'My Json None'},
+            'my_json_none': {
+                'contentMediaType': 'application/json',
+                'contentSchema': {'const': None},
+                'title': 'My Json None',
+                'type': 'string',
+            },
         },
         'required': ['my_none', 'my_none_list', 'my_none_dict', 'my_json_none'],
     }


Code review comments:
diff_hunk: @@ -1609,13 +1609,13 @@ def json_schema(self, schema: core_schema.JsonSchema) -> JsonSchemaValue:
         Returns:
             The generated JSON schema.
         """
+        content_core_schema = schema['schema'] if 'schema' in schema else core_schema.any_schema()
path: pydantic/json_schema.py
text: @user1:
I would like the following code more but it will always call `core_schema.any_schema()`

```suggestion
        content_core_schema = schema.get('schema', core_schema.any_schema())
```

It would be nice to have a constant like `core_schema.DEFAULT_SCHEMA` that will hold this value and allow the above to be as efficient as `if..in`.

diff_hunk: @@ -1609,13 +1609,13 @@ def json_schema(self, schema: core_schema.JsonSchema) -> JsonSchemaValue:
         Returns:
             The generated JSON schema.
         """
+        content_core_schema = schema['schema'] if 'schema' in schema else core_schema.any_schema()
path: pydantic/json_schema.py
text: @author:
```suggestion
    content_core_schema = schema.get('schema') or core_schema.any_schema()
```
I think this is the zero-overhead version of your suggestion, which works since all schemas must have a type key so `schema.get('schema')` will be falsy only if it is `None` (and having the value None is not meaningfully different from not being present)