Repository: pydantic/pydantic
Pull request: Fix regression in core schema generation for indirect definition references
Closes https://github.com/pydantic/pydantic/issues/8688

Problem statement:
Discriminator attribute missing from model schema in a forwardref in pydantic 2.6
### Initial Checks

- [X] I confirm that I'm using Pydantic V2

### Description

from the sample code below, the `Submodel` schema is missing the `discriminator` tag for the field `steps` , this is not the case when using version 2.5

#### 2.5
```
steps": {
          "default": null,
          "discriminator": {
            "mapping": {
              "stepA": "#/$defs/Step_A",
              "stepB": "#/$defs/Step_B"
            },
            "propertyName": "type"
          },
          "oneOf": [
            {
              "$ref": "#/$defs/Step_A"
            },
            {
              "$ref": "#/$defs/Step_B"
            }
          ],
          "title": "Steps"
        },
```

#### 2.6
```
       "steps": {
          "anyOf": [
            {
              "$ref": "#/$defs/Step_A"
            },
            {
              "$ref": "#/$defs/Step_B"
            }
          ],
          "default": null,
          "title": "Steps"
        },
```

### Example Code

```Python
import json
from typing import Literal, Union

from pydantic import BaseModel, Field


class Step_A(BaseModel):
    type: Literal["stepA"]
    count: int


class Step_B(BaseModel):
    type: Literal["stepB"]
    value: float


class MyModel(BaseModel):
    type: Literal["mixed"]
    sub_models: list["SubModel"]
    steps: Union[Step_A, Step_B] = Field(
        default=None,
        discriminator="type",
    )


class SubModel(MyModel):
    type: Literal["mixed"]
    blending: float


with open("main.json", "w") as out:
    json.dump(MyModel.model_json_schema(), out, indent=2)
```


### Python, Pydantic & OS Version

```Text
pydantic version: 2.6.0
        pydantic-core version: 2.16.1
          pydantic-core build: profile=release pgo=true
                 install path: /scratch/miniconda3/envs/pydantic26/lib/python3.11/site-packages/pydantic
               python version: 3.11.7 | packaged by conda-forge | (main, Dec 23 2023, 14:43:09) [GCC 12.3.0]
                     platform: Linux-3.10.0-1160.45.1.el7.x86_64-x86_64-with-glibc2.17
             related packages: typing_extensions-4.9.0
                       commit: unknown
```

Hints from issue discussion:
@vigneshmanick,

Thanks for reporting this. We'll look into a fix to include in a patch release soon!
Huh, I thought we tried to fix this via: https://github.com/pydantic/pydantic/issues/8271. Looks like indeed it hasn't yet been fixed.

@StrawHatDrag0n, any ideas based on your contribution to the aforementioned fix?
Hi @vigneshmanick 

I am seeing the discriminator field in the output. 

`main.json`

```
{
  "$defs": {
    "Step_A": {
      "properties": {
        "type": {
          "const": "stepA",
          "title": "Type"
        },
        "count": {
          "title": "Count",
          "type": "integer"
        }
      },
      "required": [
        "type",
        "count"
      ],
      "title": "Step_A",
      "type": "object"
    },
    "Step_B": {
      "properties": {
        "type": {
          "const": "stepB",
          "title": "Type"
        },
        "value": {
          "title": "Value",
          "type": "number"
        }
      },
      "required": [
        "type",
        "value"
      ],
      "title": "Step_B",
      "type": "object"
    },
    "SubModel": {
      "properties": {
        "type": {
          "const": "mixed",
          "title": "Type"
        },
        "sub_models": {
          "items": {
            "$ref": "#/$defs/SubModel"
          },
          "title": "Sub Models",
          "type": "array"
        },
        "steps": {
          "anyOf": [
            {
              "$ref": "#/$defs/Step_A"
            },
            {
              "$ref": "#/$defs/Step_B"
            }
          ],
          "default": null,
          "title": "Steps"
        },
        "blending": {
          "title": "Blending",
          "type": "number"
        }
      },
      "required": [
        "type",
        "sub_models",
        "blending"
      ],
      "title": "SubModel",
      "type": "object"
    }
  },
  "properties": {
    "type": {
      "const": "mixed",
      "title": "Type"
    },
    "sub_models": {
      "items": {
        "$ref": "#/$defs/SubModel"
      },
      "title": "Sub Models",
      "type": "array"
    },
    "steps": {
      "default": null,
      "discriminator": {
        "mapping": {
          "stepA": "#/$defs/Step_A",
          "stepB": "#/$defs/Step_B"
        },
        "propertyName": "type"
      },
      "oneOf": [
        {
          "$ref": "#/$defs/Step_A"
        },
        {
          "$ref": "#/$defs/Step_B"
        }
      ],
      "title": "Steps"
    }
  },
  "required": [
    "type",
    "sub_models"
  ],
  "title": "MyModel",
  "type": "object"
}
```


Looking at your example I see that the `steps` field you are referring to might be different ones in 2.5 and 2.6

In 2,5 it contains `oneOf`
```
steps": {
          "default": null,
          "discriminator": {
            "mapping": {
              "stepA": "#/$defs/Step_A",
              "stepB": "#/$defs/Step_B"
            },
            "propertyName": "type"
          },
          "oneOf": [
            {
              "$ref": "#/$defs/Step_A"
            },
            {
              "$ref": "#/$defs/Step_B"
            }
          ],
          "title": "Steps"
        },
```

While in the 2.6 it contains `anyOf`
```
       "steps": {
          "anyOf": [
            {
              "$ref": "#/$defs/Step_A"
            },
            {
              "$ref": "#/$defs/Step_B"
            }
          ],
          "default": null,
          "title": "Steps"
        },
```

Let me know if this helps

Yeah, the issue here is that the core schema is different. In 2.5, `steps` is a `tagged-union`, whereas it's just a plain `union` in 2.6.

I've traced this issue back to here: https://github.com/pydantic/pydantic/commit/8060fa1cff965850e5e08a67ca73d5272dcdcf9f.

Will dig into this further 👍 
@alexmojaki,

Could you please take a look at this issue, given that it's associated with the implementation of the `eval_type_backport` logic? Thanks!
Ah actually, looks like the issue was here: https://github.com/pydantic/pydantic/commit/1fe90b4007badaf49b9bfe8878e9a05a2718ccf0

Commit message: Fix regression in core schema generation for indirect definition references

Patch to review:
diff --git a/pydantic/_internal/_core_utils.py b/pydantic/_internal/_core_utils.py
index 2c2f027af4e..432de5f5655 100644
--- a/pydantic/_internal/_core_utils.py
+++ b/pydantic/_internal/_core_utils.py
@@ -228,7 +228,7 @@ def _handle_ser_schemas(self, ser_schema: core_schema.SerSchema, f: Walk) -> cor
     def handle_definitions_schema(self, schema: core_schema.DefinitionsSchema, f: Walk) -> core_schema.CoreSchema:
         new_definitions: list[core_schema.CoreSchema] = []
         for definition in schema['definitions']:
-            if 'schema_ref' and 'ref' in definition:
+            if 'schema_ref' in definition and 'ref' in definition:
                 # This indicates a purposely indirect reference
                 # We want to keep such references around for implications related to JSON schema, etc.:
                 new_definitions.append(definition)
diff --git a/tests/test_discriminated_union.py b/tests/test_discriminated_union.py
index 24e319da306..24d1ead48d4 100644
--- a/tests/test_discriminated_union.py
+++ b/tests/test_discriminated_union.py
@@ -1763,3 +1763,90 @@ class CreateObjectDto(BaseModel):
             'type': 'object',
         },
     }
+
+
+def test_nested_discriminator() -> None:
+    """
+    The exact details of the JSON schema produced are not necessarily important; the test was added in response to a
+    regression that caused the inner union was losing its discriminator. Even if the schema changes, the important
+    thing is that the core schema (and therefore JSON schema) produced has an actual discriminated union in it.
+    """
+
+    class Step_A(BaseModel):
+        type: Literal['stepA']
+        count: int
+
+    class Step_B(BaseModel):
+        type: Literal['stepB']
+        value: float
+
+    class MyModel(BaseModel):
+        type: Literal['mixed']
+        sub_models: list['SubModel']
+        steps: Union[Step_A, Step_B] = Field(
+            default=None,
+            discriminator='type',
+        )
+
+    class SubModel(MyModel):
+        type: Literal['mixed']
+        blending: float
+
+    MyModel.model_rebuild()
+    assert MyModel.model_json_schema() == {
+        '$defs': {
+            'Step_A': {
+                'properties': {
+                    'count': {'title': 'Count', 'type': 'integer'},
+                    'type': {'const': 'stepA', 'title': 'Type'},
+                },
+                'required': ['type', 'count'],
+                'title': 'Step_A',
+                'type': 'object',
+            },
+            'Step_B': {
+                'properties': {
+                    'type': {'const': 'stepB', 'title': 'Type'},
+                    'value': {'title': 'Value', 'type': 'number'},
+                },
+                'required': ['type', 'value'],
+                'title': 'Step_B',
+                'type': 'object',
+            },
+            'SubModel': {
+                'properties': {
+                    'blending': {'title': 'Blending', 'type': 'number'},
+                    'steps': {
+                        'default': None,
+                        'discriminator': {
+                            'mapping': {'stepA': '#/$defs/Step_A', 'stepB': '#/$defs/Step_B'},
+                            'propertyName': 'type',
+                        },
+                        'oneOf': [{'$ref': '#/$defs/Step_A'}, {'$ref': '#/$defs/Step_B'}],
+                        'title': 'Steps',
+                    },
+                    'sub_models': {'items': {'$ref': '#/$defs/SubModel'}, 'title': 'Sub Models', 'type': 'array'},
+                    'type': {'const': 'mixed', 'title': 'Type'},
+                },
+                'required': ['type', 'sub_models', 'blending'],
+                'title': 'SubModel',
+                'type': 'object',
+            },
+        },
+        'properties': {
+            'steps': {
+                'default': None,
+                'discriminator': {
+                    'mapping': {'stepA': '#/$defs/Step_A', 'stepB': '#/$defs/Step_B'},
+                    'propertyName': 'type',
+                },
+                'oneOf': [{'$ref': '#/$defs/Step_A'}, {'$ref': '#/$defs/Step_B'}],
+                'title': 'Steps',
+            },
+            'sub_models': {'items': {'$ref': '#/$defs/SubModel'}, 'title': 'Sub Models', 'type': 'array'},
+            'type': {'const': 'mixed', 'title': 'Type'},
+        },
+        'required': ['type', 'sub_models'],
+        'title': 'MyModel',
+        'type': 'object',
+    }


Code review comments:
diff_hunk: @@ -1763,3 +1763,90 @@ class CreateObjectDto(BaseModel):
             'type': 'object',
         },
     }
+
+
+def test_nested_discriminator() -> None:
+    """
+    The exact details of the JSON schema produced are not necessarily important; the test was added in response to a
+    regression that caused the inner union was losing its discriminator. Even if the schema changes, the important
+    thing is that the core schema (and therefore JSON schema) produced has an actual discriminated union in it.
+    """
+
+    class Step_A(BaseModel):
+        type: Literal['stepA']
+        count: int
+
+    class Step_B(BaseModel):
+        type: Literal['stepB']
+        value: float
+
+    class MyModel(BaseModel):
+        type: Literal['mixed']
+        sub_models: list['SubModel']
path: tests/test_discriminated_union.py
text: @user1:
```suggestion
        sub_models: List['SubModel']
```

Will have to change this to make the 3.8 tests happy

diff_hunk: @@ -1763,3 +1763,90 @@ class CreateObjectDto(BaseModel):
             'type': 'object',
         },
     }
+
+
+def test_nested_discriminator() -> None:
+    """
+    The exact details of the JSON schema produced are not necessarily important; the test was added in response to a
+    regression that caused the inner union was losing its discriminator. Even if the schema changes, the important
path: tests/test_discriminated_union.py
text: @user1:
```suggestion
    regression that caused the inner union to lose its discriminator. Even if the schema changes, the important
```