Repository: koxudaxi/datamodel-code-generator

Issue:
json schema files making use of `allOf` with a reference and a `$id` throw a `KeyError`
**Describe the bug**

json schema files making use of `allOf` with a reference and a `$id` throw a `KeyError`.

My understanding of the issue is that the `references` dictionary contains
non-resolved paths as keys for `allOf` items, and therefore the reference can't
be deleted when we do resolved lookup.

**To Reproduce**

Example schema:
```json
{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://example.com/all_of_ref_self.json",
  "type": "object",
  "properties": {
    "version": {
      "allOf": [
        { "$ref": "#/$defs/version" }
      ]
    }
  },
  "$defs": {
    "version": {
      "type": "null"
    }
  }
}
```

Used commandline:
```
$ datamodel-codegen --input-file-type jsonschema --input tmp.json
Traceback (most recent call last):
  File "/home/pim/Shadow/dev/virtu/vmdesc/env/lib/python3.11/site-packages/datamodel_code_generator/__main__.py", line 386, in main
    generate(
  File "/home/pim/Shadow/dev/virtu/vmdesc/env/lib/python3.11/site-packages/datamodel_code_generator/__init__.py", line 446, in generate
    results = parser.parse()
              ^^^^^^^^^^^^^^
  File "/home/pim/Shadow/dev/virtu/vmdesc/env/lib/python3.11/site-packages/datamodel_code_generator/parser/base.py", line 1136, in parse
    self.parse_raw()
  File "/home/pim/Shadow/dev/virtu/vmdesc/env/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1603, in parse_raw
    self._parse_file(self.raw_obj, obj_name, path_parts)
  File "/home/pim/Shadow/dev/virtu/vmdesc/env/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1687, in _parse_file
    self.parse_obj(obj_name, root_obj, path_parts or ['#'])
  File "/home/pim/Shadow/dev/virtu/vmdesc/env/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1555, in parse_obj
    self.parse_object(name, obj, path)
  File "/home/pim/Shadow/dev/virtu/vmdesc/env/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 912, in parse_object
    fields = self.parse_object_fields(obj, path, get_module_name(class_name, None))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pim/Shadow/dev/virtu/vmdesc/env/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 867, in parse_object_fields
    field_type = self.parse_item(modular_name, field, [*path, field_name])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pim/Shadow/dev/virtu/vmdesc/env/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1030, in parse_item
    return self.parse_all_of(
           ^^^^^^^^^^^^^^^^^^
  File "/home/pim/Shadow/dev/virtu/vmdesc/env/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 782, in parse_all_of
    return self._parse_object_common_part(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pim/Shadow/dev/virtu/vmdesc/env/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 677, in _parse_object_common_part
    self.model_resolver.delete(path)
  File "/home/pim/Shadow/dev/virtu/vmdesc/env/lib/python3.11/site-packages/datamodel_code_generator/reference.py", line 643, in delete
    del self.references[self.resolve_ref(path)]
        ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'https://abc.com/tmp.json/version#-datamodel-code-generator-#-allOf-#-special-#'
```

**Expected behavior**
I would expect this schema to be parsed successfully (no exceptions).

**Version:**
 - OS: Linux (Archlinux, 6.5.7-arch1-1)
 - Python version: 3.11.5
 - datamodel-code-generator version: 0.17.2 to 0.23.0

**Additional context**

This behaviour seemed to appear between 0.17.1 (works) and 0.17.2 (doesn't work). In particular, with commit https://github.com/koxudaxi/datamodel-code-generator/commit/0187b4aa7955adfc5b42090274e146546a132715

This pattern seems to be parsed correctly by other validators, e.g. https://www.jsonschemavalidator.net/s/FwyNnGC3

Real world jsonschemas also have this pattern, and show the same issue, for example with [stripe/stripe-apps](https://github.com/stripe/stripe-apps/blob/954a21bc807671e72a126ac04d72e2ac88063f5d/schema/stripe-app.schema.json):

```
$ cd $(mktemp -d)
$ python -m venv env
$ . env/bin/activate
$ pip install datamodel-code-generator
$ curl https://raw.githubusercontent.com/stripe/stripe-apps/main/schema/stripe-app.schema.json -o stripe-app.schema.json
$ datamodel-codegen --input stripe-app.schema.json
The input file type was determined to be: jsonschema
This can be specificied explicitly with the `--input-file-type` option.
Traceback (most recent call last):
  File "/tmp/tmp.8CqYazcKUp/env/lib/python3.11/site-packages/datamodel_code_generator/__main__.py", line 388, in main
    generate(
  File "/tmp/tmp.8CqYazcKUp/env/lib/python3.11/site-packages/datamodel_code_generator/__init__.py", line 435, in generate
    results = parser.parse()
              ^^^^^^^^^^^^^^
  File "/tmp/tmp.8CqYazcKUp/env/lib/python3.11/site-packages/datamodel_code_generator/parser/base.py", line 1135, in parse
    self.parse_raw()
  File "/tmp/tmp.8CqYazcKUp/env/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1603, in parse_raw
    self._parse_file(self.raw_obj, obj_name, path_parts)
  File "/tmp/tmp.8CqYazcKUp/env/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1687, in _parse_file
    self.parse_obj(obj_name, root_obj, path_parts or ['#'])
  File "/tmp/tmp.8CqYazcKUp/env/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1555, in parse_obj
    self.parse_object(name, obj, path)
  File "/tmp/tmp.8CqYazcKUp/env/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 912, in parse_object
    fields = self.parse_object_fields(obj, path, get_module_name(class_name, None))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.8CqYazcKUp/env/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 867, in parse_object_fields
    field_type = self.parse_item(modular_name, field, [*path, field_name])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.8CqYazcKUp/env/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1024, in parse_item
    data_types=self.parse_one_of(
               ^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.8CqYazcKUp/env/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 641, in parse_one_of
    data_types = self.parse_list_item(name, obj.oneOf, path, obj)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.8CqYazcKUp/env/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1075, in parse_list_item
    return [
           ^
  File "/tmp/tmp.8CqYazcKUp/env/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1076, in <listcomp>
    self.parse_item(
  File "/tmp/tmp.8CqYazcKUp/env/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1041, in parse_item
    return self.parse_object(
           ^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.8CqYazcKUp/env/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 912, in parse_object
    fields = self.parse_object_fields(obj, path, get_module_name(class_name, None))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.8CqYazcKUp/env/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 867, in parse_object_fields
    field_type = self.parse_item(modular_name, field, [*path, field_name])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.8CqYazcKUp/env/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1030, in parse_item
    return self.parse_all_of(
           ^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.8CqYazcKUp/env/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 782, in parse_all_of
    return self._parse_object_common_part(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.8CqYazcKUp/env/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 677, in _parse_object_common_part
    self.model_resolver.delete(path)
  File "/tmp/tmp.8CqYazcKUp/env/lib/python3.11/site-packages/datamodel_code_generator/reference.py", line 643, in delete
    del self.references[self.resolve_ref(path)]
        ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'https://stripe.com/stripe-app.schema.json/post_install_action#-datamodel-code-generator-#-oneOf-#-special-#/1#-datamodel-code-generator-#-object-#-special-#/type#-datamodel-code-generator-#-allOf-#-special-#'
```



Discussion:

Resolving patch:
diff --git a/datamodel_code_generator/parser/jsonschema.py b/datamodel_code_generator/parser/jsonschema.py
--- a/datamodel_code_generator/parser/jsonschema.py
+++ b/datamodel_code_generator/parser/jsonschema.py
@@ -1027,6 +1027,7 @@ def parse_item(
             )
         elif item.allOf:
             all_of_path = get_special_path('allOf', path)
+            all_of_path = [self.model_resolver.resolve_ref(all_of_path)]
             return self.parse_all_of(
                 self.model_resolver.add(
                     all_of_path, name, singular_name=singular_name, class_name=True


Test patch:
diff --git a/tests/data/expected/main/main_all_of_ref_self/output.py b/tests/data/expected/main/main_all_of_ref_self/output.py
new file mode 100644
--- /dev/null
+++ b/tests/data/expected/main/main_all_of_ref_self/output.py
@@ -0,0 +1,17 @@
+# generated by datamodel-codegen:
+#   filename:  all_of_ref_self.json
+#   timestamp: 2019-07-26T00:00:00+00:00
+
+from __future__ import annotations
+
+from typing import Optional
+
+from pydantic import BaseModel
+
+
+class Version(BaseModel):
+    __root__: None
+
+
+class Model(BaseModel):
+    version: Optional[Version] = None
diff --git a/tests/data/jsonschema/all_of_ref_self.json b/tests/data/jsonschema/all_of_ref_self.json
new file mode 100644
--- /dev/null
+++ b/tests/data/jsonschema/all_of_ref_self.json
@@ -0,0 +1,17 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://example.com/all_of_ref_self.json",
+  "type": "object",
+  "properties": {
+    "version": {
+      "allOf": [
+        { "$ref": "#/$defs/version" }
+      ]
+    }
+  },
+  "$defs": {
+    "version": {
+      "type": "null"
+    }
+  }
+}
diff --git a/tests/test_main.py b/tests/test_main.py
--- a/tests/test_main.py
+++ b/tests/test_main.py
@@ -5957,3 +5957,24 @@ def test_main_dataclass_default():
                 EXPECTED_MAIN_PATH / 'main_dataclass_field_default' / 'output.py'
             ).read_text()
         )
+
+
+@freeze_time('2019-07-26')
+def test_main_all_of_ref_self():
+    with TemporaryDirectory() as output_dir:
+        output_file: Path = Path(output_dir) / 'output.py'
+        return_code: Exit = main(
+            [
+                '--input',
+                str(JSON_SCHEMA_DATA_PATH / 'all_of_ref_self.json'),
+                '--output',
+                str(output_file),
+                '--input-file-type',
+                'jsonschema',
+            ]
+        )
+        assert return_code == Exit.OK
+        assert (
+            output_file.read_text()
+            == (EXPECTED_MAIN_PATH / 'main_all_of_ref_self' / 'output.py').read_text()
+        )