Repository: zulip/zulip

Issue:
OpenAPI schema and validation is built on a fundamental misunderstanding of allOf
The `yamole` library preprocesses our OpenAPI schema by expanding `$ref` (incorrectly: #16106) and then naïvely merging all the objects in an `allOf` array together. This fails to capture the meaning of [`allOf`](https://swagger.io/docs/specification/data-models/oneof-anyof-allof-not/) according to the specification.

For example, consider

```yaml
components:
  # …
  schemas:
    # …
    JsonResponse:
      type: object
      additionalProperties: false
      properties:
        result:
          type: string
    JsonSuccess:
      allOf:
        - $ref: "#/components/schemas/JsonResponse"
        - required:
            - result
            - msg
        - properties:
            result:
              enum:
                - success
            msg:
              type: string
        - example: {"msg": "", "result": "success"}
```

In order for `{"msg": "", "result": "success"}` to validate against `JsonSuccess`, it must validate against `JsonSuccess.allOf[0]` and validate against `JsonSuccess.allOf[1]` and validate against `JsonSuccess.allOf[2]` and validate against `JsonSuccess.allOf[3]`. But it does not validate against `JsonSuccess.allOf[0] == JsonResponse` because `JsonResponse` specifies that the object must have a `result` property and no other properties (`additionalProperties: false`).

`yamole`’s merging before validation obscures this failure in a way that’s not supported by the specification: “`allOf` takes an array of object definitions that are used for **independent** validation… To be valid against `allOf`, the data provided by the client must be valid against all of the given subschemas.”

Cc @orientor @YagoGG 
Optimize yamole
This is probably an issue for @YagoGG.

It appears that yamole is quite slow in how it parses our 1700+ line zulip.yaml file.  Reading the yaml only takes 300ms for me, but then having yamole re-walk the tree with all its deep copies increases the time to 3.5 seconds, so almost a 12x difference.

This isn't a huge priority, since we cache the results internally, but it happens at startup.

It's not clear to me why we need so much `deepcopy` going on in the code.


Discussion:
Hello @zulip/server-api members, this issue was labeled with the "area: documentation (api and integrations)" label, so you may want to check it out!

<!-- areaLabelAddition -->
Very interesting! This seems likely to be the reason why openapi-generator produces (at least for Go) bindings [where most every method returns just a `JsonSuccess`](https://chat.zulip.org/#narrow/stream/127-integrations/topic/go-zulip-api/near/991574), instead of objects with the appropriate data.


Here's upstream's documentation of a pattern that resembles what we're attempting to express here:
https://swagger.io/docs/specification/data-models/inheritance-and-polymorphism/
```yaml
components:
  schemas:
    BasicErrorModel:
      type: object
      required:
        - message
        - code
      properties:
        message:
          type: string
        code:
          type: integer
          minimum: 100
          maximum: 600
    ExtendedErrorModel:
      allOf:     # Combines the BasicErrorModel and the inline model
        - $ref: '#/components/schemas/BasicErrorModel'
        - type: object
          required:
            - rootCause
          properties:
            rootCause:
              type: string
```

Happily this looks like it may not be very hard to adapt our incorrect schemas to. The differences that stand out at me are:
* `JsonResponse` shouldn't have `additionalProperties: false`. After all, the point is to have additional properties on it. Then `{"msg": "", "result": "success"}` would validate against `JsonSuccess.allOf[0] == JsonResponse`, as it should.
* Also `JsonResponse` should have its own `required: [result]`, though that isn't the critical point here.
* In `JsonSuccess`, the `required` and `properties` and `example` keys should all go with `type: object` on a single item of the `allOf` list, a sibling of the `$ref`. It doesn't make any sense for them to each be a separate item under `allOf` -- as if they were each their own schema.


@orientor do you have time to work on this restructuring?
@zulipbot claim
Removing additionalProperties will result in loss of gurantee that all properties have been documented and that there are no additional Properties which haven't been documented. On the other hand, we can save the object received from yamole into zulip.yaml file since that would be the most schematically correct spec(without any allOfs), but this will make the file really long. So which method should I proceed with?
We can use `additionalProperties: false`, but only in “leaf” schemas from which other schemas don’t “inherit”. For example, `JsonResponse` and `JsonSuccess` cannot have `additionalProperties: false`, but this can:

```diff
   /get_stream_id:
     get:
       […]
       responses:
         "200":
           description: Success.
           content:
             application/json:
               schema:
                 allOf:
                   - $ref: "#/components/schemas/JsonSuccess"
                   - properties:
+                      result: {}
+                      msg: {}
                       stream_id:
                         type: integer
                         description: |
                           The ID of the given stream.
-                  - example: {"msg": "", "result": "success", "stream_id": 15}
+                    additionalProperties: false
+                    example: {"msg": "", "result": "success", "stream_id": 15}
```
It looks like the upcoming [OpenAPI 3.1.0](https://github.com/OAI/OpenAPI-Specification/blob/3.1.0-rc0/versions/3.1.0.md) will, by way of incorporating [JSON Schema 2019-09](https://json-schema.org/draft/2019-09/release-notes.html), support an [`unevaluatedProperties`](https://json-schema.org/draft/2019-09/json-schema-core.html#rfc.section.9.3.2.4) keyword that sees through `allOf` and `$ref` to address this use case. It may be some time before our libraries support this, though.
The [json-schema-merge-allof](https://github.com/mokkabonna/json-schema-merge-allof) library claims to do `allOf` merging correctly. Comparing its output to `yamole`’s might help highlight where we’re relying on `yamole`’s incorrect behavior.
Hello @orientor, you have been unassigned from this issue because you have not updated this issue or any referenced pull requests for over 14 days.

You can reclaim this issue or claim any other issue by commenting `@zulipbot claim` on that issue.

Thanks for your contributions, and hope to see you again soon!
Hello @zulip/server-api members, this issue was labeled with the "area: documentation (api and integrations)" label, so you may want to check it out!

<!-- areaLabelAddition -->
Hmm, that's really slow.  @YagoGG can you investigate this?  Running it under `cProfile` might be helpful.
I've been doing some profiling and the results are what one could expect: the final `return deepcopy(obj)` is taking a huge proportion of parsing time.

I won't be able to take care of this in the short term, so if anyone feels like working on this feel free to do so. At least I can try to provide some feedback and help along the process.
So, here's a proposal: What if we just emit a `.json` file as a cache after processing this, and only read that `.json` file if the `.yaml` file isn't newer?  That would turn this into a 1-time cost that can happen as part of the production build process, rather than something that is slow for every process startup.  JSON parsing is way faster than YAML parsing.  

We'd still want to fix the yamole issue, but it'd move the perf problem somewhere a lot less important.
Retagging this as "documentation (api)" -- "documentation (developer)" is for folks working on Zulip core.

Resolving patch:
diff --git a/version.py b/version.py
--- a/version.py
+++ b/version.py
@@ -44,4 +44,4 @@
 #   historical commits sharing the same major version, in which case a
 #   minor version bump suffices.
 
-PROVISION_VERSION = '110.0'
+PROVISION_VERSION = '111.0'
diff --git a/zerver/openapi/openapi.py b/zerver/openapi/openapi.py
--- a/zerver/openapi/openapi.py
+++ b/zerver/openapi/openapi.py
@@ -30,6 +30,44 @@
     ("/settings/notifications", "patch"),
 }
 
+# Most of our code expects allOf to be preprocessed away because that is what
+# yamole did.  Its algorithm for doing so is not standards compliant, but we
+# replicate it here.
+def naively_merge(a: Dict[str, object], b: Dict[str, object]) -> Dict[str, object]:
+    ret: Dict[str, object] = a.copy()
+    for key, b_value in b.items():
+        if key == "example" or key not in ret:
+            ret[key] = b_value
+            continue
+        a_value = ret[key]
+        if isinstance(b_value, list):
+            assert isinstance(a_value, list)
+            ret[key] = a_value + b_value
+        elif isinstance(b_value, dict):
+            assert isinstance(a_value, dict)
+            ret[key] = naively_merge(a_value, b_value)
+    return ret
+
+def naively_merge_allOf(obj: object) -> object:
+    if isinstance(obj, dict):
+        return naively_merge_allOf_dict(obj)
+    elif isinstance(obj, list):
+        return list(map(naively_merge_allOf, obj))
+    else:
+        return obj
+
+def naively_merge_allOf_dict(obj: Dict[str, object]) -> Dict[str, object]:
+    if "allOf" in obj:
+        ret = obj.copy()
+        subschemas = ret.pop("allOf")
+        ret = naively_merge_allOf_dict(ret)
+        assert isinstance(subschemas, list)
+        for subschema in subschemas:
+            assert isinstance(subschema, dict)
+            ret = naively_merge(ret, naively_merge_allOf_dict(subschema))
+        return ret
+    return {key: naively_merge_allOf(value) for key, value in obj.items()}
+
 class OpenAPISpec():
     def __init__(self, openapi_path: str) -> None:
         self.openapi_path = openapi_path
@@ -39,29 +77,31 @@ def __init__(self, openapi_path: str) -> None:
         self._request_validator: Optional[RequestValidator] = None
 
     def check_reload(self) -> None:
-        # Because importing yamole (and in turn, yaml) takes
-        # significant time, and we only use python-yaml for our API
-        # docs, importing it lazily here is a significant optimization
-        # to `manage.py` startup.
+        # Because importing yaml takes significant time, and we only
+        # use python-yaml for our API docs, importing it lazily here
+        # is a significant optimization to `manage.py` startup.
         #
         # There is a bit of a race here...we may have two processes
         # accessing this module level object and both trying to
         # populate self.data at the same time.  Hopefully this will
         # only cause some extra processing at startup and not data
         # corruption.
-        from yamole import YamoleParser
 
-        mtime = os.path.getmtime(self.openapi_path)
-        # Using == rather than >= to cover the corner case of users placing an
-        # earlier version than the current one
-        if self.mtime == mtime:
-            return
+        import yaml
+        from jsonref import JsonRef
 
         with open(self.openapi_path) as f:
-            yamole_parser = YamoleParser(f)
-        self._openapi = yamole_parser.data
-        spec = create_spec(self._openapi)
+            mtime = os.fstat(f.fileno()).st_mtime
+            # Using == rather than >= to cover the corner case of users placing an
+            # earlier version than the current one
+            if self.mtime == mtime:
+                return
+
+            openapi = yaml.load(f, Loader=yaml.CSafeLoader)
+
+        spec = create_spec(openapi)
         self._request_validator = RequestValidator(spec)
+        self._openapi = naively_merge_allOf_dict(JsonRef.replace_refs(openapi))
         self.create_endpoints_dict()
         self.mtime = mtime
 


Test patch: