Repository: litestar-org/litestar

Issue:
Enhancement: Generate OpenAPI schema always in the same order
### Summary

Update: Turned out the seed is already static, but that elements might be generated in random order. Let's fix that.

---

The `generate_examples` is nice but the examples are re-generated for each application execution.

This makes it impossible to store the generated schema in git, for change tracking, as there will be always a diff.

Example generation utilizes `polyfactory` and `Faker`, which both allow setting the seed, I assume. Tried googling it up, tried something like `Faker.seed(123)` but no luck.

This should be made tuneable, and maybe default to a fixed seed, too.

Related:

- The order of `components.schemas` entries seems to also change over executions. Could this also retain the ordering?

### Basic Example

See https://github.com/litestar-org/litestar/issues/3058 for example.

### Drawbacks and Impact

_No response_

### Unresolved questions

_No response_

<!-- POLAR PLEDGE BADGE START -->
---
> [!NOTE]  
> While we are open for sponsoring on [GitHub Sponsors](https://github.com/sponsors/litestar-org/) and 
> [OpenCollective](https://opencollective.com/litestar), we also utilize [Polar.sh](https://polar.sh/) to engage in pledge-based sponsorship.
>
> Check out all issues funded or available for funding [on our Polar.sh dashboard](https://polar.sh/litestar-org)
> * If you would like to see an issue prioritized, make a pledge towards it!
> * We receive the pledge once the issue is completed & verified
> * This, along with engagement in the community, helps us know which features are a priority to our users.

<a href="https://polar.sh/litestar-org/litestar/issues/3059">
<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://polar.sh/api/github/litestar-org/litestar/issues/3059/pledge.svg?darkmode=1">
  <img alt="Fund with Polar" src="https://polar.sh/api/github/litestar-org/litestar/issues/3059/pledge.svg">
</picture>
</a>
<!-- POLAR PLEDGE BADGE END -->



Discussion:
@guacs 
This seeding is already being done though it is not exposed as a configurable value as seen [here](https://github.com/litestar-org/litestar/blob/f8d03194b8f541a4459c27fe51599526a73ee2ba/litestar/_openapi/schema_generation/examples.py#L27). I think it's a good idea to make this configurable as well.

If the examples being generated are still different, then it may be that the schemas are not being created in the same order each time which seems to be the case as @tuukkamustonen pointed out regarding the `components.schemas`. So, ensuring the schemas are generated in the same order for a given Litestar app should fix this as well.
@guacs Were you planning to work on this?

I started wondering _how_ the order of the elements could be random as since Python 3.7+ dicts retain insertion order and generating the schema should be deterministic?

But yeah, it does seem to generate some elements (in my long spec) in a bit different order each time.
@tuukkamustonen if you'd like to work on this, then feel free to do so.

I haven't looked into it yet, so I'm not sure just where the randomness is coming from.
I'm not sure whether this is what this issue is about, since it seems to be related to deterministic ordering (rather than defined ordering through the `Controller`), but it might be related, so I'll avoid opening a new one.

From a DX perspective I personally find very helpful to have ordered routes in the Controller, that follow a certain logic:
- Progressive nesting (`GET /` comes before `GET /:id` and before `GET /:id/nested`)
- Logical actions ordering (`GET`, `POST`, `GET id`, `PATCH id`, `DELETE id`)

Example:

```python
class MyController(Controller):
    tags = ["..."]
    path = "/"
    dependencies = {...}

    @routes.get()
    async def get_many(self):
        ...

    @routes.post()
    async def create(self, data: ...):
        ...

    @routes.get("/{resource_id:uuid}")
    async def get(self, resource_id: UUID):
        ...

    @routes.patch("/{resource_id:uuid}")
    async def update(self, resource_id: UUID):
        ...

    @routes.delete("/{resource_id:uuid}")
    async def delete(self, resource_id: UUID):
        ...

    @routes.get("/{resource_id:uuid}/nested")
    async def get_nested(self, resource_id: UUID):
        ...
```


Currently the ordering of the route definition at the Controller is not respected by the docs, so I end up having a Swagger that looks like this:

```
- GET /:id/nested/:nest_id/another
- POST /:id/nested
- GET /
- DELETE /:id
- POST /
```

Which I personally find very confusing since:
(1) It doesn't seem to follow a pre-defined logic (couldn't find any pattern for that when looking at the docs)
(2) It doesn't respect the logic that was defined on the controller

Not sure what might be causing it, I think this is generated on `_openapi/schema_generation/schema.py:SchemaCreator` maybe? Will be happy to help with some guidance, since it seems this has been refactored not long ago here: https://github.com/litestar-org/litestar/pull/2805.
@ccrvlh I'd say that's a different feature to this one so it'd be great if you could create an issue for that. I think it does follow a certain order which is GET, PUT, POST, DELETE etc. (I'm guessing the same order as seen [here](https://github.com/litestar-org/litestar/blob/3e5c179e714bb074bae13e02d10e2f3f51e24d5c/litestar/openapi/spec/path_item.py#L44)). However, this considers `/:id/nested` to have no relation to `/:id` or any of the other routes. The logic of this starts [here](https://github.com/litestar-org/litestar/blob/3e5c179e714bb074bae13e02d10e2f3f51e24d5c/litestar/_openapi/plugin.py#L37).

The difficulty here is that by the time the route reaches to OpenAPI logic, we don't know whether this was defined as part of a controller or not. So, if that order is to be maintained then we'd need to figure out a way to tell let the OpenAPIPlugin figure out the order the paths should be in. This would have to be done at the time we parse the Controller to create the individual route handlers. Even then a single path would consist of all the GET, PUT, POST etc. in that path but it should be possible to define that they should come in the following order from your example: `/`, `/:resource_id`, `/:resource_id/nested`.
@guacs thank you, opening a new issue [here](https://github.com/litestar-org/litestar/issues/3130#issue-2150259041)
@tuukkamustonen I'm not able to reproduce the issue actually. I tried with the following and the schema generated is always the same:

```python
from __future__ import annotations

from dataclasses import dataclass

from rich import print

from litestar import get, post
from litestar.openapi.config import OpenAPIConfig
from litestar.testing import create_test_client


@dataclass
class Foo:
    foo: int


@dataclass
class Bar:
    bar: str


@get("/foo")
async def get_foo() -> Foo:
    return Foo(9)


@post("/foo")
async def post_foo(foo: Foo) -> None:
    ...


@get("/bar")
async def get_bar() -> Bar:
    return Bar("bar")


with create_test_client([get_foo, get_bar, post_foo], openapi_config=OpenAPIConfig("foo", "voo", True)) as client:
    resp = client.get("/schema/openapi.json")

    print(resp.json())
```
@guacs Thanks for looking into it. Maybe it's something more exotic, or maybe I'm just wrong. I'll investigate and ping back (a bit later).
Okay, I'm able to reproduce it in _a really weird_ setup.

The random ordering occurs in `components` section of OpenAPI spec. Elements elsewhere (e.g. `paths`) seem to remain in same order.

It's related to use of `Controllers`, where the order of HTTP methods gets randomized by a use of `set()` in https://github.com/litestar-org/litestar/blob/main/litestar/utils/sequence.py#L22

Sets are not ordered, so that `unique` should return the handlers in whatever order, which leads to model registration in whatever order.

To reproduce, I had to separate the handlers _into a separate file_.

```py
# handlers.py
from dataclasses import dataclass

from litestar import get, post, Controller
from pydantic import BaseModel


@dataclass
class Payload:
    foo: int


@dataclass
class Response(BaseModel):
    bar: str


class FooController(Controller):
    @post()
    async def post(self, data: Payload) -> None: ...

    @get()
    async def get(self) -> Response: ...


# app.py
import json

from litestar import Litestar

from handlers import FooController

app = Litestar(route_handlers=[FooController])


print(json.dumps(app.openapi_schema.to_schema()["components"], indent=4))
```

(If I dump it all into a single file, the issue will not reproduce.)

When I run that once, the first time, it outputs the schema components in `Response`, `Payload` order. Run it again, and the order is reverses. Run it again, and the order remains same as second time.

If you flush the `.pyc` cache (in `__pycache__`) and re-run it, the order is `Response`, `Payload` again. And then following invocations, with the cache have it reversed again.

I don't understand how exactly that bytecode caching impacts this, but it does, at least on my computer 🤯 Maybe the `.pyc` file is ordered somehow, and re-loading it up from there somehow leads to the `set()` always giving same ordering _in this case_. As only when running without cache the order is reversed. I don't know.

In any case, I believe that `set` messes it up. And there are other `set`s around, which might have an impact, too.

Does that sound plausible?
@tuukkamustonen thanks! I was able to reproduce it and I believe you're correct in that the use of `unique` is what's causing the difference in the order of the components and that this only happens when you use a `Controller` with multiple HTTP methods.

The use of `unique` to find the unique HTTP handlers results in `route.route_handlers` possibly having a different order of handlers each time which results in a different insertion order to `route.route_handler_map`. For creating the paths portion of the OpenaAPI schema, we iterate over the `route_handler_map` which in turn causes a difference in the order of insertion of the generated schema into the `SchemaRegistry`.

I was thinking that we force the schemas to be generated in alphabetical order so that we should be able to get the same OpenAPI schema every time regardless of the order the paths (and their individual HTTP methods) are processed. WDYT?
The OpenAPI JSON spec is supposed for machines and not humans, so whatever order is fine as long as it's consistent 🤔 

Alphabetical ordering would make sense, as it would keep the spec unchanged when you refactor and move controllers/handlers around... on the other hand, if you instead refactor by renaming models in-place, then the spec would change 😄

I guess I would just go with what is simplest and fastest, though I don't see exact issues with alphabetical ordering, either 🤷🏻‍♂️

Resolving patch:
diff --git a/litestar/_openapi/datastructures.py b/litestar/_openapi/datastructures.py
--- a/litestar/_openapi/datastructures.py
+++ b/litestar/_openapi/datastructures.py
@@ -150,7 +150,8 @@ def generate_components_schemas(self) -> dict[str, Schema]:
                 self.set_reference_paths(name_, registered_schema)
                 components_schemas[name_] = registered_schema.schema
 
-        return components_schemas
+        # Sort them by name to ensure they're always generated in the same order.
+        return {name: components_schemas[name] for name in sorted(components_schemas.keys())}
 
 
 class OpenAPIContext:


Test patch:
diff --git a/tests/unit/test_openapi/test_integration.py b/tests/unit/test_openapi/test_integration.py
--- a/tests/unit/test_openapi/test_integration.py
+++ b/tests/unit/test_openapi/test_integration.py
@@ -9,7 +9,7 @@
 import yaml
 from typing_extensions import Annotated
 
-from litestar import Controller, Litestar, get, post
+from litestar import Controller, Litestar, delete, get, patch, post
 from litestar._openapi.plugin import OpenAPIPlugin
 from litestar.app import DEFAULT_OPENAPI_CONFIG
 from litestar.enums import MediaType, OpenAPIMediaType, ParamType
@@ -399,3 +399,47 @@ def get_handler(q: str) -> None:
         assert openapi_one == openapi_two
     else:
         assert openapi_one != openapi_two
+
+
+def test_components_schemas_in_alphabetical_order() -> None:
+    # https://github.com/litestar-org/litestar/issues/3059
+
+    @dataclass
+    class A:
+        ...
+
+    @dataclass
+    class B:
+        ...
+
+    @dataclass
+    class C:
+        ...
+
+    class TestController(Controller):
+        @post("/", sync_to_thread=False)
+        def post_handler(self, data: B) -> None:
+            ...
+
+        @get("/", sync_to_thread=False)
+        def get_handler(self) -> A:  # type: ignore[empty-body]
+            ...
+
+        @patch("/", sync_to_thread=False)
+        def patch_handler(self, data: C) -> A:  # type: ignore[empty-body]
+            ...
+
+        @delete("/", sync_to_thread=False)
+        def delete_handler(self, data: B) -> None:
+            ...
+
+    app = Litestar([TestController], signature_types=[A, B, C])
+    openapi_plugin = app.plugins.get(OpenAPIPlugin)
+    openapi = openapi_plugin.provide_openapi()
+
+    expected_keys = [
+        "test_components_schemas_in_alphabetical_order.A",
+        "test_components_schemas_in_alphabetical_order.B",
+        "test_components_schemas_in_alphabetical_order.C",
+    ]
+    assert list(openapi.components.schemas.keys()) == expected_keys