Repository: edgedb/edgedb-python (language: python)

Issue:
PydanticV2 breaks EdgeDB's Python CodeGen 
# Summary

EdgeDB's codegen is broken, because the codegen relies on Pydantic, but hasn't been updated to work with PydanticV2. 

This matters because it's one of the first things a new user using Python will encounter (as the bug rears its head when you try to set up the [EdgeDB-FastAPI tutorial](https://www.edgedb.com/docs/guides/tutorials/rest_apis_with_fastapi) if you don't use an old version of PyDantic)

# Detailed problems

A `get_user` query will generate codegen that looks something like this:

```
class NoPydanticValidation:
    @classmethod
    def __get_validators__(cls):
        from pydantic.dataclasses import dataclass as pydantic_dataclass
        pydantic_dataclass(cls)
        cls.__pydantic_model__.__get_validators__ = lambda: []
        return []


@dataclasses.dataclass
class GetUserByNameResult(NoPydanticValidation):
    id: uuid.UUID
    username: str


async def get_user_by_name(
    executor: edgedb.AsyncIOExecutor,
    *,
    name: str,
) -> list[GetUserByNameResult]:
    return await executor.query(
        """\
        select User {
          username
        } filter User.username = <str>$name\
        """,
        name=name,
    )
```

There are two issues, at the surface manifestation and an underlying issue:

**Surface**
`cls.__pydantic_model__.__get_validators__` should be substituted with `cls.__get_validators__` because the `__pydantic_model__` is no longer loaded as a method during `pydantic_dataclass(cls)`
 
Unfortunately, fixing that by deleting `__pydantic_model` causes a different error, namely `pydantic_core._pydantic_core.SchemaError: Error building "chain" validator:` (guessing this is because of the empty list in the validator).
 
**Underlying**
The `query` function returns an [EdgeDB Object](https://github.com/edgedb/edgedb-python/blob/master/edgedb/datatypes/object.c)... but it's validation is skipped, and it's being duck-typed as a dataclass.

This isn't easy to follow (especially the NoValidation stuff), and it causes some strange errors when Pydantic eventually tries to validate it in the FastAPI response layer.


# Potential fix
In my opinion, the cleanest way to solve this would simply be to initialize a dataclass object manually, and then use the codegen to fill in the fields. I illustrate that below. However, that would yield a key design choice on the type you use for the result objects.

```
# Option 1.
# Unfortunately, https://github.com/pydantic/pydantic/issues/7111
@dataclasses.dataclass
class GetUserByNameResult():
    id: uuid.UUID
    username: str

# Option 2.
# Unfortunately, this isn't plug and play.
@pydantic.dataclasses.dataclass
class GetUserByNameResult():
    id: uuid.UUID
    username: str

# Option 3.
class GetUserByNameResult(pydantic.BaseModel):
    id: uuid.UUID
    username: str
```

Option 1 is the best abstractly, since you don't need a pydantic dependency. However it suffers from this [pydantic bug](https://github.com/pydantic/pydantic/issues/7111), which occurs if you use it with FastAPI.

Option 2 is nice, because it circumvents the above bug (see the bug notes on this), while keeping most of the `dataclass` interface.

Option 3 is nice too -- and most importantly, it lets you use the `model_construct` method to skip validation (if you want that).

My vote would be for **option 2**, because it works well:
* The Pydantic dependency isn't a big deal -- you already have it in the current version.
* It avoids the aforementioned bug, and is linked
* It's performant to create
* If you want to change it to a regular dataclass at some point, it's a drop in replace.

Notably, I think with all of these options, **you can get rid of the `NoPydanticValidation` hack**... since the object would have been validated during these steps (which presumably shouldn't fail, if your codegen is correct).

Also note: PydanticV2 [claims to make this verification quite cheap](https://docs.pydantic.dev/latest/concepts/models/#creating-models-without-validation), but you can use `BaseModel.model_construct()` if you want to avoid that.

------------------
 # Detailed journey:

While I created the webapp described in the [python tutorial](https://www.edgedb.com/docs/guides/tutorials/rest_apis_with_fastapi), I found the CLI is erroring out. The tutorial claimed that this was due to compatibility issues with Pydantic V2 (which is partially true -- see https://github.com/pydantic/pydantic/issues/7111), but there is ALSO a problem with the edge-db codegen itself.

If you look at the edgeql query:

```
select User {
  username
} filter User.username = <str>$name
```

It produces python like:

```python
# AUTOGENERATED FROM 'app/queries/get_user_by_name.edgeql' WITH:
#     $ edgedb-py


from __future__ import annotations
import dataclasses
import edgedb
import uuid


class NoPydanticValidation:
    @classmethod
    def __get_validators__(cls):
        from pydantic.dataclasses import dataclass as pydantic_dataclass
        pydantic_dataclass(cls)
        cls.__pydantic_model__.__get_validators__ = lambda: []
        return []


@dataclasses.dataclass
class GetUserByNameResult(NoPydanticValidation):
    id: uuid.UUID
    username: str


async def get_user_by_name(
    executor: edgedb.AsyncIOExecutor,
    *,
    name: str,
) -> list[GetUserByNameResult]:
    return await executor.query(
        """\
        select User {
          username
        } filter User.username = <str>$name\
        """,
        name=name,
    )
```

A key problem is that `cls.__pydantic_model__` doesn't exist in PydanticV2... it should be replaced with `cls.__get_validators__`.

However, fixing that still causes errors if you ever try to check with Pydantic (when you try and run it through FastAPI), namely:

```
pydantic_core._pydantic_core.SchemaError: Error building "chain" validator:
  SchemaError: One or more steps are required for a chain validator
```

In my opinion, the cleanest solution would look like this. Of course, you'd tweak it to work well and abstractly with codegen (or maybe use some `Dataclass(**edgedb_object)` shorthan, etc).

The key idea is that you'd be using the code-gen to map in the individual fields of a NEW dataclass object.
```
#  AUTOGENERATED FROM 'queries/get_user_by_name.edgeql' WITH:
#    $ edgedb-py --dir queries

from __future__ import annotations
import dataclasses
import edgedb
import uuid
import pydantic

class GetUserByNameResult(pydantic.BaseModel):
    id: uuid.UUID
    username: str

async def get_user_by_name(
    executor: edgedb.AsyncIOExecutor,
    *,
    name: str,
) -> list[GetUserByNameResult]:
    r = await executor.query(
        """\
        select User {
          username
        } filter User.username = <str>$name\
        """,
        name=name,
    )
    users = [
        GetUserByNameResult(
            id=user_raw.id,
            username=user_raw.username
        )
        for user_raw in r
    ]
    return users

import asyncio
async def main() -> None:
    client = edgedb.create_async_client()
    r = await get_user_by_name(client, name="jsimonlane")
    print('ran')
    print(r)

if __name__ == "__main__":
    asyncio.run(main())
```

That should fix everything.


**Versions (please complete the following information):**
Pydantic==2.4.2
Edgedb==1.7.0

- OS: MacOS 13.3.1 (Ventura)
- `edgedb-python` version: 1.7.0
- EdgeDB CLI: EdgeDB CLI 3.5.0+e1ad387
- EdgeDB: EdgeDB 3.3+d5734dd (repl 3.5.0+e1ad387)
- Python version:3.11

**Additional context**
Add any other context about the problem here.



Discussion / hints:
Any updates?
In this conversation in the [EdgeDB discord](https://discord.com/channels/841451783728529451/1162226344025931857/1166851165548449915), @1st1 said:

> @Fantix and I will see if we can fix this on our end (or maybe we can help fix pydantic) after EdgeDB Day on November 1st

So I imagine this is in the works.

Fix patch:
diff --git a/edgedb/codegen/generator.py b/edgedb/codegen/generator.py
index 6a02352e..9b657d3a 100644
--- a/edgedb/codegen/generator.py
+++ b/edgedb/codegen/generator.py
@@ -97,8 +97,15 @@
 
 PYDANTIC_MIXIN = """\
 class NoPydanticValidation:
+    @classmethod
+    def __get_pydantic_core_schema__(cls, _source_type, _handler):
+        # Pydantic 2.x
+        from pydantic_core.core_schema import any_schema
+        return any_schema()
+
     @classmethod
     def __get_validators__(cls):
+        # Pydantic 1.x
         from pydantic.dataclasses import dataclass as pydantic_dataclass
         pydantic_dataclass(cls)
         cls.__pydantic_model__.__get_validators__ = lambda: []


Test patch:
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index 14f735c0..5dfe6041 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -88,9 +88,6 @@ jobs:
         LOOP_IMPL: ${{ matrix.loop }}
         SERVER_VERSION: ${{ matrix.edgedb-version }}
       run: |
-        if [ "${SERVER_VERSION}" = "nightly" ]; then
-          export EDGEDB_TEST_CODEGEN_ASSERT_SUFFIX=.assert4
-        fi
         if [ "${LOOP_IMPL}" = "uvloop" ]; then
             env USE_UVLOOP=1 python -m unittest -v tests.suite
         else
diff --git a/tests/codegen/test-project1/select_optional_json_async_edgeql.py.assert b/tests/codegen/test-project1/select_optional_json_async_edgeql.py.assert
index 8588dd54..cdfcd972 100644
--- a/tests/codegen/test-project1/select_optional_json_async_edgeql.py.assert
+++ b/tests/codegen/test-project1/select_optional_json_async_edgeql.py.assert
@@ -10,8 +10,15 @@ import uuid
 
 
 class NoPydanticValidation:
+    @classmethod
+    def __get_pydantic_core_schema__(cls, _source_type, _handler):
+        # Pydantic 2.x
+        from pydantic_core.core_schema import any_schema
+        return any_schema()
+
     @classmethod
     def __get_validators__(cls):
+        # Pydantic 1.x
         from pydantic.dataclasses import dataclass as pydantic_dataclass
         pydantic_dataclass(cls)
         cls.__pydantic_model__.__get_validators__ = lambda: []
diff --git a/tests/codegen/test-project2/object/link_prop_async_edgeql.py.assert b/tests/codegen/test-project2/object/link_prop_async_edgeql.py.assert
index 2185f9c7..7bff2c40 100644
--- a/tests/codegen/test-project2/object/link_prop_async_edgeql.py.assert
+++ b/tests/codegen/test-project2/object/link_prop_async_edgeql.py.assert
@@ -11,8 +11,15 @@ import uuid
 
 
 class NoPydanticValidation:
+    @classmethod
+    def __get_pydantic_core_schema__(cls, _source_type, _handler):
+        # Pydantic 2.x
+        from pydantic_core.core_schema import any_schema
+        return any_schema()
+
     @classmethod
     def __get_validators__(cls):
+        # Pydantic 1.x
         from pydantic.dataclasses import dataclass as pydantic_dataclass
         pydantic_dataclass(cls)
         cls.__pydantic_model__.__get_validators__ = lambda: []
diff --git a/tests/codegen/test-project2/object/select_object_async_edgeql.py.assert b/tests/codegen/test-project2/object/select_object_async_edgeql.py.assert
index 70c1bae8..56d040a2 100644
--- a/tests/codegen/test-project2/object/select_object_async_edgeql.py.assert
+++ b/tests/codegen/test-project2/object/select_object_async_edgeql.py.assert
@@ -10,8 +10,15 @@ import uuid
 
 
 class NoPydanticValidation:
+    @classmethod
+    def __get_pydantic_core_schema__(cls, _source_type, _handler):
+        # Pydantic 2.x
+        from pydantic_core.core_schema import any_schema
+        return any_schema()
+
     @classmethod
     def __get_validators__(cls):
+        # Pydantic 1.x
         from pydantic.dataclasses import dataclass as pydantic_dataclass
         pydantic_dataclass(cls)
         cls.__pydantic_model__.__get_validators__ = lambda: []
diff --git a/tests/codegen/test-project2/object/select_objects_async_edgeql.py.assert b/tests/codegen/test-project2/object/select_objects_async_edgeql.py.assert
index b293172a..d193082b 100644
--- a/tests/codegen/test-project2/object/select_objects_async_edgeql.py.assert
+++ b/tests/codegen/test-project2/object/select_objects_async_edgeql.py.assert
@@ -10,8 +10,15 @@ import uuid
 
 
 class NoPydanticValidation:
+    @classmethod
+    def __get_pydantic_core_schema__(cls, _source_type, _handler):
+        # Pydantic 2.x
+        from pydantic_core.core_schema import any_schema
+        return any_schema()
+
     @classmethod
     def __get_validators__(cls):
+        # Pydantic 1.x
         from pydantic.dataclasses import dataclass as pydantic_dataclass
         pydantic_dataclass(cls)
         cls.__pydantic_model__.__get_validators__ = lambda: []
diff --git a/tests/codegen/test-project2/parpkg/select_args_async_edgeql.py.assert b/tests/codegen/test-project2/parpkg/select_args_async_edgeql.py.assert
index 7485f3aa..10327852 100644
--- a/tests/codegen/test-project2/parpkg/select_args_async_edgeql.py.assert
+++ b/tests/codegen/test-project2/parpkg/select_args_async_edgeql.py.assert
@@ -10,8 +10,15 @@ import uuid
 
 
 class NoPydanticValidation:
+    @classmethod
+    def __get_pydantic_core_schema__(cls, _source_type, _handler):
+        # Pydantic 2.x
+        from pydantic_core.core_schema import any_schema
+        return any_schema()
+
     @classmethod
     def __get_validators__(cls):
+        # Pydantic 1.x
         from pydantic.dataclasses import dataclass as pydantic_dataclass
         pydantic_dataclass(cls)
         cls.__pydantic_model__.__get_validators__ = lambda: []
diff --git a/tests/codegen/test-project2/parpkg/subpkg/my_query_async_edgeql.py.assert b/tests/codegen/test-project2/parpkg/subpkg/my_query_async_edgeql.py.assert
index cd7d6bb2..d9aa5a01 100644
--- a/tests/codegen/test-project2/parpkg/subpkg/my_query_async_edgeql.py.assert
+++ b/tests/codegen/test-project2/parpkg/subpkg/my_query_async_edgeql.py.assert
@@ -16,8 +16,15 @@ MyScalar = int
 
 
 class NoPydanticValidation:
+    @classmethod
+    def __get_pydantic_core_schema__(cls, _source_type, _handler):
+        # Pydantic 2.x
+        from pydantic_core.core_schema import any_schema
+        return any_schema()
+
     @classmethod
     def __get_validators__(cls):
+        # Pydantic 1.x
         from pydantic.dataclasses import dataclass as pydantic_dataclass
         pydantic_dataclass(cls)
         cls.__pydantic_model__.__get_validators__ = lambda: []
diff --git a/tests/test_codegen.py b/tests/test_codegen.py
index 801b00ae..6b041dbd 100644
--- a/tests/test_codegen.py
+++ b/tests/test_codegen.py
@@ -27,7 +27,8 @@
 from edgedb import _testbase as tb
 
 
-ASSERT_SUFFIX = os.environ.get("EDGEDB_TEST_CODEGEN_ASSERT_SUFFIX", ".assert")
+# Use ".assert" for EdgeDB 3.x and lower
+ASSERT_SUFFIX = os.environ.get("EDGEDB_TEST_CODEGEN_ASSERT_SUFFIX", ".assert4")
 
 
 class TestCodegen(tb.AsyncQueryTestCase):