Repository: mirumee/ariadne

Issue:
GraphQLTransportWSHandler sends invalid errors message breaking `graphql-ws` clients on error (should be an array)
Lately I switched over to fully using the GraphQLTransportWSHandler for everything (including mutations and queries). However, I am receiving the following error on the client (`"graphql-ws": "^5.11.2",`).

```plain
index.ts:43 Error: "error" message expects the 'payload' property to be an array of GraphQL errors, but got {"message":".......","locations":[{"line":1,"column":36}],"extensions":{"exception":null}}
    at validateMessage (common.mjs:128:23)
    at parseMessage (common.mjs:168:12)
    at socket2.onmessage (client.mjs:202:37)
```

After checking `graphql-ws` source code and protocol specification  I think Apollo is not sending errors correctly:
https://github.com/enisdenjo/graphql-ws/blob/fbb763a662802a6a2584b0cbeb9cf1bde38158e0/src/utils.ts#L57-L66
https://github.com/enisdenjo/graphql-ws/blob/fbb763a662802a6a2584b0cbeb9cf1bde38158e0/PROTOCOL.md#error

The client expects an array of errors, but instead Apollo only sends a single error?
https://github.com/mirumee/ariadne/blob/3218cff61ce12dc649b1f425c26e23715c3241a1/ariadne/asgi/handlers/graphql_transport_ws.py#L318-L332

https://github.com/mirumee/ariadne/blob/3218cff61ce12dc649b1f425c26e23715c3241a1/ariadne/asgi/handlers/graphql_transport_ws.py#L368-L379
GraphQLTransportWSHandler sends invalid errors message breaking `graphql-ws` clients on error (should be an array)
Lately I switched over to fully using the GraphQLTransportWSHandler for everything (including mutations and queries). However, I am receiving the following error on the client (`"graphql-ws": "^5.11.2",`).

```plain
index.ts:43 Error: "error" message expects the 'payload' property to be an array of GraphQL errors, but got {"message":".......","locations":[{"line":1,"column":36}],"extensions":{"exception":null}}
    at validateMessage (common.mjs:128:23)
    at parseMessage (common.mjs:168:12)
    at socket2.onmessage (client.mjs:202:37)
```

After checking `graphql-ws` source code and protocol specification  I think Apollo is not sending errors correctly:
https://github.com/enisdenjo/graphql-ws/blob/fbb763a662802a6a2584b0cbeb9cf1bde38158e0/src/utils.ts#L57-L66
https://github.com/enisdenjo/graphql-ws/blob/fbb763a662802a6a2584b0cbeb9cf1bde38158e0/PROTOCOL.md#error

The client expects an array of errors, but instead Apollo only sends a single error?
https://github.com/mirumee/ariadne/blob/3218cff61ce12dc649b1f425c26e23715c3241a1/ariadne/asgi/handlers/graphql_transport_ws.py#L318-L332

https://github.com/mirumee/ariadne/blob/3218cff61ce12dc649b1f425c26e23715c3241a1/ariadne/asgi/handlers/graphql_transport_ws.py#L368-L379


Discussion:
I'll mark this as a bug and put this on my radar for next release.
I only got to test this just now, but @rafalp the fix is not completely according spec and will still throw errors:
```
GraphQL websocket error:  Error: "error" message expects the 'payload' property to be an array of GraphQL errors, but got [{"errors":[{"message":"Unknown argument 'id' on field 'Mutation.test'.","locations":[{"line":2,"column":20}],"extensions":{"exception":null}}]}]
```
```json
{
    "type": "error",
    "id": "c243e373-4048-4108-a8d5-d5a23ed8e796",
    "payload": [
        {
            "errors": [
                {
                    "message": "Unknown argument 'id' on field 'Mutation.test'.",
                    "locations": [
                        {
                            "line": 2,
                            "column": 20
                        }
                    ],
                    "extensions": {
                        "exception": null
                    }
                }
            ]
        }
    ]
}
```


The [spec](https://github.com/enisdenjo/graphql-ws/blob/fbb763a662802a6a2584b0cbeb9cf1bde38158e0/PROTOCOL.md#error) mentions that when `type == 'error'` the payload is directly a list of errors, while currently it is a list of objects that contains the "errors" key with a list of errors.

``` typescript
import { GraphQLError } from 'graphql';

interface ErrorMessage {
  id: '<unique-operation-id>';
  type: 'error';
  payload: GraphQLError[];
}
```
Reopening this
If I understand the spec right, if there's an error, instead of `NEXT` we should send `ERROR` message, because this protocol is not supporting partialy correct results?
It seems to me that there are three places in `GraphQLTransportWSHandler` where errors can be returned to the client, and neither of those can create the error response you are mentioning:

https://github.com/mirumee/ariadne/blob/dfffa9018088566069b4f56fbee11ef875f70525/ariadne/asgi/handlers/graphql_transport_ws.py#L325

`type` is `ERROR` and `payload` is list of error objects, not `{"errors": [...]}`. Looks okay.

https://github.com/mirumee/ariadne/blob/dfffa9018088566069b4f56fbee11ef875f70525/ariadne/asgi/handlers/graphql_transport_ws.py#L374

`type` is `ERROR` and `payload` is a list of result responses. Looks like this is the culprit. But I'll need further study.

Third place is where `NEXT` is returned with `payload.errors` instead of data:

https://github.com/mirumee/ariadne/blob/dfffa9018088566069b4f56fbee11ef875f70525/ariadne/asgi/handlers/graphql_transport_ws.py#L502
I can reproduce this. This error only occurs for `query` and `mutation` operations sent to GraphQL Transport WS.
I'll mark this as a bug and put this on my radar for next release.
I only got to test this just now, but @rafalp the fix is not completely according spec and will still throw errors:
```
GraphQL websocket error:  Error: "error" message expects the 'payload' property to be an array of GraphQL errors, but got [{"errors":[{"message":"Unknown argument 'id' on field 'Mutation.test'.","locations":[{"line":2,"column":20}],"extensions":{"exception":null}}]}]
```
```json
{
    "type": "error",
    "id": "c243e373-4048-4108-a8d5-d5a23ed8e796",
    "payload": [
        {
            "errors": [
                {
                    "message": "Unknown argument 'id' on field 'Mutation.test'.",
                    "locations": [
                        {
                            "line": 2,
                            "column": 20
                        }
                    ],
                    "extensions": {
                        "exception": null
                    }
                }
            ]
        }
    ]
}
```


The [spec](https://github.com/enisdenjo/graphql-ws/blob/fbb763a662802a6a2584b0cbeb9cf1bde38158e0/PROTOCOL.md#error) mentions that when `type == 'error'` the payload is directly a list of errors, while currently it is a list of objects that contains the "errors" key with a list of errors.

``` typescript
import { GraphQLError } from 'graphql';

interface ErrorMessage {
  id: '<unique-operation-id>';
  type: 'error';
  payload: GraphQLError[];
}
```
Reopening this
If I understand the spec right, if there's an error, instead of `NEXT` we should send `ERROR` message, because this protocol is not supporting partialy correct results?
It seems to me that there are three places in `GraphQLTransportWSHandler` where errors can be returned to the client, and neither of those can create the error response you are mentioning:

https://github.com/mirumee/ariadne/blob/dfffa9018088566069b4f56fbee11ef875f70525/ariadne/asgi/handlers/graphql_transport_ws.py#L325

`type` is `ERROR` and `payload` is list of error objects, not `{"errors": [...]}`. Looks okay.

https://github.com/mirumee/ariadne/blob/dfffa9018088566069b4f56fbee11ef875f70525/ariadne/asgi/handlers/graphql_transport_ws.py#L374

`type` is `ERROR` and `payload` is a list of result responses. Looks like this is the culprit. But I'll need further study.

Third place is where `NEXT` is returned with `payload.errors` instead of data:

https://github.com/mirumee/ariadne/blob/dfffa9018088566069b4f56fbee11ef875f70525/ariadne/asgi/handlers/graphql_transport_ws.py#L502
I can reproduce this. This error only occurs for `query` and `mutation` operations sent to GraphQL Transport WS.

Resolving patch:
diff --git a/ariadne/asgi/handlers/graphql_transport_ws.py b/ariadne/asgi/handlers/graphql_transport_ws.py
--- a/ariadne/asgi/handlers/graphql_transport_ws.py
+++ b/ariadne/asgi/handlers/graphql_transport_ws.py
@@ -367,15 +367,22 @@ async def get_results():
                 yield result
 
             # if success then AsyncGenerator is expected, for error it will be List
-            results_producer = get_results() if success else [result]
+            if success:
+                results_producer = get_results()
+            else:
+                results_producer = result["errors"]
 
         if not success:
-            results_producer = cast(List[dict], results_producer)
+            if not isinstance(results_producer, list):
+                error_payload = cast(List[dict], [results_producer])
+            else:
+                error_payload = results_producer
+
             await websocket.send_json(
                 {
                     "type": GraphQLTransportWSHandler.GQL_ERROR,
                     "id": operation_id,
-                    "payload": [results_producer[0]],
+                    "payload": error_payload,
                 }
             )
         else:


Test patch:
diff --git a/tests/asgi/__snapshots__/test_websockets_graphql_transport_ws.ambr b/tests/asgi/__snapshots__/test_websockets_graphql_transport_ws.ambr
new file mode 100644
--- /dev/null
+++ b/tests/asgi/__snapshots__/test_websockets_graphql_transport_ws.ambr
@@ -0,0 +1,23 @@
+# serializer version: 1
+# name: test_invalid_query_error_is_handled_using_websocket_connection_graphql_transport_ws
+  list([
+    dict({
+      'locations': list([
+        dict({
+          'column': 17,
+          'line': 1,
+        }),
+      ]),
+      'message': "Cannot query field 'error' on type 'Query'.",
+    }),
+    dict({
+      'locations': list([
+        dict({
+          'column': 23,
+          'line': 1,
+        }),
+      ]),
+      'message': "Cannot query field 'other' on type 'Query'.",
+    }),
+  ])
+# ---
diff --git a/tests/asgi/test_websockets_graphql_transport_ws.py b/tests/asgi/test_websockets_graphql_transport_ws.py
--- a/tests/asgi/test_websockets_graphql_transport_ws.py
+++ b/tests/asgi/test_websockets_graphql_transport_ws.py
@@ -153,6 +153,30 @@ def test_mutation_can_be_executed_using_websocket_connection_graphql_transport_w
         assert response["id"] == "test3"
 
 
+def test_invalid_query_error_is_handled_using_websocket_connection_graphql_transport_ws(
+    client_graphql_transport_ws, snapshot
+):
+    with client_graphql_transport_ws.websocket_connect(
+        "/", ["graphql-transport-ws"]
+    ) as ws:
+        ws.send_json({"type": GraphQLTransportWSHandler.GQL_CONNECTION_INIT})
+        response = ws.receive_json()
+        assert response["type"] == GraphQLTransportWSHandler.GQL_CONNECTION_ACK
+        ws.send_json(
+            {
+                "type": GraphQLTransportWSHandler.GQL_SUBSCRIBE,
+                "id": "test2",
+                "payload": {
+                    "operationName": "Invalid",
+                    "query": "query Invalid { error other }",
+                },
+            }
+        )
+        response = ws.receive_json()
+        assert response["type"] == GraphQLTransportWSHandler.GQL_ERROR
+        assert snapshot == response["payload"]
+
+
 def test_custom_query_parser_is_used_for_subscription_over_websocket_transport_ws(
     schema,
 ):