Repository: deepset-ai/haystack

Issue:
REST API fails when embeddings returned flag is enabled in document store
Good morning Haystack devs. Not sure if this is really a bug or I'm doing something wrong, so feel free to change the type if this issue to something else if it's not a bug.

**Describe the bug**
When the document store is forced to return the embeddings the REST API request fails because the type of the embedding is a Numpy array (`dtype=float32`) but the Pydantic dataclass of `DocumentSerialized` expects a `List[float]` and I think there is no automatic conversion between both types.

**Error message**
```
[2021-11-26 08:48:59 +0000] [10] [ERROR] Exception in ASGI application
haystack-api_1  | Traceback (most recent call last):
haystack-api_1  |   File "/usr/local/lib/python3.7/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 375, in run_asgi
haystack-api_1  |     result = await app(self.scope, self.receive, self.send)
haystack-api_1  |   File "/usr/local/lib/python3.7/dist-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
haystack-api_1  |     return await self.app(scope, receive, send)
haystack-api_1  |   File "/usr/local/lib/python3.7/dist-packages/fastapi/applications.py", line 208, in __call__
haystack-api_1  |     await super().__call__(scope, receive, send)
haystack-api_1  |   File "/usr/local/lib/python3.7/dist-packages/starlette/applications.py", line 112, in __call__
haystack-api_1  |     await self.middleware_stack(scope, receive, send)
haystack-api_1  |   File "/usr/local/lib/python3.7/dist-packages/starlette/middleware/errors.py", line 181, in __call__
haystack-api_1  |     raise exc
haystack-api_1  |   File "/usr/local/lib/python3.7/dist-packages/starlette/middleware/errors.py", line 159, in __call__
haystack-api_1  |     await self.app(scope, receive, _send)
haystack-api_1  |   File "/usr/local/lib/python3.7/dist-packages/starlette/middleware/cors.py", line 84, in __call__
haystack-api_1  |     await self.app(scope, receive, send)
haystack-api_1  |   File "/usr/local/lib/python3.7/dist-packages/starlette/exceptions.py", line 82, in __call__
haystack-api_1  |     raise exc
haystack-api_1  |   File "/usr/local/lib/python3.7/dist-packages/starlette/exceptions.py", line 71, in __call__
haystack-api_1  |     await self.app(scope, receive, sender)
haystack-api_1  |   File "/usr/local/lib/python3.7/dist-packages/starlette/routing.py", line 656, in __call__
haystack-api_1  |     await route.handle(scope, receive, send)
haystack-api_1  |   File "/usr/local/lib/python3.7/dist-packages/starlette/routing.py", line 259, in handle
haystack-api_1  |     await self.app(scope, receive, send)
haystack-api_1  |   File "/usr/local/lib/python3.7/dist-packages/starlette/routing.py", line 61, in app
haystack-api_1  |     response = await func(request)
haystack-api_1  |   File "/usr/local/lib/python3.7/dist-packages/fastapi/routing.py", line 243, in app
haystack-api_1  |     is_coroutine=is_coroutine,
haystack-api_1  |   File "/usr/local/lib/python3.7/dist-packages/fastapi/routing.py", line 137, in serialize_response
haystack-api_1  |     raise ValidationError(errors, field.type_)
haystack-api_1  | pydantic.error_wrappers.ValidationError: 5 validation errors for QueryResponse
haystack-api_1  | response -> documents -> 0 -> embedding
haystack-api_1  |   value is not a valid list (type=type_error.list)
haystack-api_1  | response -> documents -> 1 -> embedding
haystack-api_1  |   value is not a valid list (type=type_error.list)
haystack-api_1  | response -> documents -> 2 -> embedding
haystack-api_1  |   value is not a valid list (type=type_error.list)
haystack-api_1  | response -> documents -> 3 -> embedding
haystack-api_1  |   value is not a valid list (type=type_error.list)
haystack-api_1  | response -> documents -> 4 -> embedding
haystack-api_1  |   value is not a valid list (type=type_error.list)
```


**To Reproduce**
Just start the REST API with `return_embedding: True` set in the document store

**System:**
 - Haystack version (commit or version number): 1.0RC1 (latest version from Github)
 - DocumentStore: Milvus
 - Reader: Farm
 - Retriever: DPR



Discussion:
Hi @LorenzBuehmann you're right that the embeddings are calculated as a numpy array
https://github.com/deepset-ai/haystack/blob/56e4e8486f346ba899dcf8231857c8e20bed9643/haystack/document_stores/elasticsearch.py#L937
whereas the `DocumentSerialized` defines the embedding as a List of float
https://github.com/deepset-ai/haystack/blob/c0892717a061eef0d9a0a185b0ec1b53f0b467d3/rest_api/schema.py#L34
I will have a closer look.
Hi @LorenzBuehmann I was able to reproduce your error.
I am working on a fix and tried adding the following two lines
`from numpy.typing import NDArray` 
`from numpy import float32 as np_float32` 
in `haystack/rest_api/schema.py` and change the line with the type of `embedding` in `DocumentSerialized` to:
`embedding: Optional[NDArray[np_float32]]`
However, that change doesn't fix the problem yet.
Hi @julian-risch ,

yes, forgot to say that I also tried this as a quick fix - but then the JSON encoder/decoder (of I think Pydantic) got into trouble. Looks like a custom 

Just to clarify why I enabled the embeddings, I added another pipeline but with generative QA, and that one takes the existing embeddings of the retrieved documents into account. Technically it should not be necessary to return those by the document store as a retriever is defined and could be user, but for whatever reasoner the generator was not aware of the retriever. Is this not possible via pipeline?

This is the pipeline definition, in particular the one named "generate" fails when the `return_embedding` is not set to `True` although a retriever is declared:

``` yaml
version: '0.9'

components:
  - name: DocumentStore
    type: MilvusDocumentStore
    params:
      milvus_url: 
      sql_url: 
      index: 
  - name: Retriever
    type: DensePassageRetriever
    params:
      document_store: DocumentStore 
      top_k: 5
  - name: Reader  
    type: FARMReader
    params:
      model_name_or_path: deepset/roberta-base-squad2
  - name: Generator
    type: RAGenerator
    params:
      model_name_or_path: facebook/rag-sequence-nq
      top_k: 1
pipelines:
  - name: query    # extractive QA Pipeline
    type: Query
    nodes:
      - name: Retriever
        inputs: [Query]
      - name: Reader
        inputs: [Retriever]
  - name: generate    # generative QA Pipeline
    type: Query
    nodes:
      - name: Retriever
        inputs: [Query]
      - name: Generator
        inputs: [Retriever]
```
it fails at https://github.com/deepset-ai/haystack/blob/master/haystack/nodes/answer_generator/transformers.py#L178-L194 with

```
AttributeError("_prepare_passage_embeddings need a DPR instance as self.retriever to embed document")
``` 
@julian-risch  Numpy arrays are not supported in json and therefore also not in the API responses. 
Actually the `NumpyEncoder` should take care of this conversion from ndarray to list[float]: https://github.com/deepset-ai/haystack/blob/c5540d05ed7b62f08b0be9a9ea20ad4d46e21086/haystack/schema.py#L172

Probably it's not used correctly when generating the FastAPI response :thinking: 
Let's address this issue in the next sprint.

Resolving patch:
diff --git a/rest_api/controller/search.py b/rest_api/controller/search.py
--- a/rest_api/controller/search.py
+++ b/rest_api/controller/search.py
@@ -2,6 +2,7 @@
 import time
 import json
 from pathlib import Path
+from numpy import ndarray
 
 from fastapi import APIRouter
 
@@ -72,6 +73,12 @@ def _process_request(pipeline, request) -> QueryResponse:
             params[key]["filters"] = _format_filters(params[key]["filters"])
 
     result = pipeline.run(query=request.query, params=params,debug=request.debug)
+    
+    # if any of the documents contains an embedding as an ndarray the latter needs to be converted to list of float
+    for document in result['documents'] or []:
+        if isinstance(document.embedding, ndarray):
+            document.embedding = document.embedding.tolist()
+    
     end_time = time.time()
     logger.info(json.dumps({"request": request, "response": result, "time": f"{(end_time - start_time):.2f}"}, default=str))
 


Test patch: