Repository: deepset-ai/haystack

Issue:
Table format of API search request different
**Is your feature request related to a problem? Please describe.**
Currently the table format of the search and get document REST API response, is different than commonly used in haystack. Commonly, we use a list representation (see screenshot). Though, when using the REST API the table is represented in a dict format. This can lead to confusion and is not ideal. 

**Describe the solution you'd like**
The response format should be adapted to the commonly used representation. 

**Additional context**

![Screenshot from 2023-01-11 17-38-43](https://user-images.githubusercontent.com/87523290/212682712-68ecc868-f1fa-40e6-bbd5-8b6640a01161.png)
![Screenshot from 2023-01-11 17-39-20](https://user-images.githubusercontent.com/87523290/212682715-b017bc06-97ae-4062-9945-489c89f714be.png)




Discussion:
@ju-gu Is the recommendation to update the response so the table comes back in a list of lists format? 
@sjrl  I would say this is the most straight forward solution here. We also could go generally for a dict format, but this would mean update all converters etc. Wdyt? I know we talked about Dict format being more handy as you can just load it immediately into a Dataframe but I am not sure whether that is enough for a change. @bogdankostic Do you have an opinion on that?
The "standard" table format in Haystack is the list of lists, so I'd say it's most sensible to adapt the output of the Rest API to that format. If we think we should adapt the table format in Haystack, we should follow the proposal process.

To change the representation of tables in the Rest API, we need to change [this line](https://github.com/deepset-ai/haystack/blob/main/rest_api/rest_api/schema.py#L19).

Resolving patch:
diff --git a/rest_api/rest_api/schema.py b/rest_api/rest_api/schema.py
--- a/rest_api/rest_api/schema.py
+++ b/rest_api/rest_api/schema.py
@@ -16,7 +16,10 @@
 
 
 BaseConfig.arbitrary_types_allowed = True
-BaseConfig.json_encoders = {np.ndarray: lambda x: x.tolist(), pd.DataFrame: lambda x: x.to_dict(orient="records")}
+BaseConfig.json_encoders = {
+    np.ndarray: lambda x: x.tolist(),
+    pd.DataFrame: lambda x: [x.columns.tolist()] + x.values.tolist(),
+}
 
 
 PrimitiveType = Union[str, int, float, bool]


Test patch:
diff --git a/rest_api/test/test_rest_api.py b/rest_api/test/test_rest_api.py
--- a/rest_api/test/test_rest_api.py
+++ b/rest_api/test/test_rest_api.py
@@ -412,10 +412,7 @@ def test_query_with_dataframe(client):
         response = client.post(url="/query", json={"query": TEST_QUERY})
         assert 200 == response.status_code
         assert len(response.json()["documents"]) == 1
-        assert response.json()["documents"][0]["content"] == [
-            {"col1": "text_1", "col2": 1},
-            {"col1": "text_2", "col2": 2},
-        ]
+        assert response.json()["documents"][0]["content"] == [["col1", "col2"], ["text_1", 1], ["text_2", 2]]
         assert response.json()["documents"][0]["content_type"] == "table"
         # Ensure `run` was called with the expected parameters
         mocked_pipeline.run.assert_called_with(query=TEST_QUERY, params={}, debug=False)