Repository: deepset-ai/haystack

Issue:
Shaper join_documents drops all metadata
**Describe the bug**
join_documents function in Shaper drops all metadata.  Seems this is known, because it actually states the following in the documentation:
https://docs.haystack.deepset.ai/reference/shaper-api#join_documents

![image](https://user-images.githubusercontent.com/6450799/226701541-51423bcb-b18d-4801-872a-abb70b84c997.png)

**Expected behavior**
Keep the metadata when the documents have been merged.

**Additional context**
You can _kinda_ get around this by "shaping the Shaper" (thanks Roland).  It's a bit chaotic, but if you rename the `documents` invocation value before sending it to the join_documents function, you can retain the metadata, however it means your output JSON contains your documents twice (in both `orig_documents` and `documents`).

```python
rshaper = Shaper(
    func="rename", inputs={"value": "documents"}, outputs=["orig_documents"]
)

shaper = Shaper(
    func="join_documents", inputs={"documents": "orig_documents"}, outputs=["documents"]
)

pipe = Pipeline()
pipe.add_node(component=retriever, name="Retriever", inputs=["Query"])
pipe.add_node(component=rshaper, name="rShaper", inputs=["Retriever"])
pipe.add_node(component=shaper, name="Shaper", inputs=["rShaper"])
pipe.add_node(component=prompt_node, name="prompt_node", inputs=["Shaper"])
```

This will put the contents of `documents` into `orig_documents`, which is then passed down to the `join_documents` Shaper, which then joins the documents and places them into an output called `documents`.


Discussion:
Hi @recrudesce , yes, we are aware that all metadata is dropped when joining documents. The reason is that currently, `join_documents` transforms a list of documents into a list containing a single Document. While we can just concatenate the text content of the input documents, it's not well defined what to do with the meta data. For example, one could concatenate the string meta fields too but what with meta fields that are a date or an int? One could also check whether all input documents have the same value for one of the meta fields and in that case use that value also for the joined document. For example, if all input documents have the same date, the joined document could have the same date.
https://github.com/deepset-ai/haystack/blob/b6371c95a81c2fd723ae64a3c7ffd9ae5fe85870/haystack/nodes/other/shaper.py#L88

Could you please explain what behavior you would expect and why it would make sense for your use case?
Hi. I don't know what the desired output is.

In case we just want to keep the common metadata,
in the past we have developed a similar method for `DocumentMerger`:
https://github.com/deepset-ai/haystack/blob/7bb6499c298855133aa75c457f583bcc30efd97d/haystack/nodes/other/document_merger.py#L70
> Could you please explain what behavior you would expect and why it would make sense for your use case?

It allows us to return a list of "further reading" to people who query our dataset.  If we have to merge the documents to be able to pass it properly to the promptnode, we are no longer able to provide this additional data.

>  it's not well defined what to do with the meta data.

Provide them in a new, separate value called `document_metadata` or similar.  Then the output is joined documents in `documents`, and all the original metadata in `document_metadata`

Seems bizarre to just drop it.  And you've also identified that it needs to be fixed, else why would someone have written that in the documentation ?
Not sure if this would be an ok implementation but one idea might be:
For each document we join, create a meta field with their name or id, within which we simply list their own metadatas. I agree with @recrudesce that this comes across as an issue that we have marked 'to fix' on our docs. So let's either fix it or remove this mark and make it clear that all 'join documents' would remove meta fields. 

So my suggestion is:
```
'meta': {
  'name_or_id_of_document_1': {'name':'abc', 'date': 2023},
  'name_or_id_of_document_2': {'name':'xyz', 'date': 2023},
}
```

Again, this is just an idea. 

Resolving patch:
diff --git a/haystack/nodes/other/shaper.py b/haystack/nodes/other/shaper.py
--- a/haystack/nodes/other/shaper.py
+++ b/haystack/nodes/other/shaper.py
@@ -100,7 +100,7 @@ def join_documents(
     - $id: The ID of the document.
     - $META_FIELD: The value of the metadata field called 'META_FIELD'.
 
-    All metadata is dropped. (TODO: fix)
+    All metadata is dropped.
 
     Example:
 
@@ -124,7 +124,7 @@ def join_documents_and_scores(documents: List[Document]) -> Tuple[List[Document]
     """
     Transforms a list of documents with scores in their metadata into a list containing a single document.
     The resulting document contains the scores and the contents of all the original documents.
-    All metadata is dropped. (TODO: fix)
+    All metadata is dropped.
     Example:
     ```python
     assert join_documents_and_scores(


Test patch: