Repository: deepset-ai/haystack
Pull request: fixed join_docs.py concatenate
### Related Issues

- fixes #4916 

### Proposed Changes:

I merged the two lists keeping only the document with the highest score

### How did you test it?

Manual verification.

### Checklist

- I have read the [contributors guidelines](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md) and the [code of conduct](https://github.com/deepset-ai/haystack/blob/main/code_of_conduct.txt)

Problem statement:
JoinDocuments should use highest score when multiple retrievers recall the same document
**Is your feature request related to a problem? Please describe.**
The JoinDocuments node currently uses the document from the last retriever if their are multiple retrievers and they recall the same document. The first retriever could have the highest score and be more useful. That's why I propose to use the document with the highest score if there are duplicate documents recalled

Hints from issue discussion:
Hi @Koenlaermans I completely agree! It's a well defined, small issue so I will also add the Contributions Wanted Label here in case you ore somebody else would like to contribute to Haystack by fixing this issue. The starting point for a fix is here in the code: https://github.com/deepset-ai/haystack/blob/56d033e7e77841b517cf2a75c03714c3f6cc7830/haystack/nodes/other/join_docs.py#L62 
Thank you for reporting this!
Hi @julian-risch 
I submitted a PR, i hope i did a good job, I'm waiting for the review.

Commit message: Merge branch 'main' into dev/join_docs

Patch to review:
diff --git a/haystack/nodes/other/join_docs.py b/haystack/nodes/other/join_docs.py
index 4185873a7c..7ce0de819d 100644
--- a/haystack/nodes/other/join_docs.py
+++ b/haystack/nodes/other/join_docs.py
@@ -1,11 +1,10 @@
-from collections import defaultdict
 import logging
+from collections import defaultdict
 from math import inf
+from typing import List, Optional
 
-from typing import Optional, List
-
-from haystack.schema import Document
 from haystack.nodes.other.join import JoinNode
+from haystack.schema import Document
 
 logger = logging.getLogger(__name__)
 
@@ -64,7 +63,7 @@ def run_accumulated(self, inputs: List[dict], top_k_join: Optional[int] = None):
         document_map = {doc.id: doc for result in results for doc in result}
 
         if self.join_mode == "concatenate":
-            scores_map = self._concatenate_results(results)
+            scores_map = self._concatenate_results(results, document_map)
         elif self.join_mode == "merge":
             scores_map = self._calculate_comb_sum(results)
         elif self.join_mode == "reciprocal_rank_fusion":
@@ -118,11 +117,22 @@ def run_batch_accumulated(self, inputs: List[dict], top_k_join: Optional[int] =
 
             return output, "output_1"
 
-    def _concatenate_results(self, results):
+    def _concatenate_results(self, results, document_map):
         """
         Concatenates multiple document result lists.
+        Return the documents with the higher score.
         """
-        return {doc.id: doc.score for result in results for doc in result}
+        list_id = list(document_map.keys())
+        scores_map = {}
+        for idx in list_id:
+            tmp = []
+            for result in results:
+                for doc in result:
+                    if doc.id == idx:
+                        tmp.append(doc)
+            item_best_score = max(tmp, key=lambda x: x.score)
+            scores_map.update({idx: item_best_score.score})
+        return scores_map
 
     def _calculate_comb_sum(self, results):
         """
diff --git a/releasenotes/notes/fix-joinDocuments-concatenate-56a7cdba00a7248e.yaml b/releasenotes/notes/fix-joinDocuments-concatenate-56a7cdba00a7248e.yaml
new file mode 100644
index 0000000000..15e507e92a
--- /dev/null
+++ b/releasenotes/notes/fix-joinDocuments-concatenate-56a7cdba00a7248e.yaml
@@ -0,0 +1,37 @@
+---
+prelude: >
+    Replace this text with content to appear at the top of the section for this
+    release. This is equivalent to the "Highlights" section we used before.
+    The prelude might repeat some details that are also present in other notes
+    from the same release, that's ok. Not every release note requires a prelude,
+    use it only to describe major features or notable changes.
+upgrade:
+  - |
+    List upgrade notes here, or remove this section.
+    Upgrade notes should be rare: only list known/potential breaking changes,
+    or major changes that require user action before the upgrade.
+    Notes here must include steps that users can follow to 1. know if they're
+    affected and 2. handle the change gracefully on their end.
+features:
+  - |
+    List new features here, or remove this section.
+enhancements:
+  - |
+    List new behavior that is too small to be
+    considered a new feature, or remove this section.
+issues:
+  - |
+    List known issues here, or remove this section. For example, if some change is experimental or known to not work in some cases, it should be mentioned here.
+deprecations:
+  - |
+    List deprecations notes here, or remove this section. Deprecations should not be used for something that is removed in the release, use upgrade section instead. Deprecation should allow time for users to make necessary changes for the removal to happen in a future release.
+security:
+  - |
+    Add security notes here, or remove this section.
+fixes:
+  - |
+    Add normal bug fixes here, or remove this section.
+preview:
+  - |
+    Add changes to Haystack version 2, or remove this section.
+    Haystack version 2 can be found under haystack/preview.


Code review comments:
diff_hunk: @@ -0,0 +1,37 @@
+---
+prelude: >
+    Replace this text with content to appear at the top of the section for this
+    release. This is equivalent to the "Highlights" section we used before.
+    The prelude might repeat some details that are also present in other notes
+    from the same release, that's ok. Not every release note requires a prelude,
+    use it only to describe major features or notable changes.
+upgrade:
+  - |
+    List upgrade notes here, or remove this section.
+    Upgrade notes should be rare: only list known/potential breaking changes,
+    or major changes that require user action before the upgrade.
+    Notes here must include steps that users can follow to 1. know if they're
+    affected and 2. handle the change gracefully on their end.
+features:
+  - |
+    List new features here, or remove this section.
+enhancements:
+  - |
+    List new behavior that is too small to be
+    considered a new feature, or remove this section.
+issues:
+  - |
+    List known issues here, or remove this section. For example, if some change is experimental or known to not work in some cases, it should be mentioned here.
+deprecations:
+  - |
+    List deprecations notes here, or remove this section. Deprecations should not be used for something that is removed in the release, use upgrade section instead. Deprecation should allow time for users to make necessary changes for the removal to happen in a future release.
+security:
+  - |
+    Add security notes here, or remove this section.
+fixes:
+  - |
+    Add normal bug fixes here, or remove this section.
+preview:
+  - |
+    Add changes to Haystack version 2, or remove this section.
+    Haystack version 2 can be found under haystack/preview.
path: releasenotes/notes/fix-joinDocuments-concatenate-56a7cdba00a7248e.yaml
text: @user1:
Hey @author, this file is just a template :slightly_smiling_face: You should check which description fits your PR better and keep only that entry.

For example, in your case it would be something like this:
```suggestion
enhancements:
  -  Make JoinDocuments return only the document with the highest score if there are duplicate documents in the list.
```

@author:
done