Repository: deepset-ai/haystack
Pull request: feat: Add `meta_fields_to_embed` to `TransformersSimilarityRanker`
### Related Issues

- fixes #6563 

### Proposed Changes:

 <!--- In case of a bug: Describe what caused the issue and how you solved it -->
 <!--- In case of a feature: Describe what did you add and how it works -->
Add `metadata_fields_to_embed` following the implementation in `SentenceTransformersDocumentEmbedder` to be able to embed metadata fields along with the content of the document. 


### How did you test it?

<!-- unit tests, integration tests, manual verification, instructions for manual tests -->
Added a new unit test to check the behavior. Updated existing tests for serializing the object.

### Notes for the reviewer

<!-- E.g. point out section where the reviewer  -->

### Checklist

- I have read the [contributors guidelines](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md) and the [code of conduct](https://github.com/deepset-ai/haystack/blob/main/code_of_conduct.txt)
- I have updated the related issue with new insights and changes
- I added unit tests and updated the docstrings
- I've used one of the [conventional commit types](https://www.conventionalcommits.org/en/v1.0.0/) for my PR title: `fix:`, `feat:`, `build:`, `chore:`, `ci:`, `docs:`, `style:`, `refactor:`, `perf:`, `test:`.
- I documented my code
- I ran [pre-commit hooks](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md#installation) and fixed any issue


Problem statement:
feat: Add embed_meta_fields support to rankers like in Haystack v1
**Is your feature request related to a problem? Please describe.**
We have found that `embed_meta_fields` greatly improves ranking when metadata is relevant while searching. This is a feature that we use a lot from v1 that would be great to also have in v2.

**Describe the solution you'd like**
Add support for `embed_meta_fields` as an init parameter to rankers like `TransformersSimilarityRanker`.

Hints from issue discussion:

Commit message: Add release notes

Patch to review:
diff --git a/haystack/components/rankers/transformers_similarity.py b/haystack/components/rankers/transformers_similarity.py
index cf849f5824..883014a263 100644
--- a/haystack/components/rankers/transformers_similarity.py
+++ b/haystack/components/rankers/transformers_similarity.py
@@ -40,6 +40,8 @@ def __init__(
         device: str = "cpu",
         token: Union[bool, str, None] = None,
         top_k: int = 10,
+        metadata_fields_to_embed: Optional[List[str]] = None,
+        embedding_separator: str = "\n",
     ):
         """
         Creates an instance of TransformersSimilarityRanker.
@@ -51,6 +53,8 @@ def __init__(
             If this parameter is set to `True`, the token generated when running
             `transformers-cli login` (stored in ~/.huggingface) is used.
         :param top_k: The maximum number of Documents to return per query.
+        :param metadata_fields_to_embed: List of meta fields that should be embedded along with the Document content.
+        :param embedding_separator: Separator used to concatenate the meta fields to the Document content.
         """
         torch_and_transformers_import.check()
 
@@ -62,6 +66,8 @@ def __init__(
         self.token = token
         self.model = None
         self.tokenizer = None
+        self.metadata_fields_to_embed = metadata_fields_to_embed or []
+        self.embedding_separator = embedding_separator
 
     def _get_telemetry_data(self) -> Dict[str, Any]:
         """
@@ -89,6 +95,8 @@ def to_dict(self) -> Dict[str, Any]:
             model_name_or_path=self.model_name_or_path,
             token=self.token if not isinstance(self.token, str) else None,  # don't serialize valid tokens
             top_k=self.top_k,
+            metadata_fields_to_embed=self.metadata_fields_to_embed,
+            embedding_separator=self.embedding_separator,
         )
 
     @component.output_types(documents=List[Document])
@@ -116,7 +124,14 @@ def run(self, query: str, documents: List[Document], top_k: Optional[int] = None
                 f"The component {self.__class__.__name__} wasn't warmed up. Run 'warm_up()' before calling 'run()'."
             )
 
-        query_doc_pairs = [[query, doc.content] for doc in documents]
+        query_doc_pairs = []
+        for doc in documents:
+            meta_values_to_embed = [
+                str(doc.meta[key]) for key in self.metadata_fields_to_embed if key in doc.meta and doc.meta[key]
+            ]
+            text_to_embed = self.embedding_separator.join(meta_values_to_embed + [doc.content or ""])
+            query_doc_pairs.append([query, text_to_embed])
+
         features = self.tokenizer(
             query_doc_pairs, padding=True, truncation=True, return_tensors="pt"
         ).to(  # type: ignore
diff --git a/releasenotes/notes/transformers-similarity-embed-meta-7f2b15988549f805.yaml b/releasenotes/notes/transformers-similarity-embed-meta-7f2b15988549f805.yaml
new file mode 100644
index 0000000000..ecf567d652
--- /dev/null
+++ b/releasenotes/notes/transformers-similarity-embed-meta-7f2b15988549f805.yaml
@@ -0,0 +1,5 @@
+---
+enhancements:
+  - |
+    Add metadata_fields_to_embed following the implementation in SentenceTransformersDocumentEmbedder to be able to
+    embed metadata fields along with the content of the document.
diff --git a/test/components/rankers/test_transformers_similarity.py b/test/components/rankers/test_transformers_similarity.py
index 09de17c2d8..62c97dc3ab 100644
--- a/test/components/rankers/test_transformers_similarity.py
+++ b/test/components/rankers/test_transformers_similarity.py
@@ -1,4 +1,6 @@
+from unittest.mock import MagicMock, patch
 import pytest
+import torch
 
 from haystack import Document, ComponentError
 from haystack.components.rankers.transformers_similarity import TransformersSimilarityRanker
@@ -15,6 +17,8 @@ def test_to_dict(self):
                 "top_k": 10,
                 "token": None,
                 "model_name_or_path": "cross-encoder/ms-marco-MiniLM-L-6-v2",
+                "metadata_fields_to_embed": [],
+                "embedding_separator": "\n",
             },
         }
 
@@ -30,9 +34,37 @@ def test_to_dict_with_custom_init_parameters(self):
                 "model_name_or_path": "my_model",
                 "token": None,  # we don't serialize valid tokens,
                 "top_k": 5,
+                "metadata_fields_to_embed": [],
+                "embedding_separator": "\n",
             },
         }
 
+    @patch("torch.sort")
+    def test_embed_metadata(self, mocked_sort):
+        mocked_sort.return_value = (None, torch.tensor([0]))
+        embedder = TransformersSimilarityRanker(
+            model_name_or_path="model", metadata_fields_to_embed=["meta_field"], embedding_separator="\n"
+        )
+        embedder.model = MagicMock()
+        embedder.tokenizer = MagicMock()
+
+        documents = [Document(content=f"document number {i}", meta={"meta_field": f"meta_value {i}"}) for i in range(5)]
+
+        embedder.run(query="test", documents=documents)
+
+        embedder.tokenizer.assert_called_once_with(
+            [
+                ["test", "meta_value 0\ndocument number 0"],
+                ["test", "meta_value 1\ndocument number 1"],
+                ["test", "meta_value 2\ndocument number 2"],
+                ["test", "meta_value 3\ndocument number 3"],
+                ["test", "meta_value 4\ndocument number 4"],
+            ],
+            padding=True,
+            truncation=True,
+            return_tensors="pt",
+        )
+
     @pytest.mark.integration
     @pytest.mark.parametrize(
         "query,docs_before_texts,expected_first_text",


Code review comments:
diff_hunk: @@ -40,6 +40,8 @@ def __init__(
         device: str = "cpu",
         token: Union[bool, str, None] = None,
         top_k: int = 10,
+        metadata_fields_to_embed: Optional[List[str]] = None,
path: haystack/components/rankers/transformers_similarity.py
text: @user1:
I think we should change it to `meta_fields_to_embed`, all dataclasses are standardizing to `meta` instead of `metadata`.

@user2:
Agree, even if we have other components using `metadata`. I opened an issue for that: https://github.com/deepset-ai/haystack/issues/6570

@author:
changed!