Repository: run-llama/llama_index
Pull request: Added support for multiple docs in SentenceTransformersFinetuneEngine. Resolves Issue #10297
# Description
In the SentenceTransformersFinetuneEngine, only the first document corresponding to a query was included in the examples. In the changes made, all the documents will considered. The intution is that if all the (query, relevant_doc) pairs are considered as positive samples, it might result in better embeddings.

Fixes #10297 

## Type of Change

Please delete options that are not relevant.

- [ ] New feature (non-breaking change which adds functionality)

# How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

- [ ] Added new unit/integration tests
- [ ] Added new notebook (that tests end-to-end)
- [ ] I stared at the code and made sure it makes sense

# Suggested Checklist:

- [ ] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] I have added Google Colab support for the newly added notebooks.
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I ran `make format; make lint` to appease the lint gods


Problem statement:
[Feature Request]: Support for more than one relevant document in SentenceTransformersFinetuneEngine
### Feature Description

In ```llama_index/finetuning/embeddings/sentence_transformer.py```, it can be seen that for every query, only one relevant document is added to the list ```examples```.
```python
# TODO: support more than 1 doc per query
examples: Any = []
for query_id, query in dataset.queries.items():
    node_id = dataset.relevant_docs[query_id][0]
    text = dataset.corpus[node_id]
    example = InputExample(texts=[query, text])
    examples.append(example)
self.examples = examples
```
Is it not possible to add every pair of ```(query, relevant_doc)``` ?
For example:
```python
examples: Any = []
for query_id, query in dataset.queries.items():
    for doc in dataset.relevant_docs[query_id]:
        node_id = doc
        text = dataset.corpus[node_id]
        example = InputExample(texts=[query, text])
        examples.append(example)
self.examples = examples
```

### Reason

_No response_

### Value of Feature

Because of the suggested change, every pair of ```(query, relevant_doc)``` will be considered as a positive sample, which might result in better embeddings.

Hints from issue discussion:

Commit message: Added support for multiple docs in SentenceTransformersFinetuneEngine

Patch to review:
diff --git a/llama_index/finetuning/embeddings/sentence_transformer.py b/llama_index/finetuning/embeddings/sentence_transformer.py
index ead2eb3c02..d769259cb7 100644
--- a/llama_index/finetuning/embeddings/sentence_transformer.py
+++ b/llama_index/finetuning/embeddings/sentence_transformer.py
@@ -35,13 +35,14 @@ def __init__(
         self.model_output_path = model_output_path
         self.model = SentenceTransformer(model_id)
 
-        # TODO: support more than 1 doc per query
+
         examples: Any = []
         for query_id, query in dataset.queries.items():
-            node_id = dataset.relevant_docs[query_id][0]
-            text = dataset.corpus[node_id]
-            example = InputExample(texts=[query, text])
-            examples.append(example)
+            for doc in dataset.relevant_docs[query_id]:
+                node_id = doc
+                text = dataset.corpus[node_id]
+                example = InputExample(texts=[query, text])
+                examples.append(example)
         self.examples = examples
 
         self.loader: DataLoader = DataLoader(examples, batch_size=batch_size)


Code review comments:
diff_hunk: @@ -35,13 +35,14 @@ def __init__(
         self.model_output_path = model_output_path
         self.model = SentenceTransformer(model_id)
 
-        # TODO: support more than 1 doc per query
+
         examples: Any = []
         for query_id, query in dataset.queries.items():
-            node_id = dataset.relevant_docs[query_id][0]
-            text = dataset.corpus[node_id]
-            example = InputExample(texts=[query, text])
-            examples.append(example)
+            for doc in dataset.relevant_docs[query_id]:
path: llama_index/finetuning/embeddings/sentence_transformer.py
text: @user1:
if we use all relevant docs for a query, it might be slower. 
am thinking should we add option to decide whether fine tune on one query to doc pair or all query to docs pairs

@author:
@user1 I agree with your proposal. I'll try to implement it and create a new pull request.