Repository: deepset-ai/haystack

Issue:
Feature: allow decreasing dataset size in `eval_beir()`
**Describe the solution you'd like**
It would be nice to allow the user to decrease the dataset when evaluating pipelines using [`eval_beir()`](https://github.com/deepset-ai/haystack/blob/7290196c32abf44e269caa03e1d2178c77992b0e/haystack/pipelines/base.py#L727) to do a quick or reduced test, for instance. Since the downloading and loading of the dataset is done inside `eval_beir()`, there's no possible workaround.

**Describe alternatives you've considered**
The possibilities could be to shrink the dataset like:

```python
corpus, queries, qrels = corpus[:size], queries[:size], qrels[:size]
```

Or to use sampling with a fixed seed so we make sure it's better shuffled, among other possibilities.


Discussion:
@ugm2 Thank you for this suggestion, Unai! I agree that this would be a nice additional feature. Some datasets are very large and running an evaluation on the full dataset would take a long time. Having the option to first do a quick evaluation run on a small subset would be great.
I think this is a good issue for community contributions. So if you or anyone else from the community would like to work on it, just write a comment here and start by creating a draft PR linking to this issue. Our contributor guidelines can be found [here](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md). 🙂 

Resolving patch:
diff --git a/haystack/pipelines/base.py b/haystack/pipelines/base.py
--- a/haystack/pipelines/base.py
+++ b/haystack/pipelines/base.py
@@ -6,6 +6,7 @@
 from datetime import timedelta
 from functools import partial
 from hashlib import sha1
+import itertools
 from typing import Dict, List, Optional, Any, Set, Tuple, Union
 
 
@@ -732,6 +733,7 @@ def eval_beir(
         query_params: dict = {},
         dataset: str = "scifact",
         dataset_dir: Path = Path("."),
+        num_documents: Optional[int] = None,
         top_k_values: List[int] = [1, 3, 5, 10, 100, 1000],
         keep_index: bool = False,
     ) -> Tuple[Dict[str, float], Dict[str, float], Dict[str, float], Dict[str, float]]:
@@ -745,6 +747,8 @@ def eval_beir(
         :param query_params: The params to use during querying (see pipeline.run's params).
         :param dataset: The BEIR dataset to use.
         :param dataset_dir: The directory to store the dataset to.
+        :param num_documents: Maximum number of documents to load from given dataset. If set to None (default)
+                             or to a value larger than the number of documents in the dataset, the full dataset is loaded.
         :param top_k_values: The top_k values each metric will be calculated for.
         :param keep_index: Whether to keep the index after evaluation.
                            If True the index will be kept after beir evaluation. Otherwise it will be deleted immediately afterwards.
@@ -765,6 +769,28 @@ def eval_beir(
         logger.info("Dataset downloaded here: %s", data_path)
         corpus, queries, qrels = GenericDataLoader(data_path).load(split="test")  # or split = "train" or "dev"
 
+        # crop dataset if `dataset_size` is provided and is valid
+        if num_documents is not None and 0 < num_documents < len(corpus):
+            logger.info(f"Cropping dataset from {len(corpus)} to {num_documents} documents")
+            corpus = dict(itertools.islice(corpus.items(), num_documents))
+            # Remove queries that don't contain the remaining documents
+            corpus_ids = set(list(corpus.keys()))
+            qrels_new = {}
+            for query_id, document_rel_dict in qrels.items():
+                document_rel_ids_intersection = list(corpus_ids & set(list(document_rel_dict.keys())))
+                # If there are no remaining documents related to the query, delete the query
+                if len(document_rel_ids_intersection) == 0:
+                    del queries[query_id]
+                # If there are remaining documents, update qrels
+                else:
+                    qrels_new[query_id] = {_id: qrels[query_id][_id] for _id in document_rel_ids_intersection}
+            qrels = qrels_new
+        elif num_documents is not None and (num_documents < 1 or num_documents > len(corpus)):
+            logging.warning(
+                f"'num_documents' variable should be lower than corpus length and have a positive value, but it's {num_documents}."
+                " Dataset size remains unchanged."
+            )
+
         # check index before eval
         document_store = index_pipeline.get_document_store()
         if document_store is not None:


Test patch: