Repository: UKPLab/sentence-transformers
Issue category: Bug Report

Issue:
NanoBeirEvaluator takes in empty dataset
I noticed that NanoBeir takes in an empty dataset and returns no error during instantiation. But when the model gets passed, the error kinda seems confusing.
So -
1. is it fine to allow the evaluator to take in an empty dataset.
2. Should we change the error message saying "invalid datset or something"

Who can help?
@tomaarsen 

Snippet reproduction
```
from sentence_transformers import SentenceTransformer
from sentence_transformers.evaluation import NanoBEIREvaluator

datasets = []

#model = SentenceTransformer("sentence-transformers-testing/stsb-bert-tiny-safetensors")

evaluator = NanoBEIREvaluator( # this bit here returns no error
    dataset_names=datasets,
)

results = evaluator(model) #this raised an error
```

#####################################################################

Error log
```
{
	"name": "KeyError",
	"message": "'cosine_ndcg@10'",
	"stack": "---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[4], line 1
----> 1 evaluator(model)

sentence_transformers\\evaluation\\NanoBEIREvaluator.py:351, in NanoBEIREvaluator.__call__(self, model, output_path, epoch, steps, *args, **kwargs)
    348 if not self.primary_metric:
    349     if self.main_score_function is None:
    350         score_function = max(
--> 351             [(name, agg_results[f\"{name}_ndcg@{max(self.ndcg_at_k)}\"]) for name in self.score_function_names],
    352             key=lambda x: x[1],
    353         )[0]
    354         self.primary_metric = f\"{score_function}_ndcg@{max(self.ndcg_at_k)}\"
    355     else:

sentence-transformers\\sentence_transformers\\evaluation\\NanoBEIREvaluator.py:351, in <listcomp>(.0)
    348 if not self.primary_metric:
    349     if self.main_score_function is None:
    350         score_function = max(
--> 351             [(name, agg_results[f\"{name}_ndcg@{max(self.ndcg_at_k)}\"]) for name in self.score_function_names],
    352             key=lambda x: x[1],
    353         )[0]
    354         self.primary_metric = f\"{score_function}_ndcg@{max(self.ndcg_at_k)}\"
    355     else:

KeyError: 'cosine_ndcg@10'"
}
```




Hints:
Hello!

Well spotted, I think we should indeed add an error stating that `dataset_names` cannot be an empty list. It can be `None`, which will default to a full list, but it shouldn't be empty.

- Tom Aarsen
cool, ill be away for a few days im going back to school for my graduation :)

ill be back and raise a PR for this.
Congratulations! 🤗

Patch:
diff --git a/sentence_transformers/evaluation/NanoBEIREvaluator.py b/sentence_transformers/evaluation/NanoBEIREvaluator.py
index ec29ad094..82907e3ed 100644
--- a/sentence_transformers/evaluation/NanoBEIREvaluator.py
+++ b/sentence_transformers/evaluation/NanoBEIREvaluator.py
@@ -420,6 +420,8 @@ def _load_dataset(self, dataset_name: DatasetNameType, **ir_evaluator_kwargs) ->
         )
 
     def _validate_dataset_names(self):
+        if len(self.dataset_names) == 0:
+            raise ValueError("dataset_names cannot be empty. Use None to evaluate on all datasets.")
         if missing_datasets := [
             dataset_name for dataset_name in self.dataset_names if dataset_name.lower() not in dataset_name_to_id
         ]:


Test patch: