Repository: arthurbr11/sentence-transformers
Pull request #22: Add `max_active_dims` to get_config_dict for sparse evaluators
Author: login: tomaarsen
Created: 2025-05-21T13:37:44Z | Merged: 2025-05-22T20:05:13Z

Hello!

## Pull Request overview
* Add max_active_dims to get_config_dict for sparse evaluators
* Update SparseNanoBEIR evaluator to set the name automatically based on `max_active_dims` (just like truncate_dim in NanoBEIREvaluator)
* Minor updates to train_csr_nq.py

## Details
The first change is required for the model card generation to distinguish multiple instances of the same evaluator but with different max_active_dims. You can see the effect in the model cards here: 
## Before
* https://huggingface.co/tomaarsen/csr-mxbai-embed-large-v1-nq-og (trained with 4 NanoBEIR evaluators at different `max_active_dims`)
* Only 1 Sparse Information Retrieval section, but it has some datasets ~5 times with no extra information about what differentiates them
* 5 Sparse Nano BEIR sections (4 for the small NanoBEIR, 1 for the bigger "test" one), no way to differentiate the first 4.
* Separate from the model_card: the evaluation results only had e.g. `eval_NanoBEIR_mean_dot_ndcg@10`, despite there being 4 different evaluators that produce values under this key. Turns out, the later ones overrode the earlier ones. 

## After
* https://huggingface.co/tomaarsen/csr-mxbai-embed-large-v1-nq (trained with 2 NanoBEIR evaluators at different `max_active_dims`)
* 3 Sparse Information Retrieval sections, each with clear `max_active_dims` that distinguishes them
* Each Sparse Information Retrieval section is followedby 1 Sparse Nano BEIR section, and they're distinguishable by name and `max_active_dims` setting
* Separate from the model_card: the evaluation results now has e.g. `eval_NanoBEIR_mean_256_dot_ndcg@10` & `eval_NanoBEIR_mean_128_dot_ndcg@10`.

This is what gets logged now, for example:
```python
{'epoch': 0.19,
 'eval_NanoBEIR_mean_128_dot_accuracy@1': 0.38666666666666666,
 'eval_NanoBEIR_mean_128_dot_accuracy@10': 0.7600000000000001,
 'eval_NanoBEIR_mean_128_dot_accuracy@3': 0.56,
 'eval_NanoBEIR_mean_128_dot_accuracy@5': 0.6466666666666666,
 'eval_NanoBEIR_mean_128_dot_map@100': 0.3759154850419881,
 'eval_NanoBEIR_mean_128_dot_mrr@10': 0.4984550264550265,
 'eval_NanoBEIR_mean_128_dot_ndcg@10': 0.4683952600569259,
 'eval_NanoBEIR_mean_128_dot_precision@1': 0.38666666666666666,
 'eval_NanoBEIR_mean_128_dot_precision@10': 0.12933333333333333,
 'eval_NanoBEIR_mean_128_dot_precision@3': 0.23777777777777773,
 'eval_NanoBEIR_mean_128_dot_precision@5': 0.18133333333333335,
 'eval_NanoBEIR_mean_128_dot_recall@1': 0.25823832372017863,
 'eval_NanoBEIR_mean_128_dot_recall@10': 0.545567985119501,
 'eval_NanoBEIR_mean_128_dot_recall@3': 0.4013041425863714,
 'eval_NanoBEIR_mean_128_dot_recall@5': 0.4541189701912864,
 'eval_NanoBEIR_mean_128_row_non_zero_mean_corpus': 128.0,
 'eval_NanoBEIR_mean_128_row_non_zero_mean_query': 128.0,
 'eval_NanoBEIR_mean_128_row_sparsity_mean_corpus': 0.96875,
 'eval_NanoBEIR_mean_128_row_sparsity_mean_query': 0.96875,
 'eval_NanoBEIR_mean_256_dot_accuracy@1': 0.4066666666666667,
 'eval_NanoBEIR_mean_256_dot_accuracy@10': 0.7933333333333333,
 'eval_NanoBEIR_mean_256_dot_accuracy@3': 0.6333333333333334,
 'eval_NanoBEIR_mean_256_dot_accuracy@5': 0.6933333333333334,
 'eval_NanoBEIR_mean_256_dot_map@100': 0.411975022853921,
 'eval_NanoBEIR_mean_256_dot_mrr@10': 0.5336455026455026,
 'eval_NanoBEIR_mean_256_dot_ndcg@10': 0.5106517684276061,
 'eval_NanoBEIR_mean_256_dot_precision@1': 0.4066666666666667,
 'eval_NanoBEIR_mean_256_dot_precision@10': 0.14533333333333331,
 'eval_NanoBEIR_mean_256_dot_precision@3': 0.2644444444444444,
 'eval_NanoBEIR_mean_256_dot_precision@5': 0.19866666666666669,
 'eval_NanoBEIR_mean_256_dot_recall@1': 0.27524688781444745,
 'eval_NanoBEIR_mean_256_dot_recall@10': 0.57487956690743,
 'eval_NanoBEIR_mean_256_dot_recall@3': 0.4579472437423752,
 'eval_NanoBEIR_mean_256_dot_recall@5': 0.5068607615246549,
 'eval_NanoBEIR_mean_256_row_non_zero_mean_corpus': 256.0,
 'eval_NanoBEIR_mean_256_row_non_zero_mean_query': 256.0,
 'eval_NanoBEIR_mean_256_row_sparsity_mean_corpus': 0.9375,
 'eval_NanoBEIR_mean_256_row_sparsity_mean_query': 0.9375,
 'eval_NanoMSMARCO_128_dot_accuracy@1': 0.34,
 'eval_NanoMSMARCO_128_dot_accuracy@10': 0.8,
 'eval_NanoMSMARCO_128_dot_accuracy@3': 0.64,
 'eval_NanoMSMARCO_128_dot_accuracy@5': 0.7,
 'eval_NanoMSMARCO_128_dot_map@100': 0.5095921396792782,
 'eval_NanoMSMARCO_128_dot_mrr@10': 0.49940476190476185,
 'eval_NanoMSMARCO_128_dot_ndcg@10': 0.572841390405141,
 'eval_NanoMSMARCO_128_dot_precision@1': 0.34,
 'eval_NanoMSMARCO_128_dot_precision@10': 0.08,
 'eval_NanoMSMARCO_128_dot_precision@3': 0.21333333333333332,
 'eval_NanoMSMARCO_128_dot_precision@5': 0.14,
 'eval_NanoMSMARCO_128_dot_recall@1': 0.34,
 'eval_NanoMSMARCO_128_dot_recall@10': 0.8,
 'eval_NanoMSMARCO_128_dot_recall@3': 0.64,
 'eval_NanoMSMARCO_128_dot_recall@5': 0.7,
 'eval_NanoMSMARCO_128_row_non_zero_mean_corpus': 128.0,
 'eval_NanoMSMARCO_128_row_non_zero_mean_query': 128.0,
 'eval_NanoMSMARCO_128_row_sparsity_mean_corpus': 0.96875,
 'eval_NanoMSMARCO_128_row_sparsity_mean_query': 0.96875,
 'eval_NanoMSMARCO_256_dot_accuracy@1': 0.36,
 'eval_NanoMSMARCO_256_dot_accuracy@10': 0.8,
 'eval_NanoMSMARCO_256_dot_accuracy@3': 0.68,
 'eval_NanoMSMARCO_256_dot_accuracy@5': 0.72,
 'eval_NanoMSMARCO_256_dot_map@100': 0.5341810113741303,
 'eval_NanoMSMARCO_256_dot_mrr@10': 0.5251904761904761,
 'eval_NanoMSMARCO_256_dot_ndcg@10': 0.5931942309850274,
 'eval_NanoMSMARCO_256_dot_precision@1': 0.36,
 'eval_NanoMSMARCO_256_dot_precision@10': 0.08,
 'eval_NanoMSMARCO_256_dot_precision@3': 0.22666666666666668,
 'eval_NanoMSMARCO_256_dot_precision@5': 0.14400000000000002,
 'eval_NanoMSMARCO_256_dot_recall@1': 0.36,
 'eval_NanoMSMARCO_256_dot_recall@10': 0.8,
 'eval_NanoMSMARCO_256_dot_recall@3': 0.68,
 'eval_NanoMSMARCO_256_dot_recall@5': 0.72,
 'eval_NanoMSMARCO_256_row_non_zero_mean_corpus': 256.0,
 'eval_NanoMSMARCO_256_row_non_zero_mean_query': 256.0,
 'eval_NanoMSMARCO_256_row_sparsity_mean_corpus': 0.9375,
 'eval_NanoMSMARCO_256_row_sparsity_mean_query': 0.9375,
 'eval_NanoNFCorpus_128_dot_accuracy@1': 0.4,
 'eval_NanoNFCorpus_128_dot_accuracy@10': 0.72,
 'eval_NanoNFCorpus_128_dot_accuracy@3': 0.5,
 'eval_NanoNFCorpus_128_dot_accuracy@5': 0.58,
 'eval_NanoNFCorpus_128_dot_map@100': 0.1055996264376909,
 'eval_NanoNFCorpus_128_dot_mrr@10': 0.4830555555555556,
 'eval_NanoNFCorpus_128_dot_ndcg@10': 0.27713696775848146,
 'eval_NanoNFCorpus_128_dot_precision@1': 0.4,
 'eval_NanoNFCorpus_128_dot_precision@10': 0.228,
 'eval_NanoNFCorpus_128_dot_precision@3': 0.31999999999999995,
 'eval_NanoNFCorpus_128_dot_precision@5': 0.272,
 'eval_NanoNFCorpus_128_dot_recall@1': 0.024714971160535826,
 'eval_NanoNFCorpus_128_dot_recall@10': 0.11670395535850293,
 'eval_NanoNFCorpus_128_dot_recall@3': 0.043912427759114324,
 'eval_NanoNFCorpus_128_dot_recall@5': 0.06235691057385921,
 'eval_NanoNFCorpus_128_row_non_zero_mean_corpus': 128.0,
 'eval_NanoNFCorpus_128_row_non_zero_mean_query': 128.0,
 'eval_NanoNFCorpus_128_row_sparsity_mean_corpus': 0.96875,
 'eval_NanoNFCorpus_128_row_sparsity_mean_query': 0.96875,
 'eval_NanoNFCorpus_256_dot_accuracy@1': 0.42,
 'eval_NanoNFCorpus_256_dot_accuracy@10': 0.74,
 'eval_NanoNFCorpus_256_dot_accuracy@3': 0.56,
 'eval_NanoNFCorpus_256_dot_accuracy@5': 0.6,
 'eval_NanoNFCorpus_256_dot_map@100': 0.14550655980989144,
 'eval_NanoNFCorpus_256_dot_mrr@10': 0.5030793650793651,
 'eval_NanoNFCorpus_256_dot_ndcg@10': 0.322530026305723,
 'eval_NanoNFCorpus_256_dot_precision@1': 0.42,
 'eval_NanoNFCorpus_256_dot_precision@10': 0.26599999999999996,
 'eval_NanoNFCorpus_256_dot_precision@3': 0.34666666666666657,
 'eval_NanoNFCorpus_256_dot_precision@5': 0.3,
 'eval_NanoNFCorpus_256_dot_recall@1': 0.045740663443342384,
 'eval_NanoNFCorpus_256_dot_recall@10': 0.12463870072229012,
 'eval_NanoNFCorpus_256_dot_recall@3': 0.0738417312271257,
 'eval_NanoNFCorpus_256_dot_recall@5': 0.0905822845739647,
 'eval_NanoNFCorpus_256_row_non_zero_mean_corpus': 256.0,
 'eval_NanoNFCorpus_256_row_non_zero_mean_query': 256.0,
 'eval_NanoNFCorpus_256_row_sparsity_mean_corpus': 0.9375,
 'eval_NanoNFCorpus_256_row_sparsity_mean_query': 0.9375,
 'eval_NanoNQ_128_dot_accuracy@1': 0.42,
 'eval_NanoNQ_128_dot_accuracy@10': 0.76,
 'eval_NanoNQ_128_dot_accuracy@3': 0.54,
 'eval_NanoNQ_128_dot_accuracy@5': 0.66,
 'eval_NanoNQ_128_dot_map@100': 0.5125546890089949,
 'eval_NanoNQ_128_dot_mrr@10': 0.5129047619047619,
 'eval_NanoNQ_128_dot_ndcg@10': 0.5552074220071551,
 'eval_NanoNQ_128_dot_precision@1': 0.42,
 'eval_NanoNQ_128_dot_precision@10': 0.08,
 'eval_NanoNQ_128_dot_precision@3': 0.18,
 'eval_NanoNQ_128_dot_precision@5': 0.132,
 'eval_NanoNQ_128_dot_recall@1': 0.41,
 'eval_NanoNQ_128_dot_recall@10': 0.72,
 'eval_NanoNQ_128_dot_recall@3': 0.52,
 'eval_NanoNQ_128_dot_recall@5': 0.6,
 'eval_NanoNQ_128_row_non_zero_mean_corpus': 128.0,
 'eval_NanoNQ_128_row_non_zero_mean_query': 128.0,
 'eval_NanoNQ_128_row_sparsity_mean_corpus': 0.96875,
 'eval_NanoNQ_128_row_sparsity_mean_query': 0.96875,
 'eval_NanoNQ_256_dot_accuracy@1': 0.44,
 'eval_NanoNQ_256_dot_accuracy@10': 0.84,
 'eval_NanoNQ_256_dot_accuracy@3': 0.66,
 'eval_NanoNQ_256_dot_accuracy@5': 0.76,
 'eval_NanoNQ_256_dot_map@100': 0.5562374973777413,
 'eval_NanoNQ_256_dot_mrr@10': 0.5726666666666667,
 'eval_NanoNQ_256_dot_ndcg@10': 0.6162310479920677,
 'eval_NanoNQ_256_dot_precision@1': 0.44,
 'eval_NanoNQ_256_dot_precision@10': 0.09,
 'eval_NanoNQ_256_dot_precision@3': 0.22,
 'eval_NanoNQ_256_dot_precision@5': 0.15200000000000002,
 'eval_NanoNQ_256_dot_recall@1': 0.42,
 'eval_NanoNQ_256_dot_recall@10': 0.8,
 'eval_NanoNQ_256_dot_recall@3': 0.62,
 'eval_NanoNQ_256_dot_recall@5': 0.71,
 'eval_NanoNQ_256_row_non_zero_mean_corpus': 256.0,
 'eval_NanoNQ_256_row_non_zero_mean_query': 256.0,
 'eval_NanoNQ_256_row_sparsity_mean_corpus': 0.9375,
 'eval_NanoNQ_256_row_sparsity_mean_query': 0.9375,
 'eval_base_loss': 0.0852,
 'eval_loss': 0.33621135354042053,
 'eval_reconstruction_loss': 0.2489,
 'eval_runtime': 97.7658,
 'eval_samples_per_second': 10.229,
 'eval_sequential_score': 0.5106517684276061,
 'eval_steps': 0.0,
 'eval_steps_per_second': 0.164}
```
Quite a lot, but users can filter later in W&B etc.

- Tom Aarsen


Diff:
diff --git a/examples/sparse_encoder/training/retrievers/train_csr_nq.py b/examples/sparse_encoder/training/retrievers/train_csr_nq.py
index 2b5642aeb..389af09be 100644
--- a/examples/sparse_encoder/training/retrievers/train_csr_nq.py
+++ b/examples/sparse_encoder/training/retrievers/train_csr_nq.py
@@ -40,16 +40,18 @@ def main():
         model_card_data=SparseEncoderModelCardData(
             language="en",
             license="apache-2.0",
-            model_name="csr-mpnet-base trained on Natural Questions",
+            model_name="Sparse CSR model trained on Natural Questions",
         ),
     )
 
-    model._first_module().requires_grad_(False)  # print the proportion of parameters that are trainable
+    # Freeze the first module of the model, i.e. the encoder, & print the number of (trainable) parameters
+    model[0].requires_grad_(False)
     num_params = sum(p.numel() for p in model.parameters())
     num_trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
     print(
-        f"Total parameters: {num_params}, Trainable parameters: {num_trainable_params}, Trainable ratio: {num_trainable_params / num_params:.2%}"
+        f"Total parameters: {num_params:,}, Trainable parameters: {num_trainable_params:,}, Trainable ratio: {num_trainable_params / num_params:.2%}"
     )
+
     # 2a. Load the NQ dataset: https://huggingface.co/datasets/sentence-transformers/natural-questions
     logging.info("Read the Natural Questions training dataset")
     full_dataset = load_dataset("sentence-transformers/natural-questions", split="train").select(range(100_000))
@@ -67,6 +69,7 @@ def main():
     for k_dim in [128, 256]:
         evaluators.append(evaluation.SparseNanoBEIREvaluator(["msmarco", "nfcorpus", "nq"], max_active_dims=k_dim))
     dev_evaluator = SequentialEvaluator(evaluators, main_score_function=lambda scores: scores[-1])
+    dev_evaluator(model)
 
     # 5. Define the training arguments
     short_model_name = model_name if "/" not in model_name else model_name.split("/")[-1]
@@ -83,13 +86,14 @@ def main():
         bf16=True,  # Set to True if you have a GPU that supports BF16
         batch_sampler=BatchSamplers.NO_DUPLICATES,  # MultipleNegativesRankingLoss benefits from no duplicate samples in a batch
         load_best_model_at_end=True,
-        metric_for_best_model="eval_NanoBEIR_mean_dot_ndcg@10",
+        metric_for_best_model="eval_NanoBEIR_mean_256_dot_ndcg@10",
         # Optional tracking/debugging parameters:
         eval_strategy="steps",
-        eval_steps=309,
+        eval_steps=300,
         save_strategy="steps",
-        save_steps=309,
-        logging_steps=10,
+        save_steps=300,
+        save_total_limit=3,
+        logging_steps=100,
         run_name=run_name,  # Will be used in W&B if `wandb` is installed
     )
 
diff --git a/sentence_transformers/evaluation/NanoBEIREvaluator.py b/sentence_transformers/evaluation/NanoBEIREvaluator.py
index 8dd0fd0d4..a473b8334 100644
--- a/sentence_transformers/evaluation/NanoBEIREvaluator.py
+++ b/sentence_transformers/evaluation/NanoBEIREvaluator.py
@@ -297,18 +297,17 @@ def __call__(
             self.score_function_names = [model.similarity_fn_name]
             self._append_csv_headers(self.score_function_names)
 
+        num_underscores_in_name = self.name.count("_")
         for evaluator in tqdm(self.evaluators, desc="Evaluating datasets", disable=not self.show_progress_bar):
             logger.info(f"Evaluating {evaluator.name}")
             evaluation = evaluator(model, output_path, epoch, steps)
-            for k in evaluation:
-                if self.truncate_dim:
-                    dataset, _, metric = k.split("_", maxsplit=2)
-                else:
-                    dataset, metric = k.split("_", maxsplit=1)
+            for full_key, metric_value in evaluation.items():
+                splits = full_key.split("_", maxsplit=num_underscores_in_name)
+                metric = splits[-1]
                 if metric not in per_metric_results:
                     per_metric_results[metric] = []
-                per_dataset_results[dataset + "_" + metric] = evaluation[k]
-                per_metric_results[metric].append(evaluation[k])
+                per_dataset_results[full_key] = metric_value
+                per_metric_results[metric].append(metric_value)
 
         agg_results = {}
         for metric in per_metric_results:
diff --git a/sentence_transformers/sparse_encoder/evaluation/SparseBinaryClassificationEvaluator.py b/sentence_transformers/sparse_encoder/evaluation/SparseBinaryClassificationEvaluator.py
index d9df01b3d..fa3f7196c 100644
--- a/sentence_transformers/sparse_encoder/evaluation/SparseBinaryClassificationEvaluator.py
+++ b/sentence_transformers/sparse_encoder/evaluation/SparseBinaryClassificationEvaluator.py
@@ -185,3 +185,9 @@ def store_metrics_in_model_card_data(
         self, model: SparseEncoder, metrics: dict[str, Any], epoch: int = 0, step: int = 0
     ) -> None:
         model.model_card_data.set_evaluation_metrics(self, metrics, epoch=epoch, step=step)
+
+    def get_config_dict(self) -> dict[str, Any]:
+        config_dict = super().get_config_dict()
+        if self.max_active_dims is not None:
+            config_dict["max_active_dims"] = self.max_active_dims
+        return config_dict
diff --git a/sentence_transformers/sparse_encoder/evaluation/SparseEmbeddingSimilarityEvaluator.py b/sentence_transformers/sparse_encoder/evaluation/SparseEmbeddingSimilarityEvaluator.py
index 8423ff677..eb71a5f37 100644
--- a/sentence_transformers/sparse_encoder/evaluation/SparseEmbeddingSimilarityEvaluator.py
+++ b/sentence_transformers/sparse_encoder/evaluation/SparseEmbeddingSimilarityEvaluator.py
@@ -160,3 +160,9 @@ def store_metrics_in_model_card_data(
         self, model: SparseEncoder, metrics: dict[str, Any], epoch: int = 0, step: int = 0
     ) -> None:
         model.model_card_data.set_evaluation_metrics(self, metrics, epoch=epoch, step=step)
+
+    def get_config_dict(self) -> dict[str, Any]:
+        config_dict = super().get_config_dict()
+        if self.max_active_dims is not None:
+            config_dict["max_active_dims"] = self.max_active_dims
+        return config_dict
diff --git a/sentence_transformers/sparse_encoder/evaluation/SparseInformationRetrievalEvaluator.py b/sentence_transformers/sparse_encoder/evaluation/SparseInformationRetrievalEvaluator.py
index ed16ddb5e..19a4e4aee 100644
--- a/sentence_transformers/sparse_encoder/evaluation/SparseInformationRetrievalEvaluator.py
+++ b/sentence_transformers/sparse_encoder/evaluation/SparseInformationRetrievalEvaluator.py
@@ -255,3 +255,9 @@ def store_metrics_in_model_card_data(
         self, model: SparseEncoder, metrics: dict[str, Any], epoch: int = 0, step: int = 0
     ) -> None:
         model.model_card_data.set_evaluation_metrics(self, metrics, epoch=epoch, step=step)
+
+    def get_config_dict(self) -> dict[str, Any]:
+        config_dict = super().get_config_dict()
+        if self.max_active_dims is not None:
+            config_dict["max_active_dims"] = self.max_active_dims
+        return config_dict
diff --git a/sentence_transformers/sparse_encoder/evaluation/SparseMSEEvaluator.py b/sentence_transformers/sparse_encoder/evaluation/SparseMSEEvaluator.py
index 19ce6cc84..79e954e6b 100644
--- a/sentence_transformers/sparse_encoder/evaluation/SparseMSEEvaluator.py
+++ b/sentence_transformers/sparse_encoder/evaluation/SparseMSEEvaluator.py
@@ -162,3 +162,9 @@ def store_metrics_in_model_card_data(
         step: int = 0,
     ) -> None:
         model.model_card_data.set_evaluation_metrics(self, metrics, epoch=epoch, step=step)
+
+    def get_config_dict(self) -> dict[str, Any]:
+        config_dict = super().get_config_dict()
+        if self.max_active_dims is not None:
+            config_dict["max_active_dims"] = self.max_active_dims
+        return config_dict
diff --git a/sentence_transformers/sparse_encoder/evaluation/SparseNanoBEIREvaluator.py b/sentence_transformers/sparse_encoder/evaluation/SparseNanoBEIREvaluator.py
index 2a5841965..b07d3040b 100644
--- a/sentence_transformers/sparse_encoder/evaluation/SparseNanoBEIREvaluator.py
+++ b/sentence_transformers/sparse_encoder/evaluation/SparseNanoBEIREvaluator.py
@@ -3,7 +3,7 @@
 import logging
 import os
 from collections import defaultdict
-from typing import TYPE_CHECKING, Callable
+from typing import TYPE_CHECKING, Any, Callable
 
 import numpy as np
 
@@ -192,6 +192,14 @@ def __init__(
             query_prompts=query_prompts,
             corpus_prompts=corpus_prompts,
         )
+        if self.max_active_dims is not None:
+            self.name += f"_{self.max_active_dims}"
+
+    def _get_human_readable_name(self, dataset_name: DatasetNameType) -> str:
+        human_readable_name = super()._get_human_readable_name(dataset_name)
+        if self.max_active_dims is not None:
+            human_readable_name += f"_{self.max_active_dims}"
+        return human_readable_name
 
     def _append_csv_headers(self, similarity_fn_names):
         super()._append_csv_headers(similarity_fn_names)
@@ -242,3 +250,9 @@ def _load_dataset(
         ir_evaluator_kwargs["max_active_dims"] = self.max_active_dims
         ir_evaluator_kwargs.pop("truncate_dim", None)
         return super()._load_dataset(dataset_name, **ir_evaluator_kwargs)
+
+    def get_config_dict(self) -> dict[str, Any]:
+        config_dict = super().get_config_dict()
+        if self.max_active_dims is not None:
+            config_dict["max_active_dims"] = self.max_active_dims
+        return config_dict
diff --git a/sentence_transformers/sparse_encoder/evaluation/SparseRerankingEvaluator.py b/sentence_transformers/sparse_encoder/evaluation/SparseRerankingEvaluator.py
index 96cf7ace2..5c6cb8e9a 100644
--- a/sentence_transformers/sparse_encoder/evaluation/SparseRerankingEvaluator.py
+++ b/sentence_transformers/sparse_encoder/evaluation/SparseRerankingEvaluator.py
@@ -196,3 +196,9 @@ def store_metrics_in_model_card_data(
         self, model: SparseEncoder, metrics: dict[str, Any], epoch: int = 0, step: int = 0
     ) -> None:
         model.model_card_data.set_evaluation_metrics(self, metrics, epoch=epoch, step=step)
+
+    def get_config_dict(self) -> dict[str, Any]:
+        config_dict = super().get_config_dict()
+        if self.max_active_dims is not None:
+            config_dict["max_active_dims"] = self.max_active_dims
+        return config_dict
diff --git a/sentence_transformers/sparse_encoder/evaluation/SparseTranslationEvaluator.py b/sentence_transformers/sparse_encoder/evaluation/SparseTranslationEvaluator.py
index 5e4afb293..078392730 100644
--- a/sentence_transformers/sparse_encoder/evaluation/SparseTranslationEvaluator.py
+++ b/sentence_transformers/sparse_encoder/evaluation/SparseTranslationEvaluator.py
@@ -150,3 +150,9 @@ def store_metrics_in_model_card_data(
         self, model: SparseEncoder, metrics: dict[str, Any], epoch: int = 0, step: int = 0
     ) -> None:
         model.model_card_data.set_evaluation_metrics(self, metrics, epoch=epoch, step=step)
+
+    def get_config_dict(self) -> dict[str, Any]:
+        config_dict = super().get_config_dict()
+        if self.max_active_dims is not None:
+            config_dict["max_active_dims"] = self.max_active_dims
+        return config_dict
diff --git a/sentence_transformers/sparse_encoder/evaluation/SparseTripletEvaluator.py b/sentence_transformers/sparse_encoder/evaluation/SparseTripletEvaluator.py
index df30e8533..a8054dc6d 100644
--- a/sentence_transformers/sparse_encoder/evaluation/SparseTripletEvaluator.py
+++ b/sentence_transformers/sparse_encoder/evaluation/SparseTripletEvaluator.py
@@ -183,3 +183,9 @@ def store_metrics_in_model_card_data(
         self, model: SparseEncoder, metrics: dict[str, Any], epoch: int = 0, step: int = 0
     ) -> None:
         model.model_card_data.set_evaluation_metrics(self, metrics, epoch=epoch, step=step)
+
+    def get_config_dict(self) -> dict[str, Any]:
+        config_dict = super().get_config_dict()
+        if self.max_active_dims is not None:
+            config_dict["max_active_dims"] = self.max_active_dims
+        return config_dict


Reviews:
user: login: copilot-pull-request-reviewer[bot]
body: ## Pull Request Overview

This pull request adds support for tracking the max_active_dims parameter in the configuration dictionary for various sparse evaluators and ensures that evaluators’ names are automatically updated to include this parameter. In addition, it makes minor updates to the training script to freeze the encoder module correctly and adjust logging metrics.
- Added get_config_dict implementations to include max_active_dims when present.
- Updated SparseNanoBEIREvaluator to modify its name and human-readable name based on max_active_dims.
- Modified train_csr_nq.py to freeze the model’s first module using indexing, update model naming, and adjust evaluation metric settings.

### Reviewed Changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

<details>
<summary>Show a summary per file</summary>

| File                                                                                           | Description                                                               |
| ---------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- |
| sentence_transformers/sparse_encoder/evaluation/SparseTripletEvaluator.py                      | Added max_active_dims to the config dict                                  |
| sentence_transformers/sparse_encoder/evaluation/SparseTranslationEvaluator.py                    | Added max_active_dims to the config dict                                  |
| sentence_transformers/sparse_encoder/evaluation/SparseRerankingEvaluator.py                      | Added max_active_dims to the config dict                                  |
| sentence_transformers/sparse_encoder/evaluation/SparseNanoBEIREvaluator.py                       | Updated evaluator name and human-readable name based on max_active_dims     |
| sentence_transformers/sparse_encoder/evaluation/SparseMSEEvaluator.py                           | Added max_active_dims to the config dict                                  |
| sentence_transformers/sparse_encoder/evaluation/SparseInformationRetrievalEvaluator.py           | Added max_active_dims to the config dict                                  |
| sentence_transformers/sparse_encoder/evaluation/SparseEmbeddingSimilarityEvaluator.py            | Added max_active_dims to the config dict                                  |
| sentence_transformers/sparse_encoder/evaluation/SparseBinaryClassificationEvaluator.py           | Added max_active_dims to the config dict                                  |
| sentence_transformers/evaluation/NanoBEIREvaluator.py                                          | Adjusted splitting of metric keys using dynamic underscore count          |
| examples/sparse_encoder/training/retrievers/train_csr_nq.py                                      | Updated module freezing, model naming and logging format                  |
</details>





state: COMMENTED
author_association: NONE
submitted_at: 2025-05-21T13:38:13Z

Review comments:
diff_hunk: @@ -196,6 +196,14 @@ def __init__(
             query_prompts=query_prompts,
             corpus_prompts=corpus_prompts,
         )
+        if self.max_active_dims:
path: sentence_transformers/sparse_encoder/evaluation/SparseNanoBEIREvaluator.py
user: login: Copilot
body: For consistency and clarity, consider checking against None (i.e., 'if self.max_active_dims is not None:') to avoid potential issues if 0 is ever passed as max_active_dims.
```suggestion
        if self.max_active_dims is not None:
```
created_at: 2025-05-21T13:38:12Z
author_association: NONE
diff_hunk: @@ -40,16 +40,18 @@ def main():
         model_card_data=SparseEncoderModelCardData(
             language="en",
             license="apache-2.0",
-            model_name="csr-mpnet-base trained on Natural Questions",
+            model_name="Sparse CSR model trained on Natural Questions",
         ),
     )
 
-    model._first_module().requires_grad_(False)  # print the proportion of parameters that are trainable
+    # Freeze the first module of the model, i.e. the encoder, & print the number of (trainable) parameters
+    model[0].requires_grad_(False)
path: examples/sparse_encoder/training/retrievers/train_csr_nq.py
user: login: Copilot
body: [nitpick] Consider adding a comment or documentation note that freezing the encoder via model[0] assumes the encoder is the first module, to guide future maintainers if the model structure changes.
created_at: 2025-05-21T13:38:13Z
author_association: NONE

PR comments:
user: login: arthurbr11
created_at: 2025-05-21T13:49:18Z
body: I looked a bit at the model card and i'm confused as i don't see the `max_active_dims` mention in the `Output Dimensionality: 4096 dimensions`. Do you know why ? You did train it with a `max_active_dims` set right ? Also in addition to this we might want to mention it in the introduction sentences ?
author_association: OWNER
user: login: tomaarsen
created_at: 2025-05-21T14:40:35Z
body: I didn't do that part in this PR. This is just improving the evaluators.

I do agree that we'll want to add something like this, but that's more for another PR:
```
Output Dimensionality: 4096 dimensions (256 active)
```

- Tom Aarsen
author_association: COLLABORATOR
user: login: arthurbr11
created_at: 2025-05-21T21:24:29Z
body: I'm wandering for the last `SparseNanoBEIREvaluator`  that we are doing (the one on all the dataset )  if it's not confusing to not have the dimension as we have it in all the other ? 
Should it seems reasonable to you to put the `max_active_dims`of the SparseNanoBEIR equal to the one of the `CSRSparsity` if the module is present and if the `max_active_dims` isn't pass to the evaluator ? I specify only for `CSRSparsity` as for Splade model we might want to evaluate a model with a given `max_active_dims` without having it during the evaluation. 
Let me know if it make sense for you ! 
author_association: OWNER
user: login: tomaarsen
created_at: 2025-05-22T08:18:43Z
body: I don't really see the benefit in complicating the code/having the `max_active_dims` (unexpectedly) automatically set in certain settings, considering the evaluation will be the same. I think `max_active_dims` should just be for additional restrictions, it's not needed if the model is already restricted to e.g. 256 dims due to its architecture I think.

- Tom Aarsen
author_association: COLLABORATOR
user: login: arthurbr11
created_at: 2025-05-22T08:24:25Z
body: Yes that make sense, but then we definitely want to mention it in the model card before. Something telling that without mention of specific value this model is by default having this as `max_active_dims`. Because else it’s confusing when we see the `SparseNanoBEIR` without any mention of dim I feel.
author_association: OWNER