Repository: huggingface/text-embeddings-inference (Rust)

Issue:
ModernBert reranker - Different results from vs transformers library
### System Info

Transformers version - 4.51.3
TEI - Latest main branch
```
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.20             Driver Version: 570.133.20     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA L40S                    On  |   00000000:36:00.0 Off |                    0 |
| N/A   35C    P0             79W /  350W |   20049MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
```


## Description
I'm very green to this field so apologies in advance if I ask some stupid questions or make some assumptions that are incorrect.
___

I have a Modernbert reranker initially run on the `Transformers` library and saw that `TEI` would perform inference faster so I migrated them over. What I observed was different outputs, initially I thought this was fine but the differences are quite large so I stepped into the different processes from tokenisation to pooling strategy and the activation function. What I found was two areas where the two libraries differ that produce a significant difference between the two libraries ouputs:

1. Tokenisation - Modernbert on TEI does not add additional padding (maybe this is fine?)
2. The pooling strategy used by default is CLS, where as in `Transformers` it uses the value in `config.json` and it just so happens Toma's and (the reranker I want to use is mean)

I tested both libraries on the same machine where TEI was run based on the Dockerfile-cuda. One small note is that I'm running TEI on `float16` (I didn't have enough memory to run with float32) but I don't think this would drastically change the results.

I made a simple high level example:

```python
query = "How do I sell my shirt?"
texts = [
    "You can sell your shirt by going to the sell page and clicking the sell button.",
    "Ketchup is a condiment that is made from tomatoes.",
    "You can sell your apple in the store.",
    "How you can sell your clothes online."
]
pairs = [[query, text] for text in texts]


# Transformers: 
[{'index': 0, 'score': 0.9871538},
 {'index': 3, 'score': 0.60760754},
 {'index': 2, 'score': 0.0041681635},
 {'index': 1, 'score': 2.5670563e-05}]

# TEI:
[{'index': 0, 'score': 0.9993761},
 {'index': 3, 'score': 0.29017562},
 {'index': 2, 'score': 0.0047737225},
 {'index': 1, 'score': 1.2219076e-05}]
```

### Model
The model that i'm running: https://huggingface.co/tomaarsen/reranker-ModernBERT-large-gooaq-bce

```
{
  "model_id": "../models/toma-reranker",
  "model_sha": null,
  "model_dtype": "float16",
  "model_type": {
    "reranker": {
      "id2label": {
        "0": "LABEL_0"
      },
      "label2id": {
        "LABEL_0": 0
      }
    }
  },
  "max_concurrent_requests": 512,
  "max_input_length": 8192,
  "max_batch_tokens": 16384,
  "max_batch_requests": null,
  "max_client_batch_size": 32,
  "auto_truncate": false,
  "tokenization_workers": 32,
  "version": "1.7.0",
  "sha": null,
  "docker_label": null
}
```


### Information

- [x] Docker
- [ ] The CLI directly

### Tasks

- [x] An officially supported command
- [ ] My own modifications

### Reproduction

Steps to reproduce:

Transformers version - 4.51.3
TEI - Latest main branch
Python - 3.11.7

1. Download [Modernbert reranker model](https://huggingface.co/tomaarsen/reranker-ModernBERT-large-gooaq-bce/tree/main)
2. Run `CrossEncoder` model from Transformers with the above model (float32)
3. Build and run Dockerfile-cuda from TEI (no flash attention so we only run ModernBert)
4. Execute both on the example:

```python
query = "How do I sell my shirt?"
texts = [
    "You can sell your shirt by going to the sell page and clicking the sell button.",
    "Ketchup is a condiment that is made from tomatoes.",
    "You can sell your apple in the store.",
    "How you can sell your clothes online."
]
pairs = [[query, text] for text in texts]
```

5. Sort transformer results so we can compare with TEI
```
transformers_sorted = sorted(
    ({"index": i, "score": score} for i, score in enumerate(response)),
    key=lambda x: x["score"],
    reverse=True
)
```

6. Observe results:
```
Transformers: 
[{'index': 0, 'score': 0.9871538},
 {'index': 3, 'score': 0.60760754},
 {'index': 2, 'score': 0.0041681635},
 {'index': 1, 'score': 2.5670563e-05}]
TEI:
[{'index': 0, 'score': 0.9993761},
 {'index': 3, 'score': 0.29017562},
 {'index': 2, 'score': 0.0047737225},
 {'index': 1, 'score': 1.2219076e-05}]
```

### Expected behavior

The results from Transformers and TEI to be similar.


Discussion: I'm getting different results on multiple runs but with Mean pooling I'm getting much closer results:

```
Transformers: 
[{'index': 0, 'score': 0.9871538},
 {'index': 3, 'score': 0.60760754},
 {'index': 2, 'score': 0.0041681635},
 {'index': 1, 'score': 2.5670563e-05}]

TEI:
[{'index': 0, 'score': 0.9871293},
 {'index': 3, 'score': 0.60696447},
 {'index': 2, 'score': 0.008187911},
 {'index': 1, 'score': 2.586743e-05}]
```



Fix (patch):
diff --git a/backends/candle/src/models/flash_modernbert.rs b/backends/candle/src/models/flash_modernbert.rs
index 4d4c8bb3..95b0cfb1 100644
--- a/backends/candle/src/models/flash_modernbert.rs
+++ b/backends/candle/src/models/flash_modernbert.rs
@@ -260,7 +260,11 @@ impl FlashModernBertModel {
 
         let (pool, classifier) = match model_type {
             ModelType::Classifier => {
-                let pool = Pool::Cls;
+                let pool: Pool = config
+                    .classifier_pooling
+                    .as_deref()
+                    .and_then(|s| Pool::from_str(s).ok())
+                    .unwrap_or(Pool::Cls);
 
                 let classifier: Box<dyn ClassificationHead + Send> =
                     Box::new(ModernBertClassificationHead::load(vb.clone(), config)?);
diff --git a/backends/candle/src/models/modernbert.rs b/backends/candle/src/models/modernbert.rs
index 12a9072a..ba913560 100644
--- a/backends/candle/src/models/modernbert.rs
+++ b/backends/candle/src/models/modernbert.rs
@@ -7,6 +7,7 @@ use candle::{DType, Device, IndexOp, Module, Result, Tensor, D};
 use candle_nn::{Embedding, VarBuilder};
 use serde::Deserialize;
 use text_embeddings_backend_core::{Batch, ModelType, Pool};
+use std::str::FromStr;
 
 // https://github.com/huggingface/transformers/blob/main/src/transformers/models/modernbert/configuration_modernbert.py
 #[derive(Debug, Clone, PartialEq, Deserialize)]
@@ -484,7 +485,11 @@ impl ModernBertModel {
     pub fn load(vb: VarBuilder, config: &ModernBertConfig, model_type: ModelType) -> Result<Self> {
         let (pool, classifier) = match model_type {
             ModelType::Classifier => {
-                let pool = Pool::Cls;
+                let pool: Pool = config
+                    .classifier_pooling
+                    .as_deref()
+                    .and_then(|s| Pool::from_str(s).ok())
+                    .unwrap_or(Pool::Cls);
 
                 let classifier: Box<dyn ClassificationHead + Send> =
                     Box::new(ModernBertClassificationHead::load(vb.clone(), config)?);
diff --git a/backends/core/src/lib.rs b/backends/core/src/lib.rs
index 3be719c1..7b927787 100644
--- a/backends/core/src/lib.rs
+++ b/backends/core/src/lib.rs
@@ -78,6 +78,23 @@ impl fmt::Display for Pool {
     }
 }
 
+impl std::str::FromStr for Pool {
+    type Err = String;
+
+    fn from_str(s: &str) -> Result<Self, Self::Err> {
+        match s.trim().to_lowercase().as_str() {
+            "cls" => Ok(Pool::Cls),
+            "mean" => Ok(Pool::Mean),
+            "splade" => Ok(Pool::Splade),
+            "last_token" => Ok(Pool::LastToken),
+            _ => Err(format!(
+                "Invalid pooling method '{}'. Valid options: cls, mean, splade, last_token", 
+                s
+            )),
+        }
+    }
+}
+
 #[derive(Debug, Error, Clone)]
 pub enum BackendError {
     #[error("No backend found")]