Repository: huggingface/tokenizers (language: rust)

Issue:
PreTrainedTokenizerFast `char_to_token` `token_to_char` not working as expected
### System Info

- `transformers` version: 4.44.0
- Platform: macOS-13.6.9-arm64-arm-64bit
- Python version: 3.11.4
- Huggingface_hub version: 0.23.4
- Safetensors version: 0.4.3
- Accelerate version: 0.32.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.4.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: no

### Who can help?
@ArthurZucker

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3.1-8B-Instruct")
text = "the quick brown fox jumps over the lazy dog"
out = tokenizer(text)
out.char_to_token(0)
```
This returns None for any char index not just 0

Also, `token_to_char` doesnt return expected results:
`out.token_to_chars(4)` returns 
`CharSpan(start=15, end=15)`
instead of `CharSpan(start=15, end=19)`

### Expected behavior

```
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3.1-8B-Instruct")
text = "the quick brown fox jumps over the lazy dog"
out = tokenizer(text)
out.char_to_token(0)
```
should return `1`

`out.token_to_chars(4)`
should return `CharSpan(start=15, end=19)`


Discussion / hints:
I think this is related to huggingface/transformers#25082 and is more related to `tokenizers` than `transformers`
I don't have a fix, but it's a but indeed
So should I open the issue in that repo? this is really needed for  huggingface/transformers#30650 
Yeah, it's basically the same as https://github.com/huggingface/tokenizers/issues/1553, since the offsets are wrong, the char to token that just uses them is also outputing wrong outputs. Let me transfer the issue! 
Any progress on this one?

Fix patch:
diff --git a/tokenizers/src/models/bpe/model.rs b/tokenizers/src/models/bpe/model.rs
index 1585da761..9cdd6213b 100644
--- a/tokenizers/src/models/bpe/model.rs
+++ b/tokenizers/src/models/bpe/model.rs
@@ -462,7 +462,11 @@ impl BPE {
     fn tokenize_with_cache(&self, sequence: &str) -> Result<Vec<Token>> {
         if self.ignore_merges {
             if let Some(id) = self.vocab.get(sequence) {
-                return Ok(vec![Token::new(*id, sequence.to_string().clone(), (0, 0))]);
+                return Ok(vec![Token::new(
+                    *id,
+                    sequence.to_string().clone(),
+                    (0, sequence.len()),
+                )]);
             }
         }
         if let Some(ref hit) = self.cache.as_ref().and_then(|c| c.get(sequence)) {
@@ -941,10 +945,13 @@ mod tests {
             .build()
             .unwrap();
         let tokens = bpe.tokenize(".:.:").unwrap();
-        assert_eq!(tokens, vec![Token::new(0u32, ".:.:".into(), (0, 0))]);
+        assert_eq!(tokens, vec![Token::new(0u32, ".:.:".into(), (0, 4))]);
 
         let tokens = bpe.tokenize("Ġbelirtilen").unwrap();
-        assert_eq!(tokens, vec![Token::new(1u32, "Ġbelirtilen".into(), (0, 0))]);
+        assert_eq!(
+            tokens,
+            vec![Token::new(1u32, "Ġbelirtilen".into(), (0, 12))]
+        );
 
         bpe.ignore_merges = false;
 
diff --git a/tokenizers/src/tokenizer/mod.rs b/tokenizers/src/tokenizer/mod.rs
index d92e04a1d..49bc539a2 100644
--- a/tokenizers/src/tokenizer/mod.rs
+++ b/tokenizers/src/tokenizer/mod.rs
@@ -1181,11 +1181,10 @@ where
                 };
 
                 trainer.feed(
-                    sequences.map(|s| {
+                    sequences.inspect(|s| {
                         if let Some(progress) = &progress {
                             progress.inc(s.len() as u64)
                         }
-                        s
                     }),
                     |seq| {
                         let normalized = self.do_normalize(seq.as_ref())?;
@@ -1233,11 +1232,10 @@ where
         };
 
         trainer.feed(
-            sequences.map(|s| {
+            sequences.inspect(|_s| {
                 if let Some(progress) = &progress {
                     progress.inc(1)
                 }
-                s
             }),
             |seq| {
                 let normalized = self.do_normalize(seq.as_ref())?;


Test patch: