RobertaTokenizer has incorrect character offsets
Author: Craigacp
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.40.1
- Platform: macOS-14.6.1-arm64-arm-64bit
- Python version: 3.9.19
- Huggingface_hub version: 0.23.4
- Safetensors version: 0.4.3
- Accelerate version: 0.31.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.3.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I have byte offsets for a tagging task which I am trying to train a transformer with. Using the `Salesforce/codet5-base` tokenizer (which uses RobertaTokenizer) I tokenize the input and then call `char_to_token` to find out which tokens my byte offset lands in so I can build the label tensor. For some inputs `char_to_token` returns `None`, even when the character range in question appears in a token.

Running the following code:
```python
        test_string = """("    " +"""
        tok = AutoTokenizer.from_pretrained("Salesforce/codet5-base", local_files_only=True)

        enc = tok(text=test_string, truncation=False, add_special_tokens=False)

        char_mapping = []
        for i in range(len(enc["input_ids"])):
            char_mapping.append((i, enc.token_to_chars(i), tok.decode(enc["input_ids"][i])))
        print(f"\nchar_mapping = {char_mapping}")

        tok_mapping = []
        for i in range(len(test_string)):
            tok_mapping.append((i, enc.char_to_token(i)))
        print(f"\ntok_mapping = {tok_mapping}")
```
produces this output:
```python
char_mapping = [(0, CharSpan(start=0, end=2), '("'), 
                (1, CharSpan(start=5, end=5), '   '), 
                (2, CharSpan(start=6, end=7), ' "'), 
                (3, CharSpan(start=8, end=9), ' +')]

tok_mapping = [(0, 0), (1, 0), (2, None), (3, None), (4, None), (5, None), (6, 2), (7, None), (8, 3)]
```
Note that token 1 has a zero length character span despite it containing 3 space characters. The corresponding `char_to_token` call for those characters produces `None`. Also tokens 2 and 3 each contain 2 characters, but have single character spans, and again the corresponding `char_to_token` call for the start characters of those tokens returns `None`.

Part of my tagging task involves tagging string literals with a "string literal" label, but whenever the string contains whitespace I hit these `None` values and I can't find the correct token to tag.

### Expected behavior

`char_to_token` should return the correct token id when the token contains the specified character offset.

Comments:
cc @itazap as well!

Hey @Craigacp ! Thanks for providing a reproducer 😊 `trim_offsets=False` might be your fix! With this, I get the following output below, please let me know if it works for you!

```python
AutoTokenizer.from_pretrained("Salesforce/codet5-base", trim_offsets=False)

# output
char_mapping = 
[(0, CharSpan(start=0, end=2), '("'), 
(1, CharSpan(start=2, end=5), '   '), 
(2, CharSpan(start=5, end=7), ' "'), 
(3, CharSpan(start=7, end=9), ' +')]

tok_mapping = [(0, 0), (1, 0), (2, 1), (3, 1), (4, 1), (5, 2), (6, 2), (7, 3), (8, 3)]

```

Ok, it looks like that behaviour has been there for a long time, but it's quite confusing. I understand trimming it if the tokenizer itself added the whitespace, but given there's whitespace in the input I'd expect it to leave that alone by default. It makes the behaviour of `char_to_token` hard to operate as I need to check if each character is whitespace before doing the mapping, though `token_to_chars` is still fairly straightforward.

The end offset looks like it's correct, and using that I built something that works around the issue but it makes several assumptions. I guess I'll check if supplying `trim_offsets` in my larger run fixes the issue.

RoberteTokenizer has `trim_offsets=True` by default, perhaps that was a feature in the original implementation of this model by its authors (I would have to look into the original paper) and result in the tokens being trimmed, so matching the `char` to a `token` can only matched to a trimmed 'final' token, which can be different from the input due to post-processing.  I would agree that the behaviour with `trim_offests=False` is more intuitive in this case and hope that supplying the param works for your case! 

Is it possible to modify the documentation of `char_to_token` (https://huggingface.co/docs/transformers/en/main_classes/tokenizer#transformers.BatchEncoding.char_to_token) so it says that it can return `None` if the whitespace is trimmed (and if there are any other situations where this happens).

yes sounds good! The docs would automatically be updated after updating the docstring here: https://github.com/huggingface/transformers/blob/9230d78e76611cfa38c845213021aeb185362d10/src/transformers/tokenization_utils_base.py#L588-L590 

Perhaps it can be udpated with adding `, returns None if whitespace-only token and whitespace is trimmed (trim_offsets=True)`
 
 @ArthurZucker wdyt? 

yeah sure!

@Craigacp do you want to open a PR for that? 

Sure. Are there any other options which could cause the return value to be None?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@Craigacp there should not be other cases 😅 

Ok, I'll make a PR for the docs update.