(sort of) a bug with token offsets: some special tokens have (0, 0) offsets regardless of their position in the document
Author: viktor-shcherb
Pull request: no
Labels: name: Usage

name: bug

### System Info

- `transformers` version: 4.46.3
- Platform: macOS-14.4-arm64-arm-64bit
- Python version: 3.11.9
- Huggingface_hub version: 0.26.3
- Safetensors version: 0.4.5
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.5.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
xlm_tok = AutoTokenizer.from_pretrained('facebook/xlm-v-base')

xlm_tok('test', return_offsets_mapping=True)
```
Output:
```
{'input_ids': [0, 1340, 2], 'attention_mask': [1, 1, 1], 'offset_mapping': [(0, 0), (0, 4), (0, 0)]}
```

### Expected behavior

The special tokens should have the offset that reflects their position in the document. In this case, (4, 4) instead of (0, 0).

Why is it a remotely a big deal? To simplify the code that iterates through the offsets. For example this piece of code will not work now:

```python
def tokenize(example: dict, tokenizer: PreTrainedTokenizer, tokenizer_name: str, max_length: int = 512) -> dict:
    ner_tags: list[int] = example['ner_tags']
    example_words: list[str] = example['tokens']
    text = ' '.join(example_words)
    
    # map words to positions in text
    word_positions: list[int] = example.get('word_positions', [])
    
    if len(word_positions) != len(example_words):
        text_iterator = 0
        for word in example_words:
            while text[text_iterator:text_iterator + len(word)] != word:
                text_iterator += 1
                assert text_iterator < len(text)
            
            word_positions.append(text_iterator)
    
    encoding: BatchEncoding = tokenizer(text, return_offsets_mapping=True, truncation=True, max_length=max_length)
    num_sub_tokens = len(encoding.offset_mapping)
    
    sub_token_iterator = 0
    sub_token_ner_tags: list[int] = []
    for word_id, ner_tag in enumerate(ner_tags):
        word_start = word_positions[word_id]
        word_end = word_start + len(example_words[word_id])
        
        # there may be some empty space between words. the sub tokens that include this empty space receive O label
        # we compare with the end ([1]) to ensure that 0-length tokens are labelled as O (for example <CLS>)
        while sub_token_iterator < num_sub_tokens and  encoding.offset_mapping[sub_token_iterator][1] <= word_start:
            sub_token_iterator += 1
            sub_token_ner_tags.append(0)  # 0 = O
            
        ext_tag = ner_tags_ext[ner_tag]
        
        if sub_token_iterator < num_sub_tokens:
            # the first sub token of a word receives original label, the rest receive extended label
            sub_token_ner_tags.append(ner_tag)
            sub_token_iterator += 1
        
        # again, we need to be careful about 0-length tokens, so we compare start ([0]) with the word end
        while sub_token_iterator < num_sub_tokens and encoding.offset_mapping[sub_token_iterator][0] < word_end:
            sub_token_iterator += 1
            sub_token_ner_tags.append(ext_tag)
    
    # any tokens at the end (like <SEP>) receive O tokens
    while sub_token_iterator < num_sub_tokens:
        sub_token_iterator += 1
        sub_token_ner_tags.append(0)
        
    return {
        'word_positions': word_positions,
        f'{tokenizer_name}_sub_tokens': encoding.input_ids,
        f'{tokenizer_name}_sub_token_offsets': encoding.offset_mapping,
        f'{tokenizer_name}_sub_token_ner_tags': sub_token_ner_tags,
    }
```

It produces entities like `Peter Blackburn
` for `"Peter Blackburn"` text. Not a big deal, but annoying to work around and hard to catch.

Comments:
cc @arthurzucker @itazap for tokenizers

Hey! Sorry but the position of the special token is that there are not special tokens in `test`. 
If you want to have the offset, they need to be in the document: 
```python 
>>> xlm_tok('
test', return_offsets_mapping=True, add_special_tokens=False)
{'input_ids': [0, 1340], 'attention_mask': [1, 1], 'offset_mapping': [(0, 3), (3, 7)]}
```

Thank you for the clarification!

I feel like having 0-length sort of "ethereal" special tokens with correct (start, end) positions would be a lot more logical. I don't see a case where having (0, 0) positions for any special token is more practical.

But anyway, this could be just a documentation issue, I don't see this tokenization edge case documented anywhere https://huggingface.co/docs/transformers/en/main_classes/tokenizer#transformers.PreTrainedTokenizer.__call__.return_offsets_mapping

Agree with you, the idea is that these token simply do not exist in the string and as such we cannot add a correct offset. ex: for the last one, 7,7 would just mean the same as if it was actually part of the text. 

You can open a pr to update the documentation if you want! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.