Encoding.word_to_tokens() returns None within valid sequence
Author: carschno
Pull request: no

### System Info

- `transformers` version: 4.23.1
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.10.6
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.12.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes(?)
- Using distributed or parallel set-up in script?: no

### Who can help?

@SaulLu @sgugger @stevhliu 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Tokenize a sentence -> `BatchEncoding`
2. Iterate over `word_ids`
3. Call `word_to_chars(word_index)`
4. `TypeError` is raised at arbitrary word index (see output below)

```
MODEL_NAME = "DTAI-KULeuven/robbertje-1-gb-non-shuffled"
MODEL_MAX_LENGTH = 512

from transformers import AutoTokenizer


tokenizer = AutoTokenizer.from_pretrained(
    MODEL_NAME, model_max_length=MODEL_MAX_LENGTH, truncation=True
)
text = "Dit is een goede tekst."

encoding = tokenizer(text)


for word_index in range(len(encoding.word_ids())):
    if word_index is not None:
        print(word_index)
        char_span = encoding.word_to_chars(word_index)

0
1
2
3
4
5
6
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
tokenization_test.ipynb Cell 3 in <cell line: 1>()
      [2](vscode-notebook-cell:/tokenization_test.ipynb#W2sZmlsZQ%3D%3D?line=1) if word_index is not None:
      [3](vscode-notebook-cell:/tokenization_test.ipynb#W2sZmlsZQ%3D%3D?line=2)     print(word_index)
----> [4](vscode-notebook-cell:/tokenization_test.ipynb#W2sZmlsZQ%3D%3D?line=3)     char_span = encoding.word_to_chars(word_index)

File ~/opt/anaconda3/envs/SoS/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:615, in BatchEncoding.word_to_chars(self, batch_or_word_index, word_index, sequence_index)
    613     batch_index = 0
    614     word_index = batch_or_word_index
--> 615 return CharSpan(*(self._encodings[batch_index].word_to_chars(word_index, sequence_index)))

TypeError: transformers.tokenization_utils_base.CharSpan() argument after * must be an iterable, not NoneType
```

The word index is valid:
```
encoding.word_ids()[word_index:word_index+10]
[164, 165, 166, 166, 166, 166, 167, 168, 168, 168]
```

On further investigation, I have noticed that there is a work-around by validating there is a word-to-token mapping for the word index:
```
if word_index is not None and encoding.word_to_tokens(word_index) is not None:
    [...]
```
So the underlying issue seems to be that `word_to_tokens()` sometimes returns None, although it seems counter-intuitive that there are words in a texts that do not have corresponding tokens.


### Expected behavior

`BatchEncoding.word_to_tokens()` should not output `None`; or it should be [documented](https://huggingface.co/docs/transformers/main_classes/tokenizer#transformers.BatchEncoding.word_to_tokens) why/if this can happen. 

Comments:
Hi @carschno , 
Thank you very much for bringing the problem to our attention! It is indeed information filled in the docstring of the method but there is a problem when rendering this documentation on our site. I'm trying to fix it in this PR #20450 .

I see, it is a documentation issue. Thanks for looking into it!

I still think it is counter-intuitive that there can be words without corresponding tokens. I can imagine some special cases, but perhaps it would be a good occasion to elaborate or exemplify those cases in the documentation a bit?

> I still think it is counter-intuitive that there can be words without corresponding tokens. I can imagine some special cases, but perhaps it would be a good occasion to elaborate or exemplify those cases in the documentation a bit?

Thanks for your feedback! We can indeed give an example in the documentation. It is a case that occurs in particular when we ask to the tokenizer to add special tokens to match a template. For example, if it is asked to add a class token at the beginning of the sentence, this token class token does not correspond to anything in the initial raw sentence.

Thanks again, that makes sense! However, the example case you describe does not really fit the case I have encountered. No template has been involved there.

Originally, I came across the issue in a long, OCR'd text with many special characters and erroneous tokens due to OCR errors. But in the example I used in my investigations (and pasted here), this is not the case either.


> But in the example I used in my investigations (and pasted here), this is not the case either.

I could be wrong but it seems to me that your example uses a template. We can see it by running the following code:
```python
print(encoding.word_ids())
print(tokenizer.convert_ids_to_tokens(encoding.input_ids))
```
which gives:
```bash
[None, 0, 1, 2, 3, 4, 5, None]
['
', 'Dit', 'Ġis', 'Ġeen', 'Ġgoede', 'Ġtekst', '.', '
']
```
Here we can see that the `None` correspond to the "template" tokens `'
'` and `'
'`.


I suppose you are right. I had some doubts because in my aforementioned original text (long erroneous), this occurred somewhere in the middle of the text. I will try to reproduce, but I guess there might have been special tokens as well due to longer sequences of whitespace and/or punctuation.