issue with opus100/en-fr dataset 
State: closed | Author: dorost1234 (NONE)

Hi
I am running run_mlm.py code of huggingface repo with opus100/fr-en pair, I am getting this error, note that this error occurs for only this pairs and not the other pairs. Any idea why this is occurring? and how I can solve this? 

Thanks a lot  @lhoestq for your help in advance.

`
thread '<unnamed>' panicked at 'index out of bounds: the len is 617 but the index is 617', /__w/tokenizers/tokenizers/tokenizers/src/tokenizer/normalizer.rs:382:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
 63%|██████████████████████████████████████████████████████████▊                                   | 626/1000 [00:27<00:16, 22.69ba/s]

Traceback (most recent call last):
  File "run_mlm.py", line 550, in <module>
    main()
  File "run_mlm.py", line 412, in main
    in zip(data_args.dataset_name, data_args.dataset_config_name)]
  File "run_mlm.py", line 411, in <listcomp>
    logger) for dataset_name, dataset_config_name\
  File "/user/dara/dev/codes/seq2seq/data/tokenize_datasets.py", line 96, in get_tokenized_dataset
    load_from_cache_file=not data_args.overwrite_cache,
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/datasets/dataset_dict.py", line 448, in map
    for k, dataset in self.items()
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/datasets/dataset_dict.py", line 448, in <dictcomp>
    for k, dataset in self.items()
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1309, in map
    update_data=update_data,
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 204, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/datasets/fingerprint.py", line 337, in wrapper
    out = func(self, *args, **kwargs)
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1574, in _map_single
    batch, indices, check_same_num_examples=len(self.list_indexes()) > 0, offset=offset
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1490, in apply_function_on_filtered_inputs
    function(*fn_args, effective_indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
  File "/user/dara/dev/codes/seq2seq/data/tokenize_datasets.py", line 89, in tokenize_function
    return tokenizer(examples[text_column_name], return_special_tokens_mask=True)
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 2347, in __call__
    **kwargs,
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 2532, in batch_encode_plus
    **kwargs,
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/transformers/tokenization_utils_fast.py", line 384, in _batch_encode_plus
    is_pretokenized=is_split_into_words,
pyo3_runtime.PanicException: index out of bounds: the len is 617 but the index is 617

`

Comments: @lhoestq  I also deleted the cache and redownload the file and still the same issue, I appreciate any help on this. thanks 

Here please find the minimal code to reproduce the issue @lhoestq  note this only happens with MT5TokenizerFast

```
from datasets import load_dataset
from transformers import MT5TokenizerFast

def get_tokenized_dataset(dataset_name, dataset_config_name, tokenizer):
    datasets = load_dataset(dataset_name, dataset_config_name, script_version="master")
    column_names = datasets["train"].column_names
    text_column_name = "translation"
    def process_dataset(datasets):
        def process_function(examples):
            lang = "fr"
            return {"src_texts": [example[lang] for example in examples[text_column_name]]}
        datasets = datasets.map(
            process_function,
            batched=True,
            num_proc=None,
            remove_columns=column_names,
            load_from_cache_file=True,
        )
        return datasets
    datasets = process_dataset(datasets)
    text_column_name = "src_texts"
    column_names = ["src_texts"]
    def tokenize_function(examples):
            return tokenizer(examples[text_column_name], return_special_tokens_mask=True)
    tokenized_datasets = datasets.map(
            tokenize_function,
            batched=True,
            num_proc=None,
            remove_columns=column_names,
            load_from_cache_file=True
    )

if __name__ == "__main__":
     tokenizer_kwargs = {
        "cache_dir": None,
        "use_fast": True,
        "revision": "main",
        "use_auth_token": None
     }
     tokenizer = MT5TokenizerFast.from_pretrained("google/mt5-small", **tokenizer_kwargs)
     get_tokenized_dataset(dataset_name="opus100", dataset_config_name="en-fr", tokenizer=tokenizer)
~     
```

as per https://github.com/huggingface/tokenizers/issues/626 this looks like to be the tokenizer bug, I therefore, reported it there https://github.com/huggingface/tokenizers/issues/626 and  I am closing this one.