Problem with using custom tokenizers with run_mlm.py
Author: antmarakis
Pull request: no

Hi! I have an issue with running the `run_mlm.py` script with a tokenizer I myself trained. If I use pretrained tokenizers everything works.

Versions:

```
python: 3.8.3
transformers: 3.5.1
tokenizers: 0.9.4
torch: 1.7.0
```

This is how I train my tokenizer:

```
from tokenizers import BertWordPieceTokenizer

tokenizer = BertWordPieceTokenizer(lowercase=False, strip_accents=False, clean_text=True)
tokenizer.train(files=['/mounts/data/proj/antmarakis/wikipedia/wikipedia_en_1M.txt'], vocab_size=350, special_tokens=[
    "[PAD]",
    "[UNK]",
    "[CLS]",
    "[SEP]",
    "[MASK]",
])

tokenizer.save_model('wikipedia_en')
```

The above results in a vocab.txt file.

And this is how I try to train my model (using the `run_mlm.py` script):

```
python run_mlm.py \
--model_type bert \
--config_name bert_custom.json \
--train_file wikipedia_en_1M.txt \
--tokenizer_name wikipedia_en \
--output_dir lm_temp \
--do_train \
--num_train_epochs 1 \
--overwrite_output_dir
```

If I use a pretrained model/tokenizer, this script works (that is, I replace `config_name` and `tokenizer_name` with `model_name_or_path roberta-base` or something). But using the above code, I get the following error message:

```
Traceback (most recent call last):
  File "run_mlm.py", line 392, in <module>
    main()
  File "run_mlm.py", line 334, in main
    tokenized_datasets = tokenized_datasets.map(
  File "/mounts/Users/cisintern/antmarakis/.local/lib/python3.8/site-packages/datasets/dataset_dict.py", line 283, in map
    {
  File "/mounts/Users/cisintern/antmarakis/.local/lib/python3.8/site-packages/datasets/dataset_dict.py", line 284, in <dictcomp>
    k: dataset.map(
  File "/mounts/Users/cisintern/antmarakis/.local/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1240, in map
    return self._map_single(
  File "/mounts/Users/cisintern/antmarakis/.local/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 156, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/mounts/Users/cisintern/antmarakis/.local/lib/python3.8/site-packages/datasets/fingerprint.py", line 163, in wrapper
    out = func(self, *args, **kwargs)
  File "/mounts/Users/cisintern/antmarakis/.local/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1525, in _map_single
    writer.write_batch(batch)
  File "/mounts/Users/cisintern/antmarakis/.local/lib/python3.8/site-packages/datasets/arrow_writer.py", line 278, in write_batch
    pa_table = pa.Table.from_pydict(typed_sequence_examples)
  File "pyarrow/table.pxi", line 1531, in pyarrow.lib.Table.from_pydict
  File "pyarrow/array.pxi", line 295, in pyarrow.lib.asarray
  File "pyarrow/array.pxi", line 195, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 107, in pyarrow.lib._handle_arrow_array_protocol
  File "/mounts/Users/cisintern/antmarakis/.local/lib/python3.8/site-packages/datasets/arrow_writer.py", line 100, in __arrow_array__
    if trying_type and out[0].as_py() != self.data[0]:
  File "pyarrow/array.pxi", line 949, in pyarrow.lib.Array.__getitem__
  File "pyarrow/array.pxi", line 362, in pyarrow.lib._normalize_index
IndexError: index out of bounds
```

This approach used to work for previous versions, but after I upgraded to the latest releases this doesn't seem to work anymore and I do not know where it broke. Any help would be appreciated!

Comments:
I have simplified the code to show that it is definitely the pretrained tokenizer that breaks the execution:

```
python run_mlm.py \
--model_name_or_path bert-base-cased \
--train_file data.txt \
--tokenizer_name custom_tokenizer \
--output_dir output \
--do_train \
--num_train_epochs 1 \
--overwrite_output_dir

```

This seems like an issue that concerns the new mlm script, `tokenizers` and `datasets` so I'll ping the holy trinity that may have an idea where the error comes from: @sgugger @n1t0 @lhoestq  

Looks like the tokenizer returns an empty batch of elements, which causes an `IndexError` ?

is this issue resolved? ran into the same error.

From recent experience, I think this might happen if no `model_max_length` is set for the tokenizer.

In the directory where your tokenizer files live, do you mind adding another file called `tokenizer_config.json`, with the following information: `{"model_max_length": 512}`?

Thank you.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.