run_mlm.py crashes when saving model checkpoint
Author: avacaondata
Pull request: no
Labels: name: wontfix

## Environment info
  
- `transformers` version: 4.0.1
- Platform: Google Cloud
- Python version: 3.6.10
- PyTorch version (GPU?): 1.7
- Tensorflow version (GPU?):
- Using GPU in script?: NO. Using TPU
- Using distributed or parallel set-up in script?: YES

### Who can help
@LysandreJik  @mfuntowicz @sgugger

## Information

I'm trying to train al Albert model from scratch, with a custom tokenizer, on Google Cloud TPUS. The problem arises when saving the model checkpoints, more specifically when trying to save the tokenizer. I'm using your example script run_mlm.py. 

The problem arises when using:
* [ ] the official example scripts: run_mlm.py

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: Masked Language Modelling.

## To reproduce

Steps to reproduce the behavior:

1. Run run_mlm.py with the following params: python transformers/examples/xla_spawn.py --num_cores 8 \
    transformers/examples/language-modeling/run_mlm.py \
    --model_type albert \
    --train_file texts_train.txt  \
    --validation_file good_texts_valid.txt \
    --output_dir modelo_prueba \
    --tokenizer_name ./tokenizadores/definitivo  \
    --overwrite_output_dir \
    --line_by_line \
    --pad_to_max_len \
    --do_train \
    --do_eval \
    --evaluation_strategy steps \
    --per_device_train_batch_size 32 \
    --per_device_eval_batch_size 32 \
    --learning_rate 1e-3 \
    --max_steps 500 \
    --save_steps 100 \
    --save_total_limit 15 \
    --overwrite_cache \
    --max_seq_length 512 \
    --eval_accumulation_steps 10 \
    --logging_steps 100 \
    --config_name ./config/albert-base-v2.json \

At step 100, the following error arises:

```
INFO|trainer.py:1141] 2020-12-21 15:46:34,157 >> Saving model checkpoint to modelo_prueba/checkpoint-100                                                                                                                                     
[INFO|configuration_utils.py:281] 2020-12-21 15:46:34,158 >> Configuration saved in modelo_prueba/checkpoint-100/config.json
[INFO|modeling_utils.py:741] 2020-12-21 15:46:34,556 >> Model weights saved in modelo_prueba/checkpoint-100/pytorch_model.bin
Exception in device=TPU:0: expected str, bytes or os.PathLike object, not NoneType
Traceback (most recent call last):
  File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 330, in _mp_start_fn
    _start_fn(index, pf_cfg, fn, args)
  File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 324, in _start_fn
    fn(gindex, *args)
  File "/home/alejandro_vaca/transformers/examples/language-modeling/run_mlm.py", line 405, in _mp_fn
    main()
  File "/home/alejandro_vaca/transformers/examples/language-modeling/run_mlm.py", line 379, in main
    trainer.train(model_path=model_path)
  File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/transformers/trainer.py", line 777, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/transformers/trainer.py", line 848, in _maybe_log_save_evaluate
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/transformers/trainer.py", line 869, in _save_checkpoint
    self.save_model(output_dir)
  File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/transformers/trainer.py", line 1135, in save_model
    self._save_tpu(output_dir)
  File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/transformers/trainer.py", line 1157, in _save_tpu
    self.tokenizer.save_pretrained(output_dir)
  File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 1972, in save_pretrained
    filename_prefix=filename_prefix,
  File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/transformers/tokenization_utils_fast.py", line 524, in _save_pretrained
    vocab_files = self.save_vocabulary(save_directory, filename_prefix=filename_prefix)
  File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/transformers/models/albert/tokenization_albert_fast.py", line 252, in save_vocabulary
    if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file):
  File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/posixpath.py", line 378, in abspath
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType
```


## Expected behavior

The expected behavior is that the script doesn't crash. Moreover, it's completely unnecessary to save the tokenizer in trainer.py, as the tokenizer is already trained and doesn't need to be saved again. 



Comments:
We don't have your tokenizer, so the reproducer you give us does not work. I tried on my side to run the same command with a saved tokenizer and a saved config file and it works without any trouble.

> Moreover, it's completely unnecessary to save the tokenizer in trainer.py, as the tokenizer is already trained and doesn't need to be saved again.

It is necessary to allow users to resume training from the latest checkpoint.

Yeah, it's necessary to allow users to resume training, but that concerns the model only, not the tokenizer. The tokenizer is trained prior to training the model and doesn't change during training. I'll upload the tokenizer so that you can reproduce the issue. 

I'm trying to post my tokenizer but Github doesn't let me, it has too many characters....
Can I send it to you via email ? @sgugger 

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.