Please add the functionality to save tokenizer model for run_language_modeling.py
Author: vincentwen1995
Pull request: no

# 🚀 Feature request

Please add the feature to save the tokenizer model during training to the checkpoint folders.
## Motivation

When I tried out the script for [fine-tuning with language modeling](transformers/examples/language_modeling/run_language_modeling.py), I realized that the generated checkpoints during training cannot allow continue training, because under the checkpoint folders, the corresponding tokenizer model is not saved (including the files: tokenizer_config.json, special_tokens_map.json, vocab.txt). As I checked the [script](https://github.com/huggingface/transformers/blob/0866669e751bef636fa693b704a28c1fea9a17f3/examples/language-modeling/run_language_modeling.py#L250), and noticed that the tokenizer model is only saved after the training process.
## Your contribution

Haven't looked into the code in details so it might be best to have someone familiar with the Trainer class to integrate this.

Comments:
This would be good indeed (cc @julien-c). In the meantime I think you can specify 
```
--tokenizer_name=$TOKENIZER_NAME_OR_PATH
```
 so that it always loads the initial tokenizer (which does not change during training).

> This would be good indeed (cc @julien-c). In the meantime I think you can specify
> 
> ```
> --tokenizer_name=$TOKENIZER_NAME_OR_PATH
> ```
> 
> so that it always loads the initial tokenizer (which does not change during training).

Thank you!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I'm trying to figure out how to use a custom made or create a tokenizer using this script and having significant difficulty. There does not seem to be any documentation on how to do this. I attempted to follow [this ](https://huggingface.co/blog/how-to-train) example but it makes no mention of how the tokenizer gets loaded / used. I get errors like:

```
OSError: Can't load config for 'EsperBERTo-small'. Make sure that:

- 'EsperBERTo-small' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'EsperBERTo-small' is the correct path to a directory containing a config.json file
```

Hi @EdwardRaff ,
I'm facing the same issue while trying to train BERT model from scratch on my own dateset. Did you figure out how to solve it?
```

OSError: Can't load config for './models/BuckBERTer-small/'. Make sure that:

- './models/BuckBERTer-small/' is a correct model identifier listed on 'https://huggingface.co/models'

- or './models/BuckBERTer-small/' is the correct path to a directory containing a config.json file
```

@EdwardRaff @FerchichiNourchene `run_language_modeling.py` doesn't work if a tokenizer is specified, but it does not contain the model configuration files. [This](https://stackoverflow.com/a/64795300/3950710) workaround worked for me.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

@vincentwen1995, did you manage to get the tokenizers somehow? One year later and it seems like the `Trainer` is still not saving `tokenizer_config.json` in the checkpoint folders. 
Where is it even saved?