Saving and loading tokenizers with torch.save fails
Author: mittalsuraj18
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Albert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Load albert base tokenizer using `AutoTokenizer.from_pretrained`
2. Save it to a file using `torch.save`
3. Delete `~/.cache/torch/transformers` directory
4. Now try to load from the file using `torch.load`
5. Loading fails as the cached file does not exist

```
import transformers
import torch
token = transformers.AutoTokenizer.from_pretrained("albert-base-v2")
torch.save({"token":token}, "./token.pt")
```
Delete `~/.cache/torch/` directory
Then Run
```
import torch
torch.load("./token.pt")
```

## Expected behavior
Tokenizer should load successfully.

## Environment info


- `transformers` version: 2.11.0
- Platform: Linux-4.19.104-microsoft-standard-x86_64-with-debian-bullseye-sid
- Python version: 3.7.6
- PyTorch version (GPU?): 1.3.1+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


Comments:
I met save issue before.  Based on my experience, it is more related to PyTorch. 
See https://pytorch.org/tutorials/beginner/saving_loading_models.html, you should use state_dict to save and load model.

But @FacingBugs, this is an issue with the tokenizers and not the models 

For tokenizer, using Transformer package provided  API:  
```
tokenizer.save_pretrained(your_output_model_dir)
```

@FacingBugs actually I have raised this bug because it was causing an issue in another library which uses this package
https://github.com/flairNLP/flair/issues/1712
And since `torch.save` is mostly used to persist the models and dependencies for pytorch based learning, I believe the fix should be implemented in the transformers library itself rather than other dependent libraries which may add on top of transformers to provide their custom pytorch models in which case `torch.save` would mostly be used to save the models.

I think according to the PyTorch documentation, ```torch.save()``` is not recommended. 

I cite this from the documentation:
"However in this case, the serialized data is bound to the specific classes and the exact directory structure used, so it can break in various ways when used in other projects, or after some serious refactors."


For my personal experience with the Transformer package, the ```xx.save_pretrained()``` works for most of the cases (models, tokenizers, configs).  For the tokenizer, I think the package actually saved several other files besides the vocab file.  I think using the save_pretrained method should be the best practice. 

Hope this can help you.

Wouldn’t this make persisting other models on top of transformers difficult because now we have to save and track multiple files instead of a single file?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.