Option to serialize tokenizers in memory instead of to a directory
Author: justinaustin
Pull request: no

# 🚀 Feature request
Hello!

It would be useful to serialize tokenizers to bytes in memory instead of having to write to a directory. [`PreTrainedModel.save_pretrained`](https://huggingface.co/transformers/internal/tokenization_utils.html#transformers.tokenization_utils_base.PreTrainedTokenizerBase.save_pretrained) only takes in a `save_directory`, where ideally it would also accept a [`BytesIO`](https://docs.python.org/3/library/io.html#io.BytesIO) or similar to serialize in memory (or there would be another method to call for this).

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

We're trying to integrate `transformers` with some existing software that handles signing serialized models for security. We're able to save to a directory on the local filesystem and then read the bytes back into memory, but it would be nice to avoid that and serialize directly to an in-memory BytesIO object. Also, this would expose a more flexible API that would make it easier for others using this great package to export their models.

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->
Unfortunately, I'm not familiar with the internals of the package and just wanted to see the feasibility of this. I'm happy to help however I can, especially if the change wasn't too involved. Thank you!

Comments:
Hi @justinaustin, how about pickling it?

The following code works for me:
```
import pickle
import torch
import transformers
from transformers import BertTokenizer, BertModel

print('PyTorch version:', torch.__version__, ' transformers version:', transformers.__version__)

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased", additional_special_tokens=["[SP]"])
ids_1 = tokenizer("Hello World[SP]!")

pickled_tokenizer = pickle.dumps(tokenizer)
tokenizer = pickle.loads(pickled_tokenizer)
ids_2 = tokenizer("Hello World[SP]!")

print(ids_1 == ids_2) # Output: True
```

Thank you for the suggestion @qqaatw ! Pickling is a great short-term solution for us. We hesitate to use pickle long-term because of brittleness between different Python versions and potential security concerns when deserializing.

It's just unfortunate that there's already a great serialization framework within transformers that takes care serializing to well-defined json and text files but you're unable to use if you want to avoid writing intermediate files to disk.


I understand your concern. However, I think currently pickling is the only way to serialize the entire tokenizer as a byte object, and this approach is tested in the unit test of `transformers`. Additionally, you may find that the loading and saving utilities of PyTorch also leverage `pickle` package to process serialization so I believe pickling still remain somewhat of reliabilities to complete these general things.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.