Tokenizers save_pretrained doesn't work with custom vocabs (v3.0.2)
Author: mozharovsky
Pull request: no
Labels: name: Core: Tokenization

# 🐛 Bug

## Information

I'm using an instance of `RobertaTokenizerFast` with my custom vocab (pretrained using `tokenizers` library). When I try to save the tokenizer using `tokenizer.save_pretrained(<path>)` method a type error occurs. 

I did a bit of investigation why this happens and got a workaround https://github.com/huggingface/transformers/issues/5393#issuecomment-654342933

## To reproduce

Steps to reproduce the behavior:
1. Choose any tokenizer that adds `AddedToken` to kwargs on init (e.g. `RobertaTokenizerFast` which adds a `mask_token` in the following manner)
2. Train a custom vocab using `tokenizers` library or download a pretrained vocab
3. Create an instance of chosen tokenizer by passing vocab and merges files to its constructor
4. Call `save_pretrained(<path>)` method

```python
tokenizer_kwargs = RobertaTokenizerFast.from_pretrained("roberta-base").init_kwargs
vocab_file = tokenizer_kwargs.get("vocab_file")
merges_file = tokenizer_kwargs.get("merges_file")

tokenizer = RobertaTokenizerFast(vocab_file, merges_file)
tokenizer.save_pretrained(".")
```

This will result in the following error:
```python
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
 in 
----> 1 tokenizer.save_pretrained(".")

~/app/.venv/lib/python3.7/site-packages/transformers/tokenization_utils_base.py in save_pretrained(self, save_directory)
   1360 
   1361         with open(tokenizer_config_file, "w", encoding="utf-8") as f:
-> 1362             f.write(json.dumps(tokenizer_config, ensure_ascii=False))
   1363 
   1364         with open(special_tokens_map_file, "w", encoding="utf-8") as f:

~/.pyenv/versions/3.7.6/lib/python3.7/json/__init__.py in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw)
    236         check_circular=check_circular, allow_nan=allow_nan, indent=indent,
    237         separators=separators, default=default, sort_keys=sort_keys,
--> 238         **kw).encode(obj)
    239 
    240 

~/.pyenv/versions/3.7.6/lib/python3.7/json/encoder.py in encode(self, o)
    197         # exceptions aren't as detailed.  The list call should be roughly
    198         # equivalent to the PySequence_Fast that ''.join() would do.
--> 199         chunks = self.iterencode(o, _one_shot=True)
    200         if not isinstance(chunks, (list, tuple)):
    201             chunks = list(chunks)

~/.pyenv/versions/3.7.6/lib/python3.7/json/encoder.py in iterencode(self, o, _one_shot)
    255                 self.key_separator, self.item_separator, self.sort_keys,
    256                 self.skipkeys, _one_shot)
--> 257         return _iterencode(o, 0)
    258 
    259 def _make_iterencode(markers, _default, _encoder, _indent, _floatstr,

~/.pyenv/versions/3.7.6/lib/python3.7/json/encoder.py in default(self, o)
    177 
    178         """
--> 179         raise TypeError(f'Object of type {o.__class__.__name__} '
    180                         f'is not JSON serializable')
    181 

TypeError: Object of type AddedToken is not JSON serializable
```

## Expected behavior

I expect the same behavior as if you would save a pretrained tokenizer from the hub:

```python
tokenizer = RobertaTokenizerFast.from_pretrained("roberta-base")
tokenizer.save_pretrained(".")
```

Which in its turn results in producing 4 files:

```
('/app/vocab.json',
 '/app/merges.txt',
 '/app/special_tokens_map.json',
 '/app/added_tokens.json')
```

## Possible solution

A possible approach would be to pass hooks to json methods for (de)serializing objects of complex type like `AddedToken`. Generally, a solution must preserve the type information but since types are a subject of change it's unclear whether such generalization needs to be followed at all.

This is a sketch for the solution described above.
```python
def deserialize_json_object(json_obj: Dict[Text, Any]) -> Any:
    classname = json_obj.get("__class__", "dict")
    try:
        obj = eval(classname)(**json_obj)
    except NameError:
        obj = json_obj
    return obj

def serialize_object_to_json(obj: Any) -> Dict[Text, Any]:
    get_state = getattr(obj, "__getstate__", None)
    if callable(get_state):
        json_obj = get_state()
        json_obj["__class__"] = type(obj).__name__
    else:
        json_obj = obj.__dict__
    return json_obj

json.load(<obj>, object_hook=deserialize_json_object)

json.dumps(<obj>, default=serialize_object_to_json)
```

Another solution would be to map `AddedToken` instances to dicts in `tokenizer_config` similar to constructing a `write_dict` object here:
https://github.com/huggingface/transformers/blob/d6b0b9d451e7ffe0da72a5f532cc9bec1563e801/src/transformers/tokenization_utils_base.py#L1368

This will require to deserialize dicts into `AddedToken` instances on loading vocabs and symbols. 

## Environment info

- `transformers` version: 3.0.2
- Platform: macOS Catalina 10.15.5
- Python version: 3.7.6
- PyTorch version (GPU?): 1.5.1 (No)
- Tensorflow version (GPU?): N/A
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Comments:
Fixed by #6026

> Fixed by #6026

I am still experiencing this issue when running transformers 3.1

If you load from pretrained as follows:
```py
tokenizer = RobertaTokenizer.from_pretrained("path/to/tokenizer/folder")
```
then the tokenizer init_kwargs will load appropriately. The init_kwargs takes the form of a dictionary as follows:
```py
{
merges_file: "file/path/...",
model_max_length: 512,
vocab_file: "file/path/...",
}
```
In such a scenario the tokenizer can be saved using the save_pretrained functionality as intended.

However, when defining the tokenizer using the vocab_file and merge_file arguments, as follows:
```py
tokenizer = RobertaTokenizer(vocab_file='file/path/vocab.json', merges_file='file_path/merges.txt')
```
the resulting init_kwargs appears to default to:
```py
{
bos_token: AddedToken(bos_token, lstrip=False, rstrip=False) 
eos_token = AddedToken(eos_token, lstrip=False, rstrip=False)
sep_token = AddedToken(sep_token, lstrip=False, rstrip=False)
cls_token = AddedToken(cls_token, lstrip=False, rstrip=False)
unk_token = AddedToken(unk_token, lstrip=False, rstrip=False) 
pad_token = AddedToken(pad_token, lstrip=False, rstrip=False) 
}
```
which i see is defined within the RobertaTokenizer class of tokenization_roberta, but should be assigned along with the vocab_file and merges_file which does not appear to be the case.

This means that within the save_pretrained() function, the lines that are causing the issue are:
```py
tokenizer_config = copy.deepcopy(self.init_kwargs)
if len(self.init_inputs) > 0:
    tokenizer_config["init_inputs"] = copy.deepcopy(self.init_inputs)
for file_id in self.vocab_files_names.keys():
    tokenizer_config.pop(file_id, None)

with open(tokenizer_config_file, "w", encoding="utf-8") as f:
    f.write(json.dumps(tokenizer_config, ensure_ascii=False))

```
The solution #6026 looks like it addresses a separate part of the save_pretrained function, and hasn't stopped this error being raised when I run the above scenario? I am running transformers = 3.1, with tokenizers = 0.8.1rc2

Hello! If you check the init method of the `RobertaTokenizer`, you'll see it does not only expect the `vocab_file` and `merges_file`, but it also accepts the special tokens you mention:

```py
    def __init__(
        self,
        vocab_file,
        merges_file,
        errors="replace",
        bos_token="
",
        eos_token="
",
        sep_token="
",
        cls_token="
",
        unk_token="<unk>",
        pad_token="<pad>",
        mask_token="<mask>",
        add_prefix_space=False,
        **kwargs
    ):
```

This should allow you to specify which special tokens should be used.

If this does not solve your issue, do you mind opening a new issue with your specific problem, including the code that raises the error and the full stack-trace? This will help us help you. Thank you!

Hi @LysandreJik , thanks for the quick reply. Unfortunately defining the other arguments didn't appear to solve the issue. I have open a new issue: #8306 which includes code to recreate the problem