'use_fast=True' results in 'TypeError' when trying to save tokenizer via AutoTokenizer
Author: lingdoc
Pull request: no
Labels: name: Fast Tokenizers

# 🐛 Bug

## Information

Model I am using: all/any

Language I am using the model on: English

The problem arises when using:
* [x] the official example scripts: `AutoTokenizer.from_pretrained([model], use_fast=True)`

After updating to Transformers v2.10.0, when setting `use_fast=True` as in `tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased', use_fast=True)`, when trying to save the model by using `tokenizer.save_pretrained(path)` I get the following error and the process quits:
```
File "../python3.6/site-packages/transformers/tokenization_utils.py", line 1117,
  in save_pretrained
    vocab_files = self.save_vocabulary(save_directory)
  File "../python3.6/site-packages/transformers/tokenization_utils.py", line 2657,
  in save_vocabulary
    files = self._tokenizer.save(save_directory)
  File "../python3.6/site-packages/tokenizers/implementations/base_tokenizer.py",
  line 328, in save
    return self._tokenizer.model.save(directory, name=name)
TypeError
```
When I omit the `use_fast=True` flag, the tokenizer saves fine.

The tasks I am working on is:
* [x] my own task or dataset: Text classification

## To reproduce

Steps to reproduce the behavior:

1. Upgrade to `transformers==2.10.0` (requires `tokenizers==0.7.0`)
2. Load a tokenizer using `AutoTokenizer.from_pretrained()` with flag `use_fast=True`
3. Train for one epoch on any dataset, then try to save the tokenizer.


## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
The tokenizer/file should save into the chosen path, as it does with the regular tokenizer.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.10.0
- Platform: Linux-5.0.0-37-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.8
- PyTorch version (GPU?): 1.4.0+cu100 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


Comments:
Hi @lingdoc, 

Thanks for reporting this. Unfortunately, I'm not able to reproduce currently ... Loading, then training and finally saving works as espected on my side, with various tokenizers.

```python
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased', use_fast=True)

# Some training ...

tokenizer.save_pretrained("test_bert_tokenizer")
('test_bert_tokenizer\\vocab.txt',
 'test_bert_tokenizer\\special_tokens_map.json',
 'test_bert_tokenizer\\added_tokens.json')
```

Can you give the path you're trying to save ? Just to make sure we're ending having a `None` somewhere in the `save_pretrained` that would explain the `TypeError` raised.

Thanks!

Hm, this is really strange. I can't reproduce it either. Maybe something was wonky with my virtualenv - now it works fine! Next time I'll try a restart & run before I post.

Oops, I closed it too soon. I'm still getting the issue. Models I have tried:

`bert-base-uncased`
`distibert-base-uncased`
`google/electra-small-discriminator`

The path I am trying to save to is `"output/model_out"` - but it's generated using `Path()`, in case that makes a difference (not sure why it would make a difference for saving the `fast` tokenizer and not the regular one though).

Ok, that seems to be the issue after all - when I explicitly cast the `Path()`-generated path to `str`, it saves fine. I guess the regular tokenizer/save function does this somehow but the `fast` version doesn't..

Thanks for digging this further. 

I'll check what is the behaviour discrepancy between both version of the tokenizers when using `Path()` and I'll post here 👍 

For what it is worth, I have this exact same issue with the `"distilroberta-base"` tokenizer when `use_fast=True`. Casting my `Path` object to a `str` solved the issue.