BertTokenizerFast does not load custom vocab created from Tokenizer library
Author: questpavan
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
Bert ["iuliaturc/bert_uncased_L-2_H-128_A-2"]
https://huggingface.co/iuliaturc/bert_uncased_L-2_H-128_A-2

Language I am using the model on (English, Chinese ...):
English

The problem arises when using:
my own modified scripts: (give details below)



The tasks I am working on is:
my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
see comments in the code 

1. Train model and tokenizer on new tokenizers library 
2. try to add new tokens to tokenizer
3. Resize token embeddings for model
 

### Problem 1 : 
If Loading with AutoTokenizer, can not use BertTokenizerFast but able to use custom vocabs

4. save tokenizer using save_tokenizer
5. Load using  AutoTokenizer

### Problem 2:
If Loading with BertTokenizerFast, can use the functioanlity of BertTokenizerFast but losed the custom vocabs

4. save tokenizer using save_tokenizer
5. Load using  BertTokenizerFast

```python


# Loaded Model and Tokenizer Default 
tokenizer = BertTokenizerFast.from_pretrained("""iuliaturc/bert_uncased_L-2_H-128_A-2""")
model = AutoModel.from_pretrained("""iuliaturc/bert_uncased_L-2_H-128_A-2""")

# Added Custom vocab to the tokenizer

def add_vocab_to_model(df, model, tokenizer, old_vocab, vocab_size=30000):
    """Adds new vocab to tokenizer and randomly initialises rows for new vocab in the model"""
    PATH = Path('./lm_data')
    PATH.mkdir(exist_ok=True)

    df.dropna(inplace=True)
    lm_text = ' '.join(df['text'])
    lm_text = lm_text.replace("■","")
    lm_text = re.sub(r'[^\x00-\x7F]+',' ', lm_text)
    lm_text = re.sub(r'[^\u0000-\u007F]+',' ', lm_text)

    with open('./lm_data/data.txt', mode='w') as f:
        f.write(lm_text)

    paths = [str(x) for x in Path("./lm_data/").glob("**/*.txt")]

    # Initialize a tokenizer
    tokenizer_new = BertWordPieceTokenizer(old_vocab, lowercase=True)

    # Customize training
    tokenizer_new.train(files=paths, vocab_size=vocab_size, min_frequency=20)
    tokenizer_new.save(".", "./lm_data/new")

    new_vocab = open('./lm_data/new-vocab.txt', 'r').read().split('\n')
    new_vocab.remove('')

    print('Adding new tokens to vocab')
    n_orig_tokens = len(tokenizer)
    tokenizer.add_tokens(new_vocab)
    print('Original no. of tokens: %s'%n_orig_tokens)
    print('Final no. of tokens: %s'%len(tokenizer))

    print('Initialised random emb for new tokens')
    model.resize_token_embeddings(len(tokenizer))

    return model, tokenizer


# Just needs to pass a dataframe which is having a column with name "text"
model1, tokenizer1 = add_vocab_to_model(dataframe, model, tokenizer, 'bert-base-uncased-vocab.txt', vocab_size=10000)
#Adding new tokens to vocab
#Original no. of tokens: 30522
#Final no. of tokens: 34340
#Initialised random emb for new tokens


print(len(tokenizer1))
# Got the result : 34340
#Finalized Result

print(type(tokenizer1))
# transformers.tokenization_bert.BertTokenizerFast
# Final Tokenizer


# Problem 1 : 
# If Loading with AutoTokenizer, can not use BertTokenizerFast but able to use custom vocabs


# Saved and Loaded Model Again
tokenizer1.save_pretrained("./tokenizer")
tokenizer2 = AutoTokenizer.from_pretrained("./tokenizer")

print(len(tokenizer2)
# Got 34340
# Which is Correct.

print(type(tokenizer2))
# transformers.tokenization_bert.BertTokenizer
# Not able to load BertTokenizerFast defaulting back to BertTokenizer


# Problem 2:
# If Loading with BertTokenizerFast, can use the functioanlity of BertTokenizerFast but losed the custom vocabs


# Saved and Loaded Model Again
tokenizer1.save_pretrained("./tokenizer")
tokenizer3 = BertTokenizerFast.from_pretrained("./tokenizer")

print(len(tokenizer3)
# Got 30522
# Losing the added custom vocabs

print(type(tokenizer3))
# transformers.tokenization_bert.BertTokenizerFast
#which is correct

```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->




## Expected behavior
1. If Loading tokenizer using AutoTokenizer it should load the BertTokenizerFast 
2. If Loading tokenizer using BertTokenizerFast, it should maintain custom vocabs


<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.9.1
- Platform: Windows
- Python version:  3.7.7
- PyTorch version (GPU?): 1.5.1 
- Tensorflow version (GPU?): NA
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Comments:
Hi,
Can anyone from Huggingface team look into this?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Similar to #6025 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.