Issue 1:
🐛 Bug

Model I am using (Bert, XLNet....): Bert

Language I am using the model on (English, Chinese....): English

The problem arise when using:

the official example scripts: modeling_bert.py

The tasks I am working on is:

my own task or dataset: fine-tuning Bert with added new tokens to vocabulary

To Reproduce

Steps to reproduce the behavior:

Running "run_lm_finetuning.py" with added tokens to vocabulary.

new_vocab_list = ['token_1', 'token_2', 'token_3']
tokenizer.add_tokens(new_vocab_list)
logger.info("vocabulary size after adding: " + str(len(tokenizer)))
model.resize_token_embeddings(len(tokenizer))
logger.info("size of model.cls.predictions.bias: " + str(len(model.cls.predictions.bias)))

Expected behavior

The result should be:

vocabulary size after adding: 31119

size of model.cls.predictions.bias: 31119

But actually the result is:

vocabulary size after adding: 31119

size of model.cls.predictions.bias: 31116

Environment

OS: Ubuntu

Python version: 3.6

PyTorch version: 1.3.1

PyTorch Transformers version (or branch): 2.2.1

Using GPU: yes

Distributed or parallel setup: no

Additional context

I have found the problem to be: for BERT model, the class "BertLMPredictionHead" has two separate attributes "decoder" and "bias". When adding new tokens, the code "model.resize_token_embeddings(len(tokenizer))" only updates the size of "decoder" and its bias if it has (this bias is different from the "BertLMPredictionHead.bias"). The attribute "BertLMPredictionHead.bias" is not updated and therefore, causes the error.

I have added the updating-bias code in my "modeling_bert.py". And if you want, I can merge my branch to your code. However, if I misunderstand something, please notice me too.

Thank you very much for your code base.

Issue 2:
🐛 Bug

Model I am using : BertForMaskedLM

Language I am using the model on (English, Chinese....): multilingual

The problem arise when using:

the official example scripts: BertForMaskedLM

The tasks I am working on is:

my own task or dataset: Finetuning with newly added tokens.

To Reproduce

Steps to reproduce the behavior:

# Get model
model = BertForMaskedLM.from_pretrained("bert-base-multilingual-cased")

# define input
input = torch.tensor([[1,2,3,4,5]])

# before experiment
print(f"BERT num embeddings before\t: {model.bert.embeddings.word_embeddings.num_embeddings}")
print(f"LM Decoder num embedding before\t: {model.cls.predictions.decoder.out_features}")
print(f"MLM Loss\t\t\t: {model(input_ids=input, masked_lm_labels=input)[0]}")

# change embedding size for experiment
model.resize_token_embeddings(119547 + 5)
print(f"\nBERT num embeddings after\t: {model.bert.embeddings.word_embeddings.num_embeddings}")
print(f"LM Decoder num embedding after\t: {model.cls.predictions.decoder.out_features}")
# print failuire
try:
print(f"MLM Loss\t\t\t: {model(input_ids=input, masked_lm_labels=input)[0]}")
except RuntimeError:
print(" ---- Forward Pass failed ---- ")

This is the output that I see

BERT num embeddings before	: 119547
LM Decoder num embedding before	: 119547
MLM Loss	: 18.269268035888672

BERT num embeddings after	: 119552
LM Decoder num embedding after	: 119547
---- Forward Pass failed ----

If you see the LM decoder did not change the embedding size, hence won't ever predict the new tokens.

The expectation would be that resize_token_embeddings handles the FC inside LM decoder.

I tried also using BertConfig

config = BertConfig.from_pretrained("bert-base-multilingual-cased")
config.vocab_size = 119547 + 5
model = BertForMaskedLM.from_pretrained("bert-base-multilingual-cased", config=config)

Which naturally results in error while loading state_dict

RuntimeError: Error(s) in loading state_dict for BertForMaskedLM:
size mismatch for bert.embeddings.word_embeddings.weight: copying a param with shape torch.Size([119547, 768]) from checkpoint, the shape in current model is torch.Size([119554, 768]).
size mismatch for cls.predictions.bias: copying a param with shape torch.Size([119547]) from checkpoint, the shape in current model is torch.Size([119554]).
size mismatch for cls.predictions.decoder.weight: copying a param with shape torch.Size([119547, 768]) from checkpoint, the shape in current model is torch.Size([119554, 768]).