resize_token_embeddings doesn't work as expected for BertForMaskedLM
Author: praateekmahajan
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using : **BertForMaskedLM**

Language I am using the model on (English, Chinese....): **multilingual**

The problem arise when using:
* [x] the official example scripts: **BertForMaskedLM**

The tasks I am working on is:
* [x] my own task or dataset: **Finetuning with newly added tokens**.

## To Reproduce

Steps to reproduce the behavior:

```python
# Get model
model = BertForMaskedLM.from_pretrained("bert-base-multilingual-cased")

# define input
input = torch.tensor([[1,2,3,4,5]])

# before experiment
print(f"BERT num embeddings before\t: {model.bert.embeddings.word_embeddings.num_embeddings}")
print(f"LM Decoder num embedding before\t: {model.cls.predictions.decoder.out_features}")
print(f"MLM Loss\t\t\t: {model(input_ids=input, masked_lm_labels=input)[0]}")

# change embedding size for experiment
model.resize_token_embeddings(119547 + 5)
print(f"\nBERT num embeddings after\t: {model.bert.embeddings.word_embeddings.num_embeddings}")
print(f"LM Decoder num embedding after\t: {model.cls.predictions.decoder.out_features}")
# print failuire
try:
  print(f"MLM Loss\t\t\t: {model(input_ids=input, masked_lm_labels=input)[0]}")
except RuntimeError:
  print(" ---- Forward Pass failed ---- ")
```

This is the output that I see
```
BERT num embeddings before	: 119547
LM Decoder num embedding before	: 119547
MLM Loss			: 18.269268035888672

BERT num embeddings after	: 119552
LM Decoder num embedding after	: 119547
 ---- Forward Pass failed ---- 
```

If you see the LM decoder did not change the embedding size, hence won't ever predict the new tokens.

The expectation would be that `resize_token_embeddings` handles the FC inside LM decoder.

I tried also using BertConfig 

```python
config = BertConfig.from_pretrained("bert-base-multilingual-cased")
config.vocab_size = 119547 + 5
model = BertForMaskedLM.from_pretrained("bert-base-multilingual-cased", config=config)
```

Which naturally results in error while loading `state_dict`

```
RuntimeError: Error(s) in loading state_dict for BertForMaskedLM:
	size mismatch for bert.embeddings.word_embeddings.weight: copying a param with shape torch.Size([119547, 768]) from checkpoint, the shape in current model is torch.Size([119554, 768]).
	size mismatch for cls.predictions.bias: copying a param with shape torch.Size([119547]) from checkpoint, the shape in current model is torch.Size([119554]).
	size mismatch for cls.predictions.decoder.weight: copying a param with shape torch.Size([119547, 768]) from checkpoint, the shape in current model is torch.Size([119554, 768]).
```

Comments:
Hi! May I know what version are you using? On the current master branch, this is the result:

```py
import torch
from transformers import BertForMaskedLM

input_ids = torch.tensor([[1,2,3,4,5]])
print(model.bert.embeddings.word_embeddings.num_embeddings)  # 119547
print(model.cls.predictions.decoder.out_features)  # 119547
model.resize_token_embeddings(119547 + 5)
print(model.bert.embeddings.word_embeddings.num_embeddings)  # 119552
print(model.cls.predictions.decoder.out_features)  # 119552
```

Hmm, I am also on 2.1.1. Looks like in your example you're not loading the model from a pretrained model. (Though I am not sure how that makes a difference, but looks like it does)


```python
model = BertForMaskedLM.from_pretrained(model_type, cache_dir="/tmp/bert_models")

print(transformers.__version__) #2.1.1
print(model.bert.embeddings.word_embeddings.num_embeddings)  # 119547
print(model.cls.predictions.decoder.out_features)  # 119547
model.resize_token_embeddings(119547 + 5)
print(model.bert.embeddings.word_embeddings.num_embeddings)  # 119552
print(model.cls.predictions.decoder.out_features)  # 119547
```

And as per the current master also I don't see where we copy over the params from the old `cls` to the new updated `cls` with the new number of tokens as we do it for the embedding layer [here](https://github.com/huggingface/transformers/blob/de890ae67d43e1e5d031a815dab5dfed081e9a95/transformers/modeling_utils.py#L196).


In the next comment, I can share the temporary hack that I have.

```python
class CustomBertForMaskedLM(BertForMaskedLM):
    def __init__(self, config):
        super().__init__(config)

    def resize_embedding_and_fc(self, new_num_tokens):
      # Change the FC 
      old_fc = self.cls.predictions.decoder
      self.cls.predictions.decoder = self._get_resized_fc(old_fc, new_num_tokens)
      
      # Change the bias
      old_bias = self.cls.predictions.bias
      self.cls.predictions.bias = self._get_resized_bias(old_bias, new_num_tokens)
      
      # Change the embedding
      self.resize_token_embeddings(new_num_tokens)
    
    
    def _get_resized_bias(self, old_bias, new_num_tokens):
        old_num_tokens = old_bias.data.size()[0]
        if old_num_tokens == new_num_tokens:
            return old_bias

        # Create new biases
        new_bias = nn.Parameter(torch.zeros(new_num_tokens))
        new_bias.to(old_bias.device)

        # Copy from the previous weights
        num_tokens_to_copy = min(old_num_tokens, new_num_tokens)
        new_bias.data[:num_tokens_to_copy] = old_bias.data[:num_tokens_to_copy]
        return new_bias
    
    def _get_resized_fc(self, old_fc, new_num_tokens):

        old_num_tokens, old_embedding_dim = old_fc.weight.size()
        if old_num_tokens == new_num_tokens:
            return old_fc

        # Create new weights
        new_fc = nn.Linear(in_features=old_embedding_dim, out_features=new_num_tokens)
        new_fc.to(old_fc.weight.device)

        # initialize all weights (in particular added tokens)
        self._init_weights(new_fc)

        # Copy from the previous weights
        num_tokens_to_copy = min(old_num_tokens, new_num_tokens)
        new_fc.weight.data[:num_tokens_to_copy, :] = old_fc.weight.data[:num_tokens_to_copy, :]
        return new_fc
```


Using this, I can achieve the output as expected.

![image](https://user-images.githubusercontent.com/7589415/68256992-a3f7d580-ffe6-11e9-9496-5624152b85c2.png)


Indeed I had forgotten the line about the model initialization, but it is initialized from a pretrained checkpoint as you have done. Running your code gives me the same results:

```py
from transformers import BertForMaskedLM
import torch

model = BertForMaskedLM.from_pretrained("bert-base-cased")

print(model.bert.embeddings.word_embeddings.num_embeddings)  # 28996
print(model.cls.predictions.decoder.out_features)  # 28996
model.resize_token_embeddings(119547 + 5)
print(model.bert.embeddings.word_embeddings.num_embeddings)  # 119552
print(model.cls.predictions.decoder.out_features)  # 119552
```

 but please note I am running on the **master branch**, not on the official pypi release. In the official release the predictions layer indeed isn't correctly tied, but it has been fixed since then with #1721.

You can install from master with: `pip install git+https://github.com/huggingface/transformers`.

As for where the tying happens, it is [in the superclass `PreTrainedModel`](https://github.com/huggingface/transformers/blob/master/transformers/modeling_utils.py#L114) which ties the input embeddings and output embeddings if there are any.

@LysandreJik This issue is only partially fixed. In _tie_or_clone_weights(), it tries to fix output_embeddings.bias, but output_embeddings is actually self.cls.predictions.decoder, which has no bias term. Instead, self.cls.predictions has a bias term. This bias term is not fixed, and exceptions will occur. 

I'll have to use @praateekmahajan 's hack temporarily 😃 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This is still an issue, I'm reopening it.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@LysandreJik do you know if its `wontfix` or just in your backlog. Lmk if I can help somehow. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.