Some layers of pretrained Albert model albert-base-v2 didn't match the architecture of AlbertForMaskedLM in latest transfomers 4.1.1.
Author: BlueHeart0621
Pull request: no

 albert: @LysandreJik 
## Information

Model I am using is Albert:

The problem arises when using:
* [x] my own modified scripts: (give details below)

When I load pretrained albert-base-v2 model, I find some of the medata of the model.state_dict can not match the latest AlbertForMaskedLM model of transformers. And it seem to be that the pretrained model didn't repretrain after the albert code change.

I find the AlbertAttention class in transformer 2.2.0 is:
```python
class AlbertAttention(BertSelfAttention):
    def __init__(self, config):
        super(AlbertAttention, self).__init__(config)

        self.output_attentions = config.output_attentions
        self.num_attention_heads = config.num_attention_heads
        self.hidden_size = config.hidden_size 
        self.attention_head_size = config.hidden_size // config.num_attention_heads
        self.dropout = nn.Dropout(config.attention_probs_dropout_prob)
        ......
```
It has one layer `self.dropout`. However, the AlbertAttention class in transformer 4.1.1 is:
```python
class AlbertAttention(nn.Module):
    def __init__(self, config):
        super().__init__()
        if config.hidden_size % config.num_attention_heads != 0 and not hasattr(config, "embedding_size"):
            raise ValueError(
                "The hidden size (%d) is not a multiple of the number of attention "
                "heads (%d)" % (config.hidden_size, config.num_attention_heads)
            )

        self.num_attention_heads = config.num_attention_heads
        self.hidden_size = config.hidden_size
        self.attention_head_size = config.hidden_size // config.num_attention_heads
        self.all_head_size = self.num_attention_heads * self.attention_head_size

        self.query = nn.Linear(config.hidden_size, self.all_head_size)
        self.key = nn.Linear(config.hidden_size, self.all_head_size)
        self.value = nn.Linear(config.hidden_size, self.all_head_size)

        self.attention_dropout = nn.Dropout(config.attention_probs_dropout_prob)
        self.output_dropout = nn.Dropout(config.hidden_dropout_prob)
        ......
```
It has two layer `self.attention_dropout ` and `self.output_dropout`.

When I load pretrained model of albert, I find it still maintain the architecture of that in transformers 2.2.0. So those unmatched layer cannot load pretrained parameters, which make the model load from albert-base-v2 only has very low accuracy when training.

Comments:
Hello! When you speak of `unmatched layers`, do you mean the dropout layers? These layers have no weights.

Furthermore, when setting the verbosity level to `INFO` and loading the `albert-base-v2` weights in current `master`'s `AlbertForMaskedLM`:

```py
>>> from transformers import AlbertForMaskedLM
>>> from transformers import logging
>>> logging.set_verbosity_info()
>>> model = AlbertForMaskedLM.from_pretrained("albert-base-v2")
[...]
All model checkpoint weights were used when initializing AlbertForMaskedLM.

All the weights of AlbertForMaskedLM were initialized from the model checkpoint at albert-base-v2.
If your task is similar to the task the model of the checkpoint was trained on, you can already use AlbertForMaskedLM for predictions without further training.
```

This tells you that all weights were correctly initialized. It would seem the issue comes from somewhere else, or maybe I have misunderstood your issue? Could you expand on how you identify the "unmatched layers that cannot load pretrained parameters"?

Thanks. I made a mistake. The dropout layer is turely no weights. 
But I still have a question. I am using the AlbertForMaskLM on cloth dataset, and I load the pretrained model albert-base-v2, the train accuracy is start from 0.28; I load the pretrained model albert-xxlarge-v2, the train accuracy is start from 0.79. Is it normal?
Thanks a lot.

I do not have any experience with the CLOTH dataset, but taking a quick look at it it seems to be a cloze task, which is one of the pre-training objectives of the ALBERT model. It isn't surprising to me that the largest ALBERT model gets better results with no fine-tuning.

Yes, the larger pretrained deserve better performance. But the base model is only start from 0.28, which mean just like  randomly to choose answer in cloze(random is 0.25). And after convergence, the accuracy can reach 0.77. It seem to be the pretrained model doesn't learn any prior, just like from scratch. 
Anyway, thanks a lot.