Warnings about some weights that were not initialized in Greek BERT
Author: m-nlp-q
Pull request: no

Hello,

in order to use the Greek BERT model, I use `AutoModel` class, and specifically

`greek_bert = AutoModel.from_pretrained("nlpaueb/bert-base-greek-uncased-v1")`

When I load the model with the above command I get this warning: 
```

Some weights of the model checkpoint at nlpaueb/bert-base-greek-uncased-v1 were not used when initializing BertModel: ['cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.bias', 'cls.predictions.decoder.bias', 'cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.weight']
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). 
```

Why is this warning thrown?

Comments:
It tells you that you are initializing a `BertModel` without the heads that were used for pre-training (namely next sentence prediction and masked language modeling). That's not a problem, as you probably don't need these weights for a downstream task of interest (such as question answering or sequence classification). 

Thank you @NielsRogge for your quick reply,

Indeed, I don't need those weighs for my task, but how can you tell that these are the weights of the heads used in pre-training?

You can see it based on their names:

`['cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.bias', 'cls.predictions.decoder.bias', 'cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.weight']`

=> `cls.seq_relationship` refers to the linear layer used for next sentence prediction.
=> `cls.predictions` refers to the masked language modeling head. It consists of a `transform` layer followed by a `decoder` (which maps to the vocabulary). 

The author of Greek BERT probably used the `BertForPreTraining` model for pre-training the BERT model. You can see the definition of the heads [here](https://github.com/huggingface/transformers/blob/bd3b599c12cfcf5ef517c5ffe526afbdbaa92539/src/transformers/models/bert/modeling_bert.py#L1011).

I see. I saw the names too, but I wasn't sure that these weights correspond to these heads just by reading the names.

Thank you @NielsRogge, you have been very helpful.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.