BertForMaskedLM cannot be initialized from BERT checkpoints
Author: DavidNemeskey
Pull request: no

When I try to load a BERT model from a TF checkpoint (via `transformers-cli convert`) into a `BertForMaskedLM`, I get the following warning:
```
Some weights of BertForMaskedLM were not initialized from the model checkpoint at `SZTAKI-HLT/hubert-base-cc` and are newly initialized:
['cls.predictions.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight',
'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.weight']

You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

The model also performs poorly (as in: completely randomly) in masked LM. If I load a "named" model, such as `bert-base-cased`, I do not get the warning and masked LM works OK.

This is all to be expected if the tensors mentioned in the warning are indeed not part of the converted model. The question then is twofold:
1. Why aren't they? MaskedLM is one of the training tasks for a BERT model, and users rightly expect that it works (I have already received two reports for my model to that effect); i.e. that they can initialize a `BertForMaskedLM` model from a BERT checkpoint / HF model without any problems.
2. How can I convert the model so that it includes said tensors? To my knowledge, there are not options in `transformers-cli convert` that would enable me to do so.
3. The [documentation](https://huggingface.co/transformers/converting_tensorflow_models.html) should warn people of this (and better yet, describe how to convert all tensors).

## Environment info
- `transformers` version: 4.3.2
- Platform: Linux-5.4.0-60-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.9
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): 2.3.0 (False)
- Using GPU in script?: N/A
- Using distributed or parallel set-up in script?: N/A

### Who can help
@LysandreJik
@sgugger

## Information

Model I am using (Bert, XLNet ...): SZTAKI-HLT/hubert-base-cc (BERT)

The problem arises when using:
* [X] the official example scripts: `transformer-cli convert`
* [ ] my own modified scripts:

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: masked LM
* [ ] my own task or dataset:

## To reproduce

Steps to reproduce the behavior:

1. `BertForMaskedLM.from_pretrained('SZTAKI-HLT/hubert-base-cc')`
2. Observer warning messages
3. Try to use it for masked LM

## Expected behavior

Conversion: the ability to convert tensors for the training tasks
Usage: no warning messages and same MLM / NSP performance as with the official TF BERT code

Comments:
Note that the way to load TF weights in a PyTorhc model while using the hub is just to do:
```
model = BertForMaskedLM.from_pretrained("SZTAKI-HLT/hubert-base-cc", from_tf=True)
```
It does look like this model has some weights missing (I still get a warning but with a few less weights than you) but if that's blocking you, there is nothing we can do about it: you should contact the author of the model on the hub (the same weights are missing in the TF version).

@sgugger **I** am the author of the model. :) And yes, the weights are missing, hence this issue; as described above, I used `transformers-cli convert` to convert the original BERT TF (1.5) checkpoint to Pytorch, and the script apparently did not convert all the weights.

I might be wrong about this, but I was under the impression that the `from_tf=True` is to be used when the model has both PT and TF versions uploaded to the hub, not for importing an original BERT checkpoint.

Ah sorry I misunderstood your problem, sorry! I though you were trying to use the `transformers-cli` on the PyTorch model file of the hub. Not sure what the problem is with the conversion script. Maybe @LysandreJik will have an idea?

Hello! Thank your for opening an issue. As you both have said, there seems to be an error with the conversion.

Do you mind letting me know how you obtained your checkpoint? For example, is it one of the checkpoints available on google-research/bert, or is a custom one?

All the checkpoints available on google-research/bert should convert without any issue.

OK, I have experimented a bit and it seems that actually conversion works -- it is possible that I created a `BertModel` and saved my model as that instead of `BertForPretraining`. In any case, I have updated my model(s) and the issue is moot.

Before closing it, however, one final question. I created the Pytorch model first. When I converted that to TF2 via
```
TFBertForPreTraining.from_pretrained('SZTAKI-HLT/hubert-base-cc', from_pt=True)
```
, I got the following warning:
```
Some weights of the PyTorch model were not used when initializing the TF 2.0 model TFBertForPreTraining: ['cls.predictions.decoder.bias', 'bert.embeddings.position_ids']
```
The model, when used for masked LM, behaves identically to the Pytorch model, so I am wondering why they store different tensors (given that these tensors come from the original TF checkpoint) and if the model will be alright without them.

Glad you could convert it!!

These warnings aren't important, the bias is included in another weight and the position IDs are a buffer that does not need to be created in TensorFlow.

I'm addressing these warnings in #10397.