Repository: huggingface/transformers

Issue:
Language modelling head has zero weights in pretrained TFMobileBertForMaskedLM
## Description
The `TFMobileBertForMaskedLM` example returns all zero logits while `MobileBertForMaskedLM` example works fine.
https://huggingface.co/transformers/model_doc/mobilebert.html#tfmobilebertformaskedlm

I checked language modeling head weights for both models and found that TF pretrained model has zero weights.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.0
- Platform: Google Colab
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101
- Tensorflow version (GPU?): 2.4.1
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@jplu @patrickvonplaten

## Information

Model I am using (Bert, XLNet ...): TFMobileBertForMaskedLM

The problem arises when using:
- [x] the official example scripts: (give details below)

## To reproduce

```
from transformers import MobileBertForMaskedLM
from transformers import TFMobileBertForMaskedLM

# PyTorch
model = MobileBertForMaskedLM.from_pretrained('google/mobilebert-uncased')
print(list(model.cls.parameters())[0])

# Parameter containing:
# tensor([-7.2946, -7.4302, -7.5401,  ..., -7.4850, -7.4503, -2.7798],
#       requires_grad=True)

# TensorFlow
model = TFMobileBertForMaskedLM.from_pretrained('google/mobilebert-uncased')
print(model.layers[1].get_weights()[0])

# array([0., 0., 0., ..., 0., 0., 0.], dtype=float32)
```
## Expected behavior
Language modeling head for TFMobileBertForMaskedLM should have the same weights as of MobileBertForMaskedLM



Discussion:
Hello!

Indeed there is an issue in the naming for `TFMobileBertForMaskedLM`. This will be fixed in the next release.

Resolving patch:
diff --git a/src/transformers/models/mobilebert/modeling_tf_mobilebert.py b/src/transformers/models/mobilebert/modeling_tf_mobilebert.py
--- a/src/transformers/models/mobilebert/modeling_tf_mobilebert.py
+++ b/src/transformers/models/mobilebert/modeling_tf_mobilebert.py
@@ -1105,7 +1105,6 @@ class TFMobileBertForMaskedLM(TFMobileBertPreTrainedModel, TFMaskedLanguageModel
     _keys_to_ignore_on_load_unexpected = [
         r"pooler",
         r"seq_relationship___cls",
-        r"predictions___cls",
         r"cls.seq_relationship",
     ]
 
@@ -1113,10 +1112,10 @@ def __init__(self, config, *inputs, **kwargs):
         super().__init__(config, *inputs, **kwargs)
 
         self.mobilebert = TFMobileBertMainLayer(config, add_pooling_layer=False, name="mobilebert")
-        self.mlm = TFMobileBertMLMHead(config, name="mlm___cls")
+        self.predictions = TFMobileBertMLMHead(config, name="predictions___cls")
 
     def get_lm_head(self):
-        return self.mlm.predictions
+        return self.predictions.predictions
 
     def get_prefix_bias_name(self):
         warnings.warn("The method get_prefix_bias_name is deprecated. Please use `get_bias` instead.", FutureWarning)
@@ -1179,7 +1178,7 @@ def call(
             training=inputs["training"],
         )
         sequence_output = outputs[0]
-        prediction_scores = self.mlm(sequence_output, training=inputs["training"])
+        prediction_scores = self.predictions(sequence_output, training=inputs["training"])
 
         loss = None if inputs["labels"] is None else self.compute_loss(inputs["labels"], prediction_scores)
 


Test patch: