Logit explosion in MobileBertForNextSentencePrediction example from documentation (and all others tried)
Author: mmistele
Pull request: no

## Environment info
- `transformers` version: 4.11.3
- Platform: Darwin-19.6.0-x86_64-i386-64bit
- Python version: 3.6.8
- PyTorch version (GPU?): 1.9.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@vshampor 

## Information

Model I am using (Bert, XLNet ...): MobileBertForNextSentencePrediction

The problem arises when using:
* [x] the official example scripts: (give details below)
Using the example code provided in https://huggingface.co/transformers/model_doc/mobilebert.html#mobilebertfornextsentenceprediction.

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: Next Sentence Prediction

## To reproduce

Steps to reproduce the behavior:

Run the code from the official example script in the documentation:
```
>>> from transformers import MobileBertTokenizer, MobileBertForNextSentencePrediction
>>> import torch

>>> tokenizer = MobileBertTokenizer.from_pretrained('google/mobilebert-uncased')
>>> model = MobileBertForNextSentencePrediction.from_pretrained('google/mobilebert-uncased')

>>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
>>> next_sentence = "The sky is blue due to the shorter wavelength of blue light."
>>> encoding = tokenizer(prompt, next_sentence, return_tensors='pt')

>>> outputs = model(**encoding, labels=torch.LongTensor([1]))
>>> loss = outputs.loss
>>> logits = outputs.logits
```
Printing `logits`, we get `tensor([[2.7888e+08, 2.7884e+08]], grad_fn=<AddmmBackward>)` - strangely huge for both classes, and one that leads to a softmax score of 1 for the "is next sentence" class—the opposite of the correct answer, which is strange for an example from the documentation.

I ran it on a handful of related prompt and next sentence pairs, then on a larger set from my own NSP dataset, and got the same strange behavior: logits of about 2e+08 for both classes, and higher for the first class in the 3rd or 4th significant figure, no matter the prompt and sentence pair. Given the sizes, it leads to a softmax score of 1 "is the next sentence" (the first class) and 0 for the other no matter what the first and second sentence is, no matter how unrelated the second sentence is.

## Expected behavior
For comparison, the logits produced on the same example using BertForNextSentencePrediction with bert-base-uncased instead on this example are `tensor([[-3.0729,  5.9056]], grad_fn=<AddmmBackward>)`. I would expect that for an example with 'next sentence' from the "Not following the prompt" category like this, MobileBertForNextSentencePrediction with the default pretrained model would get this right, and have logits in a similar ballpark - not huge positive values like the ones pictured.

I posted about this on HuggingFace Hub discussion board, but it got immediately taken down by the bot for some reason. Linking here in case admins approve it: https://discuss.huggingface.co/t/next-sentence-prediction-with-google-mobilebert-uncased-producing-massive-near-identical-logits-10-8-for-its-documentation-example-and-2k-others-tried/10750/1.

Comments:
@vshampor Any updates?

@mmistele sorry for the late reply.

The behaviour you are experiencing seems to be natural for the model that is basically *untrained* for the downstream task you are trying to apply it to. The reason is that you are loading the wrong checkpoint for the `MobileBertForNextSentencePrediction` object, and this is even highlighted in the console output:

```
Some weights of the model checkpoint at google/mobilebert-uncased were not used when initializing MobileBertForNextSentencePrediction: ['cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.bias', 'cls.predictions.dense.weight', 'cls.predictions.transform.dense.weight']
```

The regular `google/mobilebert-uncased` contains the MobileBERT backbone and the language modeling head used for pre-training the backbone, but not the trained weights for the prediction head corresponding to the downstream "next sentence prediction" task. That `google/mobilebert-uncased` would be used by itself in the MobileBertForNextSentencePrediction docs by HF seems to be a bug by oversight. Summoning @LysandreJik who helped me integrate this model originally, but I don't see MobileBERT pre-trained for NSP on the HF hub so that the docs could be immediately made consistent.

Thank you so much for investigating! And for integrating this model into HF in the first place.

I'm not so sure the console output has to do with missing weights - the [source code](https://github.com/huggingface/transformers/blob/62ccbe0960019aceb4e36b1ee929ed2349e9653e/src/transformers/modeling_utils.py#L1597) for that log line comes from an if clause around the model checkpoint having  _extra_ weights that the model doesn't expect or need, i.e. the weights that `google/mobilebert-uncased` has that aren't used by MobileBertForNextSentencePrediction.

I wonder... if google/mobilebert-uncased is missing weights for the prediction head, why is there no log line from line [1610](https://github.com/huggingface/transformers/blob/62ccbe0960019aceb4e36b1ee929ed2349e9653e/src/transformers/modeling_utils.py#L1610) of modeling_utils.py, a few lines below the lines that logged the console output you posted?
```
if len(missing_keys) > 0:
  logger.warning(
    f"Some weights of {model.__class__.__name__} were not initialized from the model checkpoint at {pretrained_model_name_or_path} "
    f"and are newly initialized: {missing_keys}\n"
    f"You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference."
  )
```

Took another look at the pre-training objective for MobileBERT and it turns out that the NSP task is actually one of the two objectives (the other being masked LM). So the prediction head for NSP *should* be present in the checkpoint meant for pre-training, and it even seems to be loadable into the NextSentencePrediction model, judging by the lack of output by the line of code you highlighted.

Checked the code and yes, the MobileBertForPreTraining and MobileBertForNextSentencePrediction are crafted in such a way, state_dict-wise, that PreTraining is loadable into the NextSentencePrediction; the LM head won't be loaded, (but it shouldn't get used anyway). My theory doesn't explain the current state of affairs, then - the example should be working since the NSP from pretraining should have been transferred into the NSP-specific model. Will try and investigate further.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hello i know this long went stale, was there ever a consensus on this? Is this intended? Is there a way to fix this? I found it to cause my training to be unstable.