Albert pre-train from scratch convergence problem
Author: yl-to
Pull request: no

# 🐛 Bug
Albert pre-train convergence problem
- The model training loss converged at 6.6 when using AlbertForMaskedLM as model class
- negative training loss when using AlbertForPretrain as model class

notice: I was deliberately set the eval dataset the same as training set for checking training loss at last run.
## Information
Using AlbertForMaskedLM as model class, figure showed below:
![image](https://user-images.githubusercontent.com/23205976/88234831-7ca56f00-cc2e-11ea-8ab7-75ce3e841b88.png)


Using AlbertForPretrain as model class, figure showed below:
![image](https://user-images.githubusercontent.com/23205976/88230876-0867cd00-cc28-11ea-90aa-a7412dd9c0c6.png)

Besides, when I was using the official `run_lanugage_modeling.py`, the training loss on wikiText-2 is also not converge to 0, it converged at 6.6 for several epochs.

Model I am using (Bert, XLNet ...):
Albert
Language I am using the model on (English, Chinese ...):
English
The problem arises when using:
* [x] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official task: wikiText-2
* [ ] my own task or dataset: (give details below)

## To reproduce
```
from transformers import (
    AlbertConfig,
    AlbertTokenizer,
    BertTokenizer,
    AlbertForPreTraining,
    AlbertForMaskedLM,
    LineByLineTextDataset,
    TextDataset,
    DataCollatorForLanguageModeling,
    Trainer,
    TrainingArguments
)
import math

albert_base_configuration = AlbertConfig(
    hidden_size=768,
    num_attention_heads=12,
    intermediate_size=3072,
)

tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')
# model = AlbertForPreTraining(config=albert_base_configuration)
model = AlbertForMaskedLM(config=albert_base_configuration)
train_dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path="/home/ubuntu/data_local/wikitext-2-raw/wiki.train.raw",
    block_size=512,
)
eval_dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path="/home/ubuntu/data_local/wikitext-2-raw/wiki.test.raw",
    block_size=512,
)

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=True, mlm_probability=0.15
)

training_args = TrainingArguments(
    output_dir="./results/new",
    overwrite_output_dir=True,
    num_train_epochs=5,
    per_gpu_train_batch_size=5,
    save_steps=10_000,
    save_total_limit=1,
    logging_steps=100,
    learning_rate=1.76e-3
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    prediction_loss_only=True,
)

trainer.train()
trainer.save_model("./results/new")

eval_output = trainer.evaluate()
perplexity = math.exp(eval_output["eval_loss"])
print({"loss": eval_output["eval_loss"]})
result = {"perplexity": perplexity}
print(result)
```
Steps to reproduce the behavior:

1.Download Albert here https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/
2.Run the script

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
- The training loss should converge to 0 in tiny datasets.
- Cross entropy should be always positive and eventually converge to zero.
<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: v3.02
- Platform: AWS instance
- Python version: 3.7.7
- PyTorch version (GPU?): 1.5.1 with GPU
- Tensorflow version (GPU?): None
- Using GPU in script?: Yes, 8 GPU in total
- Using distributed or parallel set-up in script? Not sure.


Comments:
@patil-suraj @julien-c @sgugger @sshleifer 

I tried different learning rate like 1.76e-3, 5e-5.(https://github.com/huggingface/transformers/issues/4727)
different dataset type: TextDataset, line_by_lineDataset.
None of them works.

I don't think your script will work like this, since the model requires two sets of labels (labels and sentence_order_label). The `DataCollatorForLanguageModeling` will generate the labels, but you will need to add something that generates the pair of sentences and adds the sentence_order_label.

> I don't think your script will work like this, since the model requires two sets of labels (labels and sentence_order_label). The `DataCollatorForLanguageModeling` will generate the labels, but you will need to add something that generates the pair of sentences and adds the sentence_order_label.

@sgugger Thanks for the reply! The `AlbertForPreTraining` model did requires 2 different labels. I am planning to create a PR for that, before that,
**Could I have any hints about why the `AlbertForMaskedLM` model didn't converge to 0?** Actually this is my major concern, I am planing to train the Albert using larger datasets like the whole wikipedia, if the MLM task even not converge on small dataset(say, wikiText-2), I think it impede my way to scale up the dataset.

@sgugger I used larger batch batch size the lower the learning rate did help the model converged to 0 in a very tiny text data(not wikiText-2), I repeated several articles several time to create that text dataset.

> I don't think your script will work like this, since the model requires two sets of labels (labels and sentence_order_label). The `DataCollatorForLanguageModeling` will generate the labels, but you will need to add something that generates the pair of sentences and adds the sentence_order_label.

@sgugger hi，I want to train a albert tiny model. Could you tell me is there any methods or class that can be used to generate the pair of sentences and adds the sentence_order_label ? Thanks a lot.

I have gotten it after read your commit. I think the class LineByLineWithSOPTextDataset in "transformers/data/datasets/language_modeling.py" can solve my problem. Thank yuo again,hh.

I also resolved with larger batch size (96) and lower learning_rate (5e-5). Thank you.