why My Albert pretrain loss can't decrease?
Author: wa008
Pull request: no
Labels: name: Migration

# 📚 Migration

## Information

<!-- Important information -->

Model i am using Albert

Language I am using the model on just digits(Desensitized Chinese)

The problem arises when using: Albert Trainer

## Details
The loss can decrease normal when I take this config to RoBerta, I use Albert replace RoBerta, The loss can't decrease, I don't know what's the problem, please help

```
%%time
bert_file = './albert'
from transformers import Trainer, TrainingArguments
from transformers import LineByLineTextDataset, DataCollatorForLanguageModeling
from transformers import AlbertConfig, AlbertForMaskedLM

config = AlbertConfig(
    hidden_size = 768, 
    num_attention_heads = 12,
    intermediate_size = 3072, 
    vocab_size = vocab_size + 10
)
model = AlbertForMaskedLM(config=config)

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=True, mlm_probability=0.15
)

train_dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path="./my_wiki_file",
    block_size=128,
)

training_args = TrainingArguments(
    output_dir=bert_file,
    overwrite_output_dir=True,
    num_train_epochs=40,
    per_device_train_batch_size=64,
    save_steps=100000,
    save_total_limit=2,
    prediction_loss_only=False,
)

%%time
trainer = Trainer(
    model = model,
    args = training_args,
    data_collator = data_collator,
    train_dataset = train_dataset
)
trainer.train()
```
This is result
```
Step	Training Loss
500	6.687300
1000	4.034700
1500	3.826200
2000	3.777200
2500	3.788800
3000	3.751100
3500	3.780000
4000	3.772900
4500	3.795800
5000	3.737000
5500	3.782300
6000	3.775600
6500	3.821400
7000	3.730200
7500	3.751700
8000	3.787000
8500	3.824500
9000	3.746300
9500	3.782600
10000	3.770600
```

## Environment info
- `transformers` version:4.5.0.dev0
- Platform:Kaggle notebook
- Python version:3.7
- PyTorch version (GPU?):
- torch version: 1.8.0
- Using GPU in script?:YES
- Using distributed or parallel set-up in script?:NO


## Checklist

- [ ] I have read the migration guide in the readme.
 ([pytorch-transformers](https://github.com/huggingface/transformers#migrating-from-pytorch-transformers-to-transformers);
  [pytorch-pretrained-bert](https://github.com/huggingface/transformers#migrating-from-pytorch-pretrained-bert-to-transformers))
- [ ] I checked if a related official extension example runs on my machine.


Comments:
Hi! ALBERT is known to have issues converging in some cases as all layers are shared. See the following issues for similar issues and potential resolutions:

https://github.com/huggingface/transformers/issues/5984
https://github.com/huggingface/transformers/issues/4727
https://github.com/huggingface/transformers/issues/2553

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.