[Questions & Help] The loss doesn't decrease correctly while training BERT from scratch
Author: ghrua
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

I am now using [huggingface/transformers](https://github.com/huggingface/transformers) to train a BERT model on **1m** wiki data **from scratch**, but the training loss looks so weird. Before showing the details of the training process, I would first share the scripts and configs I used:

```
python run_language_modeling.py --output_dir $OUTPUT_DIR \
                                --model_type bert \
                                --mlm \
                                --config_name $CONFIG_AND_DATA_DIR \
                                --tokenizer_name $CONFIG_AND_DATA_DIR \
                                --do_train \
                                --do_eval \
                                --num_train_epochs 20 \
                                --learning_rate 1e-4 \
                                --save_steps 250 \
                                --per_gpu_train_batch_size 64 \
                                --evaluate_during_training \
                                --seed 404 \
                                --block_size 256 \
                                --train_data_file $DATA_DIR/train.txt \
                                --eval_data_file $DATA_DIR/valid.txt \
                                --evaluate_during_training \
                                --logging_steps 250 > log.bert
```

where [run_language_modeling.py](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_language_modeling.py) is the python script provided by huggingface. I didn't change the config of BERT, except for the vocabulary size. The vocabulary, or the tokenizer, was trained using [huggingface/tokenizers](https://github.com/huggingface/tokenizers).


I put the log of training loss [here](https://gist.github.com/ghrua/01fd859707923f80f1e16af5c2bd3f6a). We can see that, after 20 epochs, the training loss decreases from `7.84` to `7.60`. That's so weird, since the number of lines of the raw data is just 1 million, the training loss should decrease sharply with such an amount of data. Note that I also used the same python script to train a GPT-2 from scratch with the same data, and it worked very well, the loss decreased as expected.


I have tried several ways to address this issue:

1. Set the batch size as big as the GPU can afford. Since BERT just predicts 15% tokens at each time step, big batch size may give the model more error signals while training. However, it doesn't help.
2. Maybe the learning rate is too small. I tried to adjust the `learning_rate` to 5e-4, unfortunately, the converged loss even became worse.
3. Maybe the `vocab.txt` has some problem, which is extracted from my training data using the toolkit [huggingface/tokenizer](https://github.com/huggingface/tokenizers). Then, I used the `vocab.txt` and `config.json` downloaded from this repo to run the python script, but it met the same problem.
4. I also run the same script on 2k data. I trained the BERT model for 200 epochs, and the converged training loss was at around `6.8`, which cannot even overfit on a toy dataset.

Thanks for your kindly help!!!

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:
[Link to the question asked on SO](https://stackoverflow.com/questions/61873435/the-loss-doesnt-decrease-correctly-while-training-bert-from-scratch)


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.