MNLI finetuning results affected by values of max_steps
Author: leo-liuzy
Pull request: no
Labels: name: wontfix

# 🐛 Bug
If you follow my bash script and change MAX from 40000 to 50000, the performance is dramatically affected. However, 50000 may not be the smallest number that have this behavior and I am not sure if this is a bug or something expected; the performance is severely affected, so I decided to report.
## Information

Model I am using (Bert, XLNet ...): bert-base-uncased
Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Set MAX=40000, Run my provided code
2. Set MAX=50000, Run my provided code
3.

```
export GLUE_DIR=downstream_datasets/glue
export TASK_NAME=QQP
SAVING_PERIOD=1000
MAX=40000
BS=32
SEED=42
#EPOCH=1
export CUDA_VISIBLE_DEVICES=0
for TASK_NAME in MNLI # QQP SST-2
do
  echo $TASK_NAME
  python run_glue.py \
    --model_type bert \
    --model_name_or_path bert-base-uncased \
    --task_name $TASK_NAME \
    --do_train \
    --save_steps $SAVING_PERIOD \
    --logging_steps $SAVING_PERIOD \
    --max_steps $MAX \
    --do_eval \
    --data_dir $GLUE_DIR/$TASK_NAME \
    --max_seq_length 128 \
    --per_gpu_train_batch_size $BS \
    --per_gpu_eval_batch_size $BS \
    --learning_rate 2e-5 \
    --seed $SEED \
    --output_dir model_output/"$TASK_NAME"_maxlen128_seed"$SEED"_savesteps"$SAVING_PERIOD"_maxsteps"$MAX"/ \
    --logging_dir runs/"$TASK_NAME"_maxlen128_seed"$SEED"_savesteps"$SAVING_PERIOD"_maxsteps"$MAX"/ \
    --logging_first_step \
    --evaluate_during_training
done
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
From my limited experiments, in the tensorboard dashboard:
MAX=40000: the acc will keep climbing.
MAX=50000: the acc will fluctuate throughout training. It can not exceed 40%.

I suspect MNLI's performance is affecteed by learning rate set by optimizer and scheduler, which are affected by the max_steps. However, this behaviour **does not** hold on other similar-sized datasets like QQP and SST-2. This inconsistency is confusing.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.8.0
- Platform: Linux-5.3.0-46-generic-x86_64-with-debian-buster-sid
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.0+cu101 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: single gpu


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.