A question about the test accuracy of BERT-based-uncased model on the MNLI dataset
Author: 14H034160212
Pull request: no
Labels: name: wontfix

# 🐛 Bug
I am curious about how accuracy you guys can reach by using the BERT-based-uncased model on the MNLI task? I got 61% test accuracy on the MNLI-matched dataset and 62% accuracy on the MNLI-unmatched dataset.
## Information

Model I am using Bert-based-uncased:

Language I am using the model on English:

The problem arises when using:
* I use the exactly the same official example scripts: (give details below)
https://github.com/huggingface/transformers#quick-tour
https://github.com/huggingface/transformers#run_gluepy-fine-tuning-on-glue-tasks-for-sequence-classification

The tasks I am working on is:
* An official GLUE task: (MNLI)

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: v2.11.0
- Platform: Linux
- Python version: 3.7
- PyTorch version (GPU?): 1.5.0
- Using GPU in script?: 2 Tesla V100 GPUs
- Using distributed or parallel set-up in script?: No


Comments:
Same, I can't replicate the results in the README file.

> Same, I can't replicate the results in the README file.

I also try to use BERT-large-uncased model and get 0.6209 eval_mnli/acc and 0.6281 eval_mnli-mm/acc.

FYI, I can get 84.25 eval_mnli/acc by using a non-zero number of warmup steps. . 
Specifically, adding these lines in `trainer.py`:
```
warmup_steps = float(num_training_steps)*0.1
scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=warmup_steps, num_training_steps=num_training_steps)
```

> FYI, I can get 84.25 eval_mnli/acc by using a non-zero number of warmup steps. .
> Specifically, adding these lines in `trainer.py`:
> 
> ```
> warmup_steps = float(num_training_steps)*0.1
> scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=warmup_steps, num_training_steps=num_training_steps)
> ```

Many thanks for your sharing. Cheers!

> FYI, I can get 84.25 eval_mnli/acc by using a non-zero number of warmup steps. .
> Specifically, adding these lines in `trainer.py`:
> 
> ```
> warmup_steps = float(num_training_steps)*0.1
> scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=warmup_steps, num_training_steps=num_training_steps)
> ```
Hi, Chaitanya. I try to fix the code, but I got an error when running the new program. Here is a double check that does the path of the ``trainer.py`` is ``src/transformers/trainer.py``? And the lines you added are beginning from the line 312 in the ``trainer.py``? Does the original code is ``scheduler = get_linear_schedule_with_warmup(
             optimizer, num_warmup_steps=self.args.warmup_steps, num_training_steps=num_training_steps
         )``? Many Thanks.
![微信截图_20200704002130](https://user-images.githubusercontent.com/23516191/86468851-5fe2ed80-bd8c-11ea-9ae7-5fdf26b77919.png)


Adding a non-zero number of warmup steps in `src/transformers/trainer.py` is the only change I made. This error seems to occur because you have an output label that is outside the range [0, n_classes) while computing the loss. 
Did you make any other changes to the trainer/dataset? And are you training with the original MNLI dataset?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.