the results of run_squad.py is terrible
Author: ppyu
Pull request: no
Labels: name: wontfix

# ❓ the results of run_squad.py is terrible
## I runned run_squad.py and the results is as folllows:
`Results: {'exact': 40.25098964036048, 'f1': 44.224848486777795, 'total': 11873, 'HasAns_exact': 80.61740890688259, 'HasAns_f1': 88.57652261867625, 'HasAns_total': 5928, 'NoAns_exact': 0.0, 'NoAns_f1': 0.0, 'NoAns_total': 5945, 'best_exact': 50.11370336056599, 'best_exact_thresh': 0.0, 'best_f1': 50.11370336056599, 'best_f1_thresh': 0.0}`

_**`exact`,`f1` and `HasAns_exact`,`HasAns_f1` are very different!**_

## The startup args are as follows:

export SQUAD_DIR=./datasets/squad

python run_squad.py \
  --squad_model bert_qa_model_squad \
  --model_type bert \
  --model_name_or_path bert-base-uncased \
  --do_train \
  --do_eval \
  --do_lower_case \
  --train_file $SQUAD_DIR/train-v2.0.json \
  --predict_file $SQUAD_DIR/dev-v2.0.json \
  --per_gpu_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir ./models/debug_squad/



Comments:
Maybe I not use the argument `--version_2_with_negative` while training on Squad V2.0 ?

Indeed, not using that clarg would result in such bad results as you're not taking into account impossible answers. Can you let us know if you obtain better results with `--version_2_with_negative`?

> Indeed, not using that clarg would result in such bad results as you're not taking into account impossible answers. Can you let us know if you obtain better results with `--version_2_with_negative`?

After using `--version_2_with_negative`, the result is also not satisfactory.
Both exact and f1 are not exceed **80%**.

The results are as follows:
{"exact_20000": 73.88191695443443, "f1_20000": 76.9165287314905, "total_20000": 11873, "HasAns_exact_20000": 68.60661268556005, "HasAns_f1_20000": 74.68453873633385, "HasAns_total_20000": 5928, "NoAns_exact_20000": 79.1421362489487, "NoAns_f1_20000": 79.1421362489487, "NoAns_total_20000": 5945, "best_exact_20000": 73.88191695443443, "best_exact_thresh_20000": 0.0, "best_f1_20000": 76.91652873149043,}

After the 25000th steps of training, the performance got worse.


That's already a lot better! What is this clarg `--squad_model bert_qa_model_squad`? It's not in the `run_squad.py` script?




> That's already a lot better! What is this clarg `--squad_model bert_qa_model_squad`? It's not in the `run_squad.py` script?

Oh ,I have adjusted some code from `run_squad.py` so that it could load and train my custom model from clarg.
And `bert_qa_model_squad` is the model whose code is copied from `BertForQuestionAnswering` in `transformers`.
So that I can design my custom model and then transmit it to the `run_squad.py` script.

Does your script differ in any other way to the `run_squad.py` script? Does your model differ in any way to the `BertForQuestionAnswering` model in `transformers`? Have you tried running the standard command on the `transformers` script and model?

```
python run_squad.py
--model_name_or_path bert-base-uncased
--do_train
--do_eval
--do_lower_case
--train_file $SQUAD_DIR/train-v2.0.json
--predict_file $SQUAD_DIR/dev-v2.0.json
--version_2_with_negative
--per_gpu_train_batch_size 12
--learning_rate 3e-5
--num_train_epochs 2.0
--max_seq_length 384
--doc_stride 128
--output_dir ./models/debug_squad/
```

> Does your script differ in any other way to the `run_squad.py` script? Does your model differ in any way to the `BertForQuestionAnswering` model in `transformers`? Have you tried running the standard command on the `transformers` script and model?
> 
> ```
> python run_squad.py
> --model_name_or_path bert-base-uncased
> --do_train
> --do_eval
> --do_lower_case
> --train_file $SQUAD_DIR/train-v2.0.json
> --predict_file $SQUAD_DIR/dev-v2.0.json
> --version_2_with_negative
> --per_gpu_train_batch_size 12
> --learning_rate 3e-5
> --num_train_epochs 2.0
> --max_seq_length 384
> --doc_stride 128
> --output_dir ./models/debug_squad/
> ```

sorry, I will try later and tell you the results.

> Does your script differ in any other way to the `run_squad.py` script? Does your model differ in any way to the `BertForQuestionAnswering` model in `transformers`? Have you tried running the standard command on the `transformers` script and model?
> 
> ```
> python run_squad.py
> --model_name_or_path bert-base-uncased
> --do_train
> --do_eval
> --do_lower_case
> --train_file $SQUAD_DIR/train-v2.0.json
> --predict_file $SQUAD_DIR/dev-v2.0.json
> --version_2_with_negative
> --per_gpu_train_batch_size 12
> --learning_rate 3e-5
> --num_train_epochs 2.0
> --max_seq_length 384
> --doc_stride 128
> --output_dir ./models/debug_squad/
> ```
I have run the official `run_squad.py` script，and the result is as follows:
10/16/2020 19:34:00 - INFO - main -   Results: {'exact': 72.60170133917292, 'f1': 75.79599520268259, 'total': 11873, 'HasAns_exact': 72.2165991902834, 'HasAns_f1': 78.61434734167507, 'HasAns_total': 5928, 'NoAns_exact': 72.9857022708158, 'NoAns_f1': 72.9857022708158, 'NoAns_total': 5945, 'best_exact': 72.60170133917292, 'best_exact_thresh': 0.0, 'best_f1': 75.79599520268256, 'best_f1_thresh': 0.0}


i have ran the official run_squad.py script, but got a 'cached_train_bert-base-uncased_384' not a result. Did i something wrong?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.