Squad 2.0
Author: abeljim
Pull request: yes

Added Squad 2.0 support. It has been tested on Bert large with a null threshold of zero with the result of
{
  "exact": 75.26320222353239,
  "f1": 78.41636742280099,
  "total": 11873,
  "HasAns_exact": 74.51079622132254,
  "HasAns_f1": 80.82616909765808,
  "HasAns_total": 5928,
  "NoAns_exact": 76.01345668629101,
  "NoAns_f1": 76.01345668629101,
  "NoAns_total": 5945
}
I believe the score will match google's 83 with a null threshold between -1 and -5 
run with [--version_2_with_negative] flag for SQuAD 2.0
[--null_score_diff_threshold $NULL_Threshold] to change threshold default value 0.0  
Tested Squad 1.1 with BERT base and it seems not  to break it results: 
{"exact_match": 79.73509933774834, "f1": 87.67221720784892}



Comments:
The new run_squad.py can train and predict , but can't predict only.
![image](https://user-images.githubusercontent.com/17742385/50643749-e48f5880-0fa9-11e9-9258-9872c6ddaaba.png)




in the predict only mode, len(nbest) is always 1

my scripts for predicting is 


python3 run_squad.py \
  --bert_model bert-large-uncased_up \
  --do_predict \
  --do_lower_case \
  --train_file squad/train-v2.0.json \
  --predict_file squad/dev-v2.0.json \
  --learning_rate 3e-5 \
  --num_train_epochs 2 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir squad2_diff \
  --train_batch_size 24 \
  --fp16 \
  --loss_scale 128 \
  --version_2_with_negative \
  --null_score_diff_threshold -2.6929588317871094
  
  


I've found the reason. Previously trained models are needed for prediction, but your code will check whether output_dir already exists and error will be reported if it exists, which is unreasonable.

the output_dir during evaluation should be consistent with the training process

the f1 without thersh is  80.8 and the best_f1_thesh is 81.2

Thanks for verifying the model works. I think predict not working by itself is not because of my code. I can create another pull request and fix it there. I look into it tomorrow morning.

Hi @abeljim, thanks for this, it looks nice!

Would you mind creating a separate example (e.g. called `run_squad_2.py`) instead of modifying `run_squad.py`? It will be less error prone and easier to document/maintain.

@thomwolf Yeah no problem I will work on that today 

> @zhaoguangxiang i trained as your script, like following:
> python run_squad.py 
> --bert_model bert-large-uncased 
> --do_train 
> --do_predict 
> --do_lower_case 
> --train_file $SQUAD_DIR/train-v2.0.json 
> --predict_file $SQUAD_DIR/dev-v2.0.json 
> --learning_rate 3e-5 
> --num_train_epochs 2.0 
> --max_seq_length 384 
> --doc_stride 128 
> --output_dir /tmp/squad3 
> --train_batch_size 1 
> --loss_scale 128 
> --version_2_with_negative 
> --null_score_diff_threshold -2.6929588317871094
> but i got the result like this:
> {
> "exact": 10.595468710519667,
> "f1": 13.067377840768328,
> "total": 11873,
> "HasAns_exact": 0.5398110661268556,
> "HasAns_f1": 5.490718134858687,
> "HasAns_total": 5928,
> "NoAns_exact": 20.62237174095879,
> "NoAns_f1": 20.62237174095879,
> "NoAns_total": 5945
> }
> i don't know why i got the wrong results, really need your help. thx.

the train_batch_size in your script is too small.




> my scripts for predicting is
> 
> python3 run_squad.py
> --bert_model bert-large-uncased_up
> --do_predict
> --do_lower_case
> --train_file squad/train-v2.0.json
> --predict_file squad/dev-v2.0.json
> --learning_rate 3e-5
> --num_train_epochs 2
> --max_seq_length 384
> --doc_stride 128
> --output_dir squad2_diff
> --train_batch_size 24
> --fp16
> --loss_scale 128
> --version_2_with_negative
> --null_score_diff_threshold -2.6929588317871094

Hi, How to find the best `null_score_diff_threshold` ?