F1 and EM scores output for run_squad.py
Author: mingkkkkk
Pull request: no
Labels: name: Help wanted

Hi,

I was doing prediction after fine-tuning the bert-base model and I was wondering whether the f1 and em scores will show automatically since I only saw the following two log outputs 
03/02/2019 22:20:05 - INFO - __main__ -   Writing predictions to: /tmp/debug_squad/predictions.json
03/02/2019 22:20:05 - INFO - __main__ -   Writing nbest to: /tmp/debug_squad/nbest_predictions.json

Where am I able to get those scores? Thanks for any help!


Comments:
Use the Squad python scripts available on their website

Is that run_squad.py? I used that one but didn’t see output scores, having
the output predictions files though. Thanks!

abeljim <notifications@github.com>于2019年3月3日 周日上午3:03写道：

> Use the Squad python scripts available on their website
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/pytorch-pretrained-BERT/issues/336#issuecomment-469011833>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/At7y4r3yC9C41y_BrZptqWiUtrmLwnuMks5vS6vtgaJpZM4baqZe>
> .
>


https://rajpurkar.github.io/SQuAD-explorer/ get the eval script for the correct version

Thanks so much, really helps!

abeljim <notifications@github.com>于2019年3月3日 周日上午3:08写道：

> https://rajpurkar.github.io/SQuAD-explorer/ get the eval script for the
> correct version
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/pytorch-pretrained-BERT/issues/336#issuecomment-469012224>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/At7y4u9BkK-km5KUXJuBfzM3loMnr7gwks5vS60wgaJpZM4baqZe>
> .
>


@abeljim  Hi Abel, 

I got a very strange problem in running the prediction only for run_squad.py and wonder if you have any idea about why this happens.
  I first ran the following codes to both do train and predict on the files:
 `python run_squad_final.py   --bert_model bert-base-uncased  --do_train   --do_predict   --do_lower_case   --train_file train-v2.0.json   --predict_file dev-v2.0.json   --train_batch_size 6   --learning_rate 3e-5   --num_train_epochs 1.0   --max_seq_length 384   --doc_stride 128   --fp16   --version_2_with_negative   --null_score_diff_threshold -1   --output_dir ./temp/
/`

 The output predictions.json file looks normal, but when I tried to delete the "--do_train" part and only do the prediction on the same file, it gives very different and strange outputs, many of the answers are repetitive as below and the scores are like only 0.1:
(The output in predictions are like:)

>  "68cf05f67fd29c6f129fe2fb9": "mands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their",
>  "f5fead9187d56af2bdbfcb921": "mands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their",
> "f9183ead5bb93aaa12ea37245": "mands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their",
>     "d583847c96cbbbfaaa99dfcad": "Rollo, agreed to swear fealty",
>     "e06fbbdb50af7ab3faefde618": "Rollo, agreed to swear fealty",
>     "cbb48eaacbbbfcccefb7aab7f": "Rollo, agreed to swear fealty",

Do you know what caused the problem?

Yeah sorry I forgot to respond. The way the it runs if the train flag is  off that it will load a pretrained version of bert and run the prediction on that. A way to get this to work is to modify the file to load a saved trained version instead. I could add this functionality but I'm busy with school for the next two weeks. Modify lines 1011 to 1025 for a quick fix