Add Roberta SQuAD model
Author: vlarine
Pull request: yes

There is the realisation of a RoBERTa SQuAD finetuning.

On 2x1080Ti on RoBERTa Base it gives:

python3 run_squad.py \
  --model_type roberta \
  --model_name_or_path roberta-base \
  --do_train \
  --do_eval \
  --train_file $SQUAD_DIR/train-v1.1.json \
  --predict_file $SQUAD_DIR/dev-v1.1.json \
  --per_gpu_train_batch_size 8 \
  --per_gpu_eval_batch_size 8 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --save_steps 2000 \
  --overwrite_output_dir \
  --verbose_logging \
  --output_dir /tmp/debug_squad/
Results: {'exact': 85.80889309366131, 'f1': 92.09291402361669, 'total': 10570, 'HasAns_exact': 85.80889309366131, 'HasAns_f1': 92.09291402361669,   'HasAns_total': 10570}

On RoBERTa Large:

python3 run_squad.py \
  --model_type roberta \
  --model_name_or_path roberta-large \
  --do_train \
  --do_eval \
  --train_file $SQUAD_DIR/train-v1.1.json \
  --predict_file $SQUAD_DIR/dev-v1.1.json \
  --per_gpu_train_batch_size 2 \
  --per_gpu_eval_batch_size 2 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --save_steps 2000 \
  --overwrite_output_dir \
  --verbose_logging \
  --output_dir /tmp/debug_squad/
Results: {'exact': 87.04824976348155, 'f1': 93.14253401654709, 'total': 10570, 'HasAns_exact': 87.04824976348155, 'HasAns_f1': 93.14253401654709,   'HasAns_total': 10570}

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1388?src=pr&el=h1) Report
> Merging [#1388](https://codecov.io/gh/huggingface/transformers/pull/1388?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/5c3b32d44d0164aaa9b91405f48e53cf53a82b35?src=pr&el=desc) will **decrease** coverage by `0.16%`.
> The diff coverage is `23.52%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1388/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1388?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1388      +/-   ##
==========================================
- Coverage   84.69%   84.52%   -0.17%     
==========================================
  Files          84       84              
  Lines       12596    12627      +31     
==========================================
+ Hits        10668    10673       +5     
- Misses       1928     1954      +26
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1388?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/1388/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3JvYmVydGEucHk=) | `61.17% <23.52%> (-10.05%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1388?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1388?src=pr&el=footer). Last update [5c3b32d...1ba42ca](https://codecov.io/gh/huggingface/transformers/pull/1388?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


I am also working on reproducing the results reported in the Roberta paper and found two issues in this PR. One issue is explained in the comment above. The other issue is that it is required to insert two sep_tokens between question tokens and answer tokens for Roberta as implemented [here](https://github.com/huggingface/transformers/blob/master/transformers/tokenization_roberta.py#L101). Therefore, `max_tokens_for_doc` should be `max_seq_length - len(query_tokens) - 4`.

In the Fairseq realisation of RoBERTa on Commonsense QA:
https://github.com/pytorch/fairseq/tree/master/examples/roberta/commonsense_qa
https://github.com/pytorch/fairseq/blob/master/examples/roberta/commonsense_qa/commonsense_qa_task.py

There is the only one sep_token between question and answer:
`
 Q: Where would I not want a fox? 
 A: hen house 
`

> In the Fairseq realisation of RoBERTa on Commonsense QA:
> There is the only one sep_token between question and answer:
> `
 Q: Where would I not want a fox? 
 A: hen house 
`

Thank you very much for your prompt reply. I did not know this. It seems to be appropriate to use single `sep_token` here because Commonsense QA is somewhat more similar to SQuAD than other tasks (e.g., GLUE).

Thanks for this @vlarine! (and @ikuyamada)

Would you agree to share the weights on our S3 as well?

Also, did you try with the same separators encoding scheme as the other RoBERTa models? 
`
 Q: Where would I not want a fox? 
 
 A: hen house 
` – did the results differ significantly?

No, I have not tried. But why there are two `
` tokens? I think more natural way is:
`
 Q: Where would I not want a fox? 
 
 A: hen house 
`

@vlarine See this docstring in `fairseq`: https://github.com/pytorch/fairseq/pull/969/files

Do you think you could try with this sep encoding scheme? Otherwise I'll do it in the next couple of days.

I would like to merge your PR soon. Any way you can give me write access to your fork, cf. https://help.github.com/en/articles/committing-changes-to-a-pull-request-branch-created-from-a-fork – so that i can add commits on top of your PR?



@vlarine @julien-c thanks for the amazing work! I can try it on SQuAD 2.0 and let you know if anything pops up there

Nice work. I also tried to add roberta into run_squad.py several days ago. Hope that my implementation would be useful. [run_squad.py with roberta](https://github.com/erenup/pytorch-transformers/pull/4) 

Folding this PR into #1386, which is close to being ready to being merged.

@vlarine @ikuyamada @pminervini @erenup Can you guys please check it out?

Closing in favor of #1386.