[examples] Add trainer support for question-answering
Author: patil-suraj
Pull request: yes

This PR adds trainer support for question-answering task. Regarding issue #4784

**TODOs**
- [ ]  Add automatic data loading. Right now it requires the user to specify data directory. Decided not to use `tfds` because I think it will be soon replaced by `nlp` here
- [ ] Add evaluation
- [ ] Test all models.

@julien-c  @patrickvonplaten 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4829?src=pr&el=h1) Report
> Merging [#4829](https://codecov.io/gh/huggingface/transformers/pull/4829?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/d2a93991158f15993eba9ab421d82766b892f948&el=desc) will **increase** coverage by `1.00%`.
> The diff coverage is `49.41%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4829/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4829?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4829      +/-   ##
==========================================
+ Coverage   76.84%   77.84%   +1.00%     
==========================================
  Files         141      142       +1     
  Lines       24685    24768      +83     
==========================================
+ Hits        18969    19281     +312     
+ Misses       5716     5487     -229     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4829?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/data/datasets/squad.py](https://codecov.io/gh/huggingface/transformers/pull/4829/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL2RhdGFzZXRzL3NxdWFkLnB5) | `47.56% <47.56%> (ø)` | |
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/4829/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `99.22% <100.00%> (ø)` | |
| [src/transformers/data/datasets/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/4829/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL2RhdGFzZXRzL19faW5pdF9fLnB5) | `100.00% <100.00%> (ø)` | |
| [src/transformers/modeling\_tf\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/4829/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9vcGVuYWkucHk=) | `20.78% <0.00%> (-74.20%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/4829/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9iZXJ0LnB5) | `73.37% <0.00%> (-25.00%)` | :arrow_down: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/4829/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `79.72% <0.00%> (-1.38%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4829/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.21% <0.00%> (+1.00%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_electra.py](https://codecov.io/gh/huggingface/transformers/pull/4829/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9lbGVjdHJhLnB5) | `95.38% <0.00%> (+68.46%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_mobilebert.py](https://codecov.io/gh/huggingface/transformers/pull/4829/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9tb2JpbGViZXJ0LnB5) | `96.72% <0.00%> (+73.10%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4829?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4829?src=pr&el=footer). Last update [d2a9399...5497ae6](https://codecov.io/gh/huggingface/transformers/pull/4829?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Hi @patil-suraj,

I think @julien-c can answer questions regarding the Trainer better :-) 

Just in case you wanted to use Weights & Biases, you should just have to do a `pip install wandb` and it should automatically track everything.

>My main question is, have you reproduced the training results that are documented in https://github.com/huggingface/transformers/blob/master/examples/question-answering/README.md ?

I didn't train `bert-base` (just trained for 1 epoch to see if the implementation was working) but instead I used it to train `electra-base`  and it gave better results than mentioned in the paper

In the paper the authors mentioned that electra-base achieves 84.5 EM and 90.8 F1. I was able to achieve 85.05 EM and 91.60 F1. Sadly didn't use wandb, you can find the colab [here](https://colab.research.google.com/drive/11yo-LaFsgggwmDSy2P8zD3tzf5cCb-DU?usp=sharing)

It uses the same code, just copy pasted in colab. But if required I can try to reproduce the documented results.

> I didn't train `bert-base` (just trained for 1 epoch to see if the implementation was working) 

I can do it tomorrow morning, I currently have a V100 on hand:)

Just a note that I tried `python run_squad_trainer.py --model_name_or_path bert-base-uncased --model_type bert --data_dir squad --output_dir /tmp/debug_squad/ --overwrite_output_dir --do_train --do_eval --evaluate_during_training --logging_steps 100`.

For some reason I don't get any evaluation metric during training (I was expecting `loss` or `eval_loss`).

> Just in case you wanted to use Weights & Biases, you should just have to do a `pip install wandb` and it should automatically track everything.

@borisdayma yes, there are no start and end positions in eval dataset which is why eval loss is not calculated. I will add that. Were you able to see training loss ?
Thanks !

> yes, there are no start and end positions in eval dataset which is why eval loss is not calculated. I will add that. Were you able to see training loss ?

Hmm, I'm pretty sure the dev-v1.1.json file has the same labels as the training one (start positions). Otherwise we wouldn't have any eval results at all in the readme. No?

pinging @LysandreJik on this:)

> @borisdayma yes, there are no start and end positions in eval dataset which is why eval loss is not calculated. I will add that. Were you able to see training loss ?

Yes, training loss was logged.

@julien-c In the two `TensorDatasets` created (one for training and one for evaluation), only the training has the correct `start_position` and `end_position`.

I believe this is because while the training dataset only has one possible answer per question, the dev and validation datasets both have multiple answers per question (usually different-lengths spans).

@LysandreJik So I guess we should update the eval dataset to pick one start_position (or the most frequent one) – how do people do it usually with SQuAD eval, do you know @thomwolf?

Maybe this can be done in a second PR though. Everyone ok with merging this (renaming `run_squad_trainer.py` to `run_squad.py`)?

@patil-suraj Can you resolve the conflicts and switch to the new `default_data_collator` now that it should work for your dict inputs?
I can take over if you don't have time, but this is the only thing standing in the way of merging this PR.

@sgugger Yes, I'll  switch to the new data collator. 

Hi @sgugger, you can take this over, I'm running short on time ;(

Thanks @sgugger :)

@sgugger can you please rename `run_squad_trainer.py` to `run_squad.py`? see also #5547