Add MultipleChoice to TFTrainer [WIP]
Author: ViktorAlm
Pull request: yes
Labels: name: TensorFlow

name: Ex: Multiple Choice

Working on getting the TF trainer running on TPUs for multiple choice. I'm quite the newb so input is needed.

- [x]  Add 1 GPU strategy
- [x]  Args for TPU Name
- [x]  Add MPC example for TFTrainer
- [x]  Fix docstrings

Examples:
GPU on Swag
https://colab.research.google.com/drive/1X9d1TY1I7r8d3rCJ9vWiy0UgvJM6BqQ4

TPU on Swag (fails, havent looked into it might be my misstake):
https://colab.research.google.com/drive/1X9d1TY1I7r8d3rCJ9vWiy0UgvJM6BqQ4#scrollTo=2r1Rng8NWzit

Glue on TPU (fails, havent looked into it, might be my mistake)
https://colab.research.google.com/drive/1f2x5wPSaBV0NbGCpMyUPXGUSMEGjSHeX#scrollTo=YDhXmegN3WWj

Comments:
Hey! 

Thanks a lot for your contribution on the trainer :+1:

Your PR looks great! Just have a small comment, see above. Don't hesitate if you have questions!

#4252 

```
05/11/2020 19:59:05 - INFO - oauth2client.transport -   Attempting refresh to obtain initial access_token
INFO:tensorflow:Found TPU system:
05/11/2020 19:59:05 - INFO - tensorflow -   Found TPU system:
INFO:tensorflow:*** Num TPU Cores: 8
05/11/2020 19:59:05 - INFO - tensorflow -   *** Num TPU Cores: 8
INFO:tensorflow:*** Num TPU Workers: 1
05/11/2020 19:59:05 - INFO - tensorflow -   *** Num TPU Workers: 1
INFO:tensorflow:*** Num TPU Cores Per Worker: 8
05/11/2020 19:59:05 - INFO - tensorflow -   *** Num TPU Cores Per Worker: 8
```
So its grabbing by name now by specifying --tpu_name

How do I see what fails with the code quality in __init__.py? I do run make style hehe

Great work with the trainer @jplu I saw your other plans for it. Would be very neat.

Ill mess around with the TFdataset and tpus on another branch, i need to read up on some stuff. Is there anywhere i can see the plans for a framework agnostic approach for the datasets?

Thanks :)

For the style I always do it manually in this following order:

```
isort --recursive examples templates tests src utils
black --line-length 119 --target-version py35 examples templates tests src utils
flake8 examples templates tests src utils
```
Then git add/commit/push.

What is currently working in your MPC example? Single GPU and CPU?

> Thanks :)
> 
> For the style I always do it manually in this following order:
> 
> ```
> isort --recursive examples templates tests src utils
> black --line-length 119 --target-version py35 examples templates tests src utils
> flake8 examples templates tests src utils
> ```
> 
> Then git add/commit/push.
> 
> What is currently working in your MPC example? Single GPU and CPU?

Yep only tested on my trusty rtx2080 and in colab on gpu.

Very Good!

You should also update the examples [README](https://github.com/huggingface/transformers/blob/master/examples/README.md) to add the check to the multiple_choice TFTrainer support :wink: 

Can you also add a TF section in the multiple_choice [README](https://github.com/huggingface/transformers/tree/master/examples/multiple-choice) plz

> Can you also add a TF section in the multiple_choice [README](https://github.com/huggingface/transformers/tree/master/examples/multiple-choice) plz

I should sleep on this. Thanks alot for the help!

Looks very good! Let me know once you think it can be merged.

@ViktorAlm I think it's great if you link to your Colab! Maybe just save it to a repo so that one can take a look at the history (and open issues if it ever breaks:)

None of the failing flake8 code quality seems to be related to me. If this works I think its done for now.

Looks good to me!

@julien-c and @LysandreJik should double check and merge if it seems good for them.

Maybe add a link to the syn dataset you added. It's [that one](https://aclweb.org/aclwiki/TOEFL_Synonym_Questions_(State_of_the_art)) right?

In the Colab ideally we would expose/expand a little bit more what happens in the example scripts (see what we did in the Colab for GLUE for instance), i.e. the calls to Trainer/TFTrainer.

We can improve/expand this in the future though @ViktorAlm @jplu 

> Maybe add a link to the syn dataset you added. It's [that one](https://aclweb.org/aclwiki/TOEFL_Synonym_Questions_(State_of_the_art)) right?

Its my own dataset which ill release once I'm done