BertForMultipleChoice and Swag dataset example.
Author: rodgzilla
Pull request: yes

Hi!

This is the code that enables Bert models to be used for Multiple Choice problems (such as [Swag](https://github.com/rowanz/swagaf) and [ROCStories](http://cs.rochester.edu/nlp/rocstories/). 

For my implementation, I use the algorithm described in #90 and issue [#38](https://github.com/google-research/bert/issues/38) from the tensorflow implementation repo.

The commentaries and the `README.md` files have been updated but I would very much appreciate if someone could check my changes.

I am also unable to test my code on multiple GPUs so I can't check whether it works or not. 

I will let a training run during the night to see what kind of result we get although I won't be able to do a proper hyper-parameter search due my computing power limitations.

I also have a question. I used `pandas` to load the Swag dataset, do I need to specify it somewhere in a file to add it as a dependency for `pip`? I have never published a module on `pip`. 

Comments:
Hi Gregory, I will take some time to review and test that this week.

Just a word on additional dependencies, I would like to keep the package as light as possible (currently it's aligned with the dependencies of AllenNLP) so if you can manage to avoid adding any additional dependency it would be better.

Did you get good results fine-tuning the model on SWAG?
We should also indicate (reproducible) numbers in the `README.md` if we want to add this example to the repo (like the other examples).

Completely forgot to run the training, I've running it right now and I should have the results by the end of the day.

My parameters are the following ones:

```bash
python examples/run_swag.py \
--do_train \
--do_eval \
--do_lower_case \
--data_dir $SWAG_DIR/data/ \
--bert_model bert-base-uncased \
--max_seq_length 100 \
--train_batch_size 4 
--learning_rate 2e-5 \
--num_train_epochs 3.0 \
--output_dir /tmp/swag
```

The batch size of 4 isn't ideal but my GPU memory is pretty already full with these parameters.



I get a 77.76% accuracy on SWAG, I would be interested in the results with a batch size of 16 like in the Bert paper.

I've added my results to the readme and precised that the difference in performance was probably caused by the difference in `training_batch_size`.

@thomwolf Any chance you could run it on multiple GPUs?

I will commit a patch later to remove the pandas dependency.

Yes, I'll give a try on a bigger machine.

You can use gradient accumulation to get bigger batch size on a single GPU, you know right? (I wrote a lengthy blurb on that [here](https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255))

I didn't know actually, this is super cool! 

I'll give it try. Thanks for the link.

I did another finetuning using a `training_batch_size` of 16 and a `gradient_accumulation_steps` of 4 and I get a much better accuracy (80.62% instead of 77.76%). I have updated the readme accordingly and if everything is okay the branch should be ready for a merge.

@thomwolf Thanks for introducing to gradient accumulation, it's quite a neat trick and I think I will use it a lot more in the future. 

Oh that's great, now we are in the ballpark of the 81.6 reported in the BERT paper!
Looks good to me, I'm merging!

still not able to get 81.6.. after 3 epochs it get to 79.97.. any help?

Probably relevant #461 

Any chance someone can post the command for only testing, after training by using a model from a checkpoint? Because even when I test it on swag original data, it complains about having label column even though there is none. I set —do_test and but it’s still not working.

Can anybody help me to understand this code?
i looking for an example of using "Bertformultiplechoice" for training swag datasets.
Thank you in advance

 When I run run_multiple_choice.py I get this error, I am not sure why? I need help here.

Error:
2020-11-30 21:53:38.693828: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
Traceback (most recent call last):
  File "./transformers/examples/multiple-choice/run_multiple_choice.py", line 26, in <module>
    import transformers
  File "/usr/local/lib/python3.6/dist-packages/transformers/__init__.py", line 20, in <module>
    from . import dependency_versions_check
  File "/usr/local/lib/python3.6/dist-packages/transformers/dependency_versions_check.py", line 21, in <module>
    from .file_utils import is_tokenizers_available
  File "/usr/local/lib/python3.6/dist-packages/transformers/file_utils.py", line 88, in <module>
    import datasets  # noqa: F401
  File "/usr/local/lib/python3.6/dist-packages/datasets/__init__.py", line 26, in <module>
    from .arrow_dataset import Dataset, concatenate_datasets
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_dataset.py", line 41, in <module>
    from .arrow_writer import ArrowWriter, TypedSequence
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_writer.py", line 26, in <module>
    from .features import Features, _ArrayXDExtensionType
  File "/usr/local/lib/python3.6/dist-packages/datasets/features.py", line 210, in <module>
    class _ArrayXDExtensionType(pa.PyExtensionType):
AttributeError: module 'pyarrow' has no attribute 'PyExtensionType'

I uninstalled pyarrow and installed it again. it fixed the error.