run_squad with xlm: Dataparallel has no attribute config. 
Author: waalge
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using XLM.

Language I am using the model on English:

The problem arise when using:
* [x] the official example scripts: run_squad.py

## To Reproduce

Steps to reproduce the behavior:

1. Azure VM with 2 GPUs 
2. run_squad with XLM
3. Everything fine until evaluation step. 

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
```
12/03/2019 15:54:26 - INFO - __main__ -   ***** Running evaluation  *****
12/03/2019 15:54:26 - INFO - __main__ -     Num examples = 10918
12/03/2019 15:54:26 - INFO - __main__ -     Batch size = 16
Evaluating: 100%|█████████████████████████████████████████████████████| 683/683 [05:16<00:00,  2.16it/s]
12/03/2019 15:59:42 - INFO - __main__ -     Evaluation done in total 316.178766 secs (0.028959 sec per example)
Traceback (most recent call last):
  File "transformers/examples/run_squad.py", line 575, in <module>
    main()
  File "transformers/examples/run_squad.py", line 564, in main
    result = evaluate(args, model, tokenizer, prefix=global_step)
  File "transformers/examples/run_squad.py", line 280, in evaluate
    model.config.start_n_top, model.config.end_n_top,
  File "/home/wallis/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 585, in __getattr__
    type(self).__name__, name))
AttributeError: 'DataParallel' object has no attribute 'config'
```
## Expected behavior

Calculate scores and prints them. 

## Possible suggestion for parallel use: 

Change that line 280 in run_squad.py to: 
```
model.module.config.start_n_top, model.module.config.end_n_top,
```
as suggested [here](https://discuss.pytorch.org/t/dataparallel-throws-an-error-attributeerror-dataparallel-object-has-no-attribute-loss/34228). 

With this change, it seems to progress but only reach another 
[error](https://github.com/huggingface/transformers/issues/1771)
so not sure. 

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: ubuntu 16.04 on azure
* Python version: 3.7
* PyTorch version: 1.3.1 
* PyTorch Transformers version (or branch): 2.2.0
* Using GPU ? yes
* Distributed of parallel setup ? I think its trying to do parallel
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
Having exactly same error when updating to transformers v2.2.1, have you fix the bug yet?


`12/05/2019 08:57:41 - INFO - __main__ -   Saving features into cached file ./datasets/SQuAD/cached_dev_xlnet-base-cased_384
12/05/2019 08:57:53 - INFO - __main__ -   ***** Running evaluation  *****
12/05/2019 08:57:53 - INFO - __main__ -     Num examples = 12551
12/05/2019 08:57:53 - INFO - __main__ -     Batch size = 32
Evaluating: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 393/393 [04:08<00:00,  1.58it/s]
12/05/2019 09:02:01 - INFO - __main__ -     Evaluation done in total 248.621486 secs (0.019809 sec per example)
Traceback (most recent call last):
  File "./examples/run_squad.py", line 578, in <module>
    main()
  File "./examples/run_squad.py", line 567, in main
    result = evaluate(args, model, tokenizer, prefix=global_step)
  File "./examples/run_squad.py", line 283, in evaluate
    model.config.start_n_top, model.config.end_n_top,
  File "/root/workspace/renqian/kzs/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 585, in __getattr__
    type(self).__name__, name))
AttributeError: 'DataParallel' object has no attribute 'config'`

The suggested fix gets pass that step. 
However, the evaluation step errors (as linked) and the model output is junk. 
By which I mean, I ran the output model on my own evaluation script and it gave junk answers. 

I don't understand why the training script run_squad is organized like it is, why the comments refer only to xlnet, yet apply to xlm also (in a separate script it says the heads of xlnet and xlm are the same...), or why xlm has two classes: ".XLMForQuestionAnswering" and "XLMForQuestionAnsweringSimple", or etc... 
I'm sure people had good reasons for all these things, but they aren't apparent to me.

I posted this as a solidarity search cos I couldn't find anyone else saying they ran into this problem. 

If someone knows where to find the script that the XLM authors used to train for squad, please share.

Hi! A very big SQuAD refactor was done these past few weeks, and the issue you're talking about with `DataParallel` was fixed. You can try the new `run_squad` script (make sure you install the library from source beforehand as it leverages several important and recent abstractions).

As for your other questions, I'll try to answer as best as I can:
>With this change, it seems to progress but only reach another
error

This error was patched as well with the new `run_squad` script.

> I don't understand why the training script run_squad is organized like it is, why the comments refer only to xlnet, yet apply to xlm also

It was organized like it was because models were added separately. I agree that as more models were added, there was a discrepancy between the comments and the code. The comments should be more understandable as of now.

> why xlm has two classes: ".XLMForQuestionAnswering" and "XLMForQuestionAnsweringSimple"

This is the case for both XLNet and XLM. Models that are used with question answering heads (like BERT or RoBERTa) usually add a simple linear layer on top of the transformer model. This linear layer gets as input the transformer outputs, and outputs logits corresponding to the beginning and end of the predicted sequence.

This is not the case with either XLNet or XLM, which use much more complex question answering heads. For example, `XLNetForQuestionAnswering` has the [following architecture](https://github.com/huggingface/transformers/blob/master/transformers/modeling_xlnet.py#L1358).

This leads to a difference in outputs: traditional question answering heads output only two values: `(start_logits, end_logits)`, while XLNet and XLM output five values: `(start_top_log_probs, start_top_index, end_top_log_probs, end_top_index, cls_logits)`

This introduces a more complex post-processing, and explains why [two methods are necessary in the `run_squad.py` script](https://github.com/huggingface/transformers/blob/master/examples/run_squad.py#L305-L314): one needs to handle the two outputs and the other one the five outputs.

The models `XXXForQuestionAnsweringSimple` use a simple dense layer like the one used by BERT/RoBERTa. Those models are not currently supported by the `run_squad` script, but they eventually will.

------

We just released the new `run_squad` script this morning and do not have the time nor compute to test it extensively on all the models supported. We would gladly appreciate it if you could share your results when using this script so that we may be aware of improvements that need to be made, especially for newly supported models like XLM.

Let me know if you have any other questions.

Hey!! 

Thanks LysandreJik for your detailed response. 

I'm pretty sure that when I ran the code I pip installed transformers from the pypi repository, 
but ran the run_squad from a clone of the git repo. Probably not the best idea. 

Yeah I saw the refactor this morning and have been going through the code. 
Its a huge improvement.  Still, there are many bits in [this](https://github.com/huggingface/transformers/blob/master/examples/run_squad.py) like:

* Line 301 queries by model type (quite clear) 
* Line 266 queries by number of output commenting on model type (huh, why?) 

I was prepping a feature request to push more of the functionality into the ``XXXForQuestionAnswering`` classes. 
And I still think it would be so much better if this was done. 

Much of the answering cleaning in the squad scripts is useful in application, 
and it would remove this persistent if... else conditioning.

I'll post the feature request and maybe we can discuss the merits/ demerits there. 

I also saw you released distil-mbert. I tried finetuning that on monday, which worked more or less (I had to drop the --evaluating_during_training as it wasn't happy with it - i didnt record the error, sorry). (Oh, and the bootstrapping to other languages proved to be wishful thinking). 

I have previously finetuned distilbert on squad using the example arguments in the docs. The model that is output, when you query it returns 2 tensors (start and end logits, maybe?) 
My finetuned distil-Mbert gives those, plus an additional tuple of three other things...  
(Which caused me issues integrating it into my own test framework. 
And I didn't bother going to find out what they were by this point. )
But it seemed inconsistent for this to happen. 

> do not have the time nor compute to test it extensively on all the models supported. We would gladly appreciate it if you could share your results when using this script so that we may be aware of improvements that need to be made, especially for newly supported models like XLM.

Very happy to. I'm current burning through my free trial accounts on various cloud compute services. 
Rather than me saying "I set up a VM, installed these things, ran this code with these parameters, it took n hours and here are my copy and pasted results/ error messages". 

How easy would it be to properly formulate/automate this? 
Ie instead have a script that takes a ip, port, username, password, and automatically sets up and experiment and formats a report of the results? 
This would probably give better quality reporting, and would make my life easier. 

There's a new script, which doesn't get this far, so I'll close this and make a new one.