Standardized head for Question Answering
Author: andrelmfarias
Pull request: no
Labels: name: wontfix

Hi,
With some colleagues, we developed a QA system that uses a whole QA pipeline (Retriever, Reader, Ranker). We use your older version of `BertForQuestionAnswering` as Reader and now we wish to update it to be compatible with your new release and to add others models as well (XLNet, XLM).

Our system uses the logits outputted by the model in order to rank the answers between different paragraphs (using probabilities outputted by the softmax layer is an incorrect approach for such systems). 

However, we understand that in your new API, the `forward` method of QA models now can only return probabilities. We suggest you add the option to output the raw logits as well. 

We could of course overcome this by using the functions `self.start_logits` and `self.end_logits`, but we think that this feature can be useful for other users as well

Comments:
Well, the output of the `BertForQuestionAnswering` model hasn't changed, the returned `start_score` and `end_score` are still scores before the softmax.

Can you point more specifically to the changes you are referring to?

I'm sorry. Indeed the implementation of `BertForQuestionAnswering` remains the same.

Actually, I was referring to the implementation of `XLNetForQuestionAnswering`, which is pretty different from `BertForQuestionAnswering` (I thought you had standardised the implementation for all QA models and I hadn't checked the `BERT` implementation before posting the issue here)

Please correct me if I am wrong, but I do not see the `forward()` method outputting the Start and End's logits here (only the softmax probas - `start_log_probs`and `end_log_probs`):

https://github.com/huggingface/pytorch-transformers/blob/268c6cc160ba046d6a91747c5f281f82bd88a4d8/pytorch_transformers/modeling_xlnet.py#L1226-L1290

Same with XLM:

https://github.com/huggingface/pytorch-transformers/blob/fec76a481d1ecfbf068d87735dd44ffc26158f6e/pytorch_transformers/modeling_xlm.py#L899-L921

https://github.com/huggingface/pytorch-transformers/blob/fec76a481d1ecfbf068d87735dd44ffc26158f6e/pytorch_transformers/modeling_utils.py#L649

Oh yes the official XLNet implementation uses a beam search for Question Answering so the output is more complex.

I'll see if I can come up with a standardized way to use both.

Thanks, I get it. Maybe caching the `start_logits` during the search and returning the logits of chosen End and Start positions would be a solution.

By the way, FYI the examples in the documentation for `XLNetForQuestionAnswering` and `XLMForQuestionAnswering` are incorrect, both show `start_scores` and `end_scores` in the outputs and the example in `XLNetForQuestionAnswering` uses `XLM` instead of `XLNet`

[`XLNetForQuestionAnswering`](https://huggingface.co/pytorch-transformers/model_doc/xlnet.html#xlnetforquestionanswering):

![image](https://user-images.githubusercontent.com/43521764/61778726-98015280-adff-11e9-878d-f6c51aa511b2.png)

[`XLMForQuestionAnswering`](https://huggingface.co/pytorch-transformers/model_doc/xlm.html#xlmforquestionanswering):

![image](https://user-images.githubusercontent.com/43521764/61778845-d39c1c80-adff-11e9-985b-d844ebf3ed4b.png)


This is silly/nitpicky - but can we change the title of this issue? Got very worried I had been using the BERT heads incorrectly until I read into the weeds of the comments..

Hello, any news on this standardized head? @thomwolf?
What do you think about my proposition about caching the `start_logits` during the Beam search and outputting both `start_logits` and `end_logits` as it is done with `BertForQuestionAnswering`?
Or do you have any other ideas?

I can try to work on that and do a PR if you need

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.