Refactor functionality of run_squad and squad_utils into XXXForQuestionAnswering
Author: waalge
Pull request: no
Labels: name: wontfix

## Request 

Push some/most/all the functionality of the squad training scripts into the class ``XXXForQuestionAnswering``.

## Alt Request. 

I'm guessing this is immediately objectionable, since ``XXXForQuestionAnswering`` is just the nice clean NN.
No messy string manipulation functions welcome here.

So perhaps I request a ``XXXQuestionAnswerHandler`` class.

# Context 

Basically, as far as I can tell, there is quite a gulf between the input/output of ``XXXForQuestionAnswering`` and actually doing squad. 
Currently, we attempt to straddle this gulf by a number of scripts, perhaps what's called glue code(?). 
These require passing around of many arguments, so many that I can't keep track of them, 
and a lot of conditionals to treat the idiosyncrasies of different models. 

I think separating the ``XXXForQuestionAnswering`` from actually being able to do a squad like task is a cause of some problems.  
If these models are really supposed to be used for question answering, 
and not simply churning through a full training/eval squad-style json, 
then these auxilliary scripts of answer selection, and answer cleaning should be fastened to the model firmly (like within some handler). 

Squad has set the defacto format for CDQA, 
and many of the steps in the scripts would be useful in wider applications.

## Context II

A massive thanks for refactoring the squad training code. 
It is so much clearer than it was but I still think there's big room for improvement. 

For me, using the previous incarnation of squad_run was ummm... constantly problematic (eg [like this](https://github.com/huggingface/transformers/issues/2038)). 
Surely 97% cos I'm a noob - some of the issues I had unfathomably basic. 
But the scripts were really not user friendly (now improved but still - see previous), and the current classes really don't give much away:

* "Hello, my dog is cute" is not a CDQA [?!](https://github.com/huggingface/transformers/blob/master/transformers/modeling_xlm.py#L869).

* The ``...Simple`` class, partly clarified by LysandreJik's [post](https://github.com/huggingface/transformers/issues/2038#issuecomment-564220238), 
but doesn't explain why these are still lingering on the master branch without appearing in the official docs, while the tensor flow analogue only has a simple...
I mean its fine, but its pretty confusing if you're already confused. 

* the documentation of the output being "well depends on the config" which although true is... not exactly useful to the uninitiated.

I got to the point of trying to refactor the training scripts myself, but made very little progress. 
Very happy to see someone else has been on the case. 

## Context III

An example functionality: allow pre-cache examples-to-feature - effectively do a sort of dry run. 

Running ``squad_run.py`` on a GPU machine saw a lot of idle time where one CPU would take an hour to cache the examples suitable to the configuration of the finetuning.
Why not build a ``examples_to_features`` method within the class? 
Then do this on your desktop before shipping if off for training _oven-ready_. 
At the moment in the script ``squad_run.py``, the caching for the training is called from ``main`` (not ``train``) and caching the evaluation was done in ``evaluate``.
I don't follow this decision. 
I tried to extract the caching function, but it was super hacky and would be addressed by the request.


## Invitation for comments

I'm sure this isn't a new idea, and hasn't happened cos its either too much work or a terrible idea. 
I tried to see what others were doing but the scripts and specifics people have used to arrive at their claimed squad scores
do not seem as available as other resources ( ... XLM? ... Roberta? - Am I missing something?) 

I'd be very happy to hear thoughts on this, 
including responses that begin 

> _"This is nonsense because ..."_ 

Thanks HF for your awesome opensource NLP lib

Comments:
Thanks a lot for your input. We're trying to continually improve our training scripts and would like to keep them efficient while keeping them understandable.

As you have noticed, we have recently refactored the glue and squad scripts somewhat, and will continue to do so. Your input is appreciated and we're keeping it in mind for the future improvements that are bound to happen (sooner rather than later).

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.