SQuAD: Implement eval in Trainer-backed run_squad_trainer
Author: julien-c
Pull request: no
Labels: name: Good First Issue

A very good first issue IMO!

See https://github.com/huggingface/transformers/pull/4829#issuecomment-645994130

> we should update the eval dataset to pick one start_position (or the most frequent one)

Optionally, use the `huggingface/nlp` library to get the eval dataset, and hook it into the Trainer.

Also referenced in https://github.com/huggingface/transformers/issues/6997#issuecomment-688747680

Comments:
Hi Julien, I would like to contribute. Could you assign me to the issue?

Yes, go for it! @sgugger and I (and others) can help if needed

Hi there! I missed that there was an already an assignee, but I took a crack at it too.

I initially tried to pick the most frequent `start_position` —however the same `start_position` may have `answer_text`s of different length (i.e one answer could be a substring of another answer). I wasn’t sure how this should be best handled in `transformers/data/processors/squad.py(737)` where the length of the `answer_text` is used to calculate the start and end positions (Maybe I could try picking the start_pos of the most frequent *unique* answer_texts?). Given that, I went with picking the answer in the first index as a first attempt. 

How were we doing it in `run_squad.py` again, @LysandreJik?

In `run_squad.py` we were leveraging the [`squad_metrics.py` evaluation script](https://github.com/huggingface/transformers/blob/master/src/transformers/data/metrics/squad_metrics.py).

If I recall correctly, it gets the sequence predicted by the model from the start and end positions. It then compares that sequence to all the possible answers.

For the evaluation I think it is imperative to compare the model prediction against all possible answers, so I don't think the evaluation can simply be solved by adding a single start/end position.

Thanks for the pointer, that makes sense. I'm interested in looking into it further, but I'll jump out and return the reigns to @ovbondarenko 

see also the `squad` metrics in the huggingface/nlp library: https://github.com/huggingface/nlp/blob/master/metrics/squad/evaluate.py

Thanks for the pointers, everyone! .. and I am sorry for hijacking the issue, @kay-wong. @LysandreJik, I agree that we might want to check if predicted answer matches any one for the possible answers. I am going to give this approach a go.

@julien-c, @LysandreJik I've spent some time trying to decide on the best way to implement the evaluation, and I've realized that I need some help. I think the fastest way would be to basically reuse `load_and_cache_examples()` and `evaluate()` functions from [run_squad.py](https://github.com/huggingface/transformers/blob/master/examples/question-answering/run_squad.py) with adapted args. But it may not be the best way.

`evaluate()` function from run_squad.py is also handling predictions using `compute_predictions_logits` from [squad.py](https://github.com/huggingface/transformers/blob/master/src/transformers/data/metrics/squad_metrics.py). I would prefer to use the output from `trainer.predict` for a squad evaluator downstream. By design [trainer](https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py) allows to substitute `compute_metrics` method by a custom evaluation method, so I am assuming this would ultimately be the preferred implementation of a custom squad evaluator for [run_squad_trainer.py](https://github.com/huggingface/transformers/blob/master/examples/question-answering/run_squad_trainer.py). Problem is, I am struggling to figure out what exactly trainer.predict is returning. 

When I call trainer.predict on dev-v2.0 it returns predictions in a form of ndarray, which is clear enough:

```
# from trainer_utils.py:
class PredictionOutput(NamedTuple): 
    predictions: np.ndarray 
    label_ids: Optional[np.ndarray] 
    metrics: Optional[Dict[str, float]] 
```

I thought that the predictions ndarray would contain computed probabilities for each question in the dev-v2.0.json dataset, but the length of this array (21454) does not match the number of examples in dev-v2.0 (11873), or the number of examples of train-v2.0.json (130319), or it is a multiple of any of the two. Could you help me understand what the trainer.predict method is returning?

Thanks a lot for any feedback!

Just realized while reading this that there is a big limitation inside Trainer prediction_loop: it expects just one output (maybe two if there is the loss) when here we have two (or even five in some cases). Will try to fix that on Monday to unlock you on this stage. Will add tests about the number of predictions outputed as well, but it should be the length of the dataset passed.

@sgugger Awesome, thank you!

Hi. Are there any updates on this?

Any progress on this issue?
Trying to define and pass in my own metric for QuestionAnswering tasks, but not quite sure how to calculate the exact loss and F1 when the `compute_metric' only seem to take in the predictions and label_ids.

Or is it possible to override say `EvalPrediction` and `PredictionOutput` to pass in the full input_ids and calculate the SQuAD metrics that way?


I would like to take up this issue?

This has been fixed in #8992 already. I forgot to close this issue as a result, but this is implemented now.