Pipeline's QnA and run_qa predictions do not match
Author: oersoy1
Pull request: no

## Environment info

- `transformers` version: 4.3.0 or 4.4.0dev (tested in both versions) 
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.7.1+cu101 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
pipelines: @LysandreJik
maintained examples (run_qa.py): @sgugger, @patil-suraj

## Information
I have noticed that the `pipeline("question-answering)` and `run_qa.py `evaluation functions (prediction + post_process) not only are built differently but also yield different results affecting the precision/recall numbers, `pipeline` being the worse performer. I ran the test on 2 different environments and got the same results.  Given the same exact model, the module and the script may affect precision/recall numbers up to 5%. The issue may manifest itself only in a set with negatives as I have not run a test on a set where negatives are not possible. 

The problem arises when using:
* [X] the official example scripts: 
`run_qa.py` and `pipeline`

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: SQuAD V2 

## To reproduce
Steps to reproduce the behavior:

1. Take any QnA model and execute `run_qa.py`on the SQuAD V2 validation set
2. Get predictions on the same set via `pipeline` with the same model and parameters
3. Compare

Here is my notebook to follow the logic and see the results on a particular QnA model.

[Colab Notebook](https://colab.research.google.com/drive/1GMetvI6e0pkUPUHiNX6f46_TBO0LUKre?usp=sharing
)
## Expected behavior
The results need to match. `run_qa.py` code appears to be optimized for training and evaluation at training. Most researchers report QnA model performance on a script like this. The inference module `pipeline` should match what `run_qa.py` is producing. 

### more observations:

1.  Pipeline produces more NULL answers than `run_qa` (Highest Concern)
2. About half the non-null answers do not match exactly (45%). Most of those are off by a character or so
3. More than negligible number of records do not match closely
4. pipeline produces shorter answers, 
5. pipeline does not take care of apostrophe cases
6. pipeline answers may be in separate places in the context.




Comments:
Just to be sure I understand completely the problem since I don't see the F1s for the full results obtained via pipeline, which is producing the best results? `pipeline` or `run_qa`? Or are they different but overall comparable?

`run_qa.py` produces the better results.  They are different on about 3000 records, 2700 of those are off by a character or so, 300  are off by a larger margin, sometimes different answers, 12 records Null (pipeline) vs Not-Null (run_qa) (false negative). And that is on SQuAD. I see a wider gap on custom datasets. "Comparable" is a subjective word so I perhaps should not comment on that. In my opinion they need to match. What we get out of run_qa.py in batch for a dataset should be the same for record i, meaning when we pass the same query and context to the model, the script and the module should deterministically return the same answer. The problem might not be exactly in the predictions but rather in the post prediction functions. 

@sgugger Hi, any update on this? Did you all get a chance to look into the problem?  by the way, after I wrote the message above, I realized "better" is also subjective.  I would prefer a 5 token answer to a 1 token answer for instance depending on the question and context. As I mentioned pipeline does not clean up the answer as well, punctuation gets carried over. Looking at custom validation sets I can tell the F1 score is lower for Pipelines, which I did it pretty much manually by writing custom code as I had a smaller set.  If there is a way to take the pipeline predictions and calculate a F1 score against the ground truths in SQuAD, I would be happy to do that and return you a result. But again, that would be just SQuAD.  better or worse, the answers don't match. Maybe the problem is in run_qa.py and this is an opportunity to get that fixed. Thanks in advance. 

The problem is that neither is easily fixable. For backward compatibility reason we can't change the way the qa pipeline works out of the blue. We are trying to think of having some "pipeline configs" to be able to switch behaviors in pre or post processing but it won't be added overnight.

Overall this is going to take some time to solve.

@sgugger thank you for the acknowledgement and clarification that there indeed is a difference.  I was not at least dreaming stuff up.  

Here is what I ended up doing:  I gutted out `run_qa.py` and created a custom inference code using the evaluation functions in that script.  Now my model answers match obviously but I ended up with a different problem. When I pass **a single** query/context pair to my custom question answerer, the latency increased by about 10x.  I profiled my code and lo and behold the problem is in the data prep part that is written with `datasets` and seems like is optimized for batch processing during training, meaning there is a cost incurred in the upstream processing before the prediction.

So now I am caught between a rock and a hard place as the model does not produce the same answers if I use the `pipeline` and I have a latency issue for single inference with the `run_qa.py` code..  Is it possible to summarize the problem with `pipeline` so that maybe I can write some code to patch that configurable part into it?  That would go a long way.  Merci beaucoup!

PS:  I also noted 2 other main differences between these two inference codes. 

1) I don't see where we can incorporate a null threshold in the `pipeline` function. I actually benefit up to 2-3 points for F1 on my custom dataset with a threshold. 
2) The pipeline can take multiple query/context pairs but it loops through them one at a time, so if I pass a batch of 16, I get 16x latency. 

I am thinking about entering separate issues as I can't see any open issues regarding them. Please let me know if I have overlooked or you all have incorporated some of this recently. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hi @sgugger @oersoy1,

I am facing the same issue. The pipeline predictions are not the same as run_qa.py predictions. Please let me know if there is any update or work around for the same.