[Longformer] longformer in question-answering pipeline
Author: patil-suraj
Pull request: yes
Labels: name: wontfix

This PR adds `LongformerForQuestionAnswering` in `QuestionAnsweringPipeline`

@patrickvonplaten @ibeltagy 

Comments:
This works well however I noticed some discrepancy in answers generated with pipeline and without  pipeline

for this example
```
question = 'Who was Jim Henson?'
text = 'Jim Henson was a nice puppet.'
```

pipeline produces `nice puppet.`
 without pipeline `a nice puppet`

@patrickvonplaten  is this expected or there's something wrong ? 

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4615?src=pr&el=h1) Report
> Merging [#4615](https://codecov.io/gh/huggingface/transformers/pull/4615?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/8cc6807e8997b8b7404c07037bd02c578da98baf&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4615/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4615?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4615      +/-   ##
==========================================
- Coverage   78.03%   78.02%   -0.01%     
==========================================
  Files         124      124              
  Lines       20647    20647              
==========================================
- Hits        16111    16110       -1     
- Misses       4536     4537       +1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4615?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/data/processors/squad.py](https://codecov.io/gh/huggingface/transformers/pull/4615/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvc3F1YWQucHk=) | `28.66% <ø> (ø)` | |
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/4615/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `39.24% <0.00%> (-0.24%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4615?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4615?src=pr&el=footer). Last update [8cc6807...8d7469b](https://codecov.io/gh/huggingface/transformers/pull/4615?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


@patrickvonplaten 
I seem to have figured out why  this is happening.

This line https://github.com/huggingface/transformers/blob/master/src/transformers/data/processors/squad.py#L103
tokenizes the doc text into individual tokens and then this line 
https://github.com/huggingface/transformers/blob/master/src/transformers/data/processors/squad.py#L134

uses the list of those tokens for encoding

While this works for other BERT models , for roberta and and longformer tokenizer, the final 
encoding results in this
`'
 Who was Jim Henson?

JimHensonwasanicepuppet
'`

Changing `span_doc_tokens` with `example.context_text` at L134 seems to solve the problem. But I'm not sure if doing this will cause other things to break. 

Thanks for the PR @patil-suraj. 

I will put this on hold for a week though since we will most likely do some major changes very soon here.

1) I think the squad preprocessing functions will probably be refactoring building on the new `nlp` library @thomwolf 

2) IMO, the function `squad_convert_examples_to_features` should in general not be used in the `QuestionAnsweringPipeline` since we only need `input_ids` and `attention_mask` for inference and some other values for the `score`. Also, we should also be able to use the pipeline for `TriviaQA` (with Longformer, we now have a pretrained model that works very well on TriviaQA). The pipeline should not be dataset specific. I think it might be a good idea to do a bigger refactoring of `QuestionAnsweringPipeline` and make it independent from the `squad_convert_examples_to_features` function. What do you think @julien-c @thomwolf @LysandreJik 

Hi @patrickvonplaten , I think we  should fix this now, as the newly launched model inference api uses qa pipeline its failing or giving weird answers for longformer qa models on model hub. This might discourage the users from using them. 

@patrickvonplaten I agree with you, especially because that method is made for heavy processing of large data, which is not the case with pipelines. It's slow to start, and uses multiprocessing by default, something we don't necessarily want with the pipelines.

Also putting @mfuntowicz in cc here

Ok we'll have conflicts here (#5496), we need to handle with care.



This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I believe this issue with roberta models for QA never got fixed. Any plans to continue working here? 

> While this works for other BERT models, for roberta and and longformer tokenizer, the final
encoding results in this 
< s> Who was Jim Henson?

JimHensonwasanicepuppet


As mentioned by @patil-suraj, we don't respect whitespace before words in the passage. Therefore, we currently use the input id for "ĠHenson" in the question, but the one for "Henson" in the passage.

The current implementation also leads to quite poor results of our QA models. For example, F1 of `deepset/roberta-base-squad2` on SQuAD 2 dev is down to 0.69 whereas it gets 0.81 with "whitespace preserving" tokenization.

A simple fix could be to add `add_prefix_space=True` here in the tokenizer call for Roberta tokenizers (or similar), but might not be the most elegant solution. 
https://github.com/huggingface/transformers/blob/28cf873036d078b47fb9dd38ac3421a7c874da44/src/transformers/data/processors/squad.py#L112

I can to do a PR for this, if that's how you want to fix it.  


A PR would be very welcome :-) 

@patrickvonplaten Added a PR https://github.com/huggingface/transformers/pull/7387