QA Pipeline: Key Error due to predicting a token in question
Author: brandenchan
Pull request: no
Labels: name: wontfix

name: Core: Pipeline

# 🐛 Bug

## Information

Model: deepset/roberta-base-squad2
Language: English
The problem arises when using: QA inference via pipeline

This seems to be a very similar issue to #5711 
The pipeline throws an exception when the model predicts a token that is not part of the document, in this case it seems to be in the question.
In the example below, the model predicts token 3 to be the start and end of the answer span. But these tokens are a part of the question I believe. Therefore, we get a key error when trying to access
feature.token_to_orig_map[3]) in here:

https://github.com/huggingface/transformers/blob/ce374ba87767d551f720242d5e64bfa976531079/src/transformers/pipelines.py#L1370-L1380

## To reproduce

```
nlp = pipeline("question-answering",model="deepset/roberta-base-squad2",
                                  tokenizer="deepset/roberta-base-squad2",
                                  device=-1)

nlp(question="Who is the father of Sansa Stark?", context="===''A Game of Thrones''===\
Sansa Stark begins the novel by being betrothed to Crown Prince Joffrey Baratheon, believing Joffrey to be a gallant prince. While Joffrey and Sansa are walking through the woods, Joffrey notices Arya sparring with the butcher's boy, Mycah. A fight breaks out and Joffrey is attacked by Nymeria (Arya's direwolf) after Joffrey threatens to hurt Arya. Sansa lies to King Robert about the circumstances of the fight in order to protect both Joffrey and her sister Arya.  Since Arya ran off with her wolf to save it, Sansa's wolf is killed instead, estranging the Stark daughters.\
During the Tourney of the Hand to honour her father Lord Eddard Stark, Sansa Stark is enchanted by the knights performing in the event.  At the request of his mother, Queen Cersei Lannister, Joffrey spends a portion of the tourney with Sansa, but near the end he commands his guard Sandor Clegane, better known as The Hound, to take her back to her quarters. Sandor explains how his older brother, Gregor, aka "Mountain that Rides" pushed his face into a brazier of hot coals, for playing with one of his wooden toys.\
After Eddard discovers the truth of Joffrey's paternity, he tells Sansa that they will be heading back to Winterfell. Sansa is devastated and wishes to stay in King's Landing, so she runs off to inform Queen Cersei of her father's plans, unwittingly providing Cersei with the information needed to arrest her father. After Robert dies, Sansa begs Joffrey to show mercy on her father and he agrees, if Ned will swear an oath of loyalty, but executes him anyway, in front of Sansa.  Sansa is now effectively a hostage in King's Landing and finally sees Joffrey's true nature, after he forces her to look at the tarred head of her now-deceased father.")
```
results in
```
Traceback (most recent call last):
  File "/Users/deepset/deepset/haystack/tutorials/Tutorial1_Basic_QA_Pipeline.py", line 145, in <module>
    prediction = finder.get_answers(question="Who is the father of Sansa Stark?", top_k_retriever=1, top_k_reader=5)
  File "/Users/deepset/deepset/haystack/haystack/finder.py", line 57, in get_answers
    top_k=top_k_reader)  # type: Dict[str, Any]
  File "/Users/deepset/deepset/haystack/haystack/reader/transformers.py", line 80, in predict
    predictions = self.model(query, topk=self.n_best_per_passage)
  File "/Users/deepset/deepset/environments/haystack/lib/python3.7/site-packages/transformers/pipelines.py", line 1316, in __call__
    for s, e, score in zip(starts, ends, scores)
  File "/Users/deepset/deepset/environments/haystack/lib/python3.7/site-packages/transformers/pipelines.py", line 1316, in <listcomp>
    for s, e, score in zip(starts, ends, scores)
KeyError: 3
```
## Expected behavior
Predictions that are pointing to tokens that are not part of the "context" (here: tokens in question) should be filtered out from possible answers.

## Environment info
transformers version: latest master (82dd96cae74797be0c1d330566df7f929214b278)
Platform: Mac OS Catalina
Python version: 3.7.5
PyTorch version (GPU?): 1.5.1, CPU
Using GPU in script?: No
Using distributed or parallel set-up in script?: No




Comments:
I also have this -- and related issues on v3.0.0, 3.0.2

on v3.0.0 I get Key Error for sequence terminal tokens -- I think this may be to do with removing special tokens but not masking their positions or due to a truncation error -- may be truncating one too few?
on v3.0.2 I get Key Error for tokens at the beginning of the sequence (Key Error : 0) -- may be an issue to do with the [CLS] token and/or tokens falling in the question span.

I think the trick is to use the attention_mask * token_type_ids * start/end_scores. This will set the logits for all tokens outside the answer to 0 and can be done easily on batch tensors/GPU. I will see if I can put together a pull request.

Platform: Mac OS Catalina, GCP linux cuda 10.1
Python version: 3.6.8
PyTorch version (GPU?): 1.5.1, GPU
Using GPU in script?: Yes
Using distributed or parallel set-up in script?: No

@brandenchan @jusjosgra thanks for reporting the issue and the steps to reproduce.

We did have an issue but it should have been fixed on master branch.
When running the snippet you provided on master I get the following: 

`{'score': 0.05008925125002861, 'start': 679, 'end': 697, 'answer': 'Lord Eddard Stark,'}`

If you can checkout the master branch and give it a try to make sure it works on your side too.

_If everything work as expected:  I need to checkout with the team when we can do a maintenance release._

I still have an error. It appears to be for an example where start and end are both predicted as 0 (a null answer).
In this case the valid results have been filtered to document characters only and so an index for 0 doesnt exist in feature.token_to_orig_map (the first index in my instance is 12, 0 doesnt exist).

So there needs to be a method to handle when the predicted span occurs outside the filtered feature dict. You could return a null object since these cases represent no answer or you could return the best answer found inside the valid span (i.e. mask the logits for non document tokens when getting the max value).

Here is one solution:

original code
```python
                 # Normalize logits and spans to retrieve the answer
                start_ = np.exp(start_ - np.log(np.sum(np.exp(start_), axis=-1, keepdims=True)))
                end_ = np.exp(end_ - np.log(np.sum(np.exp(end_), axis=-1, keepdims=True)))

                if kwargs["handle_impossible_answer"]:
                    min_null_score = min(min_null_score, (start_[0] * end_[0]).item())

                starts, ends, scores = self.decode(start_, end_, kwargs["topk"], kwargs["max_answer_len"])
                char_to_word = np.array(example.char_to_word_offset)

                # Convert the answer (tokens) back to the original text
                answers += [
                    {
                        "score": score.item(),
                        "start": np.where(char_to_word == feature.token_to_orig_map[s])[0][0].item(),
                        "end": np.where(char_to_word == feature.token_to_orig_map[e])[0][-1].item(),
                        "answer": " ".join(
                            example.doc_tokens[feature.token_to_orig_map[s] : feature.token_to_orig_map[e] + 1] 
                        ),
                    }
                    for s, e, score in zip(starts, ends, scores)
```
my update
```python
                 # Normalize logits and spans to retrieve the answer
                start_ = np.exp(start_ - np.log(np.sum(np.exp(start_), axis=-1, keepdims=True)))
                end_ = np.exp(end_ - np.log(np.sum(np.exp(end_), axis=-1, keepdims=True)))

                if kwargs["handle_impossible_answer"]:
                    min_null_score = min(min_null_score, (start_[0] * end_[0]).item())

                starts, ends, scores = self.decode(start_, end_, kwargs["topk"], kwargs["max_answer_len"])
                char_to_word = np.array(example.char_to_word_offset)

                # Convert the answer (tokens) back to the original text
                answers += [
                    {
                        "score": score.item(),
                        "start": np.where(char_to_word == feature.token_to_orig_map[s])[0][0].item(),
                        "end": np.where(char_to_word == feature.token_to_orig_map[e])[0][-1].item(),
                        "answer": " ".join(
                            example.doc_tokens[feature.token_to_orig_map[s] : feature.token_to_orig_map[e] + 1]
                        ),
                    }
                    if s in feature.token_to_orig_map and e in feature.token_to_orig_map # this condition handles the case when answer spans are outside the valid token range.
                    else {"score": min_null_score, "start": 0, "end": 0, "answer": ""}
                    for s, e, score in zip(starts, ends, scores)
```

personally I would rather get the best valid span (max over the masked logits) rather than an error/null answer. This might be a more useful use of "handle impossible answer". Returning null answers might be the best default behaviour and "best valid span" might be a good alternative although this would involve a significant refactor of decode to mask the logits appropriately.

I think there is another bug in the decode function (although I may be misunderstanding).
You compute negative log likelihoods as probabilities but in order to mask items you set them to 0. These items need to be set to a high negative number (e.g. -99) as valid values span zero.
for example:
```python
def decode(...):

...
                start_, end_ = (
                    start_ - np.abs(-99 * np.array(feature.p_mask)),
                    end_ - np.abs(-99 * np.array(feature.p_mask)),
                )

                # Mask CLS
                start_[0] = end_[0] = -99.
```

@mfuntowicz 
Got the same key error zero issue. Above code fixed it

@jusjosgra Thanks for the provided solution here, do you want to submit a PR with your fix? Tag myself and @LysandreJik as reviewers and we'll merge it into master. 

Otherwise I'll do 😄.

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.