QA pipeline run-time error when there is no answer
Author: AlecS12
Pull request: no
Labels: name: wontfix

name: Core: Pipeline

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):
Bert large uncased SQUAD /finetuned on SQUAD2.0 and my dataset 
Language I am using the model on (English, Chinese....):
English
The problem arise when using:
* [x ] the official example scripts: (give details)
```python
from transformers import pipeline
nlp_ft = pipeline('question-answering',  model='/data/bert/divorce_qa/wwm-bert-uncased-finetuned-squad', tokenizer='bert-large-uncased')
            nlp_ft({
    'question': "is it raining?",
    'context': ''
})
```
Converting examples to features: 100%|██████████| 1/1 [00:00<00:00, 2486.25it/s]
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-13-c7eb59b211f0> in <module>
      1 nlp_ft({
      2     'question': "is it raining?",
----> 3     'context': ''
      4 })

~/miniconda3/envs/nlp2/lib/python3.7/site-packages/transformers/pipelines.py in __call__(self, *texts, **kwargs)
    657                     # Retrieve the score for the context tokens only (removing question tokens)
    658                     fw_args = {k: torch.tensor(v) for (k, v) in fw_args.items()}
--> 659                     start, end = self.model(**fw_args)
    660                     start, end = start.cpu().numpy(), end.cpu().numpy()
    661 

~/miniconda3/envs/nlp2/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~/miniconda3/envs/nlp2/lib/python3.7/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, start_positions, end_positions)
   1265                             position_ids=position_ids,
   1266                             head_mask=head_mask,
-> 1267                             inputs_embeds=inputs_embeds)
   1268 
   1269         sequence_output = outputs[0]

~/miniconda3/envs/nlp2/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~/miniconda3/envs/nlp2/lib/python3.7/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask)
    687                 extended_attention_mask = attention_mask[:, None, None, :]
    688         else:
--> 689             raise ValueError("Wrong shape for input_ids (shape {}) or attention_mask (shape {})".format(input_shape, attention_mask.shape))
    690 
    691         # Since attention_mask is 1.0 for positions we want to attend and 0.0 for

ValueError: Wrong shape for input_ids (shape torch.Size([0])) or attention_mask (shape torch.Size([0]))

Another case (with topk):
```python
            nlp_ft({
    'question': "met with client  to discuss her house.",
    'context': 'snow'
}, topk=5)
```
Converting examples to features: 100%|██████████| 1/1 [00:00<00:00, 998.17it/s]
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-21-bdcc590f57c7> in <module>
      2     'question': "met with client  to discuss her house.",
      3     'context': 'snow'
----> 4 }, topk=5)

~/miniconda3/envs/nlp2/lib/python3.7/site-packages/transformers/pipelines.py in __call__(self, *texts, **kwargs)
    684                     'answer': ' '.join(example.doc_tokens[feature.token_to_orig_map[s]:feature.token_to_orig_map[e] + 1])
    685                 }
--> 686                 for s, e, score in zip(starts, ends, scores)
    687             ]
    688         if len(answers) == 1:

~/miniconda3/envs/nlp2/lib/python3.7/site-packages/transformers/pipelines.py in <listcomp>(.0)
    684                     'answer': ' '.join(example.doc_tokens[feature.token_to_orig_map[s]:feature.token_to_orig_map[e] + 1])
    685                 }
--> 686                 for s, e, score in zip(starts, ends, scores)
    687             ]
    688         if len(answers) == 1:

KeyError: 255

Is SQUAD2.0 no answer (the span is the CLS token) implemented?

A more difficult problem:
The same span 0:42 appears in the answers twice  with different probabilities:
```python
            nlp_ft({
    'question': 'not divorcing?',
    'context': 'My son loves ice cream. I hired a lawyer.  The weather is beautiful'
}, topk=20)
```
Converting examples to features: 100%|██████████| 1/1 [00:00<00:00, 608.13it/s]
[{'score': 6.8867944232487335e-15,
  'start': 24,
  'end': 42,
  'answer': 'I hired a lawyer.'},
 {'score': 5.837555144012473e-15,
  'start': 0,
  'end': 42,
  'answer': 'My son loves ice cream. I hired a lawyer.'},
 {'score': 5.104124911876544e-15,
  'start': 26,
  'end': 42,
  'answer': 'hired a lawyer.'},
 {'score': 4.480651773242456e-15,
  'start': 0,
  'end': 23,
  'answer': 'My son loves ice cream.'},
 {'score': 3.9003458779199145e-15,
  'start': 34,
  'end': 42,
  'answer': 'lawyer.'},
 {'score': 3.683094227838328e-15,
  'start': 24,
  'end': 66,
  'answer': 'I hired a lawyer. The weather is beautiful'},
 {'score': 3.637585671341453e-15, 'start': 0, 'end': 6, 'answer': 'My son'},
 {'score': 3.4290555803043746e-15,
  'start': 3,
  'end': 42,
  'answer': 'son loves ice cream. I hired a lawyer.'},
 {'score': 3.1219554896279132e-15,
  'start': 0,
  'end': 66,
  'answer': 'My son loves ice cream. I hired a lawyer. The weather is beautiful'},
 {'score': 2.963547353528702e-15,
  'start': 32,
  'end': 42,
  'answer': 'a lawyer.'},
 {'score': 2.8398875189274633e-15,
  'start': 43,
  'end': 66,
  'answer': 'The weather is beautiful'},
 {'score': 2.7297131068172943e-15,
  'start': 26,
  'end': 66,
  'answer': 'hired a lawyer. The weather is beautiful'},
 {'score': 2.631992946944041e-15,
  'start': 3,
  'end': 23,
  'answer': 'son loves ice cream.'},
 {'score': 2.4540475046027306e-15,
  'start': 17,
  'end': 42,
  'answer': 'cream. I hired a lawyer.'},
 {'score': 2.3835331682965698e-15,
  'start': 24,
  'end': 42,
  'answer': 'I hired a lawyer.'},
 {'score': 2.351548698805639e-15,
  'start': 7,
  'end': 42,
  'answer': 'loves ice cream. I hired a lawyer.'},
 {'score': 2.136764987640859e-15, 'start': 3, 'end': 6, 'answer': 'son'},
 {'score': 2.0859256871447656e-15,
  'start': 34,
  'end': 66,
  'answer': 'lawyer. The weather is beautiful'},
 {'score': 2.0203893789166256e-15,
  'start': 0,
  'end': 42,
  'answer': 'My son loves ice cream. I hired a lawyer.'},
 {'score': 1.9864633437892544e-15, 'start': 24, 'end': 25, 'answer': 'I'}]



* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS:  Ubuntu 18.04.3 LTS
* Python version: 3.7.4
* PyTorch version: 1.3
* PyTorch Transformers version (or branch): master 2.3 (or later)
* Using GPU ? no
* Distributed or parallel setup ? no
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.