Question-answering pipeline failing with Nonetype exception when selecting spans with tokens outside of the context
Author: psorianom
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: '4.6.0.dev0'
- Platform: Linux Mint 20
- Python version: 3.7.10
- PyTorch version (GPU?): GPU
- Tensorflow version (GPU?): NA
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->
@LysandreJik 

## Information

Model I am using (Bert, XLNet ...): camembert (specifically [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf))

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: 
* [x] my own task or dataset: Question Answering with own SQuAD-like dataset

## To reproduce

When using a `question-answering` pipeline, if the context is too small (or if the model can't find multiple candidates), the produced scores will be zero and thus when sorting and filtering for `topk > 1`, we may return random indices of zero score values which correspond to tokens that **are not** in the context, but in the question.  This sorting and index returning happens [here](https://github.com/huggingface/transformers/blob/95dab34d5588fb155dfed8293ac2fbb1217a95a7/src/transformers/pipelines/question_answering.py#L406).

Asking for an index that does not exist in the context returns a `None` down the line (in function `enc.word_to_chars()` [here](https://github.com/huggingface/transformers/blob/95dab34d5588fb155dfed8293ac2fbb1217a95a7/src/transformers/pipelines/question_answering.py#L376)). This bug may be related to this issue https://github.com/huggingface/transformers/issues/9843.

This suite of events finally produce this exception:
```
Traceback (most recent call last):
  File "/home/pavel/.config/JetBrains/PyCharmCE2021.1/scratches/bug_transf.py", line 25, in <module>
    print(nlp({'question': questions[0], 'context': text}, topk=20, handle_impossible_answer=True, max_seq_len=256, doc_stride=128))
  File "/home/pavel/miniconda3/envs/piaf-ml/lib/python3.7/site-packages/transformers/pipelines.py", line 1968, in __call__
    for s, e, score in zip(starts, ends, scores)
  File "/home/pavel/miniconda3/envs/piaf-ml/lib/python3.7/site-packages/transformers/pipelines.py", line 1968, in <listcomp>
    for s, e, score in zip(starts, ends, scores)
TypeError: 'NoneType' object cannot be interpreted as an integer
```

## Full Context

We are building a Retriever (ES with bm25) + Reader (QA with the above mentioned model) search engine with the haystack library. In this setting, we test with different lengths for the contexts where the QA model will find the answer.  We are also testing for different values of `topk`.
As an example, if I have a 1001 words context and I set the max length to 1000, I will split the document in two sub-documents, one with the first 1000 words and the other with the last word. Thus my second sub-document will be very small. These type of small documents will be passed to the transformers QA pipeline which will usually generate the above exception when `topk` is greater than one.


Steps to reproduce the behavior:
```python
from transformers import pipeline
nlp = pipeline('question-answering', model='etalab-ia/camembert-base-squadFR-fquad-piaf', tokenizer='etalab-ia/camembert-base-squadFR-fquad-piaf')


question = "Comment bénéficier du billet de congé annuel de la SNCF à tarif réduit ?"
context = "perle"
result = nlp({'question': question, 'context': context}, topk=20, handle_impossible_answer=True, max_seq_len=256, doc_stride=128)
print(result)
```
## Proposed Solution

Given that in `self.decode` we return the indices of the context tokens to create the answers, we could re-filter them to make sure that we will use context-tokens indices to generate the spans later on. Just like this (replacing this [line](https://github.com/huggingface/transformers/blob/95dab34d5588fb155dfed8293ac2fbb1217a95a7/src/transformers/pipelines/question_answering.py#L344)): 
```python
starts, ends, scores = self.decode(start_, end_, kwargs["topk"], kwargs["max_answer_len"])
desired_spans = np.in1d(starts, undesired_tokens.nonzero()) & np.in1d(ends, undesired_tokens.nonzero())
starts = starts[desired_spans]
ends = ends[desired_spans]
scores = scores[desired_spans]
```
I have a [branch](https://github.com/psorianom/transformers/blob/e96afad34bc872b4fc9318d45a551e0c33f3de8c/src/transformers/pipelines/question_answering.py#L346) here ready to be PRequested if you agree with this solution. 


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

I would like to have an answer with valid spans even if they are lower than the required `topk` parameter.

<!-- A clear and concise description of what you would expect to happen. -->


Comments:
Hi, thank you for opening such a detailed issue. I know @Narsil has some experience with the question answering pipeline and has probably been confronted to that issue in the past.

Nicolas, what do you think of the fix proposed above?

Either way we should work on a clearer error message.

The proposed fix seems reasonable.

A few things:

- `decode` is already supposed to do the proper filtering, so we should probably move the logic in the function and document the    changes in the docstring.
IMO it makes sense to return less than `topk` if there are not enough options available within `context`.
- Also we should probably add a tests for this use case (the low hanging fruit is adding the exact excerpt as a `slow` test).
- The `numpy` doc recommends using `isin` instead of `in1d` for new code: https://numpy.org/doc/stable/reference/generated/numpy.in1d.html

Hi @LysandreJik and @Narsil, thank you for your quick answers.
I will modify my solution as suggested. I will have to change my solution's logic because in `decode` we are not aware of the `undesired_tokens`. Maybe I could just add it as parameter or maybe some masking over the zero-valued scores would be preferable ?
I will also look into the tests.



I think adding it as an argument is fine. Maybe let's make it optional to keep backward compatibility though, @LysandreJik  ?

Sounds good! 

Great! I am working on this. I will make a PR as soon as I can.