Invalid CLS masking in question answer pipelines top K calculation
Author: antonyscerri
Pull request: no

## Environment info
- `transformers` version: 4.18.0
- Platform: Linux-5.10.102-99.473.amzn2.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.12
- Huggingface_hub version: 0.5.1
- PyTorch version (GPU?): 1.10.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
@Narsil

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Using any question and answering model which leverages CLS token and run it with pipelines. 
2. Supply an input record which has (con)text longer than the max_seq_length set to cause it to chunk the input record.
3. If you inspect the results of p_mask after line https://github.com/huggingface/transformers/blob/4975002df50c472cbb6f8ac3580e475f570606ab/src/transformers/pipelines/question_answering.py#L307
4. You will see the token position used by CLS in the model (typically the first) has not been set to False and remains True meaning it will be masked out as an undesired token later on.

This will work if the input record is not chunked. The reason is the code before that constructs p_mask doesn't work as expected when the chunk record are different lengths (when calling np.asarray) and the lines pointed to above then fail silently to correctly mask the CLS token position. 

This then causes differences in the calculations of the results answer spans and their associated probabilities.

## Expected behavior

The CLS token position should be correctly set to False so that it is considered a valid token for consideration in answer calculations regardless of whether the input record was chunked or not.

As a simple but possibly not an efficient fix I replaced the two lines in the if pointed to above with the following:

```
    for span_id in range(num_spans):
        cls_index = np.nonzero(np.array(encoded_inputs["input_ids"][span_id]) == self.tokenizer.cls_token_id)
        p_mask[span_id][cls_index] = 0
```


Comments:
Hi @antonyscerri ,

Thank you very much for the report ! 

This definitely seems like a legitimate issue.

Do you have an example where this triggers an error really in the pipeline ? (This is to craft a test to make sure this is tested against.)
I will try to find one on a dummy example, but it's always better if we can have a real world example.


Unfortunately I cannot share the data I observed it with. However any text block with a question which produces an answer should show a change in its score between it being fixed or not, assuming an appropriate model that uses CLS token is used. If you have an example where the NIL answer (based on CLS token) is the "best" answer you may see the top answer change from some other span to the NIL answer.  See below for a quick example i just put together, which i tested it using "deepset/roberta-base-squad2" model.

And sorry but for completeness i also realised my quick fix involved changing the construction of p_mask to the following (moving the asarray inside the outer array:

```
            p_mask = [
                    np.asarray([tok != 1 if question_first else 0 for tok in encoded_inputs.sequence_ids(span_id)])
                    for span_id in range(num_spans)
                ]
```

Running the following example with the original code yields the top answer of:

	Answer: orth individuals in >Europe< after Paris and the
	Score: 0.1588

With my quick fix i get:

	Answer: ><London is the capita
	Score: 0.9948

The other answer is 2nd place with a score of 0.000.

The data used was a passage taken from a wikipedia page on London and was run with the handle_impossible_answer set to True and max_seq_length=512:

```
{"context": "London is the capital and largest city of England and the United Kingdom. It stands on the River Thames in south-east England at the head of a 50-mile (80 km) estuary down to the North Sea, and has been a major settlement for two millennia. The City of London, its ancient core and financial centre, was founded by the Romans as Londinium and retains boundaries close to its medieval ones. Since the 19th century, \"London\" has also referred to the metropolis around this core, historically split between the counties of Middlesex, Essex, Surrey, Kent, and Hertfordshire, which largely comprises Greater London, governed by the Greater London Authority. The City of Westminster, to the west of the City of London, has for centuries held the national government and parliament. As one of the world's global cities, London exerts strong influence on its arts, commerce, education, entertainment, fashion, finance, health care, media, tourism, and communications, and has sometimes been called the capital of the world. Its GDP (€801.66 billion in 2017) makes it the biggest urban economy in Europe, and it is one of the major financial centres in the world. In 2019 it had the second-highest number of ultra high-net-worth individuals in Europe after Paris and the second-highest number of billionaires in Europe after Moscow. As of 2021, London has the most millionaires of any city. With Europe's largest concentration of higher education institutions, it includes Imperial College London in natural and applied sciences, the London School of Economics in social sciences, and the comprehensive University College London. The city is home to the most 5-star hotels of any city in the world. In 2012, London became the first city to host three Summer Olympic Games. London is the capital and largest city of England and the United Kingdom. It stands on the River Thames in south-east England at the head of a 50-mile (80 km) estuary down to the North Sea, and has been a major settlement for two millennia. The City of London, its ancient core and financial centre, was founded by the Romans as Londinium and retains boundaries close to its medieval ones. Since the 19th century, \"London\" has also referred to the metropolis around this core, historically split between the counties of Middlesex, Essex, Surrey, Kent, and Hertfordshire, which largely comprises Greater London, governed by the Greater London Authority. The City of Westminster, to the west of the City of London, has for centuries held the national government and parliament. As one of the world's global cities, London exerts strong influence on its arts, commerce, education, entertainment, fashion, finance, health care, media, tourism, and communications, and has sometimes been called the capital of the world. Its GDP (€801.66 billion in 2017) makes it the biggest urban economy in Europe, and it is one of the major financial centres in the world. In 2019 it had the second-highest number of ultra high-net-worth individuals in Europe after Paris and the second-highest number of billionaires in Europe after Moscow. As of 2021, London has the most millionaires of any city. With Europe's largest concentration of higher education institutions, it includes Imperial College London in natural and applied sciences, the London School of Economics in social sciences, and the comprehensive University College London. The city is home to the most 5-star hotels of any city in the world. In 2012, London became the first city to host three Summer Olympic Games.", "question": "What country is Paris the capital of?"}
```

Thank you so much for this test, definitely helps a lot ! 

Can't replicate with small random models (for obvious reasons) but at least we now have a slow tests covering this.