Overflow when normalizing start and end logits in QA pipeline
Author: fgbelidji
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.18.0
- Platform: Linux-4.19.0-20-cloud-amd64-x86_64-with-debian-10.12
- Python version: 3.7.12
- Huggingface_hub version: 0.4.0
- PyTorch version (GPU?): 1.11.0+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```


### Who can help?

@Narsil @sgugger 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

After succesfully training a question answering model using the [run_qa.py ](https://github.com/huggingface/transformers/blob/215e0681e4c3f6ade6e219d022a5e640b42fcb76/examples/pytorch/question-answering/run_qa.py) script, I tried to reuse it in for inference in a question-answering `pipeline` but the model predicts `no-answer` for all the (question, context) pairs even for those included in my evaluation file which was used when running the `run_qa.py` script. The evaluation part of `run_qa.py` was able to predict an answer for those examples. 

The issue is related to the normalization of start and end logits done in the QA pipeline https://github.com/huggingface/transformers/blob/215e0681e4c3f6ade6e219d022a5e640b42fcb76/src/transformers/pipelines/question_answering.py#L401 

`start_` and `end_` are provided `np.float32` arrays and all the logits calculated by the model were <-100  which leads to overflow when applying `exp` then `log`. The normalization differs from the one applied in `run_qa.py` file which explains this overflow error happening only while doing inference. 

https://github.com/huggingface/transformers/blob/215e0681e4c3f6ade6e219d022a5e640b42fcb76/examples/flax/question-answering/utils_qa.py#L191 


The trace showing this behavior is provided below:

``` 
> /opt/conda/lib/python3.7/site-packages/transformers/pipelines/question_answering.py(401)postprocess()
    400             # Normalize logits and spans to retrieve the answer
--> 401             start_ = np.exp(start_ - np.log(np.sum(np.exp(start_), axis=-1, keepdims=True)))
    402             end_ = np.exp(end_ - np.log(np.sum(np.exp(end_), axis=-1, keepdims=True)))

ipdb> start_.dtype
dtype('float32')

ipdb> start_.shape
(1, 512)
ipdb> start_[0, :20]
array([-10000.     , -10000.     , -10000.     , -10000.     ,
       -10000.     , -10000.     , -10000.     , -10000.     ,
       -10000.     , -10000.     , -10000.     , -10000.     ,
       -10000.     , -10000.     , -10000.     , -10000.     ,
       -10000.     ,   -155.77672,   -155.8097 ,   -155.81848],
      dtype=float32)
ipdb> n #running operation line 401

  start_ = np.exp(start_ - np.log(np.sum(np.exp(start_), axis=-1, keepdims=True)))
> /opt/conda/lib/python3.7/site-packages/transformers/pipelines/question_answering.py(402)postprocess()
    401             start_ = np.exp(start_ - np.log(np.sum(np.exp(start_), axis=-1, keepdims=True)))
--> 402             end_ = np.exp(end_ - np.log(np.sum(np.exp(end_), axis=-1, keepdims=True)))
    403 

ipdb> start_[0, :20]
array([inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf,
       inf, inf, inf, inf, inf, inf, inf

```

### Expected behavior

```shell
The LogSumExp trick should be applied similarly to https://github.com/huggingface/transformers/blob/215e0681e4c3f6ade6e219d022a5e640b42fcb76/examples/flax/question-answering/utils_qa.py#L191 and https://datumorphism.leima.is/cards/machine-learning/neural-networks/log-sum-exp-trick/ to prevent this overflow.
```


Comments:
Yes, the logsumexp trick should be used every time we compute a `softmax` in any pipeline.

Thanks for finding this ! 

This is old code that was carried over many times, proposed fix here: https://github.com/huggingface/transformers/pull/17143

@fgbelidji do you have an example where it triggers ? (Would be nice to have a test that triggers the error)

@Narsil thanks for the fix! The example code below will trigger the error:

```
from transformers import SquadExample, pipeline

context = """
Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debut album, Dangerously in Love (2003), which established her as a solo artist worldwide, earned five Grammy Awards and featured the Billboard Hot 100 number-one singles "Crazy in Love" and "Baby Boy".
"""

example = SquadExample(context_text=context,
            question_text= "When did Beyonce start becoming popular?",
            qas_id='56be85543aeaaa14008c9063',
            title='Beyoncé', 
            answer_text="in the late 1990s", 
            start_position_character=269)

model_name = "deepset/roberta-base-squad2"
model_pipeline = pipeline(model=model_name, 
                          tokenizer=model_name, 
                          revision="v1.0", 
                          task="question-answering", 
                          handle_impossible_answer=True, 
                          top_k=1)

example_preprocessed = next(model_pipeline.preprocess(example))
model_outputs = model_pipeline._forward(example_preprocessed)

model_outputs["start"] = model_outputs["start"].detach() 
model_outputs["end"] = model_outputs["end"].detach() 

output_without_error = model_pipeline.postprocess([model_outputs])

#Ensuring overflow
model_outputs["start"] = model_outputs["start"] * 100
model_outputs["end"] = model_outputs["end"]* 100

output_with_error = model_pipeline.postprocess([model_outputs])


print(f"No overflow: {output_without_error}")
print(f"Overflow: {output_with_error}")
```

Output:

```
No overflow: {'score': 0.6222478747367859, 'start': 277, 'end': 287, 'answer': 'late 1990s'}
Overflow: {'score': 0.0, 'start': 0, 'end': 0, 'answer': ''}
```