Question answering pipeline: error for long text sequences when `max_seq_len` is specified
Author: ATroxler
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.17.0
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyTorch version (GPU?): 1.11.0+cu113 (False)
- Tensorflow version (GPU?): 2.8.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no
```


### Who can help?

@Narsil

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

**code:**
```python
#!pip install transformers==4.16.0
!pip install transformers==4.17.0
from transformers import pipeline
context = 100 * "The quick brown fox jumps over the lazy dog. "
qa_pipeline = pipeline("question-answering", max_seq_len=2000)
qa_pipeline(question="what does the fox do?", context=context)
```
**exception traceback:**
```
No model was supplied, defaulted to distilbert-base-cased-distilled-squad (https://huggingface.co/distilbert-base-cased-distilled-squad)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
[<ipython-input-4-d1e4a860038f>](https://localhost:8080/#) in <module>()
      1 qa_pipeline = pipeline("question-answering", max_seq_len=2000)
----> 2 qa_pipeline(question="what does the fox do?", context=context)

10 frames
[/usr/local/lib/python3.7/dist-packages/transformers/pipelines/question_answering.py](https://localhost:8080/#) in __call__(self, *args, **kwargs)
    249         examples = self._args_parser(*args, **kwargs)
    250         if len(examples) == 1:
--> 251             return super().__call__(examples[0], **kwargs)
    252         return super().__call__(examples, **kwargs)
    253 

[/usr/local/lib/python3.7/dist-packages/transformers/pipelines/base.py](https://localhost:8080/#) in __call__(self, inputs, num_workers, batch_size, *args, **kwargs)
   1025             return self.iterate(inputs, preprocess_params, forward_params, postprocess_params)
   1026         else:
-> 1027             return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
   1028 
   1029     def run_multi(self, inputs, preprocess_params, forward_params, postprocess_params):

[/usr/local/lib/python3.7/dist-packages/transformers/pipelines/base.py](https://localhost:8080/#) in run_single(self, inputs, preprocess_params, forward_params, postprocess_params)
   1047         all_outputs = []
   1048         for model_inputs in self.preprocess(inputs, **preprocess_params):
-> 1049             model_outputs = self.forward(model_inputs, **forward_params)
   1050             all_outputs.append(model_outputs)
   1051         outputs = self.postprocess(all_outputs, **postprocess_params)

[/usr/local/lib/python3.7/dist-packages/transformers/pipelines/base.py](https://localhost:8080/#) in forward(self, model_inputs, **forward_params)
    942                 with inference_context():
    943                     model_inputs = self._ensure_tensor_on_device(model_inputs, device=self.device)
--> 944                     model_outputs = self._forward(model_inputs, **forward_params)
    945                     model_outputs = self._ensure_tensor_on_device(model_outputs, device=torch.device("cpu"))
    946             else:

[/usr/local/lib/python3.7/dist-packages/transformers/pipelines/question_answering.py](https://localhost:8080/#) in _forward(self, inputs)
    369         example = inputs["example"]
    370         model_inputs = {k: inputs[k] for k in self.tokenizer.model_input_names}
--> 371         start, end = self.model(**model_inputs)[:2]
    372         return {"start": start, "end": end, "example": example, **inputs}
    373 

[/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _call_impl(self, *input, **kwargs)
   1108         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1109                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110             return forward_call(*input, **kwargs)
   1111         # Do not call functions when jit is used
   1112         full_backward_hooks, non_full_backward_hooks = [], []

[/usr/local/lib/python3.7/dist-packages/transformers/models/distilbert/modeling_distilbert.py](https://localhost:8080/#) in forward(self, input_ids, attention_mask, head_mask, inputs_embeds, start_positions, end_positions, output_attentions, output_hidden_states, return_dict)
    853             output_attentions=output_attentions,
    854             output_hidden_states=output_hidden_states,
--> 855             return_dict=return_dict,
    856         )
    857         hidden_states = distilbert_output[0]  # (bs, max_query_len, dim)

[/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _call_impl(self, *input, **kwargs)
   1108         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1109                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110             return forward_call(*input, **kwargs)
   1111         # Do not call functions when jit is used
   1112         full_backward_hooks, non_full_backward_hooks = [], []

[/usr/local/lib/python3.7/dist-packages/transformers/models/distilbert/modeling_distilbert.py](https://localhost:8080/#) in forward(self, input_ids, attention_mask, head_mask, inputs_embeds, output_attentions, output_hidden_states, return_dict)
    546 
    547         if inputs_embeds is None:
--> 548             inputs_embeds = self.embeddings(input_ids)  # (bs, seq_length, dim)
    549         return self.transformer(
    550             x=inputs_embeds,

[/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _call_impl(self, *input, **kwargs)
   1108         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1109                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110             return forward_call(*input, **kwargs)
   1111         # Do not call functions when jit is used
   1112         full_backward_hooks, non_full_backward_hooks = [], []

[/usr/local/lib/python3.7/dist-packages/transformers/models/distilbert/modeling_distilbert.py](https://localhost:8080/#) in forward(self, input_ids)
    131         position_embeddings = self.position_embeddings(position_ids)  # (bs, max_seq_length, dim)
    132 
--> 133         embeddings = word_embeddings + position_embeddings  # (bs, max_seq_length, dim)
    134         embeddings = self.LayerNorm(embeddings)  # (bs, max_seq_length, dim)
    135         embeddings = self.dropout(embeddings)  # (bs, max_seq_length, dim)

RuntimeError: The size of tensor a (1009) must match the size of tensor b (512) at non-singleton dimension 1
```

### Expected behavior

```shell
Run through and produce a result similar to the following, like with transformers 4.16.0

{'answer': 'The quick brown fox jumps over the lazy dog',
 'end': 3418,
 'score': 0.017251048237085342,
 'start': 3375}
```
```


Comments:
I think this is expected, not a bug. The `max_seq_length` of `distilbert` is 512. Set `max_seq_length` to be larger than 512 essentially disable the truncation. When you feed a text than 512 tokens, it will raise this error

No, according ot the documentation of `transformers.QuestionAnsweringPipeline.__call__`, the parameter `max_seq_len` is the maximum length of the total sentence (context + question) *after tokenization*.
The context will be split in several chunks if needed, i.e., if it is longer than the maximum sequence length of the model.
And this is the way the pipeline behaved up to `transformers==4.17.0`.
This feature is useful to process long sequences (longer than model length).

@ATroxler 

I am not sure this was changed since 4.17.0 since the diff does concern this parameter

```diff
diff --git a/src/transformers/pipelines/question_answering.py b/src/transformers/pipelines/question_answering.py
index efab83b92..bbffa3471 100644
--- a/src/transformers/pipelines/question_answering.py
+++ b/src/transformers/pipelines/question_answering.py
@@ -5,10 +5,9 @@ from typing import TYPE_CHECKING, Dict, List, Optional, Tuple, Union
 import numpy as np
 
 from ..data import SquadExample, SquadFeatures, squad_convert_examples_to_features
-from ..file_utils import PaddingStrategy, add_end_docstrings, is_tf_available, is_torch_available
 from ..modelcard import ModelCard
 from ..tokenization_utils import PreTrainedTokenizer
-from ..utils import logging
+from ..utils import PaddingStrategy, add_end_docstrings, is_tf_available, is_torch_available, logging
 from .base import PIPELINE_INIT_ARGS, ArgumentHandler, ChunkPipeline
 
 
@@ -302,11 +301,6 @@ class QuestionAnsweringPipeline(ChunkPipeline):
                 ]
             )
 
-            # keep the cls_token unmasked (some models use it to indicate unanswerable questions)
-            if self.tokenizer.cls_token_id is not None:
-                cls_index = np.nonzero(encoded_inputs["input_ids"] == self.tokenizer.cls_token_id)
-                p_mask[cls_index] = 0
-
             features = []
             for span_idx in range(num_spans):
                 input_ids_span_idx = encoded_inputs["input_ids"][span_idx]
@@ -316,6 +310,11 @@ class QuestionAnsweringPipeline(ChunkPipeline):
                 token_type_ids_span_idx = (
                     encoded_inputs["token_type_ids"][span_idx] if "token_type_ids" in encoded_inputs else None
                 )
+                # keep the cls_token unmasked (some models use it to indicate unanswerable questions)
+                if self.tokenizer.cls_token_id is not None:
+                    cls_indices = np.nonzero(np.array(input_ids_span_idx) == self.tokenizer.cls_token_id)[0]
+                    for cls_index in cls_indices:
+                        p_mask[span_idx][cls_index] = 0
                 submask = p_mask[span_idx]
                 if isinstance(submask, np.ndarray):
                     submask = submask.tolist()
@@ -399,8 +398,11 @@ class QuestionAnsweringPipeline(ChunkPipeline):
             end_ = np.where(undesired_tokens_mask, -10000.0, end_)
 
             # Normalize logits and spans to retrieve the answer
-            start_ = np.exp(start_ - np.log(np.sum(np.exp(start_), axis=-1, keepdims=True)))
-            end_ = np.exp(end_ - np.log(np.sum(np.exp(end_), axis=-1, keepdims=True)))
+            start_ = np.exp(start_ - start_.max(axis=-1, keepdims=True))
+            start_ = start_ / start_.sum()
+
+            end_ = np.exp(end_ - end_.max(axis=-1, keepdims=True))
+            end_ = end_ / end_.sum()
 
             if handle_impossible_answer:
                 min_null_score = min(min_null_score, (start_[0, 0] * end_[0, 0]).item())
```

However there was indeed a bugfix in 4.17.0 (from 4.16.0) where `max_seq_len` was not passed (so it was basically ignored).
When ignored (or not passed) `max_seq_len == min(self.tokenizer.max_seq_len, 384)` even before that  (not sure which version bug much earlier) it was always 384.


`max_seq_len` corresponds to the maximum length of a single chunk (question + context chunk), and it will chunk indeed if full_context is too long.

So @sijunhe is indeed correct here.

Rereading the documentation of this parameter I can understand the confusion.

```
                The maximum length of the total sentence (context + question) after tokenization. The context will be
                split in several chunks (using `doc_stride`) if needed.
```

Would something like :
```
                The maximum length of the total sentence (context + question) of each chunk passed to the model. If the context is too large, it will be split in several chunks (using `doc_stride` as overlap length) if needed.
```

Be more understandable ?

The `(context + question)`  wants to convey that if the question is taking too much space, then there's less room for context so you need to be careful about extremely long questions.

Is the problem clearer to you ? Do you have any suggestions to improve even further the docs ?

Also if I understand correctly you want to limit the amount of context fed to your model right (not just chunking but really ignoring part of the text you send, which usually we try to avoid since you are sending it :) ) ? May I ask why you want to do so ? Are you using documents of arbitrary length and know that the answer should be in the beginning for instance ?

The idea is just to figure out how we could maybe cook up a nice option for this use case (while keeping the others understandable too)
 

Many thanks, @Narsil , for clarification.
Indeed, changing the documentation in the suggested manner would avoid the confusion.

My use case is to use the QA pipeline as a pre-proessor for a sequence classification task:
* STEP 1 - apply QA pipeline to extract from long input sequences the parts relevant to the task.
* STEP 2 - apply sequence classification to the extract

With my interpretation of the existing documentation, I had been under the impression that the sequences are truncated by default to a length of 384, which I wanted to override by specifying `max_seq_len=2000`.
Thanks to your explanation I understand now that I can simply omit the parameter `max_seq_len`, and the pipeline behaves eactly the way I want, i.e. it breaks the long context into chunks.

**Example:**
```python
!pip install transformers==4.18.0
from transformers import pipeline
context = 100 * "This part of the text is totally useless. " + "The quick brown fox jumps over the lazy dog."
qa_pipeline = pipeline("question-answering")
qa_pipeline(question="what does the fox do?", context=context)
```
**Result:**
```
{'answer': 'jumps over the lazy dog',
 'end': 4643,
 'score': 0.639270007610321,
 'start': 4620}
```
:-)

@ATroxler I opened an issue to clarify this documentation ! Thanks for raising the issue, and glad it works as intended !