Pipeline inference with text pair is broken
Author: fxmarty
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.20.0.dev0
- Platform: Linux-5.15.0-27-generic-x86_64-with-glibc2.35
- Python version: 3.9.12
- Huggingface_hub version: 0.4.0
- PyTorch version (GPU?): 1.11.0+cu113 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
```


### Who can help?

@narsil

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)

### Reproduction

Basically, the pipeline for text classification does not handle well input pairs that must be separated by [SEP] token.

For example, for glue's mnli dataset, we have:

```python
premise = 'The new rights are nice enough'
hypothesis = 'Everyone really likes the newest benefits '
```

Whether we pass
* `pipeline([[premise, hypothesis]], padding=True, truncation=True)`
* or `pipeline(" ".join([premise, hypothesis]), padding=True, truncation=True)`

the pipeline output is wrong.

## Detailed reproduction
If necessary, install transformers in the dev version (`pip uninstall transformers && git clone https://github.com/huggingface/transformers.git && cd transformers && pip install -e .`).

Replace https://github.com/huggingface/transformers/blob/1f13ba818e0e3b780cf9155242e2c83a27fdfa9a/src/transformers/pipelines/text_classification.py#L132-L134

by

```python
    def preprocess(self, inputs, **tokenizer_kwargs) -> Dict[str, GenericTensor]:
        return_tensors = self.framework
        tokenized_inps = self.tokenizer(inputs, return_tensors=return_tensors, **tokenizer_kwargs)
        print("tokenized_inps", tokenized_inps)
        return tokenized_inps
 ```
to be able to see what are the tokenized inputs in the pipeline.

Then run

```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from transformers import pipeline
from datasets import load_dataset

tokenizer = AutoTokenizer.from_pretrained("roberta-large-mnli")
model = AutoModelForSequenceClassification.from_pretrained("roberta-large-mnli")

pipe = pipeline(task="text-classification", tokenizer=tokenizer, model=model)

raw_datasets = load_dataset("glue", "mnli")

txt1 = raw_datasets["validation_matched"][0]["premise"]
txt2 = raw_datasets["validation_matched"][0]["hypothesis"]
    
inputs = [txt1, txt2]
    
txt = " ".join(inputs)
res = pipe(txt, padding=True, truncation=True)
print(res)
"""Output:
tokenized_inps {'input_ids': tensor([[ 0, 133, 92, 659, 32, 2579, 615, 7632, 269, 3829, 5, 8946, 1795, 1437, 2]]), 'attention_mask': ...}
[{'label': 'NEUTRAL', 'score': 0.7983464002609253}]

NOTE: theses input_ids correspond to:
'
The new rights are nice enough Everyone really likes the newest benefits 
'
"""
```

We can see that separating the premise and hypothesis by a space is a very bad idea as there is no [SEP] token between the two.

Now run:
```python
from transformers import BatchEncoding
data = raw_datasets["validation_matched"][0:1]
tokenized_inps = tokenizer(data["premise"], data["hypothesis"], padding=True, truncation=True)
tokenized_inps = BatchEncoding(tokenized_inps, tensor_type="pt")
print(tokenized_inps)
print(tokenizer.decode(tokenized_inps["input_ids"][0]))
"""Output:
{'input_ids': tensor([[ 0, 133, 92, 659, 32, 2579, 615, 2, 2, 11243, 269,  3829, 5, 8946, 1795, 1437,  2]]), 'attention_mask': ...}

The new rights are nice enough

Everyone really likes the newest benefits 

"""
```

Here, the `tokenizer` takes a `text=premise` and `text_pair=hypothesis`, and we see as expected SEP tokens between the two.

Other possibility with the pipeline:
```python
txt1 = raw_datasets["validation_matched"][0]["premise"]
txt2 = raw_datasets["validation_matched"][0]["hypothesis"]
    
inputs = [txt1, txt2]
  
res = pipe([inputs], padding=True, truncation=True)
print(res)
"""Outputs:
tokenized_inps {'input_ids': tensor([[ 0, 133, 92, 659, 32, 2579, 615, 2, 1],
        [ 0, 11243, 269, 3829, 5, 8946, 1795, 1437, 2]]), 'attention_mask': ...}
[{'label': 'NEUTRAL', 'score': 0.8978187441825867}]

Note that now input_ids is 2D! The decoding gives:

The new rights are nice enough
<pad>

Everyone really likes the newest benefits 

"""
```
There is a [CLS] token inserted in the middle, most likely this is not desirable. In fact, when we run the pipeline on several examples from the dataset, all are classified as neutral and wrong.

## Hacky solution
Use

```python
txt1 = raw_datasets["validation_matched"][0]["premise"]
txt2 = raw_datasets["validation_matched"][0]["hypothesis"]
    
inputs = [txt1, txt2]
tokenized_inps = pipe.preprocess([inputs])
res = pipe.forward(tokenized_inps)
res = pipe.postprocess(res)
print(res)
"""Output:
tokenized_inps {'input_ids': tensor([[ 0, 133, 92,659, 32, 2579, 615, 2, 2, 11243, 269, 3829, 5, 8946, 1795,  1437, 2]]), 'attention_mask': ...}
{'label': 'NEUTRAL', 'score': 0.9636728167533875}
We get the right input_ids, and the score is the same as with manually using tokenizer + model, yay!
"""
```

which gives the same proba as with using the tokenizer and model separately.

To me, the issue lies in two facts:
* It is very wrong to join two sentences with a space (as suggested in the doc https://huggingface.co/tasks/text-classification ) since we loose the info that they are different sentence.
* In case we pass the data as `pipeline([[premise, hypothesis]])`, it could be that there is some funny stuff happening in https://github.com/huggingface/transformers/blob/1f13ba818e0e3b780cf9155242e2c83a27fdfa9a/src/transformers/pipelines/pt_utils.py#L111

### Expected behavior

Pipeline for text-classification with text pair should output the same result than manually using tokenizer + model + softmax.


Comments:
Hi @fxmarty ,

Text pair was indeed never supported by the pipeline, so it's not tested against. We could definitely add support.
Without any code change you could do :
```python
res = pipe([[[txt, txt_pair]]] padding=True, truncation=True)
```

Then we could also have a fix within the pipeline (and properly crash when malformed input instead of here a silent error).


The true culprit is two difference parts of magics conflicting.

```python
tokenizer([txt, txt_pair])
```
Is guessing `txt` and `txt_pair` are actually two different texts, and outputs batched input_ids.
```python
tokenizer([[txt, txt_pair]])
```
Is guessing the first list is the batch, and the second list therefore MUST be a pair of texts.

It interacts with `pipeline` magic, which also supposes that lists are batches so 
```python
pipe([[txt, txt_pair]])
```
Is understood as a list of inference to run on, and gives only `[txt, txt_pair]` to the underlying tokenizer. which in terms treats them as a batch (which is not what we want in this case).

Since `pipeline.preprocess` is only really supposed to preprocess one item at a time (any sort of list is handled by the parent class) we can definitely change the call to the tokenizer appropriately into `tokenizer(text=txt, text_pair=txt_pair)` so will yield the correct output.

Question @lhoestq : Do `datasets` have a consistent way of expressing pairs of text so that we could maybe express the whole thing as

```python
for output in pipe(load_dataset('glue', 'mnli'))):
    print(output)
```
akin to what we did with ASR and the `Audio` column ? If it's not consistent, then the proposed fix should be enough.

@fxmarty is my explanation clear about what's happening ? I am going to propose a fix so that we support text pairs (which is not supported by every model/tokenizer out there but still pretty useful)

Cheers.

> Question @lhoestq : Do datasets have a consistent way of expressing pairs of text so that we could maybe express the whole thing as

No it doesn't. It could be pairs of question/answer, of sentence1/sentence2, of language1/language2 or any column names.

@Narsil Thanks a lot for your detailed explanation, makes sense!