Transfomers Pipline: Batching does not work for Sentence-Pair Text Classification 
Author: maximilianreimer
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.6.1
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyTorch version (GPU?): 1.11.0+cu113 (True)
- Tensorflow version (GPU?): 2.8.0 (True)
- Using GPU in script?: yes / no (Both)
- Using distributed or parallel set-up in script?: No
```


### Who can help?

@Narsil 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Run: 
```python
from transformers import pipeline
model = pipeline(
   task="text-classification", 
   model="roberta-large-mnli", # does also happen for our own fine-tunes roberta models
   device=0 # does also happen on CPU
)  
n_samples = 10000
sample = ['The earth is not flat.', 'Physicists will find it shocking, but there are plenty of people around the world who genuinely believe the Earth is flat...']
data = [sample]* n_samples

model([sample]) # works
model(data, batch_size=1) # results in the follwing error
```
Output
```
Some weights of the model checkpoint at roberta-large-mnli were not used when initializing RobertaForSequenceClassification: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']
- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

[<ipython-input-2-62b2d5a9a338>](https://localhost:8080/#) in <module>()
      6 
      7 model([sample])
----> 8 model(data, batch_size=1)

14 frames

[/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_roberta.py](https://localhost:8080/#) in forward(self, hidden_states)
    347     def forward(self, hidden_states):
    348         hidden_states = self.dense(hidden_states)
--> 349         hidden_states = self.intermediate_act_fn(hidden_states)
    350         return hidden_states
    351 

RuntimeError: CUDA out of memory. Tried to allocate 5.34 GiB (GPU 0; 14.76 GiB total capacity; 9.35 GiB already allocated; 4.00 GiB free; 9.37 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
```

### Expected behavior

```shell
The pipeline should run even with a very large list of inputs if the batch size is low enough.
```


Comments:
I am on version `4.18.0` and the following code works with and without GPU

```
from transformers import pipeline
model = pipeline(
   task="text-classification", 
   model="roberta-large-mnli", # does also happen for our own fine-tunes roberta models
   device=0 # does also happen on CPU
)  
n_samples = 1000
sample = ['The earth is not flat.', 'Physicists will find it shocking, but there are plenty of people around the world who genuinely believe the Earth is flat...']
data = [sample]* n_samples

model([sample], padding=True)
model(data, batch_size=2, padding=True)
```

Hi @maximilianreimer,

As  @sijunhe said, can you try upgrading your `transformers` version just because a lot has been done to improve the batching in more recent version.

For everyone here also another nice to have is to change the format from `list` to a `generator` which will iterate over results without having to maintain the list of all results (it also allows you to store results as they come in, allowing you to recover if sample number 10_014 fails for instance instead of having to rerun the whole thing).

```python
from transformers import pipeline

model = pipeline(
    task="text-classification",
    model="roberta-large-mnli",  # does also happen for our own fine-tunes roberta models
    device=0,  # does also happen on CPU
)
n_samples = 1000
sample = [
    "The earth is not flat.",
    "Physicists will find it shocking, but there are plenty of people around the world who genuinely believe the Earth is flat...",
]


def data():
    for i in range(n_samples):
        yield sample


out = model([sample], padding=True)
for out in model(data(), batch_size=2, padding=True):
    print(out)
```

Just a nice to have but should definitely help when processing large amounts of data.


Thanks for the helpful comments. Updating seems to fix the issue for me as well!

Closing this then.