Zero-Shot Classification - Pipeline - Batch Size
Author: bhacquin
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.21.3
- Platform: Linux-5.4.0-113-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.9.1
- PyTorch version (GPU?): 1.12.1+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@Narsil

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

class TextDataset(Dataset):

    def __init__(self, list_of_text):
        self.news = list_of_text
        
    def __len__(self):
        return len(self.news)

    def __getitem__(self, idx):
        sample = {'text' : self.news[idx]}
        return sample

classifier = pipeline("zero-shot-classification",
                      model="facebook/bart-large-mnli", device=0, framework='pt')

candidate_labels = ['advertisement','politics']
dataset = TextDataset(news_list)

for i in classifier(KeyDataset(dataset, 'text'),candidate_labels=candidate_labels, batch_size=32):
    print(i)
    break

Out : {'sequence': 'Love is Where It All Begins: adidas X Thebe Magugu Launch .... Herzogenaurach, Aug 15 2022 – Today, adidas launches its latest Tennis collection, created in partnership with contemporary South African...', 'labels': ['advertisement', 'global warming'], 'scores': [0.9311832189559937, 0.0002945002052001655]}

### Expected behavior

As I am using batch_size 32, I do expect my output to be a sequence of dicts of length 32. However, it only returns the first element each and every time. 

Comments:
> Out : {'sequence': 'Love is Where It All Begins: adidas X Thebe Magugu Launch .... Herzogenaurach, Aug 15 2022 – Today, adidas launches its latest Tennis collection, created in partnership with contemporary South African...', 'labels': ['advertisement', 'global warming'], 'scores': [0.9311832189559937, 0.0002945002052001655]}
> Expected behavior
> 
> As I am using batch_size 32, I do expect my output to be a sequence of dicts of length 32. However, it only returns the first element each and every time.

Actually everything is working as intended. The model is indeed seeing 32 items at a time, however 32 texts in this occurence is NOT a batch of 32.

In order to work on this data, there is 1 text pair constructed for each text + candidate_labels, so in your case each text generates 2 items to be processed by the model.

This pipeline is actually quite smart, and starts by outputting in a generating fashion all the items one by one, which are automatically batched (regardless if it's the same text or not) into a batch of 32 (so here 16 texts x 2 candidate labels but it would work the same with any amount of candidate labels)

It then proceeds and run the model on this batch

Then the output is iteratively debatched, to be processed 1 texts + candidate_labels at a time, yielding the exact same output as if it wasn't batched (but it was indeed batched yielding performance speedups if used on the appropriate GPU for instance ).

Does that answer your question ?

More info there: 
https://huggingface.co/docs/transformers/v4.22.2/en/main_classes/pipelines#pipeline-chunk-batching
https://github.com/huggingface/transformers/pull/14225

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.