Zero-shot classification pipeline does not appear to batch examples
Author: lsimoneau
Pull request: no

### System Info

- `transformers` version: 4.29.2
- Platform: Linux-5.15.107+-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.15.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.6.9 (gpu)
- Jax version: 0.4.10
- JaxLib version: 0.4.10

### Who can help?

@narsi

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Execute the zero-shot classification pipeline and vary the number of labels:

```
hf_pipeline = pipeline('zero-shot-classification', model="facebook/bart-large-mnli", device=device)
hf_pipeline("top 10 destinations to visit this summer", candidate_labels=["travel", "lifestyle", "technology"])
```

### Expected behavior

If we increase the number of labels, when running inference on a GPU we would expect latency to remain relatively constant as the inputs to the underlying entailment model should be batched together.

This is not explicitly documented, but the forum post announcing the zero-shot pipeline does say this:

>Assuming you’re using the same model, the pipeline is likely faster because it batches the inputs. If you pass a single sequence with 4 labels, you have an effective batch size of 4, and the pipeline will pass these through the model in a single pass.

In practice though we see latency increase more or less linearly with label count (compared against a naive implementation batching up inferences to bart-mnli for the same inputs):

![image](https://github.com/huggingface/transformers/assets/31768/ed1f8ee3-c44c-405e-b247-90704c628459)

Here is the colab used to make the graph above: https://colab.research.google.com/drive/19YiQFDcJUm8iz0azYWX35I6-vQ-bTheC?usp=sharing that demonstrates the issue.


Comments:
cc @Narsil 

> Assuming you’re using the same model, the pipeline is likely faster because it batches the inputs. If you pass a single sequence with 4 labels, you have an effective batch size of 4, and the pipeline will pass these through the model in a single pass.

This comment could be old, this has been changed 1yr+ ago.

Now the batching happens when using `pipeline(..., batch_size=4)` (should you want this batch_size).

The batching happens regardless on the number of candidate_labels, and this was changed exactly for this reason. If we batched on candidate_labels automatically, users couldn't control the memory requirements nicely, so it would OOM easily with large number of labels, and couldn't batch more than number of labels if the GPU allowed for it.

So now `batch_size=n` will batch `n` samples for all forward passes, in could be `< len(candidate_labels)` or `> len(candidate_labels)`. Meaning you have a much finer control over the batching mecanism.

This is documented at the `pipeline` level since this behavior is orthogonal to each specific pipeline's behavior.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@Narsil I am running into a similar problem that batch_size can only make a difference for up to the number of labels. i.e. It's not batching across sentences if we pass in a list of strings. In our test, we have 5 candidate_labels, and setting `batch_size=5` is almost the same as `batch_size=32`.

Looking into the implementation, it seems we will set is_last for each sentence's last label: https://github.com/Narsil/transformers/blob/main/src/transformers/pipelines/zero_shot_classification.py#L190-L195
and then the is_last will stop the accumulator when the batch is not full:https://github.com/Narsil/transformers/blob/main/src/transformers/pipelines/pt_utils.py#L270-L275
 Not sure does it mean that `batch_size` only takes effect on zero-shot-classifier when `batch_size < len(candidate_labels)`?