`ZeroShotClassificationPipeline.preprocess` simply discards all but the first item
Author: w568w
Pull request: no

### System Info

- `transformers` version: 4.37.1
- Platform: Linux-6.5.0-26-generic-x86_64-with-glibc2.35
- Python version: 3.11.5
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: 0.26.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.2+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@Narsil 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
model = transformers.pipeline(
            "zero-shot-classification",
            model="cross-encoder/nli-deberta-v3-large",
        )
for preprocessed in model.preprocess(
        ["Seq A", "Seq B"],
        candidate_labels=[
            ["Label A", "Label B"],
            ["Label C", "Label D"]
        ],
    ):
        print(preprocessed)
```

### Expected behavior

`ZeroShotClassificationPipeline.preprocess` should do text pairing:

```
"Seq A" <=> ["Label A", "Label B"]
"Seq B" <=> ["Label C", "Label D"]
```

But in fact, it outputs:

```
"Seq A" <=> ["Label A", "Label B"]
"Seq A" <=> ["Label C", "Label D"]
```

And the 2nd item, `Seq B`, is just discarded.

---

I check the source code and find something strange:

https://github.com/huggingface/transformers/blob/8c12690cecbb97e187861e386f7a0ac790e4236c/src/transformers/pipelines/zero_shot_classification.py#L208-L219

The `preprocess` function seems to only use `sequences[0]`, and ignore all other items. `postprocess` function has the similar behavior.

Is it intended, or is it the wrong way of pipeline batching with `ZeroShotClassificationPipeline`?

Comments:
cc @Rocketknight1 

Hi @w568w, there are two issues here! Firstly, I don't think the `ZeroShotClassificationPipeline` supports a different set of candidate labels for each sample. When you pass labels like this, I think the pipeline interprets each sub-list as a single candidate (in other words, `["Label A", "Label B"]` is treated as a single possible label, which is definitely not what you want). You'll need to call the pipeline once per distinct set of candidate labels instead.

Secondly, the `preprocess` function is actually working correctly, because inputs are actually separated earlier (in the shared parent class code) before being passed to it. The easiest way to see this is to checkout `transformers` from Github, insert a breakpoint in the `preprocess` function and then `pip install -e .`. Although it's a bit confusing, `preprocess()` is actually called once for `Seq A` and once for `Seq B`, and the final outputs are correct. It will give the wrong outputs only if you call it manually with a list of sequences.

> I don't think the `ZeroShotClassificationPipeline` supports a different set of candidate labels for each sample. [...] You'll need to call the pipeline once per distinct set of candidate labels instead.

I was trying to eliminate the warning saying "please do not use the pipeline sequentially", and this limitation is documented nowhere. Thanks for your explanation.

> inputs are actually separated earlier (in the shared parent class code) before being passed to it

I do not get it. Is not `preprocess` a normal function (i.e. not an overriding) that is not declared in the parent class `ChunkPipeline`? **How does "the shared parent class code" interfere with it?**

> preprocess() is actually called once for `Seq A` and once for `Seq B`

I checked the `sequence_pairs, sequences` values returned by `self._args_parser` in `preprocess`:

```
sequence_pairs: [['Seq A', "This example is ['Label A', 'Label B']."], ['Seq A', "This example is ['Label C', 'Label D']."], ['Seq B', "This example is ['Label A', 'Label B']."], ['Seq B', "This example is ['Label C', 'Label D']."]]

sequences: ['Seq A', 'Seq B']
```

And `preprocess` yields with the first #`len(candidate_labels)` items (i.e. 2 items) of `sequence_pairs`, but only uses `sequences[0]` in returning values.

So the batch processing is only implemented for one sequence and several groups of candidate labels? That would make sense. But **it is very confusing, so I suggest clarifying it in the documentation.**

And my last question is: **does such a batch process result in any performance gain?** It seems to simply generate `model_input`s serially.

It's the other way around! Batch processing should only work with one set of candidate labels, and multiple input sequences. There should be a performance gain from batching multiple inputs that way with most pipelines, but it may be different for `ZeroShotClassificationPipeline`.

> It's the other way around! Batch processing should only work with one set of candidate labels, and multiple input sequences

This sounds different from what I have observed in the code. Could you please give a code example about this?

Bump to let you know I am still caring about this issue ;)

I had already checked [the documentation about Chunk Batching](https://huggingface.co/docs/transformers/main_classes/pipelines#pipeline-chunk-batching), which does not include demo code snippets for `zero-shot-classification` tasks.

> This sounds different from what I have observed in the code.

I'm not sure I follow! What I'm saying is that:

- You can call the pipeline with one sequence, or a list of input sequences
- However, you can only supply one list of candidate labels
- Internally, what the pipeline will do is iterate over every input sequence, and for each input sequence, it will combine it with each candidate label. It will then generate per-label scores for each input sequence

In other words, if you have 10 input sequences and 4 candidate labels, then chunk batching will expand this into 40 samples to run through the pipeline. Each block of 4 (corresponding to 1 original input) will be scored together, to generate probabilities for each input sequence.

As written, the pipeline does not support candidate labels being a list of lists, with different labels per sample. If you want to do that, then the only solution is to repeatedly call the pipeline with different labels each time, and just ignore the warning!

I'm pretty confident after double-checking the code that the internal logic for the pipeline is correct. If you feel like the documentation is unclear, feel free to open a PR to improve it, but for now I don't think there's actually a bug in the code, so I'm going to politely close this issue.