4.39.3; ZeroShotClassificationPipeline broken.
Author: FanaticPythoner
Pull request: no
Labels: name: Core: Pipeline

### System Info

- `transformers` version: 4.39.3
- Platform: Linux-6.5.0-26-generic-x86_64-with-glibc2.35
- Python version: 3.10.14
- Huggingface_hub version: 0.21.4
- Safetensors version: 0.4.2
- Accelerate version: 0.28.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.2+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes.
- Using distributed or parallel set-up in script?: Using `device_map` and manually/algorithmically constructing `max_memory`.

### Who can help?

@Narsil @ArthurZucker @gante

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When upgrading from Transformers 4.38.2 to the latest release (4.39.3), the below error can be observed, wherein "discriminatorPipeline" is a zero shot classification pipeline (`pl = pipeline("zero-shot-classification", model=model, tokenizer=model)`). "model" is a `AutoModelForCausalLM`, obtained with:

```
    def load_model(self):
        model_path = "ybelkada/Mixtral-8x7B-Instruct-v0.1-bnb-4bit"
        model_class_to_use = AutoModelForCausalLM
        if self.model_config is None:
            self.model_config = AutoConfig.from_pretrained(HuggingFaceModelHandler.model_path, trust_remote_code=True)

        sortedMaxMem = dict(sorted(HuggingFaceModelHandler.maxMemory.items(), key=lambda item: item[0] if item[0] != "cpu" else 10000000))

        dtype = torch.bfloat16
        bnb_config = BitsAndBytesConfig(
                load_in_4bit=True,
                bnb_4bit_use_double_quant=True,
                bnb_4bit_quant_type="nf4",
                bnb_4bit_compute_dtype=dtype
            )

        model = model_class_to_use.from_pretrained(
                model_path,
                trust_remote_code=True,
                use_auth_token=True,
                torch_dtype=dtype,
                quantization_config=bnb_config,
                max_memory=sortedMaxMem,
                device_map="auto"
            )

        model.eval()

        return model
```

"tokenizer" is a `AutoTokenizer`, obtained with:

```
    def load_tokenizer(self):
        tokenizer_path = 'ybelkada/Mixtral-8x7B-Instruct-v0.1-bnb-4bit'
        tokenizer = AutoTokenizer.from_pretrained(tokenizer_path, trust_remote_code=True)
        
        if tokenizer.pad_token is None:
            tokenizer.pad_token_id = tokenizer.unk_token_id

        return tokenizer
```

And finally, the pipeline is called by doing:

```
        ...
        discriminatorPipeline_res = self.mph.get_free_resource("discriminatorPipeline") # Returns the zero shot classification pipeline resource proxy
        discriminatorPipeline = discriminatorPipeline_res.get() # Returns the raw hf zero shot classification pipeline object

        try:
            outputs =  discriminatorPipeline(prompt,
                                    candidate_labels=candidate_labels, # list of 2 strings
                                    do_sample=do_sample,
                                    temperature=temperature+temperatureModifier,
                                    top_k=top_k,
                                    top_p=top_p,
                                    multi_label=False,
                                    eos_token_id=eos_token_id,
                                    repetition_penalty=repetition_penalty,
                                    length_penalty=length_penalty,
                                    encoder_repetition_penalty=encoder_repetition_penalty,
                                    use_cache=use_cache)
        except Exception as e:
            print(e)
            raise e
        finally:
            discriminatorPipeline_res.release()
        ...
```

The error didn't happen at all in 4.38.2. Reverting back from 4.39 to 4.38 fixes the issue. However, I'd like to use the `/c4ai-command-r-plus` model, and it seems to requires 4.39.

> Traceback (most recent call last):
>   File "/home/user/project/HuggingFaceModelHandler.py", line 848, in discriminatorPipelineExec
>     outputs =  discriminatorPipeline(prompt,
>   File "/home/user/anaconda3/envs/cvaitmp/lib/python3.10/site-packages/transformers/pipelines/zero_shot_classification.py", line 206, in __call__
>     return super().__call__(sequences, **kwargs)
>   File "/home/user/anaconda3/envs/cvaitmp/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1198, in __call__
>     return next(
>   File "/home/user/anaconda3/envs/cvaitmp/lib/python3.10/site-packages/transformers/pipelines/pt_utils.py", line 124, in __next__
>     item = next(self.iterator)
>   File "/home/user/anaconda3/envs/cvaitmp/lib/python3.10/site-packages/transformers/pipelines/pt_utils.py", line 266, in __next__
>     processed = self.infer(next(self.iterator), **self.params)
>   File "/home/user/anaconda3/envs/cvaitmp/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1112, in forward
>     model_outputs = self._forward(model_inputs, **forward_params)
> TypeError: ZeroShotClassificationPipeline._forward() got an unexpected keyword argument 'pad_token_id'
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File "/home/user/anaconda3/envs/conda_env_name/lib/python3.10/site-packages/transformers/pipelines/zero_shot_classification.py", line 206, in __call__
>     return super().__call__(sequences, **kwargs)
>   File "/home/user/anaconda3/envs/conda_env_name/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1198, in __call__
>     return next(
>   File "/home/user/anaconda3/envs/conda_env_name/lib/python3.10/site-packages/transformers/pipelines/pt_utils.py", line 124, in __next__
>     item = next(self.iterator)
>   File "/home/user/anaconda3/envs/conda_env_name/lib/python3.10/site-packages/transformers/pipelines/pt_utils.py", line 266, in __next__
>     processed = self.infer(next(self.iterator), **self.params)
>   File "/home/user/anaconda3/envs/conda_env_name/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1112, in forward
>     model_outputs = self._forward(model_inputs, **forward_params)
> TypeError: ZeroShotClassificationPipeline._forward() got an unexpected keyword argument 'pad_token_id'

### Expected behavior

The above code should work as is, yet it does not in the newest version.

Comments:
Anyone? It's definitely a big blocker. It prevents us from using newer models.

cc @zucchini-nlp -- this issue is probably related to https://github.com/huggingface/transformers/pull/29614