Trouble fine-tuning zero-shot image classification model
Author: moon001light
Pull request: no

### System Info

transformers 4.30.2
python 3.11.4

### Who can help?

@amyeroberts @sgugger 

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am trying to fine-tune a **zero-shot** image classifier, I am following this example for reference: https://github.com/huggingface/notebooks/blob/main/examples/image_classification.ipynb

What I changed from the notebook:

The checkpoint I am starting from: `laion/CLIP-ViT-L-14-DataComp.XL-s13B-b90K`

For the model I am using `AutoModelForZeroShotImageClassification` like this:

```
model = AutoModelForZeroShotImageClassification.from_pretrained(
    model_checkpoint, 
    label2id=label2id,
    id2label=id2label,
    ignore_mismatched_sizes = True,
)
```

When I run `trainer.train()`, I get this error:

> TypeError: CLIPModel.forward() got an unexpected keyword argument 'labels'

### Expected behavior

With my changes to the notebook, it should fine-tune for zero-shot pre-trained model `laion/CLIP-ViT-L-14-DataComp.XL-s13B-b90K`

Comments:
Hi @moon001light, thanks for raising this issue.

Models that can be loaded with the `AutoXxx` API will have a shared input and output structure. In the example notebook, the model is loaded using `AutoModelForImageClassification`. Models loaded with `AutoModelForZeroShotImageClassification` have a different set of expected inputs, in particular, they don't accept `labels` and expect `input_ids`.  Here's a guide on performing the zero-shot image classification task using transformers: https://huggingface.co/docs/transformers/tasks/zero_shot_image_classification. 

Add more example scripts is on my to-do list. I'll make sure to include this task! 

Thanks for the quick reply @amyeroberts , the link you gave seems to be a high level overview. Is there anything else besides replacing `labels` with `input_ids`? Could you point me to the code so that I can see the full list of expected inputs for `AutoModelForZeroShotImageClassification`? Appreciate your patience thanks :)

@moon001light If all you want is to train this particular checkpoint, then I would look directly at the architecture it loads, which in this [case is CLIPModel](https://github.com/huggingface/transformers/blob/ea91c2adca842da3d2f87e094504fa7d66a7008a/src/transformers/models/clip/modeling_clip.py#L1082). 

More generally, the auto groups are [defined in `modeling_auto.py`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/auto/modeling_auto.py). For [AutoModelForZeroShotClassification](https://github.com/huggingface/transformers/blob/ea91c2adca842da3d2f87e094504fa7d66a7008a/src/transformers/models/auto/modeling_auto.py#L1256C7-L1256C46) the model architectures that can be used are listed under [MODEL_FOR_ZERO_SHOT_IMAGE_CLASSIFICATION_MAPPING_NAMES](https://github.com/huggingface/transformers/blob/ea91c2adca842da3d2f87e094504fa7d66a7008a/src/transformers/models/auto/modeling_auto.py#L980). 














A bit of a newbie here, after looking into it I am still not sure how to properly set `input_ids`. This is what I tried:

```
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)

def collate_fn(examples):
    pixel_values = torch.stack([example["pixel_values"] for example in examples])
    input_ids = tokenizer(examples["label"], padding="max_length", truncation=True)
    return {"pixel_values": pixel_values, "input_ids": input_ids} 
```

But my `input_ids` is wrong :( Not really sure what `input_ids` should be. Looking forward to that example script of yours so I can learn @amyeroberts . Appreciate any help I can get :)

@moon001light I suggest inspecting the objects at each step to see what they are and contain. For example, the output of 

```python
tokenizer(examples["label"], padding="max_length", truncation=True)
```

is not `input_ids`, but rather a dictionary that contains `input_ids`.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.