Contrastive Image Text example deletes `image_column` because it is doesnt have a corresponding argument in `VisionTextDualEncoderModel.forward`
Author: luisblanche-mirakl
Pull request: no

## Configuration 
Using `transformers==4.35.0` and `datasets==2.14.7`


## Problem Statement:

Running the example in  [examples/pytorch/contrastive-image-text/run_clip.py](examples/pytorch/contrastive-image-text/run_clip.py)   (tried with the COCO dataset from the example as well as with my own dataset) 

The `image_column` is removed by : https://github.com/huggingface/transformers/blob/1394e08cf099d16515c1889ab9507946489f5afe/src/transformers/trainer.py#L729 

I think this is because [`VisionTextDualEncoderModel.forward`](VisionTextDualEncoderModel.forward) expects `pixel_values` and not a path to an image so `image_path` is not in the signature.

In the example code there is this line : https://github.com/huggingface/transformers/blob/1394e08cf099d16515c1889ab9507946489f5afe/examples/pytorch/contrastive-image-text/run_clip.py#L505-L506

Which i think maybe is the problem:  we expect to process the images files on the fly but the column is deleted before we can do it because the model only accepts pixel values. 

This causes the error downstream when calling :  
https://github.com/huggingface/transformers/blob/1394e08cf099d16515c1889ab9507946489f5afe/examples/pytorch/contrastive-image-text/run_clip.py#L446-L449

We end up with  `KeyError` for `image_path`

Comments:
Hi @luisblanche-mirakl, thanks for raising this issue! 

Could you confirm the command line command being used to run the script - is it with the default values [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/contrastive-image-text#train-the-model)?

Hi @amyeroberts  thansk for answering.

I am actually using a json parameter file : 
```json
{"output_dir": "/{mypath}/clip/test",
"model_name_or_path": "{mypath}/clip-roberta",
"data_dir": "{mypath}/data/coco",
"dataset_name": "ydshieh/coco_dataset_script",
"dataset_config_name": "2017",
"image_column":"image_path" ,
"caption_column": "caption" ,
"max_seq_length": 256,
"max_train_samples": 100, 
"max_eval_samples": 100,
"preprocessing_num_workers": 4,
"do_train": true,
"do_eval": false,
"per_device_train_batch_size": 8,
"per_device_eval_batch_size": 8,
"learning_rate": 5e-5, 
"warmup_steps": 0, 
"weight_decay": 0.1,
"token": "***",
"report_to": ["tensorboard"],
"overwrite_output_dir": true
}
```

@luisblanche-mirakl OK, thanks for sharing! You can set `remove_unused_columns=False` in the TrainingArguments which should prevent this from happening. 

cc @ydshieh to check if the script needs updating

Hi,

The [code example](https://github.com/huggingface/transformers/tree/main/examples/pytorch/contrastive-image-text#train-the-model) does use `remove_unused_columns=False`.

I don't think there is something to be updated here :-), simply following the example is good.

```bash
python examples/pytorch/contrastive-image-text/run_clip.py \
    --output_dir ./clip-roberta-finetuned \
    --model_name_or_path ./clip-roberta \
    --data_dir $PWD/data \
    --dataset_name ydshieh/coco_dataset_script \
    --dataset_config_name=2017 \
    --image_column image_path \
    --caption_column caption \
    --remove_unused_columns=False \
    --do_train  --do_eval \
    --per_device_train_batch_size="64" \
    --per_device_eval_batch_size="64" \
    --learning_rate="5e-5" --warmup_steps="0" --weight_decay 0.1 \
    --overwrite_output_dir \
    --push_to_hub
``` 

Ok thanks for checking, I will try with this argument. There is still something I don't understand:  when you set `--image_column image_path` why does it still counts as an unused column ? 

As you already found, the used/unused are done by looking at the model's forward signature

https://github.com/huggingface/transformers/blob/1394e08cf099d16515c1889ab9507946489f5afe/src/transformers/trainer.py#L729

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.