`contrastive-image-text/run_clip.py` example problems
Author: AmitMY
Pull request: no

### System Info

- `transformers` version: 4.37.0.dev0
- Platform: Linux-5.15.0-88-generic-x86_64-with-glibc2.31
- Python version: 3.11.5
- Huggingface_hub version: 0.20.1
- Safetensors version: 0.4.1
- Accelerate version: 0.25.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.2+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@amyeroberts

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

The following example script has some issues: https://github.com/huggingface/transformers/blob/main/examples/pytorch/contrastive-image-text/run_clip.py

#### Minor issue:
When using `--train_file dataset.csv`, the tokenizer fails if the caption is "None", "null" or "NA"

#### Curiosity:
- There seems to be no parameter to specify the hub repository to push to.
- Also, there seems to be no place to track the experiment (like wandb)

#### Actual issue

With the following parameters
```bash
    --model_name_or_path "openai/clip-vit-base-patch32" \
    --freeze_text_model \
    --train_file "train.csv" \
    --image_column "image_path" \
    --caption_column "caption" \
    --remove_unused_columns=False \
    --do_train \
    --per_device_train_batch_size="64" \
    --per_device_eval_batch_size="64" \
    --learning_rate="5e-5" --warmup_steps="0" --weight_decay 0.1 \
    --overwrite_output_dir \
    --push_to_hub
```

I get the following error:
```bash
[INFO|trainer.py:1712] 2023-12-30 18:16:36,697 >> ***** Running training *****
[INFO|trainer.py:1713] 2023-12-30 18:16:36,697 >>   Num examples = 348,784
[INFO|trainer.py:1714] 2023-12-30 18:16:36,697 >>   Num Epochs = 3
[INFO|trainer.py:1715] 2023-12-30 18:16:36,698 >>   Instantaneous batch size per device = 64
[INFO|trainer.py:1718] 2023-12-30 18:16:36,698 >>   Total train batch size (w. parallel, distributed & accumulation) = 64
[INFO|trainer.py:1719] 2023-12-30 18:16:36,698 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:1720] 2023-12-30 18:16:36,698 >>   Total optimization steps = 16,350
[INFO|trainer.py:1721] 2023-12-30 18:16:36,698 >>   Number of trainable parameters = 88,111,361
  0%|                                                                                                                                                                                                    | 0/16350 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/home/amoryo/sign-language/signwriting-clip/signwriting_clip/transformers/examples/pytorch/contrastive-image-text/run_clip.py", line 590, in <module>
    main()
  File "/home/amoryo/sign-language/signwriting-clip/signwriting_clip/transformers/examples/pytorch/contrastive-image-text/run_clip.py", line 559, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/amoryo/conda/envs/clip/lib/python3.11/site-packages/transformers/trainer.py", line 1534, in train
    return inner_training_loop(
           ^^^^^^^^^^^^^^^^^^^^
  File "/data/amoryo/conda/envs/clip/lib/python3.11/site-packages/transformers/trainer.py", line 1860, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/amoryo/conda/envs/clip/lib/python3.11/site-packages/transformers/trainer.py", line 2737, in training_step
    loss = self.compute_loss(model, inputs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/amoryo/conda/envs/clip/lib/python3.11/site-packages/transformers/trainer.py", line 2760, in compute_loss
    outputs = model(**inputs)
              ^^^^^^^^^^^^^^^
  File "/data/amoryo/conda/envs/clip/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/amoryo/conda/envs/clip/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/amoryo/conda/envs/clip/lib/python3.11/site-packages/transformers/models/clip/modeling_clip.py", line 1108, in forward
    text_outputs = self.text_model(
                   ^^^^^^^^^^^^^^^^
  File "/data/amoryo/conda/envs/clip/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/amoryo/conda/envs/clip/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/amoryo/conda/envs/clip/lib/python3.11/site-packages/transformers/models/clip/modeling_clip.py", line 691, in forward
    hidden_states = self.embeddings(input_ids=input_ids, position_ids=position_ids)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/amoryo/conda/envs/clip/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/amoryo/conda/envs/clip/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/amoryo/conda/envs/clip/lib/python3.11/site-packages/transformers/models/clip/modeling_clip.py", line 219, in forward
    embeddings = inputs_embeds + position_embeddings
                 ~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~
RuntimeError: The size of tensor a (128) must match the size of tensor b (77) at non-singleton dimension 1
```

### Expected behavior

Example script should train, and push to hub correctly

Comments:
Hi @AmitMY, thanks for opening an issue! 

There's a few different points to address here. 

> When using --train_file dataset.csv, the tokenizer fails if the caption is "None", "null" or "NA"

Is this the same dataset being prepared in the `README.md` example or a different one? 

As a general comment, the scripts are meant to provide a simple script to show how to train a model for a specific task, but it isn't written to cover all use cases. You may need to adapt processing code to your use case.

> There seems to be no parameter to specify the hub repository to push to.

The model will be pushed to the hub under `{YOUR_USERNAME}/output_dir`. If you wish to change this, you can use `--push_to_hub_model_id` to control this

> Also, there seems to be no place to track the experiment (like wandb)

There are many different integrations available with Trainer. For wandb: 
https://docs.wandb.ai/guides/integrations/huggingface

> Actual issue

Without a copy of the data in `train.csv` it's not possible for us to help here. Could you share an example sample or publicly available dataset that triggers the same error? 

Thanks for your response. 

---

Regarding the `None` issue - 
that's just the default behavior when using a `train_file` that is a `csv`

---

About the actual issue, we can replicate it with the original example and dataset, but just changing 
```
--model_name_or_path ./clip-roberta
```
from the example to 
```
--model_name_or_path "openai/clip-vit-base-patch32"
```
(and data dir to point to where we store the dataset)

Full command:
```py
python examples/pytorch/contrastive-image-text/run_clip.py \
    --output_dir ./clip-finetuned \
    --model_name_or_path "openai/clip-vit-base-patch32" \
    --data_dir /scratch/amoryo/tmp/coco/data \
    --dataset_name ydshieh/coco_dataset_script \
    --dataset_config_name=2017 \
    --image_column image_path \
    --caption_column caption \
    --remove_unused_columns=False \
    --do_train  --do_eval \
    --per_device_train_batch_size="64" \
    --per_device_eval_batch_size="64" \
    --learning_rate="5e-5" --warmup_steps="0" --weight_decay 0.1 \
    --overwrite_output_dir \
    --push_to_hub
```

Error:

```bash
Traceback (most recent call last):
  File "/home/amoryo/sign-language/signwriting-clip/signwriting_clip/transformers/examples/pytorch/contrastive-image-text/run_clip.py", line 590, in <module>
    main()
  File "/home/amoryo/sign-language/signwriting-clip/signwriting_clip/transformers/examples/pytorch/contrastive-image-text/run_clip.py", line 559, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/amoryo/conda/envs/clip/lib/python3.11/site-packages/transformers/trainer.py", line 1534, in train
    return inner_training_loop(
           ^^^^^^^^^^^^^^^^^^^^
  File "/data/amoryo/conda/envs/clip/lib/python3.11/site-packages/transformers/trainer.py", line 1860, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/amoryo/conda/envs/clip/lib/python3.11/site-packages/transformers/trainer.py", line 2737, in training_step
    loss = self.compute_loss(model, inputs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/amoryo/conda/envs/clip/lib/python3.11/site-packages/transformers/trainer.py", line 2760, in compute_loss
    outputs = model(**inputs)
              ^^^^^^^^^^^^^^^
  File "/data/amoryo/conda/envs/clip/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/amoryo/conda/envs/clip/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/amoryo/conda/envs/clip/lib/python3.11/site-packages/transformers/models/clip/modeling_clip.py", line 1108, in forward
    text_outputs = self.text_model(
                   ^^^^^^^^^^^^^^^^
  File "/data/amoryo/conda/envs/clip/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/amoryo/conda/envs/clip/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/amoryo/conda/envs/clip/lib/python3.11/site-packages/transformers/models/clip/modeling_clip.py", line 691, in forward
    hidden_states = self.embeddings(input_ids=input_ids, position_ids=position_ids)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/amoryo/conda/envs/clip/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/amoryo/conda/envs/clip/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/amoryo/conda/envs/clip/lib/python3.11/site-packages/transformers/models/clip/modeling_clip.py", line 219, in forward
    embeddings = inputs_embeds + position_embeddings
                 ~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~
RuntimeError: The size of tensor a (128) must match the size of tensor b (77) at non-singleton dimension 1
```

Another issue, is that if following the guide to the letter, the README.md is generated with a local path as the basemodel

```md
---
base_model: /scratch/amoryo/models/signwriting-clip/reoberta-clip
---
```
which is then rejected by the hub `"base_model" with value "/scratch/amoryo/models/signwriting-clip/roberta-clip" is not valid. Use a model id from https://hf.co/models.`


cc @ydshieh 

I try to change this, and somehow it works.


```
text_inputs = tokenizer(
            captions, max_length=data_args.max_seq_length, padding="max_length", truncation= True)
```

to

```
text_inputs = tokenizer(
            captions, max_length=data_args.max_seq_length, padding=True, truncation=True)
```

> /scratch/amoryo/models/signwriting-clip/reoberta-clip

I don't have full context. But the README use `clip-roberta`, so make sure you created the local directory and using the corresponding name in the command line to launch the training (in your case, `roberta-clip`)

You are loading `openai/clip-vit-base-patch32`, so you are likely using a `CLIPTokenizer` which has `max_position_embeddings=77`. The issue could be also resolved if you specify `max_seq_length=77` in the commandline when launching the training.

Thank you for the `max_seq_length` note.

What about https://github.com/huggingface/transformers/issues/28286#issuecomment-1881147267 ? This is indeed the correct local path for the base model, but when pushed to the hub, it gives an error (if I remove this line from the README, no error)

Could you provide the full command line you used that will fail when push to hub?

Full command here: https://github.com/huggingface/transformers/issues/28286#issuecomment-1874400979
At the end of training, it creates a `README.md` file, as described here https://github.com/huggingface/transformers/issues/28286#issuecomment-1881147267 which fails to be pushed to the hub (alongside everything else)
You can easily replicate it by running the model for 10 steps, or creating a directory with `README.md` as described, and `huggingface-cli upload me/my-model README.md`

Another issue: it seems like the data does not shuffle, but shuffling is very important for a clip-like model

<img width="100%" alt="image" src="https://github.com/huggingface/transformers/assets/5757359/fab86795-5062-46e1-93f9-92db25859ff7">


I confirmed that the issue of `base_model` is reproducible. Thanks a lot - will fix it!

BTW, I get

> "base_model" with value "./clip-roberta" is not valid. Use a model id from https://hf.co/models.

But you showed

> "/scratch/amoryo/models/signwriting-clip/roberta-clip" is not valid. Use a model id from https://hf.co/models.

Just to make sure, are you specifying, in the command line, the absolute path to the local model directory `roberta-clip`?

yes, i am specifying the absolute path

The `base_model` issue is addressed in #28482

Regarding shuffling, it's a bit hidden, but if you set a breakpoint at

https://github.com/huggingface/transformers/blob/edb314ae2ba4ac0e89d6a31d48037b8943978bff/src/transformers/trainer.py#L776

you will see `RandomSampler` is used, so we are fine.

I guess that's everything. Thanks so much! Feel free to close once #28482 is one

I still find the training loss periodicity puzzling, but i have no idea. it also happens with a different base model 
![image](https://github.com/huggingface/transformers/assets/5757359/e55a7c31-5df7-45e3-9c66-af3bb6ec419b)


Fixed in #28482 as mentioned earlier