Wrong logit tensor dimensions when training CLIP
Author: npyoung
Pull request: no

### System Info

- `transformers` version: 4.42.3
- Platform: Linux-6.5.0-26-generic-x86_64-with-glibc2.17
- Python version: 3.8.19
- Huggingface_hub version: 0.23.4
- Safetensors version: 0.4.3
- Accelerate version: 0.32.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.3.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: no
- Using GPU in script?: yes
- GPU type: NVIDIA GeForce RTX 4090

### Who can help?

@sgugger @muellerzr 

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I'm following the [example script](https://github.com/huggingface/transformers/blob/main/examples/pytorch/contrastive-image-text/run_clip.py) to fine-tune CLIP on some custom data for zero-shot image classification. I am starting from the popular `openai/clip-vit-base-patch32` weights. I can get the script to run on my dataset and I can see my training loss decreasing, but...

During eval I want to add a custom accuracy metric, so I defined a `compute_metrics(eval_preds)` that I pass to the trainer.  For this model, the `eval_preds` that gets passed to my custom metrics function should be a tuple where the first element is `logits_per_image`: an `(image_batch_size, text_batch_size)` tensor as described [here](https://github.com/huggingface/transformers/blob/1082361a1978d30db5c3932d1ee08914d74d9697/src/transformers/models/clip/modeling_clip.py#L127). 

### Expected behavior

`logits_per_image` appears to have shape `(num_validation_examples, batch_size)`.

Similarly, the 2nd element in `eval_preds` should be `logits_per_text` and have transposed dimensions but it's also `(num_validation_examples, batch_size)`. So I really can't tell what's going on here - certainly not the behavior the docs describe.

Furthermore `label_ids` is an empty list. I'd expect something like `arange(num_classes)` or the indices for the current batch.

In short, I'd like to compute top1 and top5 accuracy but I can't tell what the extra rows in the logit tensors are there for.

Comments:
Hi @npyoung, thanks for opening an issue! Could you share a minimal code reproducer? 

I find it surprising the `logits_per_image` don't have the transposed dimensions of `logits_per_text` from the model output as this is how they are [defined in the model](https://github.com/huggingface/transformers/blob/1082361a1978d30db5c3932d1ee08914d74d9697/src/transformers/models/clip/modeling_clip.py#L1141) and there are no further transformations on the object. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.