`return_loss=True` in call for `TFCLIPModel` bugs out.
Author: ariG23498
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.23.1
- Platform: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.15
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.12.1+cu113 (False)
- Tensorflow version (GPU?): 2.9.2 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@patil-suraj

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

To reproduce the bug I have used the following code snippet 👇 
```python
import tensorflow as tf
from PIL import Image
import requests
from transformers import CLIPProcessor, TFCLIPModel

model = TFCLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(
    text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="tf", padding=True
)

outputs = model(
    input_ids=inputs["input_ids"],
    pixel_values=inputs["pixel_values"],
    attention_mask=inputs["attention_mask"],
    return_loss=True,
    return_dict=True,
)
```

### Expected behavior

The call should execute and we should obtain the `outputs`.

Comments:
Hi @ariG23498, thanks for reporting this issue. 

Could you give more information about the current behaviour? Specifically any tracebacks or more details about what is happening when you do execute? 

I have created a [colab notebook](https://gist.github.com/ariG23498/f736dea2f6f488d6c55fd9bb107bef13) that can help you with the traceback.

Let me know if you need something else. Thanks for the quick response @amyeroberts (as always 😃)


It looks like the problem in this issue is that you are not passing along as many images as texts. Passing `images=[image, image]` makes your reproducer pass.



@sgugger  Yes, this was the problem the whole time 😢 . The documentation has to fixed then. 

https://huggingface.co/docs/transformers/model_doc/clip

Indeed, do you want to make a PR with that?

@sgugger  Yes, I will take it up.

@sgugger Have been thinking over this, should there be same number of images as text ? I do not see any reason to restrict it this way . Let me know if I am missing something . 

> @sgugger Have been thinking over this, should there be same number of images as text ? I do not see any reason to restrict it this way . Let me know if I am missing something .

@sgugger  Any thoughts on this ? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Pinging on this isseue

@ArthurZucker, would you like to take a look at this?

@LysandreJik  @ArthurZucker  The confusion here is should the number of images equal to number of text ?

Hey, I think that this was solved, I can't reproduce it on main. You are right, the number of images should not necessarily be the same as the number of texts. 

```python 
>>> inputs["pixel_values"].shape
TensorShape([1, 3, 224, 224])
>>> inputs["input_ids"].shape
TensorShape([2, 7])
>>> outputs.loss
<tf.Tensor: shape=(1,), dtype=float32, numpy=array([nan], dtype=float32)>
```
Now the question is rather "should the loss acatually be `nan` 😅 

@ArthurZucker  oh, great, let me look at the fix. Last time I checked the way contrastive loss was flawed. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.