CLIPProcessor produces wrong output for CLIPModel when input is torch.Tensor instead of PIL.Image
Author: allenwang2333
Pull request: no

### System Info

transformers              4.38.2
torch                     2.1.1
torchaudio                2.1.1
torchvision               0.16.1
Pillow                    10.1.0
 
platform: mac
device: mps

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
transform = Compose([
    transform_rgb,
    ToTensor(),
    Resize((224, 224)),
])

train_features, train_label = next(iter(train_loader))

model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32").to(device)
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")


image_list  = [ToPILImage()(i) for i in train_features]


inputs = processor(text=text, images=image_list, return_tensors="pt", padding=True)
inputs2 = processor(text=text, images=train_features, return_tensors="pt", padding=True)

inputs['input_ids'] = inputs['input_ids'].to(device)
inputs['attention_mask'] = inputs['attention_mask'].to(device)
inputs['pixel_values'] = inputs['pixel_values'].to(device)

inputs2['input_ids'] = inputs2['input_ids'].to(device)
inputs2['attention_mask'] = inputs2['attention_mask'].to(device)
inputs2['pixel_values'] = inputs2['pixel_values'].to(device)
train_label = train_label.to(device)
with torch.inference_mode():
    outputs = model(**inputs)
    outputs2 = model(**inputs2)
logits_per_image = outputs.logits_per_image  # this is the image-text similarity score
logits_per_image2 = outputs2.logits_per_image  # this is the image-text similarity score

probs = logits_per_image.softmax(dim=1)  
pred = torch.argmax(probs, dim=1)
print((pred == train_label).sum()/len(pred))

probs2 = logits_per_image2.softmax(dim=1)  
pred2 = torch.argmax(probs2, dim=1)
print((pred2 == train_label).sum()/len(pred2))
```



### Expected behavior

It turns out that the first accuracy is 0 and the second accuracy is 0.75.  This seems to me that support of torch tensor is wrong.

Comments:
Hi @allenwang2333, thanks for opening an issue! 

Without the dataset loaded with `train_loader`, I'm not able to replicate to confirm the reported behaviour. 

However, if I adapt the script, the outputs from the models are equivalent: 

```py
import numpy as np
import torch
from torchvision.transforms import ToPILImage

from datasets import load_dataset
from transformers import CLIPProcessor, CLIPModel

train_loader = load_dataset("marmal88/skin_cancer", split="train", batch_size=1, streaming=True)
text = "a photo of a skin cancer"

device = "cpu"

train_images = []
for _ in range(10):
    train_images.append(next(iter(train_loader))["image"])

model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32").to(device)
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

image_list = [torch.tensor(np.array(image)) for image in train_images]

inputs = processor(text=text, images=image_list, return_tensors="pt", padding=True)
inputs2 = processor(text=text, images=train_images, return_tensors="pt", padding=True)

inputs['input_ids'] = inputs['input_ids'].to(device)
inputs['attention_mask'] = inputs['attention_mask'].to(device)
inputs['pixel_values'] = inputs['pixel_values'].to(device)

inputs2['input_ids'] = inputs2['input_ids'].to(device)
inputs2['attention_mask'] = inputs2['attention_mask'].to(device)
inputs2['pixel_values'] = inputs2['pixel_values'].to(device)

with torch.inference_mode():
    outputs = model(**inputs)
    outputs2 = model(**inputs2)
logits_per_image = outputs.logits_per_image  # this is the image-text similarity score
logits_per_image2 = outputs2.logits_per_image  # this is the image-text similarity score

probs = logits_per_image.softmax(dim=1)
probs2 = logits_per_image2.softmax(dim=1)
print((probs - probs2).abs().max()) # Prints 0 
```

After I replace ToTensor() with PILToTensor(), the problem is solved. Thanks for replying :)