Unable to disable the `do_resize` option in the CLIPImageProcessor
Author: adhakal224
Pull request: no

### System Info

```
- `transformers` version: 4.26.1
- Platform: Linux-3.10.0-1160.53.1.el7.x86_64-x86_64-with-glibc2.31
- Python version: 3.10.8
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.13.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no
```

### Who can help?
@amyeroberts @NielsRogge @arthur


### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I have a situation where I convert my inputs to tensor and resize them before passing them to CLIPImageProcessor. Hence, I want to disable this resize operation inside the CLIPImageProcessor. However, when i pass `False` to the `do_resize` flag, the tensors that it returns are still resized to the default 224x224 size. Here is a reproducible code:

```
from transformers import CLIPImageProcessor
batched_tensor = tensor.rand((2,3,512,512))
processor = CLIPImageProcessor.from_pretrained("openai/clip-vit-base-patch32")
processed_image = processor(list(batched_tensor), return_tensors='pt', padding=True, do_resize=False)

print(processed_image['pixel_values'].shape)

Output:
torch.Size([2, 3, 224, 224])
```
I need to use a `DataLoader` which does not accept variable sizes input (which is the case with my data) and hence I need to resize them before sending it to the CLIPImageProcessor.  
### Expected behavior

I would expect the output of the last line to be `torch.Size([2,3,512,512])`

Comments:
Hi @adhakal224, thanks for raising! The reason the output images are still being returned with 224x224 shape is that there's two operations which modify the images' size: resizing and center cropping. In order for the image to be returned in the original dimension, cropping will also have to be disabled.

```
from transformers import CLIPImageProcessor
import torch

batched_tensor = torch.rand((2,3,512,512))

processor = CLIPImageProcessor.from_pretrained("openai/clip-vit-base-patch32")
processed_image = processor(
    list(batched_tensor), 
    return_tensors='pt', 
    padding=True, 
    do_resize=False, 
    do_center_crop=False
)

print(processed_image.pixel_values.shape)

Output:
torch.Size([2, 3, 512, 512])
```

A few things to note:
* If working from the dev branch, you don't need to convert the input batch of images into a list (this is handled in the image processor)
* You will only be able to return a batch of torch arrays (`return_tensors="pt"`) if all of the input images are of the same dimension - in this case 512x512 - and `do_resize=False` and `do_center_crop=False`. 
* You mentioned `"I need to resize them before sending it to the CLIPImageProcessor."`. If you're resizing just before passing into CLIPImageProcessor, and it's a standard resizing operation e.g. [torch's resize ](https://pytorch.org/vision/main/generated/torchvision.transforms.Resize.html) you could keep `do_resize=True` instead.  Resizing is the first transformation the image processor so would be equivalent. However, it might be slower going through the image processor as it converts to PIL.Image.Image for resizing. 

Thanks for the reply @amyeroberts. Currently I have a large dataset that is saved in `WebDataset` format. It has a very large volume of images and many of them are of different sizes. When  iterating through the webdataset I read the images as numpy arrays. I am using `pytorch_lightning` and so I need to wrap the `WebDataset` with a `DataLoader` (for which I need to resize them all to a fixed size) before sending them to the `trainer`. In my current pipeline, I am using `torch.transform.ToTensor()` and `torch.transform.Resize()` to convert the images to tensor and resize them and then sending them to CLIPImageProcessor. Is there a more efficient way I can be doing this?

Below is the class that creates and returns me the `webdataset`
```
class MultiData:

    def __init__(self, wds_path):
        self.img_size = 224
        self.wds_path = wds_path
        self.dataset = wds.WebDataset(self.wds_path)
        print('Initializing dataset')

    def get_ds(self):
        self.dataset = self.dataset.shuffle(1000).decode('rgb').to_tuple("groundlevel.jpg", "overhead.jpg", "metadata.json")
        self.dataset = self.dataset.map(self.do_transforms)
        return self.dataset

    def do_transforms(self, sample):
        img, imo, json = sample
        self.transforms_img = transforms.Compose([
            transforms.ToTensor(),
            transforms.Resize(size=(224,224), interpolation=transforms.InterpolationMode.BILINEAR)
        ])
        
        self.transforms_imo = transforms.Compose([
            transforms.ToTensor()
        ])

        img = self.transforms_img(img)
        imo = self.transforms_imo(imo)
        return img, imo, json
```
Both `img` and `imo` are later passed to the processor as:
`processed_img = self.image_processor(list(img), return_tensors='pt', padding=True).to(self.device)`
`processed_imo = self.image_processor(list(imo), return_tensors='pt', padding=True).to(self.device)`

In the case of using very large datasets in PyTorch, I would recommend using torchvision transformations for the whole preprocessing pipeline in place of the image processors. The image processors are great for getting started, however they unfortunately aren't fast. We have examples of pipelines using the data stored in image processor and how to integrate them with torchvision e.g. [this one](https://github.com/huggingface/transformers/blob/main/examples/pytorch/image-classification/run_image_classification.py). 

In order to replicate the CLIP processing pipeline, I recommend looking at the [image processor code](https://github.com/huggingface/transformers/blob/main/src/transformers/models/clip/image_processing_clip.py) and the [corresponding configuration](https://huggingface.co/openai/clip-vit-base-patch32/blob/main/preprocessor_config.json) you're trying to emulate. 

In the example you posted above, it would look something like this: 
```

class MultiData:
    def __init__(self, wds_path, image_size, image_mean, image_std):
        self.wds_path = wds_path
        self.dataset = wds.WebDataset(self.wds_path)
        self.transforms_img = transforms.Compose([
            transforms.Resize(size=image_size, interpolation=transforms.InterpolationMode.BILINEAR),
            transforms.CenterCrop(size=image_size),
            transforms.ToTensor(),
            transforms.Normalize(mean=image_processor.image_mean, std=image_processor.image_std),
        ])
        print('Initializing dataset')

    def get_ds(self):
        self.dataset = self.dataset.shuffle(1000).decode('rgb').to_tuple("groundlevel.jpg", "overhead.jpg", "metadata.json")
        self.dataset = self.dataset.map(self.do_transforms)
        return self.dataset

    def do_transforms(self, sample):
        img, imo, json = sample        
        img = self.transforms_img(img)
        imo = self.transforms_imo(imo)
        return img, imo, json

image_processor = AutoImageProcessor.from_pretrained("openai/clip-vit-base-patch32")
image_size = image_processor.size["shortest_edge"]
multidata = Multidata(wds_path, image_size, image_processor.image_mean, image_processor.image_std)
```

Additional things to note:
* In your [example above](https://github.com/huggingface/transformers/issues/21620#issuecomment-1429359464), the images are being resized to (224, 224). So even if center cropping was disabled as I suggested earlier, the output images wouldn't be of dimension (512, 512). 
* For [torchvision](https://pytorch.org/vision/main/generated/torchvision.transforms.Resize.html), the resulting output size of the image will be different for `Resize(a)` versus `Resize((a, a))`.  The image processor emulates the first behaviour `Resize(a)`, where the shortest edge is resized to `a` and the longest edge resized to preserve the aspect ratio. 
* For `processed_img = self.image_processor(list(img), return_tensors='pt', padding=True).to(self.device)` - `padding` isn't a defined argument in the `image_process.preprocess` method. As such, it won't do anything and can be removed. 

I'm closing this issue, as `do_resize` was behaving as expected.