CLIPFeatureExtractor should resize images with kept aspect ratio
Author: TobiasNorlund
Pull request: yes

# What does this PR do?
Fixes #11992 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@patil-suraj
@LysandreJik
@patrickvonplaten
@sgugger

## Description

With this PR, the preprocessing should match the original preprocessing exactly. With the example from #11992 we now get the exact same results:

```python
>>> import torch
>>> import requests
>>> from transformers import CLIPProcessor, CLIPModel
>>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
>>> processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = processor(images=image, return_tensors="pt", padding=True)
>>> inputs
{'pixel_values': tensor([[[[ 0.5873,  0.5873,  0.6165,  ...,  0.0617,  0.0471, -0.0259],
          [ 0.5727,  0.5727,  0.6603,  ...,  0.1201,  0.0763,  0.0909],
          [ 0.5873,  0.5435,  0.6165,  ...,  0.0325,  0.1201,  0.0617],
          ...,
          [ 1.8719,  1.8573,  1.8719,  ...,  1.3902,  1.4340,  1.4194],
          [ 1.8281,  1.8719,  1.8427,  ...,  1.4486,  1.4340,  1.5070],
          [ 1.8573,  1.9011,  1.8281,  ...,  1.3756,  1.3610,  1.4486]],

         [[-1.3169, -1.3019, -1.3169,  ..., -1.4970, -1.4369, -1.4820],
          [-1.2418, -1.2718, -1.2268,  ..., -1.4369, -1.4669, -1.4519],
          [-1.2568, -1.3169, -1.2268,  ..., -1.4669, -1.4069, -1.4519],
          ...,
          [ 0.1239,  0.1089,  0.1239,  ..., -0.7016, -0.6865, -0.6865],
          [ 0.0789,  0.0939,  0.0488,  ..., -0.6565, -0.6865, -0.6115],
          [ 0.0939,  0.1089,  0.0038,  ..., -0.7766, -0.7316, -0.6115]],

         [[-0.4848, -0.4137, -0.3853,  ..., -0.9541, -0.8545, -0.8545],
          [-0.4137, -0.4706, -0.3711,  ..., -0.8119, -0.8545, -0.7834],
          [-0.3284, -0.4422, -0.3853,  ..., -0.8688, -0.8119, -0.8830],
          ...,
          [ 1.5771,  1.6482,  1.6340,  ...,  0.9088,  0.9514,  0.8945],
          [ 1.6198,  1.6055,  1.6055,  ...,  0.8661,  0.8092,  0.7950],
          [ 1.6624,  1.6766,  1.5487,  ...,  0.7950,  0.8661,  0.8519]]]])}
```

Comments:
Hi @TobiasNorlund , thanks a lot for spotting this.

However, this change does not actually work as expected. I processed a few images using the origin CLIP transforms and `CLIPFeatureExtractor` and compared the output. Here's the script

```python3
from PIL import Image
import os
import skimage

import torch
from clip import load

from transformers import CLIPConfig, CLIPModel, CLIPTokenizer, CLIPFeatureExtractor, CLIPProcessor

_, transforms = load("./model.pt", jit=False)

proc = CLIPProcessor.from_pretrained("./clip-vit-base-patch32/")

files = [filename for filename in os.listdir(skimage.data_dir) if filename.endswith(".png") or filename.endswith(".jpg")]

images = []
for filename in files:
    image = transforms(Image.open(os.path.join(skimage.data_dir, filename)).convert("RGB"))
    images.append(image)

hf_images = []
for filename in files:
    image = Image.open(os.path.join(skimage.data_dir, filename)).convert("RGB")
    enc = proc(images=image, return_tensors="pt")
    hf_images.append(enc.pixel_values.squeeze(0))

match = [torch.allclose(hf_image, pt_image, atol=4e-2) for hf_image, pt_image in zip(hf_images, images)]
all(match)
```

I looked into `torchvision' s` `resize` and `center_crop` implementation and turns out they are a bit different than the way we have implemented it.  Following their implem I tried overriding the `resize` and `center_crop` methods in `CLIPFeatureExtractor` which seems to be working. Here's the code 
 
```python
def center_crop(self, image, size):
    """
    Crops :obj:`image` to the given size using a center crop. Note that if the image is too small to be cropped to
    the size is given, it will be padded (so the returned result has the size asked).

    Args:
        image (:obj:`PIL.Image.Image` or :obj:`np.ndarray` or :obj:`torch.Tensor`):
            The image to resize.
        size (:obj:`int` or :obj:`Tuple[int, int]`):
            The size to which crop the image.
    """
    self._ensure_format_supported(image)
    if not isinstance(size, tuple):
        size = (size, size)

    if not isinstance(image, Image.Image):
        image = self.to_pil_image(image)

    image_width, image_height = image.size
    crop_height, crop_width = size

    crop_top = int((image_height - crop_height + 1) * 0.5)
    crop_left = int((image_width - crop_width + 1) * 0.5)

    return image.crop((crop_left, crop_top, crop_left + crop_width, crop_top + crop_height))

def resize(self, image, size, resample=Image.BICUBIC):
    width, height = image.size

    short, long = (width, height) if width <= height else (height, width)
    if short == size:
        return image

    new_short, new_long = size, int(size * long / short)

    new_w, new_h = (new_short, new_long) if width <= height else (new_long, new_short)
    return image.resize((new_w, new_h), resample)
```
With this change, there is no need to change the `__call__` method.
However, this change requires `center_crop`  to be always applied as `resize` won't always resize to an exact given size. 


Could you verify this on your end and update the PR? Thanks!

Thanks @patil-suraj !
Please have a look at the updated PR, in which I modified your `resize` method to also support non-PIL images to make tests pass.