CLIPFeatureExtractor should resize images with kept aspect ratio
Author: TobiasNorlund
Pull request: no

## Environment info
- `transformers` version: 4.6.1
- Platform: Linux-5.11.0-7614-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.8.1+cu102 (False)
- Tensorflow version (GPU?): 2.5.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

@patil-suraj 
@LysandreJik 
@patrickvonplaten 
@sgugger 

## Information

Model I am using (Bert, XLNet ...): CLIP

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

The CLIPFeatureExtractor does not replicate the behavior of the [CLIP reference implementation](https://github.com/openai/CLIP). The below code is taken from the official [huggingface transformers CLIP documentation](https://huggingface.co/transformers/model_doc/clip.html):

```
$ docker run --rm -it huggingface/transformers-cpu:4.6.1
root@02cd404c4a60:/workspace# pip install Pillow==7.2.0
root@02cd404c4a60:/workspace# python3
Python 3.6.9 (default, Jan 26 2021, 15:33:00) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> from PIL import Image
>>> import requests
>>> from transformers import CLIPProcessor, CLIPModel
>>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
>>> processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = processor(images=image, return_tensors="pt", padding=True)
>>> inputs
{'pixel_values': tensor([[[[ 0.2807,  0.3829,  0.4267,  ..., -0.2886, -0.2740, -0.2886],
          [ 0.3245,  0.3829,  0.4121,  ..., -0.2886, -0.2886, -0.3178],
          [ 0.2807,  0.3537,  0.3683,  ..., -0.3762, -0.3470, -0.3178],
          ...,
          [ 1.6384,  1.5362,  1.4194,  ...,  1.3902,  1.2880,  1.2442],
          [ 1.6092,  1.5508,  1.5070,  ...,  1.2150,  0.9814,  0.8501],
          [ 1.6092,  1.4778,  1.4924,  ...,  0.1201, -0.1280, -0.3908]],

         [[-1.3919, -1.3919, -1.3919,  ..., -1.5420, -1.5420, -1.5570],
          [-1.3469, -1.3469, -1.3469,  ..., -1.5270, -1.5120, -1.5270],
          [-1.4069, -1.3769, -1.3469,  ..., -1.5570, -1.5420, -1.5420],
          ...,
          [-0.3414, -0.4614, -0.5515,  ..., -0.6415, -0.7016, -0.7466],
          [-0.3414, -0.3864, -0.4914,  ..., -0.7316, -0.8666, -0.9267],
          [-0.3714, -0.4914, -0.5065,  ..., -1.2869, -1.3769, -1.4820]],

         [[-0.6555, -0.4990, -0.5417,  ..., -1.0110, -0.9256, -0.9541],
          [-0.6981, -0.5986, -0.5701,  ..., -1.0110, -0.9541, -1.0110],
          [-0.6128, -0.5275, -0.4990,  ..., -1.0252, -1.0394, -1.0536],
          ...,
          [ 1.3638,  1.3496,  1.1221,  ...,  1.1647,  1.0652,  0.9514],
          [ 1.3354,  1.1789,  1.2643,  ...,  0.9372,  0.7523,  0.6244],
          [ 1.3780,  1.3780,  1.2643,  ..., -0.1293, -0.5559, -0.7408]]]])}
```

## Expected behavior

Feeding the same image through the official CLIP preprocessing function gives different results:

```
$ docker run --rm -it pytorch/pytorch:1.8.1-cuda11.1-cudnn8-runtime
root@fe597c8c5e9f:/workspace# apt update && apt install git 
root@fe597c8c5e9f:/workspace# pip install ftfy regex tqdm
root@fe597c8c5e9f:/workspace# pip install git+https://github.com/openai/CLIP.git
root@fe597c8c5e9f:/workspace# python
Python 3.8.8 (default, Feb 24 2021, 21:46:12) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> import clip
>>> import requests
>>> from PIL import Image
>>> device = "cpu"
>>> model, preprocess = clip.load("ViT-B/32", device=device)
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> inputs = preprocess(Image.open(requests.get(url, stream=True).raw)).unsqueeze(0).to(device)
>>> inputs
tensor([[[[ 0.5873,  0.5873,  0.6165,  ...,  0.0617,  0.0471, -0.0259],
          [ 0.5727,  0.5727,  0.6603,  ...,  0.1201,  0.0763,  0.0909],
          [ 0.5873,  0.5435,  0.6165,  ...,  0.0325,  0.1201,  0.0617],
          ...,
          [ 1.8719,  1.8573,  1.8719,  ...,  1.3902,  1.4340,  1.4194],
          [ 1.8281,  1.8719,  1.8427,  ...,  1.4486,  1.4340,  1.5070],
          [ 1.8573,  1.9011,  1.8281,  ...,  1.3756,  1.3610,  1.4486]],

         [[-1.3169, -1.3019, -1.3169,  ..., -1.4970, -1.4369, -1.4820],
          [-1.2418, -1.2718, -1.2268,  ..., -1.4369, -1.4669, -1.4519],
          [-1.2568, -1.3169, -1.2268,  ..., -1.4669, -1.4069, -1.4519],
          ...,
          [ 0.1239,  0.1089,  0.1239,  ..., -0.7016, -0.6865, -0.6865],
          [ 0.0789,  0.0939,  0.0488,  ..., -0.6565, -0.6865, -0.6115],
          [ 0.0939,  0.1089,  0.0038,  ..., -0.7766, -0.7316, -0.6115]],

         [[-0.4848, -0.4137, -0.3853,  ..., -0.9541, -0.8545, -0.8545],
          [-0.4137, -0.4706, -0.3711,  ..., -0.8119, -0.8545, -0.7834],
          [-0.3284, -0.4422, -0.3853,  ..., -0.8688, -0.8119, -0.8830],
          ...,
          [ 1.5771,  1.6482,  1.6340,  ...,  0.9088,  0.9514,  0.8945],
          [ 1.6198,  1.6055,  1.6055,  ...,  0.8661,  0.8092,  0.7950],
          [ 1.6624,  1.6766,  1.5487,  ...,  0.7950,  0.8661,  0.8519]]]])
```

Comments:
Specifically, I believe this is due to differing logic in the image resizing preprocessing step. The original CLIP implementation uses `torchvision.transforms.Resize` ([ref](https://github.com/openai/CLIP/blob/main/clip/clip.py#L60)) to resize the image given a single integer of the desired size. According to the [documentation of `Resize`](https://pytorch.org/vision/stable/transforms.html#torchvision.transforms.Resize):

> If size is an int, smaller edge of the image will be matched to this number. i.e, if height > width, then image will be rescaled to (size * height / width, size)

However, in transformers, the image is eventually [resized in `CLIPFeatureExtractor`](https://github.com/huggingface/transformers/blob/master/src/transformers/models/clip/feature_extraction_clip.py#L146), which in turn [resizes to a square size](https://github.com/huggingface/transformers/blob/123b597f5da6dd1e54545f9cce1450dc4b401784/src/transformers/image_utils.py#L158).