Repository: huggingface/transformers

Issue:
CLIPFeatureExtractor should resize images with kept aspect ratio
## Environment info
- `transformers` version: 4.6.1
- Platform: Linux-5.11.0-7614-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.8.1+cu102 (False)
- Tensorflow version (GPU?): 2.5.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

@patil-suraj 
@LysandreJik 
@patrickvonplaten 
@sgugger 

## Information

Model I am using (Bert, XLNet ...): CLIP

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

The CLIPFeatureExtractor does not replicate the behavior of the [CLIP reference implementation](https://github.com/openai/CLIP). The below code is taken from the official [huggingface transformers CLIP documentation](https://huggingface.co/transformers/model_doc/clip.html):

```
$ docker run --rm -it huggingface/transformers-cpu:4.6.1
root@02cd404c4a60:/workspace# pip install Pillow==7.2.0
root@02cd404c4a60:/workspace# python3
Python 3.6.9 (default, Jan 26 2021, 15:33:00) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> from PIL import Image
>>> import requests
>>> from transformers import CLIPProcessor, CLIPModel
>>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
>>> processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = processor(images=image, return_tensors="pt", padding=True)
>>> inputs
{'pixel_values': tensor([[[[ 0.2807,  0.3829,  0.4267,  ..., -0.2886, -0.2740, -0.2886],
          [ 0.3245,  0.3829,  0.4121,  ..., -0.2886, -0.2886, -0.3178],
          [ 0.2807,  0.3537,  0.3683,  ..., -0.3762, -0.3470, -0.3178],
          ...,
          [ 1.6384,  1.5362,  1.4194,  ...,  1.3902,  1.2880,  1.2442],
          [ 1.6092,  1.5508,  1.5070,  ...,  1.2150,  0.9814,  0.8501],
          [ 1.6092,  1.4778,  1.4924,  ...,  0.1201, -0.1280, -0.3908]],

         [[-1.3919, -1.3919, -1.3919,  ..., -1.5420, -1.5420, -1.5570],
          [-1.3469, -1.3469, -1.3469,  ..., -1.5270, -1.5120, -1.5270],
          [-1.4069, -1.3769, -1.3469,  ..., -1.5570, -1.5420, -1.5420],
          ...,
          [-0.3414, -0.4614, -0.5515,  ..., -0.6415, -0.7016, -0.7466],
          [-0.3414, -0.3864, -0.4914,  ..., -0.7316, -0.8666, -0.9267],
          [-0.3714, -0.4914, -0.5065,  ..., -1.2869, -1.3769, -1.4820]],

         [[-0.6555, -0.4990, -0.5417,  ..., -1.0110, -0.9256, -0.9541],
          [-0.6981, -0.5986, -0.5701,  ..., -1.0110, -0.9541, -1.0110],
          [-0.6128, -0.5275, -0.4990,  ..., -1.0252, -1.0394, -1.0536],
          ...,
          [ 1.3638,  1.3496,  1.1221,  ...,  1.1647,  1.0652,  0.9514],
          [ 1.3354,  1.1789,  1.2643,  ...,  0.9372,  0.7523,  0.6244],
          [ 1.3780,  1.3780,  1.2643,  ..., -0.1293, -0.5559, -0.7408]]]])}
```

## Expected behavior

Feeding the same image through the official CLIP preprocessing function gives different results:

```
$ docker run --rm -it pytorch/pytorch:1.8.1-cuda11.1-cudnn8-runtime
root@fe597c8c5e9f:/workspace# apt update && apt install git 
root@fe597c8c5e9f:/workspace# pip install ftfy regex tqdm
root@fe597c8c5e9f:/workspace# pip install git+https://github.com/openai/CLIP.git
root@fe597c8c5e9f:/workspace# python
Python 3.8.8 (default, Feb 24 2021, 21:46:12) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> import clip
>>> import requests
>>> from PIL import Image
>>> device = "cpu"
>>> model, preprocess = clip.load("ViT-B/32", device=device)
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> inputs = preprocess(Image.open(requests.get(url, stream=True).raw)).unsqueeze(0).to(device)
>>> inputs
tensor([[[[ 0.5873,  0.5873,  0.6165,  ...,  0.0617,  0.0471, -0.0259],
          [ 0.5727,  0.5727,  0.6603,  ...,  0.1201,  0.0763,  0.0909],
          [ 0.5873,  0.5435,  0.6165,  ...,  0.0325,  0.1201,  0.0617],
          ...,
          [ 1.8719,  1.8573,  1.8719,  ...,  1.3902,  1.4340,  1.4194],
          [ 1.8281,  1.8719,  1.8427,  ...,  1.4486,  1.4340,  1.5070],
          [ 1.8573,  1.9011,  1.8281,  ...,  1.3756,  1.3610,  1.4486]],

         [[-1.3169, -1.3019, -1.3169,  ..., -1.4970, -1.4369, -1.4820],
          [-1.2418, -1.2718, -1.2268,  ..., -1.4369, -1.4669, -1.4519],
          [-1.2568, -1.3169, -1.2268,  ..., -1.4669, -1.4069, -1.4519],
          ...,
          [ 0.1239,  0.1089,  0.1239,  ..., -0.7016, -0.6865, -0.6865],
          [ 0.0789,  0.0939,  0.0488,  ..., -0.6565, -0.6865, -0.6115],
          [ 0.0939,  0.1089,  0.0038,  ..., -0.7766, -0.7316, -0.6115]],

         [[-0.4848, -0.4137, -0.3853,  ..., -0.9541, -0.8545, -0.8545],
          [-0.4137, -0.4706, -0.3711,  ..., -0.8119, -0.8545, -0.7834],
          [-0.3284, -0.4422, -0.3853,  ..., -0.8688, -0.8119, -0.8830],
          ...,
          [ 1.5771,  1.6482,  1.6340,  ...,  0.9088,  0.9514,  0.8945],
          [ 1.6198,  1.6055,  1.6055,  ...,  0.8661,  0.8092,  0.7950],
          [ 1.6624,  1.6766,  1.5487,  ...,  0.7950,  0.8661,  0.8519]]]])
```


Discussion:
Specifically, I believe this is due to differing logic in the image resizing preprocessing step. The original CLIP implementation uses `torchvision.transforms.Resize` ([ref](https://github.com/openai/CLIP/blob/main/clip/clip.py#L60)) to resize the image given a single integer of the desired size. According to the [documentation of `Resize`](https://pytorch.org/vision/stable/transforms.html#torchvision.transforms.Resize):

> If size is an int, smaller edge of the image will be matched to this number. i.e, if height > width, then image will be rescaled to (size * height / width, size)

However, in transformers, the image is eventually [resized in `CLIPFeatureExtractor`](https://github.com/huggingface/transformers/blob/master/src/transformers/models/clip/feature_extraction_clip.py#L146), which in turn [resizes to a square size](https://github.com/huggingface/transformers/blob/123b597f5da6dd1e54545f9cce1450dc4b401784/src/transformers/image_utils.py#L158).

Resolving patch:
diff --git a/src/transformers/models/clip/feature_extraction_clip.py b/src/transformers/models/clip/feature_extraction_clip.py
--- a/src/transformers/models/clip/feature_extraction_clip.py
+++ b/src/transformers/models/clip/feature_extraction_clip.py
@@ -154,3 +154,56 @@ def __call__(
         encoded_inputs = BatchFeature(data=data, tensor_type=return_tensors)
 
         return encoded_inputs
+
+    def center_crop(self, image, size):
+        """
+        Crops :obj:`image` to the given size using a center crop. Note that if the image is too small to be cropped to
+        the size is given, it will be padded (so the returned result has the size asked).
+
+        Args:
+            image (:obj:`PIL.Image.Image` or :obj:`np.ndarray` or :obj:`torch.Tensor`):
+                The image to resize.
+            size (:obj:`int` or :obj:`Tuple[int, int]`):
+                The size to which crop the image.
+        """
+        self._ensure_format_supported(image)
+        if not isinstance(size, tuple):
+            size = (size, size)
+
+        if not isinstance(image, Image.Image):
+            image = self.to_pil_image(image)
+
+        image_width, image_height = image.size
+        crop_height, crop_width = size
+
+        crop_top = int((image_height - crop_height + 1) * 0.5)
+        crop_left = int((image_width - crop_width + 1) * 0.5)
+
+        return image.crop((crop_left, crop_top, crop_left + crop_width, crop_top + crop_height))
+
+    def resize(self, image, size, resample=Image.BICUBIC):
+        """
+        Resizes :obj:`image`. Note that this will trigger a conversion of :obj:`image` to a PIL Image.
+
+        Args:
+            image (:obj:`PIL.Image.Image` or :obj:`np.ndarray` or :obj:`torch.Tensor`):
+                The image to resize.
+            size (:obj:`int` or :obj:`Tuple[int, int]`):
+                The size to use for resizing the image. If :obj:`int` it will be resized to match the shorter side
+            resample (:obj:`int`, `optional`, defaults to :obj:`PIL.Image.BILINEAR`):
+                The filter to user for resampling.
+        """
+        self._ensure_format_supported(image)
+
+        if not isinstance(image, Image.Image):
+            image = self.to_pil_image(image)
+        if isinstance(size, tuple):
+            new_w, new_h = size
+        else:
+            width, height = image.size
+            short, long = (width, height) if width <= height else (height, width)
+            if short == size:
+                return image
+            new_short, new_long = size, int(size * long / short)
+            new_w, new_h = (new_short, new_long) if width <= height else (new_long, new_short)
+        return image.resize((new_w, new_h), resample)


Test patch:
diff --git a/tests/test_modeling_clip.py b/tests/test_modeling_clip.py
--- a/tests/test_modeling_clip.py
+++ b/tests/test_modeling_clip.py
@@ -544,7 +544,8 @@ def test_inference(self):
         ).to(torch_device)
 
         # forward pass
-        outputs = model(**inputs)
+        with torch.no_grad():
+            outputs = model(**inputs)
 
         # verify the logits
         self.assertEqual(
@@ -556,6 +557,6 @@ def test_inference(self):
             torch.Size((inputs.input_ids.shape[0], inputs.pixel_values.shape[0])),
         )
 
-        expected_logits = torch.tensor([[24.5056, 18.8076]], device=torch_device)
+        expected_logits = torch.tensor([[24.5701, 19.3049]], device=torch_device)
 
         self.assertTrue(torch.allclose(outputs.logits_per_image, expected_logits, atol=1e-3))