Repository: huggingface/transformers (Python)
Date: 2023-11-22 20:44:08+00:00
Task type: Bug Fix

Issue:
`YolosImageProcessor` violates `longest_edge` constraint for certain images
### System Info

- `transformers` version: 4.35.0
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.0+cu118 (False)
- Tensorflow version (GPU?): 2.14.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.7.4 (cpu)
- Jax version: 0.4.16
- JaxLib version: 0.4.16
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@NielsRogge @amyeroberts 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction


```py
from transformers import AutoProcessor
from PIL import Image
import requests

processor = AutoProcessor.from_pretrained("Xenova/yolos-small-300") # or hustvl/yolos-small-300
url = 'https://i.imgur.com/qOp3m0N.png' # very thin image

image = Image.open(requests.get(url, stream=True).raw).convert('RGB')
output = processor(image)
print(output['pixel_values'][0].shape)  # (3, 89, 1335)
```

A shape of (3, 89, 1335) is printed out, but this shouldn't be possible due to the `longest_edge` constraint in the [config.json](https://huggingface.co/Xenova/yolos-small-300/blob/main/preprocessor_config.json#L22):
```json
"size": {
  "longest_edge": 1333,
  "shortest_edge": 800
}
```

Here is the image used:
![image](https://github.com/huggingface/transformers/assets/26504141/74c75ab1-4678-4ff0-860b-b6b35a462eb8)


### Expected behavior

The image should have the maximum edge length be at most 1333 (1335 should not be possible)


Issue discussion:
Hi @xenova, thanks for reporting! 

Looking into it 🕵️‍♀️ 

Solution patch:
diff --git a/src/transformers/models/detr/image_processing_detr.py b/src/transformers/models/detr/image_processing_detr.py
--- a/src/transformers/models/detr/image_processing_detr.py
+++ b/src/transformers/models/detr/image_processing_detr.py
@@ -82,6 +82,7 @@
 SUPPORTED_ANNOTATION_FORMATS = (AnnotationFormat.COCO_DETECTION, AnnotationFormat.COCO_PANOPTIC)
 
 
+# From the original repo: https://github.com/facebookresearch/detr/blob/3af9fa878e73b6894ce3596450a8d9b89d918ca9/datasets/transforms.py#L76
 def get_size_with_aspect_ratio(image_size, size, max_size=None) -> Tuple[int, int]:
     """
     Computes the output image size given the input image size and the desired output size.
diff --git a/src/transformers/models/yolos/image_processing_yolos.py b/src/transformers/models/yolos/image_processing_yolos.py
--- a/src/transformers/models/yolos/image_processing_yolos.py
+++ b/src/transformers/models/yolos/image_processing_yolos.py
@@ -99,7 +99,6 @@ def get_max_height_width(
     return (max_height, max_width)
 
 
-# Copied from transformers.models.detr.image_processing_detr.get_size_with_aspect_ratio
 def get_size_with_aspect_ratio(image_size, size, max_size=None) -> Tuple[int, int]:
     """
     Computes the output image size given the input image size and the desired output size.
@@ -119,16 +118,17 @@ def get_size_with_aspect_ratio(image_size, size, max_size=None) -> Tuple[int, in
         if max_original_size / min_original_size * size > max_size:
             size = int(round(max_size * min_original_size / max_original_size))
 
-    if (height <= width and height == size) or (width <= height and width == size):
-        return height, width
-
-    if width < height:
-        ow = size
-        oh = int(size * height / width)
-    else:
-        oh = size
-        ow = int(size * width / height)
-    return (oh, ow)
+    if width < height and width != size:
+        height = int(size * height / width)
+        width = size
+    elif height < width and height != size:
+        width = int(size * width / height)
+        height = size
+    width_mod = np.mod(width, 16)
+    height_mod = np.mod(height, 16)
+    width = width - width_mod
+    height = height - height_mod
+    return (height, width)
 
 
 # Copied from transformers.models.detr.image_processing_detr.get_resize_output_image_size


Test patch:
diff --git a/tests/models/yolos/test_image_processing_yolos.py b/tests/models/yolos/test_image_processing_yolos.py
--- a/tests/models/yolos/test_image_processing_yolos.py
+++ b/tests/models/yolos/test_image_processing_yolos.py
@@ -86,18 +86,28 @@ def get_expected_values(self, image_inputs, batched=False):
         if not batched:
             image = image_inputs[0]
             if isinstance(image, Image.Image):
-                w, h = image.size
+                width, height = image.size
             else:
-                h, w = image.shape[1], image.shape[2]
-            if w < h:
-                expected_height = int(self.size["shortest_edge"] * h / w)
-                expected_width = self.size["shortest_edge"]
-            elif w > h:
-                expected_height = self.size["shortest_edge"]
-                expected_width = int(self.size["shortest_edge"] * w / h)
-            else:
-                expected_height = self.size["shortest_edge"]
-                expected_width = self.size["shortest_edge"]
+                height, width = image.shape[1], image.shape[2]
+
+            size = self.size["shortest_edge"]
+            max_size = self.size.get("longest_edge", None)
+            if max_size is not None:
+                min_original_size = float(min((height, width)))
+                max_original_size = float(max((height, width)))
+                if max_original_size / min_original_size * size > max_size:
+                    size = int(round(max_size * min_original_size / max_original_size))
+
+            if width < height and width != size:
+                height = int(size * height / width)
+                width = size
+            elif height < width and height != size:
+                width = int(size * width / height)
+                height = size
+            width_mod = width % 16
+            height_mod = height % 16
+            expected_width = width - width_mod
+            expected_height = height - height_mod
 
         else:
             expected_values = []
@@ -173,6 +183,18 @@ def test_equivalence_padding(self):
             torch.allclose(encoded_images_with_method["pixel_values"], encoded_images["pixel_values"], atol=1e-4)
         )
 
+    def test_resize_max_size_respected(self):
+        image_processor = self.image_processing_class(**self.image_processor_dict)
+
+        # create torch tensors as image
+        image = torch.randint(0, 256, (3, 100, 1500), dtype=torch.uint8)
+        processed_image = image_processor(
+            image, size={"longest_edge": 1333, "shortest_edge": 800}, do_pad=False, return_tensors="pt"
+        )["pixel_values"]
+
+        self.assertTrue(processed_image.shape[-1] <= 1333)
+        self.assertTrue(processed_image.shape[-2] <= 800)
+
     @slow
     def test_call_pytorch_with_coco_detection_annotations(self):
         # prepare image and target