Repository: huggingface/transformers (Python)
Date: 2023-03-06 14:38:39+00:00
Task type: Bug Fix

Issue:
Mask2Former ImageProcessor produces different results on Mac vs Windows.
### System Info

>>> transformers.__version__
'4.27.0.dev0'

>>> Python 3.10.6

Windows vs Mac

### Who can help?

@amyeroberts

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
import torch
from transformers import AutoImageProcessor, Mask2FormerForUniversalSegmentation
processor = AutoImageProcessor.from_pretrained("facebook/mask2former-swin-large-cityscapes-instance", reduce_labels=False, ignore_index=255, do_resize=True, size=dict(width=500, height=500), do_normalize=True, image_mean=[0.485, 0.456, 0.406], image_std=[0.229, 0.224, 0.225])


device = torch.device("cpu")

image = Image.open(filename1)
image = image.convert('RGB')
image = np.array(image)
image = image.astype(np.float32)
image = image.transpose(2,0,1)
print(image.dtype, image.shape, image.mean((1, 2))) # float32 (3, 1000, 1000) [156.41327 149.47672 137.97989]

ret = processor([image], return_tensors="pt")
pixel_values = ret["pixel_values"].to(device)
print(pixel_values.dtype, pixel_values.shape, pixel_values[0].mean((1, 2)), pixel_values[0].std((1, 2)))
```

Windows
```
float32 (3, 1000, 1000) [156.41327 149.47672 137.97989]
mean = [-0.4228946  -0.17078026  0.25235963]
std = [0.81622934 0.699496   0.71027416]
```

Mac
```
float32 (3, 1000, 1000) [156.41327 149.47672 137.97989]
mean = [-1.229962  -1.1720737 -0.6407509]
std = [1.5912648 1.5453817 1.7506045]
```
### Expected behavior

Same result on Windows and Mac


Issue discussion:
Here is the image I used.
![0023](https://user-images.githubusercontent.com/590151/222617740-0088ded3-cd49-46df-aa23-0c2a30605729.jpg)

Also cc @alaradirik 
Thanks for raising this issue @nickponline and for all the details! 

Could you give details on how you're reading in the image e.g. through torchvision and the format the image is saved in? If I download the image in the comment above I get different results than in the snippet.

```
import torchvision

# Load in downloaded image
image = torchvision.io.read_image('222617740-0088ded3-cd49-46df-aa23-0c2a30605729.jpg')
image = image.numpy()
print(image.dtype, image.shape, image.sum()) # uint8 (3, 1000, 1000) 443861838
```
@amyeroberts @sgugger 

I'm reading the image with PIL

```
from PIL import Image
image = Image.open(filename)
image = image.convert('RGB')
image = np.array(image)
image = image.astype(np.float32)
image = image.transpose(2,0,1)
```

At that point I have confirmed the the `image` is identical on both Windows and Mac. Also after inference further in the code the Mac result is the worse than the windows result if that help. But it's the image processor that is generating a different result for identical inputs. 
@amyeroberts @sgugger the means and stds of the input image are different on Windows and Mac after `ImageProcessor` forward call:

Windows
```
mean = [-0.4228946  -0.17078026  0.25235963]
std = [0.81622934 0.699496   0.71027416]
```

Mac
```
mean = [-1.229962  -1.1720737 -0.6407509]
std = [1.5912648 1.5453817 1.7506045]
```
@amyeroberts @sgugger I updated the repro snippet above to make it easier to confirm.
@nickponline  - thank you very much for extra details! I'll dig into this and try to figure out what's happening 🕵️‍♀️ 
@amyeroberts @sgugger I feel the issue is here:

https://github.com/huggingface/transformers/blob/main/src/transformers/image_transforms.py#L159

The image is already in the range `[0..255]` and after the rescale and then `image.astype(np.uint8)` the arrays are different on Windows and Mac. 
Calling in backup here: https://stackoverflow.com/questions/75632469/why-does-np-astypeuint8-give-different-results-on-windows-versus-mac 😀
Confirming that this works with `Python 3.10.6+ (Mac) Numpy 1.24.2+`. ShruggingFace 🤷‍♂️. It must be a bug or change of behavior in Numpy or Python. Can close. 
@nickponline Thanks for the updates and all the work digging into this! 

Looking at the line you highlighted and conversation on stackoverflow, it seems there's two things happening, resulting in this issue: 
* Rescaling the pixel values by multiplying by 255 if the input image is of type `float32`. Resulting in pixel values between 0 and 65,025. Then casting to `uint8` [here](https://github.com/huggingface/transformers/blob/fcf813417aa34f3a0ea7d283f7d4f6b0834cf098/src/transformers/image_transforms.py#L162)
* Different overflow behaviour in numpy - as highlighted in [the stackoverflow comment](https://stackoverflow.com/a/75632979)

In this case, updating numpy will give consistent results between the OS's, however the resulting pixel_values from the image processor may not be sensible or produce good predictions from the model, depending on how the values are cast when overflow occurs.  

The first issue is tricky to handle - the logic is partly there for backwards compatibility as resizing was handled by the PIL library and, when converting to PIL images, whether to rescale the pixel values was inferred by the type. The assumption is that raw pixel values are of an int type and between 0-255; unnormalized float type pixel values have values between 0-1. 

I think there's two possible things we can do to address these issues in the future: 
* Add an additional check on pixel values before rescaling
* Raise a warning when casting to uint8 if overflow is going to occur 
I'll open a PR for these. 

As a side note, you don't need to convert your images to float before feeding into the image processor. You can pass in the PIL images directly. 

p.s. thanks for coining 'Shrugging Face' - I shall be using it in the future! 


Solution patch:
diff --git a/src/transformers/image_transforms.py b/src/transformers/image_transforms.py
--- a/src/transformers/image_transforms.py
+++ b/src/transformers/image_transforms.py
@@ -131,7 +131,8 @@ def to_pil_image(
             The image to convert to the `PIL.Image` format.
         do_rescale (`bool`, *optional*):
             Whether or not to apply the scaling factor (to make pixel values integers between 0 and 255). Will default
-            to `True` if the image type is a floating type, `False` otherwise.
+            to `True` if the image type is a floating type and casting to `int` would result in a loss of precision,
+            and `False` otherwise.
 
     Returns:
         `PIL.Image.Image`: The converted image.
@@ -156,9 +157,20 @@ def to_pil_image(
     image = np.squeeze(image, axis=-1) if image.shape[-1] == 1 else image
 
     # PIL.Image can only store uint8 values, so we rescale the image to be between 0 and 255 if needed.
-    do_rescale = isinstance(image.flat[0], (float, np.float32, np.float64)) if do_rescale is None else do_rescale
+    if do_rescale is None:
+        if np.all(0 <= image) and np.all(image <= 1):
+            do_rescale = True
+        elif np.allclose(image, image.astype(int)):
+            do_rescale = False
+        else:
+            raise ValueError(
+                "The image to be converted to a PIL image contains values outside the range [0, 1], "
+                f"got [{image.min()}, {image.max()}] which cannot be converted to uint8."
+            )
+
     if do_rescale:
         image = rescale(image, 255)
+
     image = image.astype(np.uint8)
     return PIL.Image.fromarray(image)
 


Test patch:
diff --git a/tests/test_image_transforms.py b/tests/test_image_transforms.py
--- a/tests/test_image_transforms.py
+++ b/tests/test_image_transforms.py
@@ -96,6 +96,11 @@ def test_to_pil_image_from_float(self, name, image_shape, dtype):
         # make sure image is correctly rescaled
         self.assertTrue(np.abs(np.asarray(pil_image)).sum() > 0)
 
+        # Make sure that an exception is raised if image is not in [0, 1]
+        image = np.random.randn(*image_shape).astype(dtype)
+        with self.assertRaises(ValueError):
+            to_pil_image(image)
+
     @require_tf
     def test_to_pil_image_from_tensorflow(self):
         # channels_first