Changing dtype(For half precision) not possible in rescale in image_processing_donut
Author: archit76
Pull request: no

### System Info

- `transformers` version: 4.37.2
- Platform: Linux-6.1.0-1029-oem-x86_64-with-glibc2.35
- Python version: 3.9.18
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: 0.24.1

https://github.com/huggingface/transformers/blob/main/src/transformers/models/donut/image_processing_donut.py#L441-L445
if do_rescale:
    images = [
        self.rescale(image=image, scale=rescale_factor, input_data_format=input_data_format)
        for image in images
    ]
------------------------------------------------
self.rescale calls BaseImageProcessor rescale method:
https://github.com/huggingface/transformers/blob/v4.37.2/src/transformers/image_processing_utils.py#L557-L564
def rescale(
    self,
    image: np.ndarray,
    scale: float,
    data_format: Optional[Union[str, ChannelDimension]] = None,
    input_data_format: Optional[Union[str, ChannelDimension]] = None,
    **kwargs,
) -> np.ndarray:
Which calls rescale(image, scale=scale, data_format=data_format, input_data_format=input_data_format, **kwargs)
--------------------------------------------------
rescale function has a fixed dtype:
https://github.com/huggingface/transformers/blob/v4.37.2/src/transformers/image_transforms.py#L92-L98
def rescale(
    image: np.ndarray,
    scale: float,
    data_format: Optional[ChannelDimension] = None,
    dtype: np.dtype = np.float32,
    input_data_format: Optional[Union[str, ChannelDimension]] = None,
) -> np.ndarray:

### Who can help?

@amyeroberts, Can I create a pull request with the change?

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

from transformers import DonutImageProcessor
import numpy as np

image = np.random.rand(3, 224, 224)
print(DonutImageProcessor().preprocess(images=[image], , dtype=np.float16))

output:
{'pixel_values': [array([[[-1., -1., -1., ..., -1., -1., -1.],
        [-1., -1., -1., ..., -1., -1., -1.],
        [-1., -1., -1., ..., -1., -1., -1.],
        ...,
        [-1., -1., -1., ..., -1., -1., -1.],
        [-1., -1., -1., ..., -1., -1., -1.],
        [-1., -1., -1., ..., -1., -1., -1.]],

       [[-1., -1., -1., ..., -1., -1., -1.],
        [-1., -1., -1., ..., -1., -1., -1.],
        [-1., -1., -1., ..., -1., -1., -1.],
        ...,
        [-1., -1., -1., ..., -1., -1., -1.],
        [-1., -1., -1., ..., -1., -1., -1.],
        [-1., -1., -1., ..., -1., -1., -1.]],

       [[-1., -1., -1., ..., -1., -1., -1.],
        [-1., -1., -1., ..., -1., -1., -1.],
        [-1., -1., -1., ..., -1., -1., -1.],
        ...,
        [-1., -1., -1., ..., -1., -1., -1.],
        [-1., -1., -1., ..., -1., -1., -1.],
        [-1., -1., -1., ..., -1., -1., -1.]]], dtype=float32)]}



### Expected behavior

from transformers import DonutImageProcessor
import numpy as np

image = np.random.rand(3, 224, 224)
print(DonutImageProcessor().preprocess(images=[image], dtype=np.float16))

output:
{'pixel_values': [array([[[-1., -1., -1., ..., -1., -1., -1.],
        [-1., -1., -1., ..., -1., -1., -1.],
        [-1., -1., -1., ..., -1., -1., -1.],
        ...,
        [-1., -1., -1., ..., -1., -1., -1.],
        [-1., -1., -1., ..., -1., -1., -1.],
        [-1., -1., -1., ..., -1., -1., -1.]],

       [[-1., -1., -1., ..., -1., -1., -1.],
        [-1., -1., -1., ..., -1., -1., -1.],
        [-1., -1., -1., ..., -1., -1., -1.],
        ...,
        [-1., -1., -1., ..., -1., -1., -1.],
        [-1., -1., -1., ..., -1., -1., -1.],
        [-1., -1., -1., ..., -1., -1., -1.]],

       [[-1., -1., -1., ..., -1., -1., -1.],
        [-1., -1., -1., ..., -1., -1., -1.],
        [-1., -1., -1., ..., -1., -1., -1.],
        ...,
        [-1., -1., -1., ..., -1., -1., -1.],
        [-1., -1., -1., ..., -1., -1., -1.],
        [-1., -1., -1., ..., -1., -1., -1.]]], dtype=float16)]}

Comments:
@amyeroberts 
Passing kwargs should resolve this:
https://github.com/huggingface/transformers/blob/main/src/transformers/models/donut/image_processing_donut.py#L441-L445
if do_rescale:
images = [
self.rescale(image=image, scale=rescale_factor, input_data_format=input_data_format, **kwargs)
for image in images
]

Hi @archit76, thanks for raising this issue! 

Could you clarify the expected behaviour here? Is it that you want the returned dtype of `pixel_values` to be `float16`? 

A few comments: 

1. In the provided example, the dummy image being passed in is in `float32`. When you pass this to the image processor you should get the following warning: 

```
It looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.
```

This is because `do_rescale` rescales the pixel values to have values between [0, 1], from `[0, 255]`. The dummy image generated from `np.random.rand(...)` will have values in `[0, 1)` and so rescaling isn't required and you should set `do_rescale=False`.

2. If you're using pytorch, then you can call `to` directly on the image processor outputs to cast to the desired type. 

```
from transformers import DonutImageProcessor
import numpy as np

# Image of values in the range [0, 255]
image = np.random.randint(0, 256, (3, 224, 224))

# Output a batch of torch tensors 
outputs = image_processor(image, return_tensors="pt")

# Cast to the desired type
outputs.to(torch.float16)
``` 

3. In the codebase, we try to avoid passing kwargs wherever possible. Adding in `kwargs` here is a brittle solution. 

@amyeroberts Yes, I would like to get the dtype of pixel values to be float16.
In order to use Donut model.half() at inference, it requires input type to be float16 instead of float32.

@archit76 Then calling `outputs.to(torch.float16)` should be the approach. 

@amyeroberts I already tried outputs.to(torch.float16), but the dtype is set to float32 because of
https://github.com/huggingface/transformers/blob/v4.37.2/src/transformers/image_transforms.py#L92-L98
def rescale(
image: np.ndarray,
scale: float,
data_format: Optional[ChannelDimension] = None,
dtype: np.dtype = np.float32,
input_data_format: Optional[Union[str, ChannelDimension]] = None,
) -> np.ndarray: 

@archit76 Could you provide a code snippet to reproduce this? In the example I provided, it correctly casts the input to `torch.float16`.