Pix2Struct Image processor fails when `do_normalize = False` when image is not of `np.float32` type.
Author: gbarello-uipath
Pull request: no

### System Info

Transformers version 4.30.2

### Who can help?

@younesbelkada 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoProcessor, AutoModelForSeq2SeqLM
import numpy as np
from PIL import Image

processor = AutoProcessor.from_pretrained("google/pix2struct-base")

np_image = np.random.randint(0, 255, (100,100,3)).astype(np.uint8)
pil_image = Image.fromarray(np_image)
fp32_image = np_image.astype(np.float32)

for image in [np_image, pil_image, fp32_image]:
    for do_normalize in [True, False]:
        try:
            processor(images = [image], do_normalize = do_normalize)
            print(f"Succeeded for do_normalize={do_normalize} and image type {type(image)}.")

        except Exception as e:
            print(f"Failed for do_normalize = {do_normalize} and image type {type(image)} with exception: {e}.")

    print()
```

The above script returns the following on my system:

```
Succeeded for do_normalize=True and image type <class 'numpy.ndarray'>.
Failed for do_normalize = False and image type <class 'numpy.ndarray'> with exception: "im2col_out_cpu" not implemented for 'Byte'.

Succeeded for do_normalize=True and image type <class 'PIL.Image.Image'>.
Failed for do_normalize = False and image type <class 'PIL.Image.Image'> with exception: "im2col_out_cpu" not implemented for 'Byte'.

Succeeded for do_normalize=True and image type <class 'numpy.ndarray'>.
Succeeded for do_normalize=False and image type <class 'numpy.ndarray'>.
```

The last few items on the stack in the traceback are:

```
File ~/miniconda3/envs/jupyter-env/lib/python3.11/site-packages/transformers/models/pix2struct/image_processing_pix2struct.py:68, in torch_extract_patches(image_tensor, patch_height, patch_width)
     65 requires_backends(torch_extract_patches, ["torch"])
     67 image_tensor = image_tensor.unsqueeze(0)
---> 68 patches = torch.nn.functional.unfold(image_tensor, (patch_height, patch_width), stride=(patch_height, patch_width))
     69 patches = patches.reshape(image_tensor.size(0), image_tensor.size(1), patch_height, patch_width, -1)
     70 patches = patches.permute(0, 4, 2, 3, 1).reshape(
     71     image_tensor.size(2) // patch_height,
     72     image_tensor.size(3) // patch_width,
     73     image_tensor.size(1) * patch_height * patch_width,
     74 )

File ~/miniconda3/envs/jupyter-env/lib/python3.11/site-packages/torch/nn/functional.py:4697, in unfold(input, kernel_size, dilation, padding, stride)
   4693 if has_torch_function_unary(input):
   4694     return handle_torch_function(
   4695         unfold, (input,), input, kernel_size, dilation=dilation, padding=padding, stride=stride
   4696     )
-> 4697 return torch._C._nn.im2col(input, _pair(kernel_size), _pair(dilation), _pair(padding), _pair(stride))

RuntimeError: "im2col_out_cpu" not implemented for 'Byte'
```

### Expected behavior

I expect that this processor will succeed without errors for all combinations of PIL, np.uint8 and np.float32 images and `do_normalize` in the set `{True, False}`.

Comments:
I think a reasonable solution is to move/add a check like that which happens [here](https://github.com/huggingface/transformers/blob/07998ef39926b76d3f6667025535d0859eed61c3/src/transformers/models/pix2struct/image_processing_pix2struct.py#L345) into the main body of the `process` code and just convert all image types into `float32`.

Hi @gbarello-uipath, thanks for raising this issue! 

For backwards compatibility, we'll need to keep the type casting that happens in `normalize` there. Adding equivalent logic to `preprocess` or within `extract_flattened_patches` seems like a good solution to me. Would you like to open a PR with these changes? This way you get the github contribution :) 

Thanks @amyeroberts 

Sure, I played with it a little yesterday but didn't quite get it working satisfactorily, I will see what I can do. I am a bit unclear on what the fundamental issue is though, is it just that everything needs to be cast to `float32` in order to function properly inside `normalize`? Don't want to just apply a band-aid.

@gbarello-uipath What hardware are you running on, is it CPU? 

@amyeroberts This particular problem is in running the image preprocessor and is happening on CPU yes.

@gbarello-uipath OK, thanks for confirming. 

> Sure, I played with it a little yesterday but didn't quite get it working satisfactorily, I will see what I can do
Regarding the fix, what aspect wasn't working satisfactory?

> I am a bit unclear on what the fundamental issue is though, is it just that everything needs to be cast to float32 in order to function properly inside normalize? Don't want to just apply a band-aid.

The issue is that `torch.nn.functional.unfold` can only accept inputs of a certain type. If `do_normalize=True` then any images of type `np.uint8` are cast to `float32` and so are of a compatible type before going to `extract_flattened_patches`.  

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.