ConvNextImageProcessor / ViTImageProcessor produce inf when do_rescale = False
Author: guillermojp
Pull request: no

### System Info

- `transformers` version: 4.29.1
- Platform: Linux-5.15.0-1022-aws-x86_64-with-glibc2.31
- Python version: 3.9.4
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.1+cu117 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@amyeroberts 

### Information

- [x] The official example scripts
- [X] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
import numpy as np
from transformers import AutoImageProcessor

>>> processor = AutoImageProcessor.from_pretrained("microsoft/resnet-50")
>>> inputs = np.random.randint(0, 256, size=(224,224,3)).astype("uint8")
>>> processor(inputs, return_tensors="np", do_rescale=False)["pixel_values"]
array([[[[inf, inf, inf, ..., inf, inf, inf],
         [inf, inf, inf, ..., inf, inf, inf],
         [inf, inf, inf, ..., inf, inf, inf],
         ...,
         [inf, inf, inf, ..., inf, inf, inf],
         [inf, inf, inf, ..., inf, inf, inf],
         [inf, inf, inf, ..., inf, inf, inf]],

        [[inf, inf, inf, ..., inf, inf, inf],
         [inf, inf, inf, ..., inf, inf, inf],
         [inf, inf, inf, ..., inf, inf, inf],
         ...,
         [inf, inf, inf, ..., nan, inf, inf],
         [inf, inf, inf, ..., inf, inf, inf],
         [inf, inf, inf, ..., inf, inf, inf]],

        [[inf, inf, inf, ..., inf, inf, inf],
         [inf, inf, inf, ..., inf, inf, inf],
         [inf, inf, inf, ..., inf, inf, inf],
         ...,
         [inf, inf, inf, ..., inf, inf, inf],
         [inf, inf, inf, ..., inf, inf, inf],
         [inf, inf, inf, ..., inf, inf, inf]]]])
```


### Expected behavior

AutoImageProcessor (whichever, I've tried a ConvNext and a Vit) should convert to [0 - 1] range explicitly (specifically [here](https://github.com/huggingface/transformers/blob/v4.29.1/src/transformers/models/convnext/image_processing_convnext.py#L288), if you're counting) before `do_normalize`. It's currently done implicitly, requiring `do_rescale=True` or otherwise you have undefined behaviour and only some warining.

Comments:
Hi @guillermojp, thanks for reporting this issue. 

The image processor does convert the pixel between `[0, 1]` explicitly - it's happens in its own independent rescaling step (not within other logic) and has its own flag to control this behaviour. If you want the image to have its values set between 0-1 then `do_rescale` should be set to `True`. 

The reason for `inf` is because of [this line](https://github.com/huggingface/transformers/blob/7f8b909189547944617741d8d3c6c84504701693/src/transformers/image_transforms.py#LL372C6-L372C6): the mean and std used to normalize the image are cast to the input image dtype. In this case, the image_std `[0.229, 0.224, 0.225]` when converted to `uint8` becomes `[0, 0, 0]`. Arguably this isn't obvious and perhaps we should think about possible warnings here when normalizing e.g. if the input is of an integer type. 

I think I misunderstood the documentation of the `ConvNextImageProcessor` to this regard, as the text description `do_rescale (bool, optional, defaults to True) — Whether to rescale the image by the specified scale rescale_factor. Can be overriden by do_rescale in the preprocess method.` wasn't clear to me. Maybe I'd recommend generating a more comprehensive description in the documentation.

Marking as closed