Inconsistent Normalization for ViTImageProcessor when `do_resize` is False
Author: Interpause
Pull request: no

### System Info

- `transformers` version: 4.26.1
- Platform: Linux-5.4.0-121-generic-x86_64-with-glibc2.31
- Python version: 3.10.9
- Huggingface_hub version: 0.13.2
- PyTorch version (GPU?): 2.0.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@amyeroberts

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```py
from transformers import AutoImageProcessor
from PIL import Image
import torchvision.transforms as T

im = Image.open("t.png").convert("RGB")
to_tens = T.ToTensor()
extractor = AutoImageProcessor.from_pretrained("./pretrained/facebook/vit-msn-small")
print(extractor)  # Instance of ViTImageProcessor.

# When `do_resize` is True:
x1 = extractor(im, return_tensors="pt").pixel_values
x2 = extractor(to_tens(im), return_tensors="pt").pixel_values
print(abs(x2 - x1).mean()) # Close to 0; Correct.

# When `do_resize` is False:
x1 = extractor(im, return_tensors="pt", do_resize=False).pixel_values
x2 = extractor(to_tens(im), return_tensors="pt", do_resize=False).pixel_values
print(abs(x2 - x1).mean()) # Not close to 0; Differing behaviour.

# Additional multiplication of 255 to torch.Tensor input:
x1 = extractor(im, return_tensors="pt", do_resize=False).pixel_values
x2 = extractor(to_tens(im) * 255, return_tensors="pt", do_resize=False).pixel_values
print(abs(x2 - x1).mean()) # Close to 0; Correct again.
```



### Expected behavior

Currently, when `do_resize` is False, the tensor has to be multiplied by 255 first, while when `do_resize` is True, it is not needed. The behaviour should be consistent.

Comments:
cc @amyeroberts 

Hi @Interpause, thanks for raising this issue! 

Indeed, this is a funny behaviour. This is happening because of the use of the PIL library to resize images and the rescaling behaviour that happens in `ToTensor`. 

To explain in more detail, I'll refer to the input `im` and `im_pil` and `to_tens(im)` as `im_arr` below. Where `im_pil` is a `PIL.Image.Image` with integer pixel values between 0-255, and `im_arr` an array with pixel values between 0-1. 

In the first case, when`do_resize` is `True`:
* `im_pil` and `im_arr` are converted to numpy arrays, preserving their pixel values
* When passed to `resize` the images are converted to a `PIL.Image.Image` object. `im_pil` can be converted directly. However for `im_arr`, the values have to be multiplied by 255, as PIL can only store integer pixel values between 0-255.
* Images are resized then converted back to numpy arrays. `im_arr` now is a numpy array with values between 0-255, rather than the original 0-1. This shouldn't be happening - I'll try to think about the best way to handle this and open a PR. 

For the other cases, no conversion to `PIL` is happening and this behaviour is expected. Without rescaling by 255, the input arrays are different and different outputs are expected. Rescaling `to_tens(im)` by 255 makes them equivalent and so the same output is expected.