Bug in Microsoft TROCR Large
Author: khanfarhan10
Pull request: no

## Environment info

- `transformers` version: 4.12.2
- Platform: Linux-5.11.0-1020-azure-x86_64-with-debian-bullseye-sid
- Python version: 3.7.11
- PyTorch version (GPU?): 1.10.0+cu102 (False)
- Tensorflow version (GPU?): 2.6.1 (False)
- Flax version (CPU?/GPU?/TPU?): 0.3.6 (cpu)
- Jax version: 0.2.24
- JaxLib version: 0.1.73
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: Parallel

### Who can help
 I myself solved this one. Edited a file called `image_utils.py` which was calling the shapes wrongly.

Models:
- (Microsoft TROCR Large)[https://huggingface.co/microsoft/trocr-large-printed]

## To reproduce

Steps to reproduce the behavior:
Failed to run inference on TROCR :
 
### Installation Steps :
Followed from https://github.com/microsoft/unilm/tree/master/trocr
```
conda create -n trocr python=3.7
conda activate trocr
git clone https://github.com/microsoft/unilm.git
cd unilm
cd trocr
pip install pybind11
pip install -r requirements.txt
pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" 'git+https://github.com/NVIDIA/apex.git'
```
 
### Also installed transformers from :
```
pip install transformers[all]
python -c "from transformers import pipeline; print(pipeline('sentiment-analysis')('we love you'))" # verified
```
 
 
### Python Script to Invoke Inference :
 
From https://huggingface.co/microsoft/trocr-large-printed
```
from transformers import TrOCRProcessor, VisionEncoderDecoderModel
from PIL import Image
import requests
# load image from the IAM database (actually this model is meant to be used on printed text)
url = 'https://fki.tic.heia-fr.ch/static/img/a01-122-02-00.jpg'
image = Image.open(requests.get(url, stream=True).raw)
processor = TrOCRProcessor.from_pretrained('microsoft/trocr-large-printed')
model = VisionEncoderDecoderModel.from_pretrained('microsoft/trocr-large-printed')
pixel_values = processor(images=image, return_tensors="pt").pixel_values
generated_ids = model.generate(pixel_values)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
```
### Error Message Encountered (Inside the Library Source FIle) :
```
(trocr) hello@vm-Farhan-Ubuntu20:~/work/helloassets/DockerThings/SetupTROCR$ python simple_inference.py 
Some weights of VisionEncoderDecoderModel were not initialized from the model checkpoint at microsoft/trocr-large-printed and are newly initialized: ['encoder.pooler.dense.weight', 'encoder.pooler.dense.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Traceback (most recent call last):
  File "simple_inference.py", line 20, in <module>
    pixel_values = processor(images=image, return_tensors="pt").pixel_values
  File "/home/hello/work/anaconda3/envs/trocr/lib/python3.7/site-packages/transformers/models/trocr/processing_trocr.py", line 117, in __call__
    return self.current_processor(*args, **kwargs)
  File "/home/hello/work/anaconda3/envs/trocr/lib/python3.7/site-packages/transformers/models/vit/feature_extraction_vit.py", line 141, in __call__
    images = [self.normalize(image=image, mean=self.image_mean, std=self.image_std) for image in images]
  File "/home/hello/work/anaconda3/envs/trocr/lib/python3.7/site-packages/transformers/models/vit/feature_extraction_vit.py", line 141, in <listcomp>
    images = [self.normalize(image=image, mean=self.image_mean, std=self.image_std) for image in images]
  File "/home/hello/work/anaconda3/envs/trocr/lib/python3.7/site-packages/transformers/image_utils.py", line 149, in normalize
    return (image - mean) / std
ValueError: operands could not be broadcast together with shapes (384,384) (3,)
```
## Expected behavior

Valid OCR Output.


Comments:
Hi,

Thanks for spotting. The problem is that the image is grey-scale, meaning no color channels, and the normalize method defined in `image_utils.py` assumes 3 dimensions.

You can fix it by making sure the image has 3 dimensions, i.e. 

`image = Image.open(requests.get(url, stream=True).raw).convert("RGB")`

cc @sgugger 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.




> ## Environment info
> * `transformers` version: 4.12.2
> * Platform: Linux-5.11.0-1020-azure-x86_64-with-debian-bullseye-sid
> * Python version: 3.7.11
> * PyTorch version (GPU?): 1.10.0+cu102 (False)
> * Tensorflow version (GPU?): 2.6.1 (False)
> * Flax version (CPU?/GPU?/TPU?): 0.3.6 (cpu)
> * Jax version: 0.2.24
> * JaxLib version: 0.1.73
> * Using GPU in script?: No
> * Using distributed or parallel set-up in script?: Parallel
> 
> ### Who can help
> I myself solved this one. Edited a file called `image_utils.py` which was calling the shapes wrongly.
> 
> Models:
> 
> * (Microsoft TROCR Large)[https://huggingface.co/microsoft/trocr-large-printed]
> 
> ## To reproduce
> Steps to reproduce the behavior: Failed to run inference on TROCR :
> 
> ### Installation Steps :
> Followed from https://github.com/microsoft/unilm/tree/master/trocr
> 
> ```
> conda create -n trocr python=3.7
> conda activate trocr
> git clone https://github.com/microsoft/unilm.git
> cd unilm
> cd trocr
> pip install pybind11
> pip install -r requirements.txt
> pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" 'git+https://github.com/NVIDIA/apex.git'
> ```
> 
> ### Also installed transformers from :
> ```
> pip install transformers[all]
> python -c "from transformers import pipeline; print(pipeline('sentiment-analysis')('we love you'))" # verified
> ```
> 
> ### Python Script to Invoke Inference :
> From https://huggingface.co/microsoft/trocr-large-printed
> 
> ```
> from transformers import TrOCRProcessor, VisionEncoderDecoderModel
> from PIL import Image
> import requests
> # load image from the IAM database (actually this model is meant to be used on printed text)
> url = 'https://fki.tic.heia-fr.ch/static/img/a01-122-02-00.jpg'
> image = Image.open(requests.get(url, stream=True).raw)
> processor = TrOCRProcessor.from_pretrained('microsoft/trocr-large-printed')
> model = VisionEncoderDecoderModel.from_pretrained('microsoft/trocr-large-printed')
> pixel_values = processor(images=image, return_tensors="pt").pixel_values
> generated_ids = model.generate(pixel_values)
> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
> ```
> 
> ### Error Message Encountered (Inside the Library Source FIle) :
> ```
> (trocr) koireader@vm-Farhan-Ubuntu20:~/work/koireaderassets/DockerThings/SetupTROCR$ python simple_inference.py 
> Some weights of VisionEncoderDecoderModel were not initialized from the model checkpoint at microsoft/trocr-large-printed and are newly initialized: ['encoder.pooler.dense.weight', 'encoder.pooler.dense.bias']
> You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
> Traceback (most recent call last):
>   File "simple_inference.py", line 20, in <module>
>     pixel_values = processor(images=image, return_tensors="pt").pixel_values
>   File "/home/koireader/work/anaconda3/envs/trocr/lib/python3.7/site-packages/transformers/models/trocr/processing_trocr.py", line 117, in __call__
>     return self.current_processor(*args, **kwargs)
>   File "/home/koireader/work/anaconda3/envs/trocr/lib/python3.7/site-packages/transformers/models/vit/feature_extraction_vit.py", line 141, in __call__
>     images = [self.normalize(image=image, mean=self.image_mean, std=self.image_std) for image in images]
>   File "/home/koireader/work/anaconda3/envs/trocr/lib/python3.7/site-packages/transformers/models/vit/feature_extraction_vit.py", line 141, in <listcomp>
>     images = [self.normalize(image=image, mean=self.image_mean, std=self.image_std) for image in images]
>   File "/home/koireader/work/anaconda3/envs/trocr/lib/python3.7/site-packages/transformers/image_utils.py", line 149, in normalize
>     return (image - mean) / std
> ValueError: operands could not be broadcast together with shapes (384,384) (3,)
> ```
> 
> ## Expected behavior
> Valid OCR Output.

Hi, could I please get the updated code for `image_utils.py`?

The image you provided is not a RGB image. TrOCR requires a 3 Channel image as input, I suggest changing the following line `image = Image.open(requests.get(url, stream=True).raw)`   into  `image = Image.open(requests.get(url, stream=True).raw).convert("RGB")`

if you are using opencv to read the image, you should convert the image from grayscale to RGB using `cv2.cvtColor(im,cv2.COLOR_GRAY2RGB)` before feeding it to the model.


cc @amyeroberts this is another thing which I'd like to see resolved with our image processors, to make sure people don't occur errors like `ValueError: operands could not be broadcast together with shapes (384,384) (3,)`.

@NielsRogge What do you think the behaviour should be for the image processors? We could add checks earlier in the processing pipeline to see if the image is process-able and the errors are more explicit e.g. in this case if we can only process RGB images, we check the number of channels is 3 and raise a value error otherwise? 

> in this case if we can only process RGB images, we check the number of channels is 3 and raise a value error otherwise?

Yes, exactly!