ViLT Fine-tuning Bug: ValueError: operands could not be broadcast together with shapes
Author: ZhihaoZhang97
Pull request: no

## Environment info

- `transformers` version: 4.19.0.dev0
- Platform: Linux-5.13.0-39-generic-x86_64-with-glibc2.31
- Python version: 3.9.12
- Huggingface_hub version: 0.5.1
- PyTorch version (GPU?): 1.11.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): GPU
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

Models:

- TAPAS, LayoutLM, LayoutLMv2, LUKE, ViT, ViLT BEiT, DEiT, DETR, CANINE: @NielsRogge

Library:
- Vision: @NielsRogge, @sgugger

## Information

The model I am using **ViLT**:

The problem arises when using:
* ViLT official example scripts: (give details below)
https://github.com/NielsRogge/Transformers-Tutorials/blob/master/ViLT/Fine_tuning_ViLT_for_VQA.ipynb

The tasks I am working on is:
* Fine-tune ViLT for QVA on the complete VQAv2 validation dataset

## To reproduce

Steps to reproduce the behavior:

1. Download all the required dependencies 
2. Change the data size in **Cell [29]** from **questions=questions[:100]** to **questions=questions[0:]** and **annotations=annotations[:100]** to **annotations=annotations[0:]**
3. Run the example notebook

**Replace the code below in the example notebook Cell [29] to reproduce the behavior**

```
from transformers import ViltProcessor

processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm")

dataset = VQADataset(questions=questions[0:],
                     annotations=annotations[0:],
                     processor=processor)
```


**Error Message is shown below:**
```
  1%|▌                                                        | 236/26795 [01:31<2:52:07,  2.57it/s]
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [29], in <cell line: 4>()
      4 for epoch in range(50):  # loop over the dataset multiple times
      5     print(f"Epoch: {epoch}")
----> 6     for batch in tqdm(train_dataloader, total=len(train_dataloader)):
      7         # get the inputs; 
      8         batch = {k:v.to(device) for k,v in batch.items()}
     10         # zero the parameter gradients

File ~/anaconda3/envs/hugging-face/lib/python3.8/site-packages/tqdm/std.py:1195, in tqdm.__iter__(self)
   1192 time = self._time
   1194 try:
-> 1195     for obj in iterable:
   1196         yield obj
   1197         # Update and possibly print the progressbar.
   1198         # Note: does not call self.update(1) for speed optimisation.

File ~/anaconda3/envs/hugging-face/lib/python3.8/site-packages/torch/utils/data/dataloader.py:530, in _BaseDataLoaderIter.__next__(self)
    528 if self._sampler_iter is None:
    529     self._reset()
--> 530 data = self._next_data()
    531 self._num_yielded += 1
    532 if self._dataset_kind == _DatasetKind.Iterable and \
    533         self._IterableDataset_len_called is not None and \
    534         self._num_yielded > self._IterableDataset_len_called:

File ~/anaconda3/envs/hugging-face/lib/python3.8/site-packages/torch/utils/data/dataloader.py:570, in _SingleProcessDataLoaderIter._next_data(self)
    568 def _next_data(self):
    569     index = self._next_index()  # may raise StopIteration
--> 570     data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    571     if self._pin_memory:
    572         data = _utils.pin_memory.pin_memory(data)

File ~/anaconda3/envs/hugging-face/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py:49, in _MapDatasetFetcher.fetch(self, possibly_batched_index)
     47 def fetch(self, possibly_batched_index):
     48     if self.auto_collation:
---> 49         data = [self.dataset[idx] for idx in possibly_batched_index]
     50     else:
     51         data = self.dataset[possibly_batched_index]

File ~/anaconda3/envs/hugging-face/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py:49, in <listcomp>(.0)
     47 def fetch(self, possibly_batched_index):
     48     if self.auto_collation:
---> 49         data = [self.dataset[idx] for idx in possibly_batched_index]
     50     else:
     51         data = self.dataset[possibly_batched_index]

Input In [19], in VQADataset.__getitem__(self, idx)
     19 image = Image.open(id_to_filename[annotation['image_id']])
     20 text = questions['question']
---> 22 encoding = self.processor(image, text, padding="max_length", truncation=True, return_tensors="pt")
     23 # remove batch dimension
     24 for k,v in encoding.items():

File ~/anaconda3/envs/hugging-face/lib/python3.8/site-packages/transformers/models/vilt/processing_vilt.py:91, in ViltProcessor.__call__(self, images, text, add_special_tokens, padding, truncation, max_length, stride, pad_to_multiple_of, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, return_tensors, **kwargs)
     72 encoding = self.tokenizer(
     73     text=text,
     74     add_special_tokens=add_special_tokens,
   (...)
     88     **kwargs,
     89 )
     90 # add pixel_values + pixel_mask
---> 91 encoding_feature_extractor = self.feature_extractor(images, return_tensors=return_tensors)
     92 encoding.update(encoding_feature_extractor)
     94 return encoding

File ~/anaconda3/envs/hugging-face/lib/python3.8/site-packages/transformers/models/vilt/feature_extraction_vilt.py:265, in ViltFeatureExtractor.__call__(self, images, pad_and_return_pixel_mask, return_tensors, **kwargs)
    254     images = [
    255         self._resize(
    256             image=image,
   (...)
    262         for image in images
    263     ]
    264 if self.do_normalize:
--> 265     images = [self.normalize(image=image, mean=self.image_mean, std=self.image_std) for image in images]
    267 if pad_and_return_pixel_mask:
    268     # pad images up to largest image in batch and create pixel_mask
    269     max_size = self._max_by_axis([list(image.shape) for image in images])

File ~/anaconda3/envs/hugging-face/lib/python3.8/site-packages/transformers/models/vilt/feature_extraction_vilt.py:265, in <listcomp>(.0)
    254     images = [
    255         self._resize(
    256             image=image,
   (...)
    262         for image in images
    263     ]
    264 if self.do_normalize:
--> 265     images = [self.normalize(image=image, mean=self.image_mean, std=self.image_std) for image in images]
    267 if pad_and_return_pixel_mask:
    268     # pad images up to largest image in batch and create pixel_mask
    269     max_size = self._max_by_axis([list(image.shape) for image in images])

File ~/anaconda3/envs/hugging-face/lib/python3.8/site-packages/transformers/image_utils.py:186, in ImageFeatureExtractionMixin.normalize(self, image, mean, std)
    184     return (image - mean[:, None, None]) / std[:, None, None]
    185 else:
--> 186     return (image - mean) / std

ValueError: operands could not be broadcast together with shapes (384,576) (3,)
```

## Expected behavior
The ViLT model should be fin-tuned with the provided dataset without any error.


Comments:
Hi,

Thanks for your interest in ViLT! Could you try adding `.convert("RGB")` when reading an image in the getitem method of the PyTorch dataset?

> Hi,
> 
> Thanks for your interest in ViLT! Could you try adding `.convert("RGB")` when reading an image in the getitem method of the PyTorch dataset?

Thank you for your quick reply! I will have a try now.

@NielsRogge Problem Solved! Thank you so much!