VideoMAE assumes channel_num==3
Author: tarokiritani
Pull request: no

### System Info

- `transformers` version: 4.23.1
- Platform: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.15
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.12.1+cu113 (False)
- Tensorflow version (GPU?): 2.9.2 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@NielsRogge 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

`VideoMAEForPreTraining` assumes the channel number is 3. The code below works if `num_channels = 3`.

```python
from transformers import VideoMAEForPreTraining, VideoMAEConfig
import numpy as np
import torch

num_frames = 16
num_channels = 1
config = VideoMAEConfig(num_channels=num_channels)

model = VideoMAEForPreTraining(config)
num_patches_per_frame = (model.config.image_size // model.config.patch_size) ** 2
seq_length = (num_frames // model.config.tubelet_size) * num_patches_per_frame
bool_masked_pos = torch.randint(0, 2, (1, seq_length)).bool()
model(torch.rand([1, num_frames, num_channels, 224, 224]), bool_masked_pos)
```

The above code spits out this error message:
```
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/loss.py:530: UserWarning: Using a target size (torch.Size([1, 760, 1536])) that is different to the input size (torch.Size([1, 760, 512])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  return F.mse_loss(input, target, reduction=self.reduction)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
[<ipython-input-6-46b1d4563ea7>](https://localhost:8080/#) in <module>
     11 seq_length = (num_frames // model.config.tubelet_size) * num_patches_per_frame
     12 bool_masked_pos = torch.randint(0, 2, (1, seq_length)).bool()
---> 13 model(torch.rand([1, num_frames, num_channels, 224, 224]), bool_masked_pos)

5 frames
[/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _call_impl(self, *input, **kwargs)
   1128         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1129                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1130             return forward_call(*input, **kwargs)
   1131         # Do not call functions when jit is used
   1132         full_backward_hooks, non_full_backward_hooks = [], []

[/usr/local/lib/python3.7/dist-packages/transformers/models/videomae/modeling_videomae.py](https://localhost:8080/#) in forward(self, pixel_values, bool_masked_pos, head_mask, output_attentions, output_hidden_states, return_dict)
    884 
    885         loss_fct = MSELoss()
--> 886         loss = loss_fct(logits, labels)
    887 
    888         if not return_dict:

[/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _call_impl(self, *input, **kwargs)
   1128         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1129                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1130             return forward_call(*input, **kwargs)
   1131         # Do not call functions when jit is used
   1132         full_backward_hooks, non_full_backward_hooks = [], []

[/usr/local/lib/python3.7/dist-packages/torch/nn/modules/loss.py](https://localhost:8080/#) in forward(self, input, target)
    528 
    529     def forward(self, input: Tensor, target: Tensor) -> Tensor:
--> 530         return F.mse_loss(input, target, reduction=self.reduction)
    531 
    532 

[/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py](https://localhost:8080/#) in mse_loss(input, target, size_average, reduce, reduction)
   3277         reduction = _Reduction.legacy_get_string(size_average, reduce)
   3278 
-> 3279     expanded_input, expanded_target = torch.broadcast_tensors(input, target)
   3280     return torch._C._nn.mse_loss(expanded_input, expanded_target, _Reduction.get_enum(reduction))
   3281 

[/usr/local/lib/python3.7/dist-packages/torch/functional.py](https://localhost:8080/#) in broadcast_tensors(*tensors)
     71     if has_torch_function(tensors):
     72         return handle_torch_function(broadcast_tensors, tensors, *tensors)
---> 73     return _VF.broadcast_tensors(tensors)  # type: ignore[attr-defined]
     74 
     75 

RuntimeError: The size of tensor a (512) must match the size of tensor b (1536) at non-singleton dimension 2
```

In line [886](https://github.com/huggingface/transformers/blob/bd469c40659ce76c81f69c7726759d249b4aef49/src/transformers/models/videomae/modeling_videomae.py#L886), the dimension of `labels` is 3 times as large as it should be. This dimension mismatch is caused by this [unnormalization](https://github.com/huggingface/transformers/blob/bd469c40659ce76c81f69c7726759d249b4aef49/src/transformers/models/videomae/modeling_videomae.py#L824). Since the `mean` and `std` are 3 dimensional, the `pixel_values` are broadcast in L826. I am not sure if this "unnormalization" operation is necessary. I was a bit surprised to see it because I usually do this kind of transformations during data loading.

### Expected behavior

VideoMAEForPretraining should accept tensors even if the input channel number is not 3.

Comments:
cc @NielsRogge 

Hi,

Thanks for your interest in VideoMAE. I took the unnormalization from the original implementation as can be seen here: https://github.com/MCG-NJU/VideoMAE/blob/b6af64a997da1a2f52ce1cb2f300712faa2444a1/engine_for_pretraining.py#L38-L41. 

The unnormalization is done to "undo" the normalization done during data preprocessing (as the model needs to predict raw pixel values). So I assume something similar needs to be done when working with greyscale videos; one needs to unnormalize them before calculating the loss.



If self.config.norm_pix_loss is true, normalization of each patch undoes the effect of unnormalization:
https://github.com/huggingface/transformers/blob/bd469c40659ce76c81f69c7726759d249b4aef49/src/transformers/models/videomae/modeling_videomae.py#L852-L854
Anyhow, I suppose the goal of your implementation is to replicate the original model published by the authors. For now, I will just comment out the [unnormalization](https://github.com/huggingface/transformers/blob/bd469c40659ce76c81f69c7726759d249b4aef49/src/transformers/models/videomae/modeling_videomae.py#L824-L826) to deal with my gray scale videos.