Reproducibility of VideoMAE on Kinetics-400
Author: innat
Pull request: no

### System Info

```bash
transformers: 4.30.2
torch: 2.0.0
```

### Who can help?

@NielsRogge 

Contributor of video-mae model to transformer lib.
https://github.com/huggingface/transformers/pull/17821

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

(to - do)

### Expected behavior

Summary: 

1. In official repo of video-mae, [here](https://github.com/MCG-NJU/VideoMAE/blob/main/MODEL_ZOO.md#kinetics-400), it is reported that the video-mae-ft performed 79.0 (top1) and 93.8 (top5) accuracy on kintetics-400 test dataset.
2. I have applied [huggingface implementation](https://huggingface.co/docs/transformers/main/model_doc/videomae#transformers.VideoMAEForVideoClassification) of video-mae to evaluate or validate (`model.evaluate`) the model (`small` version) and expected to get reported results. But it didn't come close. 
3. [FYI] Using the official model, score improves compare to huggingface adaptation, but it doesn't come close to the reported results. 

- Official claims: 79% accuracy
- Huggingface adaptation: 42% accuracy
- Official code: 60% accuracy.

Could you please ensure that the model and its official weights are validated properly before adding to the library? How the health of these official weights are checked? If it is ensured, I will create a reproducible code that uses huggingface video-mae.

FYI, If I use this model on some downstream task, the model gives performant results. So, it would not make sense to validate the model by using it to downstream task. Any video classification model would give that (more or less).



Comments:
Hi @innat 

The check is done in the following file, especially at

https://github.com/huggingface/transformers/blob/1c6f072db0c17c7d82bb0d3b7529d57ebc9a0f2f/src/transformers/models/videomae/convert_videomae_to_pytorch.py#L210

Our focus is to make sure the inference gives the (almost) identical to the original implementation. For the training, it involves a lot of factors, and we can't 100% guarantee, although we try our best.

The first thing to check is to see if you get (significant) difference even with inference, between the original/HF implementation.

Hello @ydshieh 
Thanks for the details response. 

I tried to evaluate VideoMAE on the benchmark data set (K400) using their [fine-tuned weight](https://github.com/MCG-NJU/VideoMAE/blob/main/MODEL_ZOO.md#kinetics-400). But it didn't achieve such high (mentioned above). Now, as HF included their model, I was hoping that HF might tested or evaluated their provided weights on those datasets. (Not by maching the output signal.)

FYI, we could still use these models and eventually they start optimize like any other classification model. But their reproducibility on the reported datasets and scores are still a questions. 

| Method   | Backbone | #Frame          | Fine-tune            | Top-1 | Top-5 |
|----------|----------|--------|----------------------|-------|-------|
| VideoMAE |ViT-S    | 16x5x3 | script/log/[checkpoint](https://drive.google.com/file/d/1ygjLRm1kvs9mwGsP3lLxUExhRo6TWnrx/view) | **79.0**  | **93.8**  |