Add VATT model
Author: johko
Pull request: no
Labels: name: New model

### Model description

Hey,
as discussed with @NielsRogge a few weeks back, I'd like to work on adding the "VATT: Transformers for Multimodal
Self-Supervised Learning from Raw Video, Audio and Text" model from Google.

It is basically three transformers(Video/Audio/Text) that are trained jointly in an unsupervised manner using contrastive loss functions. For downstreams tasks they fine-tune the Transformers separately, but also explore a version that shares the weights for all modalities.

For Pre-Traning they use text-video-audio triplets from HowTo100M and video-audio pairs from AudioSet. The authors describe how to fine-tune VATT for vision and audio classification tasks and provide weights for the fine-tuned versions.

The backbone for vision is ViT, for audio WaveFormTransformer and for text they are using BERT/T5

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

Paper: https://arxiv.org/pdf/2104.11178.pdf
GitHub: https://github.com/google-research/google-research/tree/master/vatt

Comments:
@johko have you started implementing it?

@fcakyon yes I have started, but progress is still rather slow, as that is my first model contribution and I have to figure out some stuff.


@johko I totally understand it. Interested in your implementation since I will be using VATT in my research next year :)

Are you working on a TF implementation?

> @johko I totally understand it. Interested in your implementation since I will be using VATT in my research next year :)
> 
> Are you working on a TF implementation?

Sorry for the late reply (again 🙈). Yes I'm working on a TF implementation. As the original repo is using it, I'm first doing that and then see about pytorch.

@johko, thanks for the response! I may also help with the pytorch part once you finalize the TF implementation 👍 

@fcakyon that would be great, as my expertise is more in TF 🙂

Hey @NielsRogge , I'm sorry but I think I have to stop working this for good. I'd love to finish it, but every time I think now I finally have some time to do it, something else comes around :disappointed: 

I think I just can't provide a big contribution like this atm and will rather focus on smaller things. But maybe @fcakyon wants to pick up on it.

Sorry for blocking this so long.

any news about VATT PyTorch implementation ?

Hey @johko, sorry for the very late reply, we're catching up on the model-request backlog, and thanks for the effort you put into kicking this off (no worries at all about stepping back).

Since 2022 our process for new architectures has shifted to a *Model on the Hub first* approach: the recommended way to use VATT today is to publish the modeling code + weights in a Hub repo and load with `trust_remote_code=True`, no core port required. Given the original effort wound down a couple of years ago and no PR ever landed, I'm closing this as won't-implement for now. That's not a hard no, if you, @fcakyon, or anyone else wants to revive it, the cleanest route is a `modular_vatt.py` contribution following the modular guide (https://github.com/huggingface/transformers/blob/150eb7c9ed4091294c829fa0e9466b090cb0f87f/docs/source/en/modular_transformers.md) and the new-model-addition template