[WIP] Adding Omnivore Model to HF
Author: AnugunjNaman
Pull request: yes

This PR added `Omnivore Model` to HF. The model use single backbone of `SwinTransformer` adapted for 3D and classifies `image`, `video` and `RGBD scene` using same backbone weights with different head for each classification.

**TODO and Problems**:
- [ ] Solve Docstring Test for Final Classification Model. Currently, we have only single `id2label` and `label2id`. Here we need pair of three of them for each image, video and rgbd. I have done them using `<input_type>_id2label` and `<input_type>_label2id` where `input_type` is one of  image, video and rgbd. This breaks the docstring test. 

- [ ] The above problem also create another problem where during final prediction say for image `model.config.image_id2label[pred_id]` fails when `pred_id` is integer, we need to convert it to string as during loading of pretrained weights, the config doesn't convert them into `<int, string>` pairs.

- [ ] I'm unsure on how to do the feature extractor for this model. How to load video (couldn't find something like PIL and more several type of transformations are used) and rgbd part.  @NielsRogge Can you please do this part 🙏. Sorry for the trouble.

- [ ] Finally, I want to have a review on naming of final model for classification. It's not exactly image classification model only. i have named it here OmnivoreForVisionClassification (support three vision modalities). We might need to create separate task for it since even pipeline test is broken if kept in image classification task.

Apart of these @NielsRogge Please review and suggest any other changes the changes. 🙂
@sayakpaul. You can move for TF Model from here. Although I believe once above queries and TODO are done adding TF model should be more straightforward.

The uploaded weights at my hub are correct fot SwinT only. After final changes I will port rest of them.

Comments:
@AnugunjNaman thanks for working on this. Couple of pointers from me:

* I will work on a separate PR for the TF port for a cleaner separation. 
* > The uploaded weights at my hub are correct fot SwinT only. After final changes I will port rest of them.

 Shouldn't this be added to the Facebook organization by one of the HF team members? 

* > OmnivoreForVisionClassification
  
  I think it'd be fair to have a clear separation here for RGB images, RGB-D images, and videos even if the backbone is the same. We could likely add detailed comments / documentation to let the users know that the same backbone is being used but for API consistency we've developed separate classes. WDYT @NielsRogge? 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_17772). All of your documentation changes will be reflected on that endpoint.

> Shouldn't this be added to the Facebook organization by one of the HF team members?

This will be done once everything works. I wanted to test so I uploaded a smaller model on my hub.

> * I will work on a separate PR for the TF port for a cleaner separation.

I agree it would be good to have different PR.

>   I think it'd be fair to have a clear separation here for RGB images, RGB-D images, and videos even if the backbone is the same. We could likely add detailed comments / documentation to let the users know that the same backbone is being used but for API consistency we've developed separate classes. WDYT @NielsRogge?

If we do this not sure how the training part goes, but yeah implementation will be smooth it that case :) 



> If we do this not sure how the training part goes, but yeah implementation will be smooth it that case :)

Good point. In that case, it might make sense to expose the class `OmnivoreForVisionClassification`. There's a trade-off here between API consistency and confusion. Let's see what Niels has to say. 

Some first comments:
* I like the name `OmnivoreForVisionClassification`.
* Pinging @sgugger and @LysandreJik regarding the id2label question. So for context, Omnivore is a single model that is trained on 3 modalities at the same time (images, video and single-view 3D images). The model just takes tensors of shape `(batch_size, time, num_channels, height, width)` as input and returns logits of shape `(batch_size, num_labels)`. However, the model has 3 different classification heads for the 3 types of data, and one needs to indicate which modality is provided to the model during a forward pass in order for it to know which head to use. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@AnugunjNaman @NielsRogge how is the PR going? I am developing a video classification fine-tuning framework, would love to use this model if it gets merged into main!

I’m not sure when I will be able to finish it. My current job offer was rescinded so I’m looking for a new one. Probably when that gets sorted out so maybe a month or so. Sorry mate!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.