[WIP] Add CLIPViP
Author: tensorpro
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->


Fixes #22829 


## Before submitting
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
https://github.com/huggingface/transformers/issues/22829

- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
Docs are WIP

I haven't updated the docstrings yet. 
- [X] Did you write any new necessary tests? 
Integration test still needs some work. It's basically just the CLIP integration test, it doesn't test how video retrieval would work
## Who can review?

@NielsRogge 

maybe others?

Comments:
cc @amyeroberts 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23802). All of your documentation changes will be reflected on that endpoint.

@tensorpro  Thank you very much for this PR and adding this great model!
Let us know when this is ready for review and feel free to ping me if you have any issues or questions about the implementation in the meantime. 

Thanks @amyeroberts!
This PR should be pretty close. I need to add a custom processor for CLIPViP to setup the preprocessing for videos and improve documentation. But after that, it should be ready to review.


Are there bugs in [modeling_outputs'](https://github.com/huggingface/transformers/blob/17a55534f5e5df10ac4804d4270bf6b8cc24998d/src/transformers/modeling_outputs.py#L46-L47) type hints for attentions/hidden outputs? It seems like we want `Tuple[FloatTensor,...]` instead of the`Tuple[FloatTensor]`? 
The current type hint makes it seem like we should return a tuple with a single`FloatTensor` instead of a tuple with an arbitrary number of float tensors.

@tensorpro Good spot - yes, the returned tuples can have a variable number of items. In practice, we're not running e.g. mypy against the repo so shouldn't break things but, as you point out, could be misleading. Would you like to open an issue or PR addressing this? 

Ah thanks for clearing it up! I only caught it cause my editor was complaining about the return value types.

And I'd be happy to make a PR.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Sorry about the delay, but I got around to adding a CLIPViPProcessor + improving some of the docs.

Merging with the recent changes in [24306](https://github.com/huggingface/transformers/pull/24306) was a bit confusing, since it wasn't clear from the error messages I was getting that we need to add `_set_token_in_kwargs` to the config. Would it make sense to identify these errors and add something like "you may need to add _set_token_in_kwargs" to the error messages?

Also, I think the PR is ready for review though I will be refining the docs a bit more.

I need to update the of the docstring examples that use images as examples to use videos instead, it should be ready for review after that though.

@tensorpro Great :) Ping me when that's done and I'll review 👍 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Any update on this?