TVP model
Author: jiqing-feng
Pull request: yes

The TVP model was proposed in [Text-Visual Prompting for Efficient 2D Temporal Video Grounding](https://arxiv.org/abs/2303.04995) by Yimeng Zhang, Xin Chen, Jinghan Jia, Sijia Liu, Ke Ding. The goal of this model is to incorporate trainable prompts into both visual inputs and textual features for temporal video grounding(TVG) problems. It was introduced in [this paper](https://arxiv.org/pdf/2303.04995.pdf).

The current model hub has [Intel/tvp-base](https://huggingface.co/Intel/tvp-base) and [Intel/tvp-base-ANet](https://huggingface.co/Intel/tvp-base-ANet).

BTW, the failed checks are weird because I can successfully run `pytest` under `tests/models/tvp` locally.

Comments:
cc @rafaelpadilla for first review :) 

Hi @rafaelpadilla . Thanks for your review!

I have made the Intel/demo dataset public, now you can try again. I also fixed the two grammar error.

However, I think it would be better to keep [test_image_processing_tvp.py](https://github.com/huggingface/transformers/pull/25856/files/7fbf413bae6784ffb8c1afd10af7b478999292a0#diff-36ed6889b91bcd24299bc53f06e3f56b7aef9193f9b7cc9d0965a3449c98e1a2) the same because [bridgetower](https://github.com/huggingface/transformers/blob/main/tests/models/bridgetower/test_image_processing_bridgetower.py#L43-L44), [clip](https://github.com/huggingface/transformers/blob/main/tests/models/clip/test_image_processing_clip.py#L43-L44), [videomae](https://github.com/huggingface/transformers/blob/main/tests/models/videomae/test_image_processing_videomae.py#L49-L50) do the same thing, and I could not ensure that `image_utils.OPENAI_CLIP_MEAN` will not change.

Hi @rafaelpadilla . I added some examples from [charades](https://prior.allenai.org/projects/charades) to my demo [datasets](https://huggingface.co/datasets/Intel/tvp_demo/tree/main), the corresponding text can see [here](https://huggingface.co/datasets/Intel/tvp_demo/blob/main/text-video.txt). You can run the script in this PR and replace the video and text to check the performance of this model.

Hi @amyeroberts . I have fixed most issues, only https://github.com/huggingface/transformers/pull/25856#discussion_r1317128312 remains to be discussed. Once this issue is done, we can train a small TVP model and start to change the tests.

Thanks for your support!

Hey! @amyeroberts will be OOO for a while  😉 feel free to ping me or @rafaelpadilla if you need help 

Hi, @ArthurZucker . I have 3 plans for this [issue](https://github.com/huggingface/transformers/pull/25856#discussion_r1328819736) since `downsample_in_first_stage` does not work. 

1. Keep the TVP backbone as it has differences from the original ResNet backbone.
2. Change the ResNet backbone API to support manipulating stride in some layers.
3. Read the ResNet backbone first, and then change some conv layers' stride in the TVP initialization stage.

Which plan do you prefer? Would like to hear your opinion. Thx!

BTW, I prefer option 3 because it is more convenient, we just need to add the following codes
```python
self.backbone = AutoBackbone.from_config(backbone_config)
if backbone_config.model_type == "resnet" and backbone_config.layer_type == "bottleneck":
    for i in range(1, len(backbone_config.depths)):
        self.backbone.encoder.stages[i].layers[0].layer[0].convolution.stride = (2, 2)
        self.backbone.encoder.stages[i].layers[0].layer[1].convolution.stride = (1, 1)
```
And I also see [detr](https://github.com/huggingface/transformers/blob/fe861e578f50dc9c06de33cd361d2f625017e624/src/transformers/models/detr/modeling_detr.py#L361) has the same operation that manipulates some layers in backbone, so I think it is a good way.

Hey! The list of models that have a backbone is the following:
```python
    [
        # Backbone mapping
        ("bit", "BitBackbone"),
        ("convnext", "ConvNextBackbone"),
        ("convnextv2", "ConvNextV2Backbone"),
        ("dinat", "DinatBackbone"),
        ("dinov2", "Dinov2Backbone"),
        ("focalnet", "FocalNetBackbone"),
        ("maskformer-swin", "MaskFormerSwinBackbone"),
        ("nat", "NatBackbone"),
        ("resnet", "ResNetBackbone"),
        ("swin", "SwinBackbone"),
        ("timm_backbone", "TimmBackbone"),
        ("vitdet", "VitDetBackbone"),
    ]
```
Either there is one that fits your need, or if the changes involved are too big you can go with 3, as there is a precedent for it. Not super fan of 3, and I believe 2 would be better if possible! 

> Hey! The list of models that have a backbone is the following:
> 
> ```python
>     [
>         # Backbone mapping
>         ("bit", "BitBackbone"),
>         ("convnext", "ConvNextBackbone"),
>         ("convnextv2", "ConvNextV2Backbone"),
>         ("dinat", "DinatBackbone"),
>         ("dinov2", "Dinov2Backbone"),
>         ("focalnet", "FocalNetBackbone"),
>         ("maskformer-swin", "MaskFormerSwinBackbone"),
>         ("nat", "NatBackbone"),
>         ("resnet", "ResNetBackbone"),
>         ("swin", "SwinBackbone"),
>         ("timm_backbone", "TimmBackbone"),
>         ("vitdet", "VitDetBackbone"),
>     ]
> ```
> 
> Either there is one that fits your need, or if the changes involved are too big you can go with 3, as there is a precedent for it. Not super fan of 3, and I believe 2 would be better if possible!

Hi @ArthurZucker . Thanks for your advice. Option 2 is now ready to be reviewed. Could you please have a look at [26374](https://github.com/huggingface/transformers/pull/26374). Thx!

Sure reviewing now! 

Hi @ArthurZucker @rafaelpadilla 

Would you please help me review these new changes? Thx!

BTW, would you please help me check the failed tests? It seems failed on init weights and `transformers.models.tvp.modeling_tvp.TvpForVideoGrounding.forward`, but no detailed information. Thx!

Hi @ArthurZucker @rafaelpadilla 

I have fixed some failed tests, but there are still two failed tests. The first indicates `worker 'gw0' crashed while running 'src/transformers/models/tvp/modeling_tvp.py::transformers.models.tvp.modeling_tvp.TvpForVideoGrounding.forward'`, the second indicates `test_push_to_hub - KeyError: 'url'`. 

Would you please help me take a look at the failed tests since there is no detailed information? Thx!

For the key error url you should rebase on main! For the forward pass, the model might be too big so the PR documentation test is not able to run it.

Hi @ArthurZucker , thanks for your review. I think I have fixed all the issues, would you please review it again? Thx!

Ok! 


Hi @ArthurZucker . I think I have fixed all the issues, would you please help review it again? Thx!

Hi @ArthurZucker @amyeroberts , would you please review the new changes? Thx

Hi @ArthurZucker @amyeroberts . Thanks for your review. Most of the comments have been resolved. There are still 2 main issues:
1. [get_resize_size](https://github.com/huggingface/transformers/pull/25856#discussion_r1379049821) was a member function in TvpProcessor in the first version but it was rejected by @amyeroberts , so I am not sure where this function should be in.
2. The sequence length in TVP consists of text input ids length, text prompt length, and visual input length. So I need to change the `test_hidden_states` and `test_attention_outputs`

Other issues are related to model doc and arg comments, please see the unresolved conversations. Thx!
BTW, the failed CI are not related to my changes

Hi @ArthurZucker @amyeroberts . The CI seems OK, would you please have a look at these unresolved comments? Thx 

Hi @ArthurZucker @amyeroberts . Thanks for your review, I am clear about all comments now and have fixed all of them. Would you please review these changes? Thx!

@jiqing-feng For the quality checks, there was a recent update and we now use `ruff` for formatting our code. Could you update the formating libraries and relint? This should make the quality checks pass: 
* `pip uninstall black`
* `pip install -e .[quality]`
* `make fixup`

Hi @ArthurZucker @amyeroberts . I think all comments have been fixed, would you please review these changes? Thx!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25856). All of your documentation changes will be reflected on that endpoint.

Thanks for this hard work! Congrats on the merge!