Add Videoprism
Author: MHRDYN7
Pull request: yes
Labels: name: New model

Fixes #39893. This pr adds the VideoPrism model by google deepmind. [Original repo](https://github.com/google-deepmind/videoprism)

Comments:
# Summary of the code so far
* The VideoPrismModel has been implemented using the modular code on top of Vivit and this is only a video encoder.
* The code design is such that the same encoder code is reused as many times as possible.
* Batches of video segments of shape (num_frames=16, H=288, W=288) are passed into the model and then the tubelet embedding class is used to convert each frame into hidden states of shape (256, 768) and therefore the whole input becomes (B*num_frame, 256, 768).
* These spatial embeddings are passed into a spatial encoder
* The outputs of the spatial encoder are reshaped to (B*256, num_frame, 768) and then passed into a temporal encoder
* The attention function has an internal attention cap implemented in the modified eager_attention, not sure if this can be somehow used along with sdpa.
  
* The VideoPrismClip model uses VideoPrismModel as a backbone for the video input, passes the embeddings through an auxiliary encoder then through an attention pooling layer.
* There is also a standard text encoder that is called inside VideoPrismClip. 
* All the exact details from the original code have been extracted and correct tensors are being returned for both the models.

@qubvel I'd request your preliminary review on the current code structure. There are detailed comments with "# ?" for ease of review, these will be removed later on.

Todos:
1. Get started with the tests
2. Implement the video and text processors
3. Add code for the video classification model

** Please note that i am currently using the preprocessing utils from the original code in my convert_weights_to_hf script. The code uses mediapy, which uses lanczos interpolation for resizing videos by default. However, lanczos interpolation is not supported in torch yet and that's why we can't get the exact same outputs if a fast video processor is used.

** The videoprism team have not released the weights for the classification head.

** The weights released on hub are in npz format, safetensors need to be uploaded there.

The following notebooks are from the original repo and the expected values of the output tensors used to validate the HF implementation have been taken from here.
Video encoder model [notebook](https://colab.research.google.com/drive/1wIwbvmfvhtSy69WLPenqGItNT7AtIn3F?usp=sharing)
Video Text model [notebook](https://colab.research.google.com/drive/1SDkoUXMc4s6vSOTGFvuJ_4_RaK05BBZe?usp=sharing)

Please note that the output tensors are slightly different for the larger models when a TPU is used. My expected tensor values are for the cpu.

Since the output of the final tensors (of HF code) matches that of the original code, the elements in the current code are aligned with the original implementation, despite that strange layernorm weight increase.

Hey @MHRDYN7 feel free to ping us once you need another review

@ArthurZucker thanks a lot for this gesture. I'll make a summary of the refactored code

@qubvel I've followed most of the code suggestions from the last review and refactored accordingly. The only places where I didn't follow the code suggestions are: 
1.  Keeping the arguments `scale_logits_by_head_dims` & `no_attention_logit_cap` inside eager_attention_forward function, please see the reason in my comment regarding that.
2. I didn't align the PositionalEmbedding class with the rope class as it is just a simple sinusoidal pos emb (the very basic one from 'attention is all you need') so I used the small function create_sinusoidal_positions from gptj and some other models (but not an exact copy).
3. Also I noticed that the l2norm from the recently added qwen3_next fits in perfectly, so borrowed that with "copied from".


cc @yonigozlan as well

@yonigozlan The original repo uses the following utility function to preprocess the input video using mediapy 
```python
def read_and_preprocess_video(  # This function from the original code
    filename: str, target_num_frames: int, target_frame_size: tuple[int, int]
):
    """Reads and preprocesses a video."""

    frames = mediapy.read_video(filename)

    # Sample to target number of frames.
    frame_indices = np.linspace(0, len(frames), num=target_num_frames, endpoint=False, dtype=np.int32)
    frames = np.array([frames[i] for i in frame_indices])

    # Resize to target size.
    original_height, original_width = frames.shape[-3:-1]
    target_height, target_width = target_frame_size
    assert original_height * target_width == original_width * target_height, (
        "Currently does not support aspect ratio mismatch."
    )
    frames = mediapy.resize_video(frames, shape=target_frame_size)

    # Normalize pixel values to [0.0, 1.0].
    frames = mediapy.to_float01(frames)

    return frames
```

Mediapy uses lanczos interpolation internally and that is not supported by torchvision. Following smolvlm, I've set the interpolation to bicubic (for now) in the VideoProcessor while trying to mimic the other transformations as closely as possible (rescale + resize). Now, for the integration tests, should I use the mediapy utility function (is mediapy supported as a dependency?) or should I go with this new simple VideoProcessor? 

Hey @MHRDYN7 ! Thanks for working on this, as you said lanczos is not yet supported in fast image processors or torchvision, so let's use bicubic for everything for a lack of better solution for now, also for the integration tests :).
Also, I'll take over from @qubvel for the reviews, so feel free to ping me once you want one!

Alright, thanks for the support @yonigozlan. I'd surely want some comments/feedback on the refactored code, as suggested by Pavel. I haven't yet refactored the conversion script to align with the mllama or dinov3 code (I'll do that at the very end, so this file might look very messy), otherwise you may take a look at all the other files. I'll get the tests done as soon as  possible. Comments (with #? ) have been kept for the tensor dimensions for better code understanding, these will be removed before the merge. 

## Here's a summary again to get you up to speed with this model.
* weights have been released for the video encoder and video text model, but not for the video classification model
* the video encoder model (vid to embedding) is named as `VideoPrismModel`
* the video text model is named as `VideoPrismClipModel`. This model uses a simple text encoder (`VideoPrismTextModel`) to get text embeds and another class named `VideoPrismVideoModel` which is built on top of `VideoPrismModel` with some auxiliary layers + attention pooling. This clip like model simply returns the similarity matrix of the text and vid embeds. Currently the helper classes `VideoPrismTextModel` and `VideoPrismVideoModel` can't be imported, please let me know if these should also be allowed to be imported from transformers (by including in `__all__`).
* the model expects batches of short video segments of 16 or 8 frames, however, I feel the video processor cannot directly create a tensor of shape (B, T, C, H, W) from a long input video, the user is generally expected to sample the frames and create the input tensor (correct if I'm wrong).
* lastly, there might be some issues with the interpolation (in both spatial and temporal dimensions), should be fixed in the next commit, otherwise if there is some inherent difference between the jax and torch interpolate (for the mode being used), this discrepancy can't be avoided.

@yonigozlan a gentle ping to bring your attention to this pr

Hi @MHRDYN7, so sorry for the delay. Reviews have been piling up recently, and I haven't had the time to check out this one. This week will be busy but I'll review next week.

reminder @yonigozlan if available, or @zucchini-nlp for video model, need a review

Hey, sorry for the delay. We've been focusing on v5 release the past month and might have missed this PR. I can review it tomorrow :)

@zucchini-nlp thank you very much for the review. I've refactored to have separate configs now and also ensured the config attributes are not changed (by having four different encoders for spatial, temporal, auxiliary & text). Other recommended refactors will follow. However, the output of hidden_states and attentions are not seen here because they were removed from various [vision models](https://github.com/search?q=repo%3Ahuggingface%2Ftransformers%20return%20BaseModelOutputWithPooling(last_hidden_state%3Dsequence_output%2C%20pooler_output%3Dpooled_output)&type=code) during the unbloating of transformers. To use the modular code, I'll need to refactor vivit and vit first (if permitted).

Great, thanks for iterating. We will work on refactoring vit-based models soon, you can leave if out for now. I will review the PR this week. Can you also make sure that CI is green?

@zucchini-nlp the CI is green (at least for now). All the necessary tests (similar to siglip) have been added.
[Note: sdpa and eager implementation cannot be equivalent due to attention capping and that is why the integration tests are done using eager.]

Tests not available for
- processors (since they have minimal differences with their parent processors, like T5Tokenizer & LlavaOneVision video processor)
- VideoClassification model (the authors did not release any checkpoint for that, only the signature is tested).
- pipeline (don't exactly remember the reason why, but also due to the fact that it was missing from vjepa2 and video llava as well).
Let me know if any important test is missing.

Currently, a place holder checkpoint (google/videoprism) is used in the docstrings because the original repo contains flax weights. The integration tests are done via my repo and the hf weights + configs need to be transferred to the correct repo under google.






kosmo2 and musicgen tests are failing, not sure if they are some how affected by the code changes in videoprism.

Also, the issue with pipeline was that there are no appropriate pipeline tags for video features and video-clip models which would correctly seek the video processor instead of the image processor. Pipeline should be fine for the video classification model, but there are no checkpoints (most likely I'll finetune one if that helps).

@zucchini-nlp a gentle ping to bring your attention to this pr. Again, some other different models are failing.

@zucchini-nlp or @yonigozlan need a review. Let's merge this asap. 

Updates since my last comment:
Refactored softplus (which now works with gpu); finetuned the vision classification model (no official checkpoint), just a few epochs on top of frozen encoder, and this model is now being used for the newly added integration test.

Failing tests are unrelated!

the ci is fully green again. Need to run the slow tests @vasqu @ArthurZucker 

@zucchini-nlp @yonigozlan @molbap I understand the team is very busy, but this PR has been stalled for too long. Need a review please and let's merge it asap. The original checkpoint had 200k downloads last month, even though it was in jax (https://huggingface.co/google/videoprism-base-f16r288). So delaying adding the model does not make sense and only prevents wider adoption.

@Rocketknight1 could you please ping the right person who may be available for this pr. Thanks!

@MHRDYN7 sorry for the delay, reviewing asap tomorrow

@zucchini-nlp I see there has been a big refactor in the docstrings recently (#44296) which is why some removed docstrings don't flow (like from SiglipText to VideoPrismText) although I see that the live docs for main branch perfectly show all the args. Not sure what's going on, if the args are now extracted automatically, should I overwrite the missing docstrings completely, or just leave those parts. Also when models are loaded, in some cases I see this in the logs https://github.com/huggingface/transformers/blob/1a50a3b13b6d17c2637fe19e94a8c459bd4208a5/src/transformers/utils/auto_docstring.py#L3654

even if those args are clearly visible in the docstrings.

@MHRDYN7 for the config classes, you need to decorate with `@auto_docstring` and add the docstring of only those args that are not common (i.e. not in `utils/auto_docstring.py::ConfigArgs`)

If args are not in common and not added, that triggers a warning that you saw. The args have to be added in the same format as in other config, like

```
class Config:
    r'''
    arg1 (int, ...): description
    arg2 (bool, ...): description
    ''''
```

@zucchini-nlp I think it's ready for next review. Not sure if this one test failure is related, but I'll look deep into it.

A little summary of the tests:
skips are mostly for 
* all the `test_eager_matches_sdpa_inference` tests (obviously sdpa can't match the eager with softcap), and 
* the output hidden states and attention checks, which rely on vivit outputs (so these tests can be enabled after the fix of vivit and other models as discussed earlier.)

The docstrings are now fine I guess, no more [ERROR] is shown while loading the models.

Also on a different note, I tried to used modular_model_detector and I think it is missing a line `self.dtype = self.model.dtype` inside CodeSimilarityAnalyzer. Not sure if I should modify that as part of this pr.

> sdpa can't match the eager with softcap

it should tho, if we add softcapping in eager and pass `softcap=self.attn_logit_softcapping` to the attention fn. See https://github.com/huggingface/transformers/blob/42ff4f324f02448dcf03ea7d619f682b4e62c010/src/transformers/models/gemma2/modeling_gemma2.py#L287-L297


> I tried to used modular_model_detector and I think it is missing a line self.dtype = self.model.dtype inside CodeSimilarityAnalyzer. Not sure if I should modify that as part of this pr.

Ah, I see, you can open a separate PR for that. We're working on updating the script soon, tho it doesn't hurt fixing until then

@zucchini-nlp everything has been addressed, please find my comments.

- New Text Embedding module
- doc update
- all checkpoints updated

let me know if anything is missing

run-slow: videoprism, vivit

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/23007741186)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/videoprism", "models/vivit"]
quantizations: []

Nice, requesting review from core maintainers. It might take some time tho, as VideoPrism is a bit old model

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39895). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/23007741186)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [2d93ace5](https://github.com/huggingface/transformers/commit/2d93ace584deeac6c08f61f028b206d483480bb8) | workflow commit (merge commit) |
| PR | [67697608](https://github.com/MHRDYN7/transformers/commit/67697608b12e2c4c83bc3e2d1a021c7b065fc125) | branch commit (from PR) |
| main | [3027515c](https://github.com/huggingface/transformers/commit/3027515ce90d1fcaa52bce069c0595cd18a00c94) | base commit (on `main`) |

### Model CI Report

❌ **[10 new failed tests from this PR](https://huggingface.co/datasets/hf-internal-testing/transformers_pr_ci/raw/2c0d5747fc8067fd2dad2355c6e6bbc0f438a7a1/2026-03-12/runs/30738-23007741186/ci_results_run_models_gpu/new_failures_with_bad_commit.json)** 😭

- [videoprism](https://github.com/huggingface/transformers/actions/runs/23007741186/job/66810015997):
    tests/models/videoprism/test_modeling_videoprism.py::VideoPrismTextModelTest::test_cpu_offload (✅ ⟹ ❌)
    tests/models/videoprism/test_modeling_videoprism.py::VideoPrismTextModelTest::test_disk_offload_bin (✅ ⟹ ❌)
    tests/models/videoprism/test_modeling_videoprism.py::VideoPrismTextModelTest::test_disk_offload_safetensors (✅ ⟹ ❌)
    tests/models/videoprism/test_modeling_videoprism.py::VideoPrismTextModelTest::test_sdpa_can_compile_dynamic (✅ ⟹ ❌)
    tests/models/videoprism/test_modeling_videoprism.py::VideoPrismTextModelTest::test_sdpa_can_dispatch_on_flash (✅ ⟹ ❌)
    tests/models/videoprism/test_modeling_videoprism.py::VideoPrismClipModelTest::test_sdpa_can_compile_dynamic (✅ ⟹ ❌)
    tests/models/videoprism/test_modeling_videoprism.py::VideoPrismClipModelTest::test_sdpa_can_dispatch_on_flash (✅ ⟹ ❌)
    tests/models/videoprism/test_modeling_videoprism.py::VideoPrismModelIntegrationTest::test_videoprism_classification_model (✅ ⟹ ❌)
    tests/models/videoprism/test_modeling_videoprism.py::VideoPrismModelIntegrationTest::test_videoprism_clip_model (✅ ⟹ ❌)
    tests/models/videoprism/test_modeling_videoprism.py::VideoPrismModelIntegrationTest::test_videoprism_vision_model (✅ ⟹ ❌)



@MHRDYN7 in the meanwhile, some failing tests ⬆️ 

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism, vivit

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism, vivit

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=39895&sha=5adef6

@MHRDYN7 could you merge/rebase on main and fix the tests? Would check it out after that :hugs: 

@vasqu yep, some recent updates in main branch around config files broke the model, so I'll refactor that and ping you.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism, vivit

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism, vivit

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism, vivit

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism, vivit

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=39895&sha=f1fab4

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism, vivit

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism, vivit

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=39895&sha=0df919

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism, vivit

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=39895&sha=8ee79a

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism, vivit

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=39895&sha=fe71d0

@vasqu this failing test is unrelated I guess, otherwise all good (for cpu). Now, the issue is the failing tests in gpu (you may check the logs from last run_slow). I ran the tests on a L4 (for fa2), and the failures are mostly 'tensor-like mismatch'.

I should give you some context here. Sdpa equivalence tests are turned off for cpu since eager uses softcap and the difference is too big. But, flash attention does handle softcap so I expected the outputs to match with decent tolerance. Not exactly sure if I'll have access to gpus for long enough, but I'll try get to the end. Also, would appreciate your expert view on this issue.

p.s. I think the modular guide doc needs to be updated after the config attributes refactor

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism, vivit

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=39895&sha=dd80ec

run-slow: videoprism

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/23863874373)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/videoprism"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/23863874373)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [bb19e5c0](https://github.com/huggingface/transformers/commit/bb19e5c0410c47fa18262360c17fd5ba2f199736) | workflow commit (merge commit) |
| PR | [dd80ec0b](https://github.com/MHRDYN7/transformers/commit/dd80ec0b2b614203f3ee10286892d2b6dad0a857) | branch commit (from PR) |
| main | [f38d6639](https://github.com/huggingface/transformers/commit/f38d6639fa6b82a401f4e2ea7fef1a3eb550c1a6) | base commit (on `main`) |

### Model CI Report

❌ **[3 new failed tests from this PR](https://huggingface.co/datasets/hf-internal-testing/transformers_pr_ci/raw/d974671c5ec9e0c00228f88b60b8ac9b267fd4bd/2026-04-01/runs/32202-23863874373/ci_results_run_models_gpu/new_failures_with_bad_commit.json)** 😭

- [videoprism](https://github.com/huggingface/transformers/actions/runs/23863874373/job/69578258068):
    tests/models/videoprism/test_modeling_videoprism.py::VideoPrismModelIntegrationTest::test_videoprism_classification_model (✅ ⟹ ❌)
    tests/models/videoprism/test_modeling_videoprism.py::VideoPrismModelIntegrationTest::test_videoprism_clip_model (✅ ⟹ ❌)
    tests/models/videoprism/test_modeling_videoprism.py::VideoPrismModelIntegrationTest::test_videoprism_vision_model (✅ ⟹ ❌)



**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism, vivit

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism, vivit

@zucchini-nlp, I investigated the three failing tests on gpu to the core. There was really no issue (except one) so added the cuda logits in the tests. The tolerance might need to be changed as these values are on a L4 machine. Also, please note the the last slow tests skipped FA2 tests (although FA is enabled for the model), but on my machine the FA2 tests still have some logits that don't fall within the tolerances. Let me know if these FA tests are supposed to be skipped or not

Yeah these tests usually fail due to numerical diff in hardware. We can help you get the correct expectations later on, so you can copy-paste

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism, vivit

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism, vivit

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism, vivit

@vasqu thank you very much the detailed review. Certainly this PR needs to wait for the VIT refactor, which should help reduce LOC a lot.  Many of the suggestions have been applied and in some cases, further consideration is required. Please see my comments and the reasoning. 

In summary, the three key points to decide are 
1. sdpa should stay or not (other models have kept it despite softcap)
2. MultiheadAttentionPoolingHead (This name has precedent + I'll implement this following the existing pattern phi4multimodal instead of your suggestions on top of my current implementation)
3. Possibly refactor ClipEmbedding model to accommodate sinusoidal pos embeds so that we can directly import that class in modular file. All the design issues pointed out here stem from that clip class and these propagate via copied from.

@MHRDYN7 Answering your concerns here but will take a look at the vit refactor to get it in first :saluting_face: 
1. Imo, it kind of depends on the sensitivity of the model. We can keep it but then need to raise our tols and indicate that in our tests with proper comments why
2. Sounds good, if there is precedence I'd rather follow that then indeed. However, definitely the version that keeps the attention interface and does not use the nn.MultiHeadAttention from torch (it does not play nicely within our designs and should be avoided)
3. If it really is that different, dw about not having perfect modular there then. These are often ideas I have, not fool proof. I'd rather not force change clip for this

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism, vivit

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=39895&sha=fe2226

Just as a small update: The vit refactor is getting closer to the finish line. We are now just making sure to fix the last details and tests :hugs: 

> Just as a small update: The vit refactor is getting closer to the finish line. We are now just making sure to fix the last details and tests 🤗

yes I am monitoring that PR, will pull the changes as soon as it is merged with main

@zucchini-nlp I tried to run the utils.check_auto.py script and there are some undesirable changes in auto_mappings.py like 
```diff
- ("detr", "DetrConfig"),
+ ("detr", "MaskFormerDetrConfig"),

-("sam3_video", "Sam3VisionConfig"),
-("sam3_vit_model", "Sam3ViTConfig"),
+("sam3_vision_model", "Sam3LiteTextVisionConfig"),
+("sam3_vit_model", "Sam3LiteTextViTConfig"),

-("sam3_vision_model", "sam3"),
-("sam3_vit_model", "sam3"),
+("sam3_vision_model", "sam3_lite_text"),
+("sam3_vit_model", "sam3_lite_text"),
```
where the values get reassigned when newer models have the model_type of existing models.
The simple fix that prevents this is changing the following lines https://github.com/huggingface/transformers/blob/85099df959f924e7f67ccc94c290af30d2dda6c0/utils/check_auto.py#L99-L101

to this

```python
    if model_type != module_name:
        if model_type not in special_mappings:
            special_mappings[model_type] = module_name 

    if model_type not in model_type_map:
        model_type_map[model_type] = config_cls_name
```

This logic of checking existing key worked for me, but I haven't fully read all the other parts of the script. Please let me if the changes are to be made, and if in a separate short PR.

possibly re #45535 cc @yonigozlan, we might need to override the model type in the configs(?). Seems like that branch was out of sync maybe with main

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism, vivit

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism, vivit

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism, vivit

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism, vivit

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=39895&sha=71b1c0

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism, vivit

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism, vivit

@vasqu request for review. Incorporated the patterns from vit refactor. Btw, commonly inheriting from vivit which directly inherits from vit (for some modules). This pattern clearly shows the hierarchy (as this model is most closely related to vivit).

- slow tests were passing on cpu, still I'm going through the tests again to see if any necessary one is skipped.
- code quality check breaks due to patterns like `del self.attribute_with_no_prev_mention` in the modular file, not really sure how to deal with that

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism, vivit

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=39895&sha=11a1b3

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism, vivit

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism

@MHRDYN7 no worries just pushed a commit to fix, just need to tell ruff to ignore things :joy: 

Taking a look / reviewing in a bit

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism

hello @vasqu, I'm really sorry for not being able to respond in the last two weeks. Almost everything is done I guess and this PR should be mergeable after your final review. However, let me note, what's left to be done.

For refresher,
4 model weights were released
`VideoPrismVisionModel`: base/large

`VideoPrismClipModel`: base/large

`VideoPrismForVideoClassification`: no weights released (integration test uses my finetuned version)

However the clip and classification models also use a separate auxiliary layer after the vision model.

Now, the only big thing left to be done is to change the video assets (and corresponding tensor chunks) used in test/conversion script to one of the videos in URLS_FOR_TESTING_DATA (tennis video most likely). This can be done immediately for the 4 weights, however, for the videoclassification model, since it was trained on ucf101-subset, the tennis video will anyway result garbage predictions. So, should I remove that model's integration test totally? (currently commented out).


Feel free to ping me whenever changes are needed until slow test run + merge



**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=39895&sha=4719ad

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, videoprism