OSKAR: Omnimodal Self-supervised Knowledge Abstraction and Representation
Authors: Mohamed O Abdelfattah
Kaouther Messaoud
Alexandre Alahi
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Self-Supervised Learning
Representation Learning
Feature Prediction
Primary area: deep_learning
TL;DR: OSKAR is a self-supervised multimodal model that predicts masked token latent features from video, skeleton, and text using momentum target encoders—outperforming specialized models across tasks without reconstruction or contrastive losses.

Abstract: We present OSKAR, the first multimodal foundation model based on bootstrapped latent feature prediction. Unlike generative or contrastive methods, it avoids memorizing unnecessary details (e.g., pixels), and does not require negative pairs, large memory banks, or hand-crafted augmentations. We propose a novel pretraining strategy: given masked tokens from multiple modalities, predict a subset of missing tokens per modality, supervised by momentum-updated uni-modal target encoders. This design efficiently utilizes the model capacity in learning high-level representations while retaining modality-specific information. Further, we propose a scalable design which decouples the compute cost from the number of modalities using a fixed representative token budget—in both input and target tokens—and introduces a parameter-efficient cross-attention predictor that grounds each prediction in the full multimodal context. We instantiate OSKAR on video, skeleton, and text modalities. Extensive experimental results show that OSKAR's unified pretrained encoder outperforms models with specialized architectures  of similar size in action recognition (rgb, skeleton, frozen, low-shot) and localization, video-text retrieval, and video question answering. Project website: https://multimodal-oskar.github.io

OSKAR: Omnimodal Self-supervised Knowledge Abstraction and Representation
Mohamed Abdelfattah"1
Kaouther Messaoud"
Alexandre Alahi
École Polytechnique Fédérale de Lausanne (EPFL), Switzerland
(firstname. Lastname)@ept]l
Pretraining Stage
Latent Space
Encode
Vision
Match
Fuse & Predict
Encode Motion
Match
[MASK]
[MASK]
standing the [MASK]
Match
Encode Language
pedestrian
[MA5K] sidewalk
Downstream Tasks
Action Recognition
Action Localization
Text-Video Retrieval
Video QA
many guys
Action:
Action:
American footbell aiming board
Caption:
perform together?
Running
Play Tennis
Answer: Three
Figure I: OSKAR is self-supervised multimodal foundation model that learns in the latent space using a fuse-thhrr thee-ctreicct strategy. It integrates multiple modalities to capture cross-modal features, matching latent predictions to targets from modality-specinee momentum encoders. This preserves uni-modal structure while enabling rich cross moosl learning, and finetuning the unified encoder surpasses specialized models across video, skeleton, and text tasks.
Abstract
We present OSKAR, the first multimodal foundation model based bootstrapped latent feature prediction. Unlike generative or contrastive methods, it avoids mem- orizing unnecessary details (e.g., pixels), and does not require negative pairs, large memory banks, or hand-crafted augmentations. We propose a novel pretraining strategy: given masked tokens from multiple modalities, predict a subset of missing tokens per modality, supervised by momentum-upden uni-modal target encoders. This design efficiently utilizes the model capacity learning high-level representa- tions while retaining modality-specific information. Further, we propose scalable design which decouples the compute cost from the number of modalities using fixed representative token budget-in both input and target tokens -nd intro- duces a parameter-efficies cross-atiention predictor that grounds each prediction in the full multimodal context. We instantiate OSK AR video, skeleton, and text modalities. Extensive experimental results show that OSKAR's unified pretrained encoder outperforms models with specialized architectures of similar size in action recognition (rgb, skeleton, frozen, low-shot) and localization, video-text retrieval, and video question answering. Project website: https://multimodll -osiiall---arreeeeeee ssttar.gttub.io
Equal contribution. Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Human perception inherently multimodal-we naturally integrate visual, motion, and linguistic cues to form coherent understanding from partial observations. In computer vision, multimodal models offer key advantages: (1) they align with human perception by leveraging visual, structural, and semantic signals; (2) they provide architectural flexibility through unified, reusable representations; and (3) they enhance robustness by fusing complementary inputs (e.g., RGB + LiDAR). Existing multimodal methods typically fall into two categories: generative and contrastive. Generative approaches [6] 60, 7 36 42 26). often based on masked autoencoding [40], focus on low-level reconstruction, which may waste capacity on irrelevant details. Contrastive methods [34 83 55 88
align high-level embeddings but rely on modality-specifff priors, handcrafted augmentations, and lack cross- moosl predictive reasoning. Therefore, we ask: Is it possible move beyond inefficient reconstruction and restrictive contrastive objectives to learn rich cross-modal representations?
To address these challenges, we explore multiple strategies for effectively routing multimodal information, culminating in OSKAR (Omnimodal Self-supervised Knowledge Abstraction and Representation), comprising three novel contributions:
(1) A new pretext task: given partial multimodal observations, use cross-modal cues to predict the latent representations of subset of the missing parts in each modality. As shown in Fig. [ OSKAR fuses visible multimodal tokens but infers the missing token representations in each modality separately. This strikes a crucial balance between cross- moosl fusion and retention of modality- specific information. Further, by learning in the latent space, the model capacity efficiently utilized in learning transferrable high-level representations instead of low-level details. Our approach is grounded in the predictive coding theory (65] 30, which posits that the brain learns by predicting internal multimodal representations and minimizing errors, rather than reconstructing raw inputs.
(2) -sdelity-specific target encoders: Unlike prior works 663 70 20 43 distilled from external teachers, OSKAR trains from scratch with momentum-updated target encoders-on per modality, These encoders co-evolve with the model and data, generating stable yet adaptive targets that align closely with the model's 's internal representation space. This design offers an interesting trade-off: with fixed momentum update rate, we get shared-weight target encoders. thus providing a flexible modality-agnostt encoder in fine-tuning; with customized update rates, we allow each modality to evolve at its own learning pace, providing multiple modallttyspeectti encoders with peak performance.
(3) Scalable design: OSKAR scales efficiently thanks to three key design choices. First, it processes a fixed total number of tokens in both the input and target, thus dissociating the compute cost from the number of modalities. Importantly, it ensures fair representation of all modalities, regardless of their raw size, through Dirichlet allocation strategy. Second, it avoids information leaks through non-overlapping masking, within and between the imputs and targets. Finally, OSKAR introduces a unified, -agnsity- roos-attenntio predictor that efficiently anchors predictions in shared multimodal context- scamllssly scaling to new modalities with limited growth in model size.
Though OSKAR supports plug-and-play extensibility, instantiate it three distinct modali- ties-video (dense), poses (sparse), and text (symmolie)-foriing a challenging testbed for eval- uation. Trained entirely with pseudo-labels and without manual annotations, OSKAR matches or surpasses specialized models on RGB- and skeleton-based action recognition (86.1% K400, 91. 1% NTU120-XSub), spatiotemporal action localization (37.9 mAP AVA), text-video retrieval (50.4 R@ MSRVTT), and video question answering (49.3% MSRVTT-QA). It also outperforms baselines in low-sample, low-parameter, and low-label settings-highlighting its efficiency and versatility.
2 Related Works
Generative architectures (GAs). Generative self- supervised models corrupt the input and train an encoder together with lightweight reconstruction head to in- in-paint the missing content in input space. MAE [40] restores masked image patches; VideoMAE [7]] extends the idea to spatio-temporal tubes; OmniMAE 33 accepts mixed image- video inputs; Other unified masked models 78 68 extend to
vision-language and follow-ups 66 60 7 33 broaden the paradigm to additional modalities. While
effective at capturing low-level details, these methods optimize pixel-level losses, often diverting model capacity toward reconstructing semantically uninformative elements like texture lighting.

---

Joint-embedding architectures (JEAs). JEAs learn by aligning representations. Given two or more views of the same instance- obtained through data augmentation or drawn from another modality-the model is trained with a contrastive 19
[89 I1 or average embedding entropy maximization [5 4 38 objective that pulls similar pairs together in feature
64 aedundancy-reducttin
space while pushing dissimilar ones apart. In cross modal context, CLIP (64] aligns whole-image
and sentence embeddings via a pure contrastive loss; More recent models [50] 23 6 46] add a
momentum teacher that self-distills latent knowledge to the student. ImageBind 34 generalises
the recipe to six sensory streams. The objective enforces only global agreement between paired embeddings. As result, JEAs excel zero-shot recognition and retrieval, yet they lack mechanism for structured cross- modsll prediction for instance, nothing in CLIP compels the model to infer a missing video patch from co-occurring caption.
Joint-embedding predictive architectures (JEPAS). JEPAS 47 blend both worlds by replacing pixel reconstruction with latent prediction. A predictor maps visible-context embeddings to the target embeddings of masked regions; the loss computed in feature space. Hence, the objective is to
learn representations that are predictive of each other. On images, I-JEPA (5)
demonstrates that
latent prediction yields strong abstraction with lower compute than MAE [40]. V-JEPA [[2 and
S-JEPA M adapt the idea to video and skeleton data, respectively. iBOT 9 and data2vec 8 9 also
fall under the JEPA framework, performing latent prediction from masked inputs to match teacher representations at the patch level in iBOT, and in a moodality-agnostee manner in data2vec.
OSKAR diverges from these methods in the following: (1) Cross-modal latent prediction: Com- pared to 5 120 OSKAR intentionally uses far fewer per-modality tokens; yet, it complements them with cross-modal cues. Hence, the model is forced to learn from fused cues from all modalities to predict the missing latent features in each. (2) Modallty-specifee target encoders: OSKAR employs separate target encoders per modality, balancing intra-modality structure with cross-modal alignment, and allowing a flexible design choice between shared or customized encoders. (3) Scalable cross- modal masking: OSKAR masks both the inputs and targets with fixed budgets to keep the compute cost manageable with increasing modalities. Further, it uses Dirichlet sampling to process dense and sparse modalities fairly, while ensuring cross-modal exclusivity to avoid trivial shortcuts.
3 Methodology
Overview. OSKAR operates in two stages: pretraining and fine-tuning. In pretraining (see Fig. 2). OSKAR optimizes a novel "fuse-then-preee"" perennttrect" task: given partially masked multimodal tokens, it fuses visible inputs via cross-mode cross- fusion encoder and predicts modality-specific latent repre- sentations for masked tokens using shared-weight per-modality predictors. Target representations generated by momentum-updateed encoders, providing stable supervision. Importantly, learning occurs entirely in the latent space, emphasizing high-level semantics over low-level reconstruction. In fine-tuning, the target encoders adapted directly to downstream tasks. All backbones are standard transformers [24], enabling seamless integration into existing transformer pipelines.
3.1 Architecture
Tokenization and Embedding. Following
60 we first tokenize raw inputs (e.g., videos, skeletons,
text) using modality-specific encoders [2
92 69 tokenization serves only to accelerate training and
is not required for the method itself, producing tokens Tm, for each modality m. These are linearly projected via 9m into shared embedding space Em R.....d of Nm modality tokens, enabling unified encoder across modalities without task-specific customizations, Each E is then augmented with learnable positional e(pos). modality e(mod), and auxiliary ((aux) signals. In particular, e(aux) are modallty-speeciff auxiliary signals proposed to resolve ambiguities (e.g.. helping the predictor establish exel-to-keypoinn correspondence and disambiguate multiple people in a scene. Without them, the model may associate skeletons with the wrong subject, especially in crowded scenes). We then apply our masking strategy (explained later) to keep only a few non-overlapping patches in the fusion and target encoders, denoted E E R N' xd and E' E N' where N and No denote the number of input and target patches, respectively.
Fusion encoder. Given E the fusion encoder's objective to exploit the complementary cues from all modalities to infer the missing information in each. To that end, we feed E to the fusion
3

---

T
E
fo
YP
Vision Projection
Vision Predictor
IC
Motion Projection
Fusion Encoder
Motion Predictor
pedestrian standing on sidewalk
Language Projection
Lang. Predictor
EMA
V
Vision
Vision Target Encoder
MSE Loss
Video Taken
Projection
Skeleton Token
Motion Projection
Motion Target Encoder
Text Token
MSE Loss
Fused Token
Dropped Token
Language Projection
Language
MSE Loss
Target Encoder
Mask Token
ga
fa
Figure 2: OSKAR Architecture. Multimodal tokens (e.g., video, skeleton, text) are projected into shared space and split into two branches: (1) fusion encoder processes visible tokens to produce fused representations. (2) modality-specific target encoders generate target embeddings. A predictor estimates masked representations from fused tokens, supervised via MSE loss against targets. Target encoders are updated via EMA of the fusion encoder and are used exclusively in fine-tuning.
transformer fe. where all modality tokens interact with each other through the inter-modee Multi-Head Self-Attention (MHSA) mechanism [72] yielding fused representations H E W (.M1x xd
The role of the predictor is to generate the latent features for subset of the missing tokens in every modality. These predictions are conditioned on predictor-specifie target-location queries Q' E NR xd which are learnable mask tokens M', augmented with positional (pes) modality E and auxiliary cues ( The predictor processes each set of queries Q of modality through transformer with alternating self-attention and cross-atie cooss-attention layers. Self-attention allows queries to share information and capture intra-modaan modality structure. while cross-attention integrates information from the fused representations H. At each cross-attention layer, Q attend to all tokens in H:
Predictor:
Y! MHCA(Q H. H),
where YP, are the predicted representations and MHCA Multi-Head Cross-Attention. Hence, Yp E R xd concatenates all per-modality predictions Y Importantly, there is single predictor network with shared weights across modalities. However, it operates like several-one per modality-whil retaining the efficiency and regularization benefits of shared weights. This design (i) allows queries to exchange information within a modality (via self-attention), (ii) grounds every prediction in the full multimodal context H (via cross- aoss-atttenton),, and (iii) adapts to modalities without additional heads, all while remaining decoupled from the fusion encoder for easy transfer to downstream tasks.
Target encoders. Unlike raw inputs (e.g., pose joints)ooften often subtle, noisy, and isolated the target encoders provide clean, high-level targets Y E Ng xd steering the model away from overfitting to spurious details. A key innovation in OSKAR the use modality-specifee target encoders- than single cross modal target encoder balance two objectives: enabling the fusion encoder to learn cross- cross-modal modal abstractions while preserving each modality's 's structure and information content. While the fusion encoder and predictor parameters (0, i!) are updated with gradients, each target encoder's parameters o. updated via exponential moving average (EMA) of 0:
om 7m (1 Am)0.
where Am is a modality-specife momentum coefficient. OSKAR supports two target encoder update strategies: (1) Shared-weigh target encoders (i.e.. same for all m): offering unified target encoder

---

network with strong downstream multimodal performance, and (2) Customized targer encoders (i.e., mpdelity-specific Am values) offering multiple target encoders with better uni-modal performance. trained with varying update rates that accommodate each modality's ss learning dynamics. For flexible evaluation, we adopt the first option by default but we study all design choices in ablations sec 5
3.2 Training
Pretraining: Preeeit Given partially masked multimodal tokens, OSKAR fuses visible inputs with cross-modal transformer and predicts modality-specific latent features for masked tokens vía a single shared predictor. Targets are produced by momentun-updattd encoders for stable supervision. Learning occurs entirely in latent space, emphasizing high-level semantics over low-level reconstruction.
Pretraining Objective. With both YP and Y' now computed, we optimize OSKAR by minimizing a Mean Square Error (MSE) loss between the predicted and target representations:
C Σ - KEN!
1 N
(3)
Crucially, by predicting in feature space, we bypass the need for task/modallty-ssecifff losses (e.g., pixel-wise MSE for images or cross-entropy for text). This grants universal flexibility: The shared latent objective delivers comparable gradients across modalities, simplifying optimization, reducing negative transfer, and supporting graceful scaling to new modalities without the loss-balancing game.
Cross-Modal Masking Strategy. Multimodality introduces three core challenges: scalability and efficiency with increasing modalities, imbalance because of modality size disparities, and trivial cross-modal shortcuts. OSKAR addresses them all with key design choices: (1) Fixed token budget: Inspired by (6. we sample fixed total budget of N tokens, but for both inputs and targets, decoupling the compute cost from the number of input modalities. (2) Adaptive budgeting: Instead of naive random sampling- wwich would let larger modalities (e.g., video) overwhelm smaller ones (e.g., wext)-ee allocate a fraction I'm of N to each modality m by drawing I'm from a symmetric Dirichlet(a)) distribution, ensuring Σm F'm.
Lower a values (e.g.. 0.1-0.5) assign all N to single modality, while higher values (2 1) promote more balanced allocations. (3) Cross-modal exclusivity: We sample Nn, I'm N tokens per modality under cross-modal spatio-temporal exclusivity constraint, which prevents trivial prediction- if video patch is visible to the fusion encoder, the corresponding skeleton joint masked from the target encoder. To reduce redundancy in sequence modalities, we mask contiguous memtiottmmporl tubes, encouraging the model to reason over extended, coherent structures rather than isolated tokens. The fusion and target encoder token sets, X and X are sampled independently and are strictly disjoint (X' n X f), preventing direct copying and enforcing meaningful prediction of unseen information. This strategy generalizes seamlessly to new modalities while maintaining computational fairness and discouraging shortcuts.
Fine-Tuning After pretraining, the target encoders are adapted directly to downstream tasks. Because all backbones are standard transformers. fine-tuning integrates seamlessly into existing transformer pipelines.
4 Experimental Results
4.1 Pretraining Details.
Datasets. We train OSKAR entirely with pseudo-labels 10M videos from OpenHuman Vid
49
(13.2M videos, 16.7K hours). Using YOLO11 [44]. we pseudo-label pose, tracking, and detection, selecting top-3 individuals in crowded videos via visibility, motion, keypoint/bbox confidence, and
center proximity. Captions are generated with MiniCPM [87) and Cog VLM 41 Following (60
77. we speed up processing by teeeooknnninng the videos, skeletons, and text using V-JEPA [12]
MoT [92]. and WordPiece [69], respectively. Downstream benchmarks include: Kinetics- V2 [37] (RGB action recognition), NTU60 [66] and NTU
400 (45) and
(54)
(skeleton action recognition). AVA 39 (action localization), MSRVTT 88 /MSVD [6//VATEX 77
video retrieval), and MSRVIT-QA 84/MSVD-OL 82 /TGIFFFrame..
52 (videoQA).
At the submission time of this paper, only 10M videos from OpenHuman Vid were publicly released.
5

---

87
Oskar L/16)
MVD (LT0-
Accuracy on K400 (86)
55V2 24
K400
A6
VIERA (MTL//L1))
84
65
55
VideoMAE (MTL)/
81
45
78
35
OmNiMAE (VITA115)
75
0
1000
2000
3000
0
50
100
150
200
250
300
350
10
20
30
40
50
Samples Seen (M)
Number of Parameters (M)
Percentage of Labels
-VideoMAE =VideoSwin -OnniMAE -ST-MAE
MVD
=ViccaMAE +VidroMAEV2 + -Oskar
3 Fixel Prediction Feature Prediction
MVIT
-Oskar
VVJEPA
(a) Sample Efficiency.
(b) Parameter Efficiency.
(c) Label Efficiency.
Figure 3: OSKAR exhibits strong scalability with (a) fewer samples, (b) less parameters, and (c)
less labels per class than comparable methods.
Pre-training. We use standard ViT-S, ViT-B, and ViT-L (24 backbones with learnable positional
encodings, All target encoders use a shared EMA update parameter (A 0.998). Models are randomly initialized and trained on 500B tokens (10B warmup) using AdamW [58] (31 = 0.9,
32 = 0.95), a base learning rate of le-4, cosine decay, batch size 8192, and weight decay 0.05. Transformers use SwiGLU (67] activations and 6íloat16 [14] precision. Each model processes N = N = 128 tokens per step, with aggressive masking (<5%, 128 of 2640 tokens visible) via non-overlapping modality masks and Dirichlet-sampee sampled allocation ratios (a 0.5). Pretraining uses 256 GH200 GPUs. Video inputs are 16 frames (stride 2), resized to 224 x 224; skeletons are temporally aligned and normalized match video dimensions.
4.2 Main Results
By default, OSKAR is fine-tuned per task using standard inputs (e.g., video only for action recogni- tion).
Action recognition. OSKAR consistently outperforms specialized video-only models across multiple
model sizes and datasets without extra cost. On K400 (Tab. [ OSKAR outperforms V-JEPA [2]
and VideoMAE [71] with the same ViT-L backbone, and scales better with limited data (Fig. 3a
and fewer parameters (Fig. 3b). It also beats MVTT (27), BEVT (77), and TimeSformer [3] with
fewer frames/parameters. For SSv2, gains reach +3.3% over VideoMAE (ViT-S) and +2.5% over
V-JEPA (ViT-L). Beyond video, OSKAR transfers effectively to skeleton- based action recognition: on
NTU60 and NTU120 20 XSub (Tab. 4. OSKAR-B achieves new state-of-the-ar results, outperforming
MOT [92] by +0.9 % and +6.1 %, respectively.
Frozen low-shot action recognition. OSKAR's frozen features deliver strong out-of-the-box perfor- mance on SSV2 without fine-tuning, particularly in label-scarce settings (Fig. Bc). With only 5%, 10%, or 50% of the labels, it consistently outperforms other models, achieving 26.5 % absolute gain over VideoMAEv2 75) and 2.6% over V-JEPA [1] at the 5% setting. Unlike V-JEPA's visual-only pretraining, OSKAR intentionally reduces the number of visible video tokens but supplements them with motion and language tokens to ground the visual features in semantic and structural information, promoting generalization and yielding consistent gains (+2.6-3.6% %) across all low-shot settings.
Spatiotemporal action localization. OSKAR transfers effectively to spatiotemporal action local- ization, consistently outperforming larger video-only models. With just 22M parameters (ViT-S), it matches SlowFast (28) (59M) and surpasses VideoMAE-S by +5.0 mAP. Scaling up, OSKAR outper- forms VideoMAE-B by +3.5 mAP (ViT-B) and, the large scale, exceeds V-JEPA and VideoMAE-H by +1.7 and +1.4 mAP, respectively-whit
using less than half the parameters of VideoMAE-H. These gains reflect OSKAR's 'ss ability to leverage multimodal pretraining to capture both semantic context and fine rreiggrined motion cues, essential for localizing actions in space and time.
Text-video retrieval. Compared to methods trained with <200M pairs, OSKAR (ViT-L) achieves 50.4 RI on MSRVTT (+2.6 over OmniVL 73)): 54.4 RI on MSVD (+4.3 over LAVENDER (51); 54.1
on VATEX (+3.7 over CLIP4CLIP 59 ). Notably, OSKAR AR performs within close margin to special-
ist models trained with 2-3x more data (e.g., only 1.0 below Slide4Video on MSR-VTTI and excels
in vidco-to-text retrieval (+1.6 to +5.4 R @ over CLIP2TV (32 /CCentrrCLIP
90 /OLCIPCClip [59]

---

Table 1: RGB-based action recognition racy (%) on Kinetics-400
Table Action detection mAP on AVA v2.2 39 all using 16x2242 resolution.
Method
Resolution
GFLOPs
Acc.
Smull Models
(<80M paramelers))
Method
PT duta
(M)
mAP
VideoMAE-S
71
16x2244
57
79.0
Small Models (<80M parameters)
Nl
80x2244
VideoMAE-S
71
K400
22.5
MVTTv1-B
27
32x224F
170
80.2
MViTv1-B 27
K600
36.3
26.1
16x2244
MVITv2-B 53
K400
34.5
26.2
Mediunn Models (80-150M parametens)
SlowFast
28
K600
59.2
27.5
OmniMAE-B 35)
16x2244
180
80.6
OSKAR-S
OpenHuman Vid
22
27.5
TimmSformer-B 1B
96xx244
2380
80.7
Medium Modela (80-150M parameters)
ST-MAE-B 29
BEVT-B 77
32x224P
16x2224
282
180
81.3
81.1
VideoMAE-B
OSKAR-B
7
OpenHumanVidd
K400
87
30.2
26.7
VideoMAE-B
16x22242
OSKAR-B
16x2244
180
82.0
Large Models (150-700M) purumeters)
VideoMAE-L
71
K400
305
34.3
Large Models (150-700M penpoeteess)
ST-MAA-L
29
K400
304
34.8
V-JEPA-L [2
OmniMAF-I. 35
VideoMAE-L
VideoSwin-L
57
32x2244
16x224P
16x2247
16x224p
597
597
604
84.0
85.6
85.2
83.1
ST-MAE-
V-JEPA-L
VideoMAR
71
VideoMix2M
K400
K700
633
304
36.2
36.5
37.3
OSKAR-L
OpenHuran
OSKAR-L
16x2244
396
86.1
Table 4: Skeleton-based action recognition ac-
Table 3: Text-to-video retrieval Recall @1 MSRVTT [84), MSVD [16), and VATEX 779
curacy (%) on NTU60 [66] and NTU120 (54].
Param.
NTU60
NTU120
Method
Pairs (M)
MSRVTT MSVD
Method
(M)
XSub
XVlew
XSub
XSet
McBERT 92
62
93.0
97.2
84.8
86.4
Methods using large-scale data (>400M samples)
S-JEPA 0
21
93.1
97.6
90.3
91.3
Cap4Video [1]
400
51.4
MaskCLR R
2
62
93.9
97.3
87.4
89.5
S4Vid-L 86
400
51.4
54.9
67.9
PC3D 25
2
94.1
97.1
86.9
90.3
CLIP-ViP
500
54.2
OSKAR
22
93.7
97.3
89.6
89.6
IntVideo 80
646
55.2
58.4
OSKAR-B
86
93.9
97.3
90.9
92.2
Methods using 200M samples
OSKAR-L
305
94.3
97.8
91.1
92.0
TeachText
29.6
25.4
53.2
Table 5: VideoQA accuracy (% on MSRVTT-
CLIP4C11PP
100
QA 82, MSVD-QA [16, and TGIF 52
Frozicn 10
5
31.0
33.7
VIOLET
138
34.5
SUPPORT 62
100
30.1
28.4
45.9
Method
Pains (M)
MSRVTT MSVD
TGIF
LAV. 51
30
40.7
50.1
Singularity
48
17
42.7
IntVideo 30
646
47.1
55.5
72.2
UMT-B (56
5
46.3
47.4
GIT2 74
12900
45.6
58.2
74.9
DRL-B 76
47.6
47.0
44.6
VALOR- Fi
433
49.2
60.0
78.7
OmniVi. 73
17
47.8
COSA 18
415
49.2
60.0
79.5
OSKAR-S
160
45.5
47.5
49.4
OSKAR
160
46.7
56.8
73.2
OSKAR-B
160
50.1
$2.4
50.1
58.3
OSKAR-L
160
50,4
54.4
54.1
OSKAR-L
160
49.3
59.7
79.0
Its multimodal pretraining- -without external teachers or massive data outp-ooutssforms specialized retrieval models of comparable parameerreter scale.
Open-ended Video Question Answering. Without QA-specific architecture modifications, OSKAR demonstrates strong gains on VideoQA benchmarks across two configurations: (I) using OSKAR's visual encoder with BERT [22) text encoder, and (2) using OSK AR both video and text encoding. On MSRVTT-QA, OSKAR-L outperforms Intern Video 80 by +2.2 points and performs par with VALOR-L [17] (+0.1). On MSVD-QA, it is on par with COSA [88 (-0.3) while surpassing Intern Video by +4.2. On TGIF, OSKAR-L performs competitively, coming within 0.5 points of
COSA. These results highlight OSKAR's effective even with ~2-4x fewer training pairs.
representations for QA tasks,
Qualitative Results. To assess OSKAR's cross-modal predictions, we freeze the pretrained fusion encoder and predictor, and train a lightweight transformer decoder to map features to joint-space coordinates. As shown in Fig. 4 OSKAR accurately reconstructs missing human poses from video and text tokens. Notably, the first column shows that bounding box embeddings effectively guide pose prediction, even cluttered scenes with multiple people. These results highlight OSKAR's strong multimodal grounding and ability to preserve spatial structure.
7

---

lab workers young man at computer woman assisting, middle-apee equlpment
peosle, l9g8, appeerancee white couch. The young man gestures; the older writes. |setting detaik]. jstmosphere/owe
[settingl, two and woman rusti cock. One black, one white sult; she wears light blue dress. latmosphere].
reattop in (settingl, man in striped sweater inspects two tall (objects), [Background] shows lobjects).
Pseudo Labels
OSKAR Predictions
Figure 4: Visualization of predicted pose features. OSKAR accurately predicts human poses from video and text, guided by bounding boxes, even in cluttered, multi-person scenes.
5
5
4
+1.9
+1.7
+0.8
Improvement
3
+3.8
+2.6
Improvement
4
+1.0
+1.2
2
1
AGA (NTUSOXSub]
VidwoRA
(NTUSCXSub) (NTUBOXSUb)
(MEVD]
(NTUSOXSUB)
(MEVD)
(NTUSOXSUb)
INTUGOXSUb)
(MSVD)
emb.)
0.98
Random
(a) MAE and BBox embeddings.
(b) EMA update parameter.
(c) Concentration Parameter a.
Figure 5: Ablations (a) MAE and bounding box embeddings, (b) the EMA update parameter, and (c) the a value of the Dirichlet distribution. Blue denotes the default setting of OSKAR. Blue bold numbers indicate the difference between default setting and lowest bar.
5 Ablation Studies
Before large-scale pre-training, ablate design choices by pre-training ViT-S on 100K OpenHu- man Vid samples, then fine-tuning on NTU60-XSul [66] for RGB (VidCls) and skeleton (SkICIs) action recognition, and MSVD [16] for VidQA, comparing training from scratch (baseline).
Effect of adding modalities during pre-training. Table shows that combining modalities con- sistently boosts performance across tasks. Pretraining with video skeleton alone yields moderate gains (e.g., +2.6 for VidCIs, +2.9 for SkICls). Adding text with video further improves VidQA results (+2.1), underscoring its value for semantic understanding. The best performance (+3.2/+4.0/++) for V/I/s//SKKIII/VIQAA comes from using all three modalities, confirming that multimodal grounding of appearance, motion, and language produces more transferable features.
Staged Multimodal Attention Routing. Ablations on OSKAR's attention routing (Table show that the best performance (+3.2 VidCIs, +4.0 SkICIs, +4.3 VidQA) comes from using this config- uration: cross-attention the fusion encoder to align complementary signals early (e.g., motion and pose), enabling richer representations; intra-modality attention in the target encoder preserve modality-specific structure, yielding clearer supervision; and the predictor's 's hybrid self-attention within modalities, then cros-altentiin to fused contextual grounding 14001//0.3/+0.5 over full self-attention). This staged strategy, integrating early and specializing late, reflects how humans process and combine sensory inputs.
first specialization with
Prediction in the input vs feature space. Figure shows that predicting feature space outper- forms input-space reconstruction (MAE) by +1.9 VidCIs, +1.7 SkiCIs, and +0.8 VidQA. These gains echo the advantages self-distillation reported in unimodal models 12 spends capacity reproducing low-level artefacts such as blur or illumination, OSKAR concentrates on semantic cues- e.gg, motion dynamics and its momentum-updated target encoders further stabilise the supervision signal.
15 38). Whereas MAE

---

Table 7: Shared customized target encoders. Update speeds: + (slow, À = 0.99998), ++ (mod- erate, 0.9998), +++ (fast, À = 0.998).
Table 6: Impact of adding video, skeleton, and text modalities during pre-training.
Video
Skeleton
Text
VidCIs
SkICKs
VidQA
Video
Skeleton
Text
VidCIs
SKICKs
VidQA
Baseline
88.2
39.3
Baseline
88.2
75.7
39.3
+2.6
+3.2
+4.00
+4.3
+2.2
+2.1
+3.2
+3.9
+++
++
+2.6
+2.9
+3.7
+3.0
++
+3.6
+2.5
+2.8
+3.9
+4.0
+3.2
+4.0
44,3
+++
++
+5.0
+5.4
+5.2
+4.5
+4.8
+5.0
Table 8: Ablation modality attention routing. "S":: separate; "C": cross-modality.
Table 9: Impact of the number of input and target tokens.
Fusion
Target
Pred.
VidCIs
SkICKs
VidQA
Input
Target
VidCIs
SkICls
VidQA
Baseline
88.2
39.3
Baseline
88.2
39.3
+1.1
+3.4
+2.8
64
+1.0
+2.9
+2.7
+1.3
+2.9
+3.2
128
128
+3.2
+4.0
+4.3
+3.4
256
+3.3
43.3
43.6
C
+1.2
+2.2
43.3
128
256
+3.1
+4.0
+4.2
+3.2
64
+3.4
+4.3
+4.6
+3.2
+4.0
+4,3
Shared-weight vs Customized target encoders. The EMA coefficient À determines how quickly the target encoder tracks the fusion encoder. At À 0, the encoders are identical, causing representation collapse and near-random performance. Too small a (e.g., 0.1), i.e., fast updates, destabilizes training, while too large value (e.g.. 0.99998) prevents the target from adapting
updates. We evaluate two target encoder variants under this trade-off. (i) Shared-weight target encoders: A single encoder with global À performs best at À 0.998 (+2.7 VidCls, +3.8 SkICIs, +2.6 VidQA; Fig.5b). (ii) Customized target encoders: Assigning modality-specifie À values-fas for video (0.998), moderate for skeleton (0.9998), and slow for text (0.99998) yields the best overall results (+5.0 VidCIs, +5.4 SkICIs, +5.2 VidQA; Table7). This asymmetry aligns with each modality's nature: videos (low variability, high redundancy) benefit from fast updates focus motion; skeletons (moderate, structured variation) require balanced updates; and text (high variability, discrete tokens) improves with slower updates to preserve semantic consistency.
Input and target number of tokens. Table shows that more tokens generally improve performance, with the best results at 64 input 256 target tokens. A 128/128 setting retains 95% of the gains while using less than half the compute, and is thus adopted as default.
Controlling Modality Mix. We analyze the impact of the Dirichlet concentration parameter on modality token sampling (Fig. 5ch. Low a skews sampling toward single modality, while high a enforces uniformity. Setting a 0.5 yields the best trade-off (+0.2 VidCIs, +1.5 SkICIs, -0.2 VidQA vs. random). Random sampling favors video due to its token volume, benefiting video tasks, but Dirichlet sampling ensures balanced modality representation, improving SkiCIs and maintaining competitive video performance. adopt a 0.5 to promote balanced, modality-aware training.
Bounding box embeddings. Adding bounding box embeddings improves performance across tasks as illustrated in Fig. [a by providing spatial cues for person-specia predictions. Removing them causes some ambiguity in multi-person settings. These consistent gains highlight the value of simple spatial priors for learning more discriminative representations.
6 Conclusion and limitations
We introduced OSKAR, a novel paradigm for multimodal self sulf-super learning that learns se- mantically rich representations via latent feature prediction. OSKAR introduces a juse--thn--peelin pretext task, modality-speciff, momentum encoders for stable supervision, and scalable masking strategy for balanced and efficient learning. Trained across video, skeleton, and text, OSKAR outper- forms specialized models diverse downstream tasks, while remaining efficient and label- -aglostgc.s.c.

---

Its modular design supports extensions to new modalities, larger datasets, and adaptive learning dynamics, offering strong foundation for future multimodal research.
While OSKAR establishes a new state of the art, some limitations offer promising directions for further improvement: (1) Expanding Modalities: Although OSKAR eurrently integrates video, skelcton, and text, adding additional modalities (e.g., audio, depth, IMU) could further enrich the learned representations and unlock new applications. (2) Scaling Data: Pretraining on even larger and more diverse datasets would likely enhance the model's generalization and transferability, particularly for complex multimodal reasoning tasks. (3) EMA Sensitivity: The performance is sensitive the choice of the EMA momentum parameter. While this reflects the delicate balance required for stable training, also highlights an opportunity to develop adaptive or learned momentum strategies to improve robustness. These considerations represent opportunities to build upon OSKAR'S 'ss significant progress and extend capabilities even further.
Acknowledgment: This research is funded by the Swiss National Science Foundation (SNSF) through the project grant: 10003 100. Computational resources were provided part of the Swiss AI Initiative by grant from the Swiss National Supercomputing Centre (CSCS) under project ID a03 on Alps.
10

---

References
Mohamed Abdelfattah and Alexandre Alahi. S-jepa: joint embedding predictive architecture for skeletal action recognition. In European Conference on Computer Vision, pages 367-384. Springer, 2024.
[2] Mohamed Abdelfattah, Mariam Hassan, and Alexandre Alahi. Maskelr: Attention- Attention-guided contrastive learning for robust action representation learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 18678-18687, 2024.
[3] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc. Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch. Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-23753 2022.
[4] Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. In European conference on computer vision, pages 456-473. Springer, 2022.
[5] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with joint-embedding predictive architecture. arXiv preprint arXiv:2301 20t.::3300..02333, 2023.
[6] Roman Bachmann. David Mizrahi, Andrei Atanov, and Amir Zamir. Multimae: Multi-modal multi-tan taski-takk masked autoencoders. In European Conference on Computer Vision, pages 348-367 Springer, 2022.
[7] Roman Bachmann, Oguzhan Fatih Kar, David Mizrahi, Ali Garjani, Mingfei Gao, David Griffiths, Jiaming Hu, Afshin Dehghan, and Amir Zamir. 4m-21: An any-to-any vision model for tens of tasks and modalities. Advances in Neural Information Processing Systems, 37:61872-61911 1, 2024.
[8] Alexei Baevski, Wei-Ning Hsu. Qiantong Xu, Arun Babu, Jiatao Gu. and Michael Auli. Data2vec: general framework for self-supervised learning in speech, vision and language. In International conference muchine learning, pages 1212-1312. PMLR, 2022.
[9] Alexei Baevski, Arun Babu, Wei-Ning Hsu, and Michael Auli Efficient self-supervised learning with contextualized target representations for vision, speech and language. In International conference on machine learning, pages 1416-1429. PMLR, 2023.
[10] Max Bain, Arsha Nagrani, Güi Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder end-to-end retrieval. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1728-1738, 2021.
[11] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: self-supervised learning. arXiv preprint 1rXiv:2105.040000 2021.
regularization for
[12] Adrien Bardes, Quentin Garrido, Jcan Ponce, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. urXiv:2404.08471 2024.
[13] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. space-time attention all you need for video understanding" In Proceedings of the International Conference on Machine Learning (ICML), 2021.
[14] Neil Burgess, Jelena Milanovic, Nigel Stephens, Konstantinos Monachopoulos, and David Mansell. Bíloat1ó processing for neural networks. In 2019 IEEE 26th Symposiun on Computer Arithmetic (ARITH), pages 88-91. IEEE, 2019.
[15] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pages 9650-9660, 2021.
[16] David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of 49th annual meeting of the association for computational linguistics: human language technologies. pages 190-200, 2011.
[17] Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Weining Wang, Jinhui Tang, and Jing Liu. Valor: lasio-auuio-languagee omni-perception pretraining model and dataset, arXiv preprint arXiv.2304.0834 2023.
[18] Sihan Chen, Xingjian He, Handong Li, Xiaojie Jin, Jiashi Feng, and Jing Liu. Cosa: Concatenated sample pretrained vision language foundation model. arXiv preprint arXiv:2306. 230.00330,,088,, 2023.
11

---

[19] Ting Chen, Simon Korbllith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PmLR, 2020.
[20] Yanbei Chen, Yongqin Xian, Koepke, Ying Shan, and Zeynep Akata, Distilling audio-visual knowledge by compositional contrastive learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 7016-7025, 2021.
[21] loana Croitoru, Simion-Vias Bogolin, Marius Leordcanu, Hailin Jin, Andrew Zisserman, Samuel Albanic, and Yang Liu. Teachtext: Crossmodal generalized distillation for text-video retrieval. In Proceedings of the IEEF/CVF international conference on computer vision, pages 1.583-11555 11593, 2021.
[22] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pages 4171-41886 2019.
[23] Xiaoyi Dong. Jianmin Bao, Yinglin Zheng. Ting Zhang, Dongdong Chen, Hao Yang. Ming Zeng. Weiming Zhang, Lu Yuan, Dong Chen, et al. Maskclip: Masked self ditf-distillation advances contrastive language-ing -imag pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10995-1 1005, 2023.
[24] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 19999 2020.
[25] Haodong Duan, Yue Zhao, Kai Chen, Dahua Lin, and Bo Dai. Revisiting skelcton-base action recognition. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 2969-2978. 2022.
[26] David Fan, Jue Wang. Shuai Liao, Zhikang Zhang, Vimal Bhat, and Xinyu Li. Text-guided video masked autoencoder. In European Conference on Computer Vision, pages 282-298. Springer, 2024.
[27] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer, Multiscale vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pages 6824 6835, 2021.
[28] Christoph Feichtenhafer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings the IEEE/CVF international conference on computer vision, pages 6202- 6211, 2019.
[29] Christoph Feichtenhofer, Yanghao Li, Kaiming He, et al. Masked autoencoders as spatiotemporal leamers. Advances in neural information processing systems, 35:35946-35958, 2022.
[30] Karl Friston. The free-energy principle: a unified brain theory? Nature reviews neuroscience, 11(2): 127-138, 2010.
[31] Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang. Lijuan Wang, and Zicheng Liu. Violet: End- to-end video-language transformers with masked visual-token modeling. arXiv preprint ar 126881 2021.
[32] Zijian Gao, Jingyu Liu, Weiqi Sun, Sheng Chen, Dedan Chang, and Lili Zhao. Clip2tv: Align, match and distill for video-text retrieval. arXiv preprint 10ii:2111.056// 2021.
[33] Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens Van Der Maaten, Armand Joulin, and Ishan Misra. Omnivore: single model for many visual modalities. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16102-16112, 2022.
[34] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15180-15190. 2023.
[35] Rohit Girdhar, Alaseldin El-Nouby, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Omnimae: Single model masked pretraining on images and vidcos. In Proceedings of the IEEE/CVF conference on compuder vision and pattern recognition, pages 10406-10417, 2023.
[36] Yuan Gong. Andrew Rouditchenko, Alexander H Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James Glass. Contrastive audio- visual masked autoencoder. arXiv preprint arXiv:2210.07839, 2022.
12

---

[37] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal. Heuna Kim. Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pages 3842-585,, 2017.
[38] Sran-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning. Advances neural information processing systems, 33:2 33:21271-21284, 2020.
[39] Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijaynaaraiimhan George Toderici. Susanna Ricco, Rahul Sukthankar. et al. Ava: A video dataset of spatio-temporally localized atomic visual actions. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 6047-6056, 2018.
[40] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[41] Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang. Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. Cogvlm2: Visual language models for image and video understanding. arXiv preprint 240iv:2408. /6500, 2024.
[42] Yao Huang, Vasu Sharma, Hu Xu, Chaitanya Ryali, Yanghao Li, Shang- Wen Li, Gargi Ghosh, Jitendra Malik, Christoph Feichtenhofer, al. Mavil: Masked audio-video learners, Advances in Neural Information Processing Systems, 16:20371-20393 2023.
[43] Fushuo Huo, Wenchao Xu, Jingcai Guo, Haozhao Wang, and Song Guo, C2kd: Bridging the modality gap for mross-modal knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16005-16015, 2024.
[44] Glenn Jocher and Jing Qiu. Ultralyties yolo 2024.
[45] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudhoendra Vjayanarasimman, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint 1rXiv:1005.0695 2017.
[46] Sanghwan Kim, Rui Xiao, Mariana-Julian Georgescu, Stephan Alaniz, and Zeynep Akata. Cosmos: Cross-modality self-distillation for vision language pre-training. CVPR, 2025.
[47] Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62(1):1-62, 2022.
[48] Jie Lei, Tamara I. Berg, and Mohit Bansal. Revealing single frame bias for video-anddllanuagge learning. arxiv preprint arXiv:2206003422 2022.
[49] Hui Li, Mingwang Xu, Yun Zhan, Shan Mu, Jiaye Li, Kaihui Cheng, Yuxuan Chen, Tan Chen, Mao Ye, Jingdong Wang. et al. Openhumanvid: A large-scale high-quality dataset for enhancing human-centri video generation. Proceedings of the IEEE international conference on computer vision, 2025.
[50] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694 9705, 2021.
[51] Linjie Li, 7he Gan, Kevin Lin, Chung- Lin, Zicheng Liu, Ce Liu, and Lijuan Wang. Lavender: Unifying video-language understanding as masked language modeling. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 231 19-23129, 2023.
[52] Yuncheng Li, Yale Song, Liangliang Cao, Joel Tetreault, Larry Goldberg, Alejandro Jaimes, and Jiebo Luo. Tgif: A new dataset and benchmark on animated gif description. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition. pages 4641-4650, 2016.
[53] Yanghao Li, Chao Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transfomerrs for classification and detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4804 4814, 2022.
13

---

[54] Jun Liu, Amir Shahroudy. Mauricio Perez, Gang Wang, Ling-Yu Yu Duan, and Alex C Kot. Nuu rgb+ d 120: A large-scale benchmark for 3d human activity understanding. IEEE transactions on pattern analysis and machine intelligence, 42(10))2684-2700 2019.
[55] Yunze Liu, Qingnan Fan. Shanghang Zhang, Hao Dong, Thomas Funkhouser, and Li Yi. Contrastive multimodal fusion with tupleinfonce. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 754-763, 2021.
[56] Ye Liu, Siyuan Li. Yang Wu, Chang-Woer Wen Chen, Ying Shan, and Xiaohu Qie. Umt: Unified multi- modal transformers for joint video moment retrieval and highlight detection. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3042-3051, 2022.
[57] Ze Liu. Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3202-32 2202-3211, 2022.
[58] Ilya Loshchilov and Frank Hutter. arXiv:1111.05101.22 2017.
Decoupled
weight decay regularization. arXiv preprint
[59] Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval. arXiv preprint arXiv: (:xxvv:204.000 204.000000000 2021.
[60] David Mizrahi, Roman Bachmann, Oguzhan Kar. Teresa Yeo, Mingfei Gao. Afshin Dehghan, and Amir Zamir. 4m: Massively multimodal masked modeling, Advances in Neural Information Processing Systems, 36:58363-58408 2023.
[61] Muhammad Ferjad Naeem, Yongqin Xian, Xiaohua Zhai, Lukas Hoyer, Luc Van Gool, and Federico Tombari. Sile: Improving vision language pretraining with -isfidiitillattin. In Computer Vision ECCV 2024, pages 38-55, Cham, 2025. Springer Nature Switzerland.
[62] Mandela Patrick, Po-Yao Huang, Yuki Asano, Florian Metze, Alexander Hauptmann, Joao Henriques, and Andrea Vedaldi. Support-set bottlenecks for video-text representation learning. arXiv preprint arXiv:2010.02824 2020.
[63] Gorjan Radevski, Dusan Grujicic, Matthew Blaschko, Marie-Francine Moens, and Tinne Tuytelaars. Multimodal distillation for egocentric action recognition, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5213-5224, 2023.
[64] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 2021.
PmLR,
[65] Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: a functional interpretation of some extra extra-classica classical receptive-ficde field effects. Nature neuroscience, 2(1):79-87, 87, 1999.
[66] Amir Shahroudy, Jun Liu, Ttan-Tsong Ng, and Gang Wang. Ntu rgb+ large scale dataset for human activity analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1010-1019, 2016.
[67] Noam Shazeer. Glu variants improve transfommer. arXiv preprint arXiv:2002.05202 2020.
[68] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15638 15650, 2022.
[69] Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. Fast wordpiece tokenization, 2021.
[70] Zineng Tang, Jaemin Cho, Hao Tan, and Mohit Bansal, Vidlankd: Improving language understanding via video-distileed knowledge transfer. Advances in Neural Information Processing Systems, 04224468-24488 2021.
[71] Zhan Tong. Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self- video pre-training. In Advances in Neural Information Processing Systems. pages 10078-10093. Curran Associates, Inc., 2022.
[72] Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems. 30, 2017.
14

---

[73] Junke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo, Luowei Zhou, Yucheng Zhao, Yujia Xie, Ce Liu, Yu- Gang Jiang, and Lu Yuan. Omnivl: One foundation model for Image-language and video-language tasks, Advances neural information processing systems, 35:5696-5710 2022.
[74] Jianfeng Wang. Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image- togee-to-tett transformer for vision and language. arXiv preprint arXiv:2205141100 2022.
[75] Limin Wang. Bingkun Huang, Zhiyu Zhao, Zhan Tong. Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomac v2: Scaling video masked autoencoders with dual masking. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14549-14560, 2023.
[76] Qiang Wang, Yanhao Zhang, Yun Zheng, Pan Pan, and Xian-Sheng Hua. Disentangled representation learning for text-video retrieval. arXiv preprint arXiv:2203.07111 2022.
[77] Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengehen Liu, Yu-Gang Jiang, Luowei Zhou, and Lu Yuan. Bevt: Bert pretraining of video transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14733-14743 2022.
[78] Wenhui Wang. Hangbo Bao. Li Dong. Johan Bjorck. Zhiliang Peng. Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, al. Image foreign language: Beit pretraining for vision and vision- laiiunle tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19175-19186, 2023.
[79] Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In Proceedings of the IEFE/CVF international conference on computer vision, pages 4581-4591, 2019.
[80] Yi Wang. Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191, 2022.
[81] Wenhao Wu, Haipeng Luo, Bo Fang. Jingdong Wang, and Wanli Ouyang. Cap4video: What can auxiliary captions do for text-video retrieval? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10704-100 107 13, 2023.
[82] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Multimedia, pages 1645-1653, 2017.
[83] Hu Xu, Gargi Ghosh, Po- Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2 100.444499.4484. 2021.
[84] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288-5296, 2016.
[85] Hongwei Xue. Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jiebo Luo. Clip- vip: Adapting pre-trained image-text model to languaee representation alignment. arXiv preprint arXiv:2209.06.0 06430, 2022.
[86] Huanjin Yao, Wenhao Wa, and Zhiheng Li. Side4video; Spatial-temporra side network for memory-efficient iidgeo transfer learning. arXiv preprint arXiv:2311.15766 2023.
[87] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint 2ixiv:2408.00 2024.
[88] Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang. Michael Maire, Ajinkya Kale, and Baldo Faieta, Multimodal contrastive training for visual representation learning. In P Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7095-700,220 2021.
[89] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International conference on machine learning, pages 12310-12320 12320. PMLR, 2021.
15

---

[90] Shuai Zhao, Linchao Zhu, Xiaohan Wang, and Yi Yang. Centerclip: Token clustering for efficient text-video retrieval. In SIGIR '22: The 45th International ACM SIGIR Conference Rexearch and Development in Information Retrieval, July 11-15, 2022, Madrid, Spain, 2022.
[91] Jinghao Zhou, Chen Wei, Huiyu Wang. Wei Shen, Cihang Xie. Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.0788 2021.
[92] Wentao Zhu, Xiaoxuan Ma, Zhaoyang Liu, Libin Liu, Wayne Wu, and Yizhou Wang. Motionbert: A unified perspective on learning human motion representations. In Proceedings of IEEE/CVF International Conference on Computer Vision, 2023.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction (Sec. D clearly state OSKAR's contributions. The impacts of these contributions are carefully studied in the experiments (Sec 4 and ablations (Sec. sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Sec we provide limitations of our work. We also include a broader impact statement in the Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
17

---

Answer:
[NA]
Justification:
The paper does not include theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Detailed pretraining and evaluation settings are provided in Sec 4 and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
18

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The datasets used in our experiments are all publicly available. An anonymized version of our code is available as part of the supplementary materials. Codes will be open- sourced to the community upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
mublic/guider SoodeSubmissioooolliee
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.cc/ppppppe wwtp::ccpppppiiicce//c/s////c/// Codeeubmissiionn ieabbbmmssioooolol.mcoo
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Detailed experimental settings are available in section and the Appendix. We also ablate on all introduced hyper- aapeer-arameters, detailed section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We fix random seed for all statistical values in experiments, and follow the best practices from domain knowledge and previous works in our experiments. are unable to run each experiment multiple
Due to the high cost of rumning experiments, times. Hence, no error bars are included in our plots.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
The information on the required compute cost is availabe in section as well as the supplementary materials.
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https:
Answer: [Yes]
Justification:
The paper abides by the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the societal impacts of our paper in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
20

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We described the potential impacts of our paper in the Appendix. We will follow the standard safeguards before releasing any models data.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We carefully cited all the previous models and datasets we used in our work, which are all accessible for research purposes.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
21

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
All our assets are well-documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not invlove any human subjects or crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our work does invlove any human subjects or crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
22

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurips: hect::/:/eerrrps.ccc//ooeeereere Cc//Confereeeees/222///.... for what should or should not be described.
23


Reviews:
[{"review_id": "tknWezpGiC", "reviewer": "Reviewer_376k", "summary": "OSKAR is a model that takes components from other architectures and incorporates them into a unified process. These components are shared embeddings, cross-model transformers, and self-distillation.  It takes inputs from text, video, and skeleton data, fuses them together, and then fills in (predicts) the masked or missing parts in each modality. This prediction is done in a higher level, latent space, rather than at a pixel or coordinate or word level. The modality-specific predictions are checked against the feature vector calculated by the encoder for the masked/missing parts, and the encoder weights are updated.\nThe paper reasonably claims that theis method is highly scalable because of a fixed token budget that is allocated across all modalities in a way determined by the modeler. The authors present experiments on action recognition, video question answering, and retrieval. THey further present ablation tests which show that OSKAR’s architecture components all contribute to the model’s performance.", "questions": "* The authors acknowledge that a limitation of the paper is that only 3 modalities were tested. How would performance change because of the fixed budget. Or is there some dynamic re-allocation of the budget across modalities as some modalities appear to be more important?\n* Can you provide cases where OSKAR did not perform as expected, or any limitations you encountered while experimenting?\n* The authors gave a reason why there are no error plots. But the results look suspiciously non-significant. It might be worth the computational cost to verify OSKAR's dominance. Or, make a statement saying even if OSKAR's gain are minor, there are other, architectural reasons for using it.", "limitations": "yes", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n* Integrates latent feature prediction, momentum-updated modality-specific supervision, and efficient cross-modal fusion.\n* Strong performance across video action recognition and question answering, and retrieval.\n* Ablation studies that are extensive which validate the architectural choices. In particular, the fusion of modalities and latent space prediction are important.\n* Practical and scalable design made possible by fixed token budget and momentum-based supervision. The fixed budget allocated across modalities makes computation manageable, and labeled data is not needed, \n\n\nWeaknesses\n* Standard errors are not reported, but performance over other, closely performing models do not seem to be statistically or significantly significant. Most of them range from 0.3 to 1.0 percentage points. I tend to look at differences under 1 or 2 points as likely not statistically significant in the absence of variance measures.\n* Ablation studies seem thorough, but they do not take into account interactions. They are also highly dependent on the order in which features were added or removed.\n* Fine-tuning of momentum may complicate generalizing to other new contexts.", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 376k for their insightful comments. We are glad that they recognize the **“strong performance”, “extensive ablation studies”, “practical and scalable design”, “manageable”** compute, and the **flexibility to train on unlabelled data** in our method. In the following, we address all concerns and questions:\n \n### **1. Statistical Significance of Performance Gains**\n>*Standard errors are not reported, but performance over other, closely performing models do not seem to be statistically or significantly significant. Most of them range from 0.3 to 1.0 percentage points. I tend to look at differences under 1 or 2 points as likely not statistically significant in the absence of variance measures. + The authors gave a reason why there are no error plots. But the results look suspiciously non-significant. It might be worth the computational cost to verify OSKAR's dominance. Or, make a statement saying even if OSKAR's gain are minor, there are other, architectural reasons for using it.*\n \nTo address the reviewer’s concerns regarding statistical significance, we conducted additional ablation experiments (with the setting described in L239–241, Sec. 5). Specifically, we pretrained with 5 different random seeds and evaluated each model 5 times with varied evaluation seeds across 3 downstream tasks (total: 5 pretrains × 15 evals = **75 runs**). We report the **mean ± standard error** of absolute percentage point improvements over the baseline.\n \nRuns|VidCls|SklCls|VidQA|\n-|------|------|-----\nBaseline|88.2 ± 0.04|75.7 ± 0.06|39.3 ± 0.05\nRand. Seed 1|+3.17 ± 0.05|+4.26 ± 0.03|+4.25 ± 0.08\nRand. Seed 2|+3.22 ± 0.07|+3.97 ± 0.08|+4.26 ± 0.08\nRand. Seed 3|+3.19 ± 0.04|+4.03 ± 0.05|+4.37 ± 0.05\nRand. Seed 4|+3.14 ± 0.06|+4.08 ± 0.09|+4.35 ± 0.07\nRand. Seed 5|+3.18 ± 0.03|+4.07 ± 0.08|+4.19 ± 0.05\nOSKAR mean over runs|+3.18 ± 0.05|+4.08 ± 0.07|+4.28 ± 0.07\nSignal-to-Noise Ratio (Gain/SE)|**~63x**|**~58x**|**~61x**\n \n \n**OSKAR’s improvements consistently exceed baseline variability.**  For example, the VidCls gain (+3.18) surpasses the baseline’s standard error (0.04) by almost **80×**, far beyond the 95% confidence interval—indicating statistically robust gains across all 75 evaluations.\n \nWhile we could not re-run similar analysis on the large scale (10M+ samples) due to the extremely high compute cost and short rebuttal window, we would like to highlight the following points:\n \n1. **Consistent and significant gains:** OSKAR either exceeds or matches prior methods across 5+ downstream tasks (Tables 1–5) with **consistency**, often **exceeding the 2-point** threshold noted by the reviewer.  For example,\n   - Improves over VideoMAE-B by **+3.5** on AVA Action Detection (Table 2)  \n   - Surpasses OmniVL (**+2.6**, MSRVTT) and Lavender (**+4.3**, MSVD) in T2V retrieval (Table 3)  \n  \n2. **Efficiency and scalability:** OSKAR is data-, parameter-, and label-efficient. For example, under limited supervision (5% of labels), it outperforms VideoMAEv2 by **+26.5%**  and V-JEPA by **+2.6%** (see Figure 3).\n \n3. **Multimodality and generalization:** OSKAR provides unified, task-agnostic pretraining and **generalizes** across diverse tasks; for example, unlike the methods in comparison, it achieves SOTA in skeleton action recognition.\n \nIn summary, OSKAR's contribution extends beyond performance improvements, its real power lies in its consistency, multimodality, generalisation, efficiency, and scalability.\n \n### **2. Interaction Effects in Ablation Studies**\n>*Ablation studies seem thorough, but they do not take into account interactions. They are also highly dependent on the order in which features were added or removed.*\n \nWe agree that analyzing component interactions is critical.  To address this, we conducted a **factorial-style ablation**, jointly varying key components:  EMA update parameter (λ), attention routing (fusion, target, predictor), and input/target token counts.\n \nRow|EMA λ|Fusion|Target|Pred|Input|Target|Avg. Gain\n--:|:---:|:-----:|:----:|:--:|:---:|:----:|:--------:\n1|+++|C|C|S|128|128|+2.2\n2|+++|C|C|S|64|256|+2.5\n**3**|**+++**|**C**|**S**|**S**|**128**|**128**|**+3.8**\n**4**|**+++**|**C**|**S**|**S**|**64**|**256**|**+4.1**\n5|++|C|C|S|128|128|+1.0\n6|++|C|C|S|64|256|+1.1\n7|++|C|S|S|128|128|+2.9\n8|++|C|S|S|64|256|+2.5\n \n- +++ and ++ denote λ = 0.998 and λ = 0.9998, respectively. \n- C and S denote cross-modal all-to-all token interactions and separate intra-modality token interactions in the attention routing, respectively. \n- Metric: average gain over baseline across three tasks as described in L239–241.\n \nThe best settings (Rows 3–4) combine fast EMA, separate target attention, and asymmetric tokens—showing our design choices are **complementary**, not just additive.  We adopt Row 3 by default for the best performance/efficiency tradeoff.  This confirms that gains hold across interaction axes, addressing concerns about **order dependence**.\n \n### **3. Momentum Tuning and Generalization**\n> *Fine-tuning of momentum may complicate generalizing to other new contexts.*\n \nWe understand the reviewer’s concern that EMA parameter λ values can impact generalization. Please note that careful tuning of customized EMA update parameters per modality is **NOT required** in OSKAR. In fact, we adopt a **shared λ** by default, which already achieves competitive performance across all downstream tasks (Tables 1–5).\n \nWhen adding a new modality or context, momentum fine-tuning—like any other hyper-parameter—remains an **optional** enhancement. Our default setting already performs well across diverse contexts. As noted in the conclusion (Lines 300–303), we view momentum tuning as a promising direction for future work and plan to investigate adaptive or learned momentum strategies to further boost generalization and robustness without manual tuning.\n\n \n### **4. Token Budget and Modality Scaling**\n> *The authors acknowledge that a limitation of the paper is that only 3 modalities were tested. How would performance change because of the fixed budget. Or is there some dynamic re-allocation of the budget across modalities as some modalities appear to be more important?*\n \nWe break down your question into two parts and run targeted ablations to answer each separately.\n \n*First, does the fixed token budget hurt performance as you add more modalities?*\n \nRow|Token Budget|Video|Skeleton|Text|VidCls|SklCls|VidQA|Avg. Gain\n-|-|--|--|--|---|---|---|--\n1|64|✓|||+1.3|—|—|+1.3\n2|64|✓|✓||+1.3|+3.0|—|+2.2\n3|64|✓|✓|✓|+1.0|+2.9|+2.7|+2.2\n4|128|✓|||+2.6|—|—|+2.6\n5|128|✓|✓||+2.5|+4.1|—|+3.3\n**6**|**128**|**✓**|**✓**|**✓**|**+3.2**|**+4.0**|**+4.3**|**+3.8**\n7|256|✓|✓|✓|+3.3|+3.3|+3.6|+3.4\n8|512|✓|✓|✓|+2.9|+3.0|+3.1|+3.0\n\nOur additional ablation studies in the table above show that:\n\n- Adding modalities under small fixed budgets (e.g., 64 tokens) does not always improve performance (Rows 1–3), likely due to token competition. For example, adding text in Row 3 (vs. Row 2) does not increase the average score, yet it **expands applicability**—enabling VideoQA support (+2.7, row 3).\n\n- As the budget increases (Rows 4–6), adding all three modalities yields noticeable gains (+2.6 → +3.8, rows 4 → 6), peaking at 128 tokens. Beyond that (Rows 7–8), improvements plateau, suggesting diminishing returns. This suggests 128 tokens is a **sweet spot**—large enough to represent all modalities well, yet small enough to remain efficient and computationally practical.\n\nThese findings suggest that **modality inclusion is helpful when sufficient capacity is available.**\n \n*Second, dynamic re-allocation of the budget:*\n\nWe conducted further ablation experiments to compare three token allocation strategies: **Balanced**, **Weighted**, and **Dirichlet** sampling. Balanced allocation gives equal tokens per modality, offering moderate gains (+2.3). Weighted allocation favors larger modalities like video, improving VidCls (+3.4) but hurting text-rich tasks like VidQA, lowering the average gain (+1.9). Dirichlet sampling (adopted by default in OSKAR) stochastically varies token ratios per batch (while ensuring token balance over the entire training), enhancing diversity and generalization, with the best overall performance (+3.8). Overall, dynamic and modality-aware strategies like Dirichlet sampling are most effective, especially in imbalanced or noisy multimodal settings.\n\n\n### **5. Limitations and Failure Cases**\n>*Can you provide cases where OSKAR did not perform as expected, or any limitations you encountered while experimenting?*\n \nOSKAR excels at understanding tasks, but its design inherently limits generation.  We explored video generation from OSKAR’s latent space, but results lacked the visual fidelity and temporal coherence of state-of-the-art diffusion models. This is expected, as OSKAR’s objective prioritizes semantic abstraction over pixel detail, enabling strong transfer to understanding tasks. Nonetheless, its powerful latent representations are promising for future work with generative decoders.\n \n***We sincerely thank Reviewer 376k for their thoughtful and constructive feedback. We hope that we sufficiently addressed their major concerns. We are committed to incorporating their feedback—particularly on statistical reporting, component interactions, and generalization—in the camera-ready version.***"}, {"review_id": "ASqXBDYC0S", "reviewer": "Reviewer_1kEh", "summary": "OSKAR introduces a scalable, self-supervised multimodal learning framework using a “fuse-then-predict” strategy in latent space. It avoids low-level reconstruction and contrastive negative pairs, leveraging modality-specific momentum encoders to predict missing features from fused video, skeleton, and text tokens efficiently. OSKAR achieves state-of-the-art results in action recognition, localization, video-text retrieval, and video QA while using fewer samples, parameters, and labels. It demonstrates effective multimodal grounding, high transferability, and scalability.", "questions": "### Question:\n\n1. In the Video QA section, only a BART-based language module was used. I am curious whether OSKAR can be effectively adapted to existing LLMs, which would further enhance the significance and effectiveness of this method.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths:\n\n1. The paper is written clearly and concisely, providing rigorous and detailed experimental results and analyses, which greatly enhance the persuasiveness of the work.\n2. OSKAR demonstrates flexible and extensible characteristics and shows promising generalization performance across various tasks, proving the effectiveness of the proposed method.\n3. The paper focuses on specific technical details, providing detailed descriptions and experimental results on the sampling strategies, customized update rates, and other implementation details.\n\n### Weaknesses:\nN/A", "was_revised": true, "final_justification": "The author provided an explanation for my question, but did not provide any new results.\nIn addition, the author responded to questions about SOTA comparison by providing a new version of scale up results in the Rebuttal of Reviewer #XjPx\nIn summary, I maintain my original rating", "rebuttal": "We thank reviewer 1kEh for their valuable feedback. We are glad that they found our paper to be **“written clearly and concisely”**, our experimental results **“rigorous and detailed”**, our technical details to be **”specific”** and **“detailed”**, and our method **“flexible and extensible”**, **“shows promising generalization performance”**, and **“demonstrates effective multimodal grounding, high transferability, and scalability.”** In the following, we discuss their raised question on LLMs: \n\n>*In the Video QA section, only a BART-based language module was used. I am curious whether OSKAR can be effectively adapted to existing LLMs, which would further enhance the significance and effectiveness of this method.*\n\nWe thank the reviewer for highlighting the potential of adapting OSKAR to large language models (LLMs). Although our current experiments use a lightweight BART decoder for VideoQA, this choice was made to isolate and evaluate OSKAR’s fused representations. In fact, OSKAR is inherently compatible with LLMs and designed to serve as a **modality-agnostic** fusion encoder for multimodal reasoning.\n\nRecent works such as BLIP-2 [1], Flamingo [2], and MiniGPT-4 [3] have shown the effectiveness of **freeze-then-adapt** approaches for LLM integration. OSKAR naturally aligns with this paradigm: Its **fuse-then-predict** architecture already learns to integrate video, skeleton, and text into a **shared semantic space**. The encoder is **decoupled from task-specific heads**, enabling its outputs to be interfaced with LLMs via adapters or direct token-level alignment.\n\nImportantly, OSKAR provides not just compatibility but added value over prior visual backbones typically used in LLM-based systems. Most existing approaches rely on frozen image-text models (e.g., BLIP-2[1] and CLIP [4]) that struggle with rich temporal dynamics or motion reasoning. In contrast, OSKAR’s pretraining explicitly fuses **temporally-aware, multimodal inputs**—including motion and language—into a coherent representation space. This makes it particularly well-suited for tasks that require temporal abstraction, action understanding, and long-range multimodal context, which are difficult to capture with static image features alone.\nThis modularity means OSKAR can act as a **drop-in fusion backbone** for future LLM-based systems—offering general, scalable multimodal representations with minimal changes.\n\nWhile LLM integration is not explored in this version due to resource constraints, we are actively pursuing this direction. We appreciate the reviewer’s insight and will add a discussion to emphasize OSKAR’s role as both a standalone multimodal learner and a valuable complement to LLM-based architectures.\n\n***We thank reviewer 1kEh for their time reading our response.***\n\n[1] Li, J., et. al (2023, July). Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML. PMLR.\n\n[2] Alayrac, J. et al. (2022). Flamingo: a visual language model for few-shot learning. NeurIPS.\n\n[3] Zhu, D., et al. (2024). Minigpt-4: Enhancing vision-language understanding with advanced large language models. ICLR\n\n[4] Radford, A. et al. (2021, July). Learning transferable visual models from natural language supervision. In ICML. PmLR."}, {"review_id": "yiIqdB2sWe", "reviewer": "Reviewer_XjPx", "summary": "This paper proposes an omni-modal self-supervised pretraining framework, named OSKAR, that simultaneously extracts representations from video, action, and text modalities, achieving competitive performance on downstream tasks such as action recognition, action localization, text-video retrieval, and video question answering (VideoQA). \nOSKAR leverages cross-modal masked prediction in latent space as its pretraining objective, consists of two branches: (1). a fusion encoder that processes visible tokens to generate fused representations; (2). modality-specific target encoders that produce target embeddings. These target encoders are updated via the exponential moving average (EMA) of the fusion encoder.", "questions": "**Q1**: For the fusion encoder, are there special tokens used as identifiers for tokens of different modalities when they are input? \\\n**Q2**: The impact of mask rate on model training and performance. The author did not discuss the mask rate in the paper. Will using too high a mask rate cause the model to fail to converge or reduce performance? \\\n**Q3**: In the self-supervised training paradigm of mask reconstruction, asymmetric encoders and decoders[1] are usually used, such as MAR. In my understanding, the modality predictor plays a role similar to the decoder in the framework of this article. The author kept the structure of the predictor consistent with the encoder in the experiment. Why is this? Please experiment or discuss this. \\\n**Q4**: Reference Weaknesses 2. Please analyze and discuss the current sota model. \\\n\n**Ref**: \\\n[1]. Masked Autoencoders Are Scalable Vision Learners.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n1. Nice writing and presentation, the reviewer can clearly understand the framework of the method.\n2. Modality-specific target encoders can effectively balance intra-modality feature modeling and inter-modality feature alignment.\n3. Multimodal representation learning is one of the current popular topics. The method proposed in this paper can effectively integrate the features of action videos and texts.\n\n**Weaknesses:**\n1. Lack of validation on large-scale data. I think effectiveness on large-scale data is the key to current representation learning, enabling it to be better applied to VLM or VLA tasks. The paper shows competitive performance on small-scale data, but there is still a gap compared to other methods that use more data.\n2. The baseline methods used for comparison are somewhat outdated and do not fully reflect the current state-of-the-art. For example, in an action recognition task, there is no Comparison of current SOTA models(InternVideo[1], Unmasked Teacher[2]，Hiera[3]). These situations also exist for other tasks:Text-to-video retrieval(VATEX[4]), VideoQA(VAST[5]).\n\n**Ref:**\\\n[1]. InternVideo: Video Foundation Models for Multimodal Understanding. ECCV24 \\\n[2]. Unmasked Teacher: Towards Training-Efficient Video Foundation Models. ICCV23\\\n[3]: Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles.ICLR25\\\n[4]. Gramian Multimodal Representation Learning and Alignment. ICLR25\\\n[5]: VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset.NeurIPS23\\", "was_revised": true, "final_justification": "Two of my most concerning issues: comparison with SOTA models and trends under data model scale. For question 1, the author provides an extensive comparison in the rebuttal; for question 2, the author shows that two models with different sizes gradually improve the performance of downstream tasks as the training data increases. Based on this, I increase the score to 4.", "rebuttal": "We thank reviewer XjPx for their insightful comments and constructive feedback. We are glad that they recognize the **“nice writing and presentation”**,  and our method being able to **“effectively integrate the features of action videos and texts”**, and achieve **“competitive performance on downstream tasks”**. We are also encouraged that they are able to **“clearly understand the framework”**. In the following, we address their concerns: \n \n### **1. Validation on Large-Scale Data**\n>*Lack of validation on large-scale data. I think effectiveness on large-scale data is the key to current representation learning, enabling it to be better applied to VLM or VLA tasks. The paper shows competitive performance on small-scale data, but there is still a gap compared to other methods that use more data.*\n \nWe acknowledge the reviewer’s concern regarding evaluation at scale. To address this, we pretrained OSKAR-G (1.3B) ahead of the rebuttal, and we evaluated it across widely accepted benchmarks.\n \nModel|Publication|#PT. samples|Parameters|K400|SSv2\n-----|-----------|------------|-----------|-----|-----\nFocusVideo [8]|CVPR'25|–|450M|87.2|70.7\nHiera  [3]|ICLR'23|–|213M|87.3|75.1\nMVD  [6]|CVPR'23|1.7M|633M|87.3|77.3\nVideoMAEv2  [11]|CVPR'23|1.4M|1B|88.5|77.0\nUniFormerV2  [12]|ICCV'23|401M|354M|88.8|72.1\nUMT  [2]|ICCV'23|47M|304M|90.6|74.7\nInternVideo  [7] |arXiv'22|113M|1.3B|91.2|77.2\nInternVideov2  [1] |ECCV'24|404M|1B|**91.6**|77.1\n**OSKAR-G**|–|**10M**|**1.3B**|90.8|**77.9**\n\n* #PT. samples denotes the number of pretraining samples\n  \nModel|Publication|#Pairs|Parameters|T2V Retrieval (VATEX)|VideoQA (MSRVTT-QA)\n-----|-----------|------|-----------|----------------------|--------------------\nUMT [2]|ICCV'23|425M|304M|–|47.9\nCOSA [9] |ICLR'24|415M|1.2B|–|49.2\nTeachCLIP [10] |CVPR'24|–|–|63.6|–\nNarvid [13] |CVPR'25|400M|~250M|68.4|–\nInternVideo [7] |arXiv'22|112.8M|1.3B|71.1|47.1\nInternVideo2s2-6B [1] |ECCV'24|404M|6B|75.5|–\nVAST [5] |NeurIPS'23|442M|1.3B|83|50.1\nGRAM [4] |ICLR'25|442M|1B|**84.4**|–\n**OSKAR-G**|–|**160M**|**1.3B**|79.6|**50.4**\n \nOSKAR-G further bridges the gap with methods using more data:  \n- On action recognition: OSKAR-G achieves **77.9%** on SSv2 (new SOTA) and scores within **0.8%** of InternVideo-v2 on K400, despite pretraining on **40x** less data.  \n- On vision-language tasks: OSKAR-G surpasses VAST on MSRVTT-QA, and comes within **5** points of GRAM on VATEX T2V retrieval, despite pretraining on only **36%** of GRAM’s data size.\n \nWhile we could not train on even larger datasets due to the limited compute budget and the small rebuttal window, we believe that OSKAR is already trained at a **decent scale** and is **competitive** with methods using more data on popular evaluation datasets.  \n \nFurther, we highlight the following aspects of OSKAR:\n \n- **Consistency**: OSKAR not only matches the performance of prior methods with similar model and data scales (Tables 1–5), but also competes closely with much larger *expert* models across 5+ downstream tasks—**with consistency.** This is achieved despite OSKAR being pretrained for general-purpose use, without any task-specific customisations or distillation from any external models.\n \n- **Scalability and efficiency**: OSKAR achieves these results while being a data-, parameter-, and label-efficient learner. For example, with only 5% of the labels, it surpasses VideoMAEv2  by **26.5%** and  V-JEPA  by **2.6%** (see main paper Figure 3).  \n \n- **Multimodality and generalization**: The major contribution of OSKAR lies in its **unified** pretraining, **generic** use, and strong performance. Unlike the methods in comparison, it has the extra benefit of working seamlessly across other tasks (ex, SOTA performance in skeleton action recognition).\n \nIn summary, in addition to performance improvements, OSKAR excels in consistency, multimodality, generalisation, efficiency, and scalability. We believe that these features make OSKAR a practical and valuable foundation for the community.\n\n### **2. SOTA Comparisons**\n>*The baseline methods used for comparison are somewhat outdated and do not fully reflect the current state-of-the-art. For example, in an action recognition task, there is no Comparison of current SOTA models(InternVideo[1], Unmasked Teacher[2]，Hiera[3]). These situations also exist for other tasks:Text-to-video retrieval(GRAM, VATEX[4]), VideoQA(VAST[5]).*\n \nWe appreciate the reviewers concern on SOTA comparison. To directly address this concern, we have now included all the suggested baselines—InternVideo, Unmasked Teacher, Hiera, GRAM, and VAST—in our comparison tables above. We also added other recent models for completeness. Across tasks, OSKAR-G is on par with these methods despite using up to 40x less data and no external distillation. \n \nWe would like to clarify that, in our main paper, we limited comparisons to methods of **similar model scale**, input resolution, and data budget, to ensure a **fair and meaningful evaluation.** We excluded methods that:  \n- Are initialized or distilled from external models, e.g., InternVideo, TeachCLIP;  \n- Employ higher input resolutions.\n- Or involve larger model or dataset scales, e.g., GRAM, VAST, InternVideo-v2.\n \nPlease note that this way of ensuring fairness by limiting comparisons is a common practice (ex, in [8]) . Additionally, in our **appendix** (Tables 12-18), we already compare to some of these recent methods mentioned by the reviewer (UMT, InternVideo) with clear notes on scale, resolution, and supervision differences to maintain transparency.  \n \nThat said, the reviewer’s concern is very well acknowledged; we are committed to updating our camera-ready version to resolve it.\n \n### **3. Use of Modality-Specific Tokens in the Fusion Encoder**\n\n>*Q1: For the fusion encoder, are there special tokens used as identifiers for tokens of different modalities when they are input?* \n \nModality identity is encoded using **learnable modality-specific embeddings**. OSKAR learns per-modality embeddings that act like soft identifiers, allowing the model to distinguish and integrate different modalities during attention. Please refer to L106-115 (main paper, methodology sec. 3) for more details.\n \n\n### **4. Impact of Mask Rate on Training and Convergence**\n>*Q2: The impact of mask rate on model training and performance. The author did not discuss the mask rate in the paper. Will using too high a mask rate cause the model to fail to converge or reduce performance?*\n \nThe mask rate is implicitly determined by the **fixed token budget**, which controls how many tokens are visible during training (ablated in Table 9). For instance, 128 visible tokens out of 2,640 (1568 video + 816 skeleton + 256 text) yields ~95% masking, which performs well across tasks. \n \nAs the reviewer correctly expected, pushing to 99% (32 visible tokens) leads to degraded performance. This aligns with prior work (e.g., VideoMAEv2 [11]), which caps masking at ~95% to combat temporal redundancy and improve efficiency without hurting performance.\n \nUnlike these methods, OSKAR’s input and target tokens are multimodal, enabling cross-modal interaction and improving generalisation across diverse tasks. We study these benefits in Table 6 (ablation section 5).\n \n### **5. Symmetric vs. Asymmetric Predictor Design**\n>*Q3: In the self-supervised training paradigm of mask reconstruction, asymmetric encoders and decoders[1] are usually used, such as MAR. In my understanding, the modality predictor plays a role similar to the decoder in the framework of this article. The author kept the structure of the predictor consistent with the encoder in the experiment. Why is this? Please experiment or discuss this.*\n \nThe reviewer correctly understands the similarities between the decoder in MAE and predictor in OSKAR. We include further ablations below (with encoder depth=8, width=512) to clarify the effect of an asymmetric predictor. We observe that a deeper predictor noticeably improves performance (+3.0 → +3.8), while increasing width has marginal effect.  \n\nPredictor depth|Predictor width|Avg gain\n---------------|----------------|---------\n4|128|+3.1\n4|256|+3.2\n4|512|+3.0\n8|128|+3.6\n8|256|+3.9\n8|512|+3.8\n \n* Metric: average gain over baseline across three tasks as described in L239-241, Sec. 5.\n \nGiven that the predictor output dimension must match the target encoder's latent dimension, we adopt a symmetric predictor to avoid redundant projection layers and maintain architectural simplicity. Importantly, unlike MAE, our predictor performs semantic latent prediction across modalities, requiring sufficient capacity for generalization.\n \n***We sincerely thank reviewer XjPx for their time reading this rebuttal. We hope we sufficiently addressed their major concerns. We are committed to incorporate their feedback on evaluation at scale, SOTA comparisons, and predictor design in the camera-ready version of our paper.***\n \n[6] Wang, R. et al. (2023). Masked video distillation: Rethinking masked feature modeling for self-supervised video representation learning. In CVPR.\n\n[7] Wang, Y. et al. (2022). Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191.\n\n[8] Wang, M. et al. (2025). Action Detail Matters: Refining Video Recognition with Local Action Queries. In CVPR.\n\n[9] Chen, S. et al. (2024). Cosa: Concatenated sample pretrained vision-language foundation model. ICLR.\n\n[10] Tian, K. et al. (2024). Holistic features are almost sufficient for text-to-video retrieval. In CVPR.\n\n[11] Wang, L. et al. (2023). Videomae v2: Scaling video masked autoencoders with dual masking. In CVPR.\n\n[12] Kunchang Li et al. (2023). UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer. In ICCV.\n\n[13] Hur, C., et al (2025). Narrating the Video: Boosting Text-Video Retrieval via Comprehensive Utilization of Frame-Level Captions. In CVPR."}, {"review_id": "qQMXSXhhz3", "reviewer": "Reviewer_5fwS", "summary": "This work is built around a novel multimodal pretraining approach based on the prediction of masked high-level representations: the multimodal tokens are projected by a separate encoder and then fused into a common representation. A predictor uses the latter to reconstruct masked tokens to be compared to target tokens projected by the corresponding target encoders. These latter are updated with moving average momentum to stabilize the training chain. \nThe proposed approach's benefits are experimented with in the context of video, skeleton, and text multimodal settings. Comprehensive experimentation and ablation studies conclude the work. Supplementary material, including the software, is useful.", "questions": "- The encoded embeddings $\\mathbf{E}$ are augmented with a learnable modality-specific auxiliary $e_{(aux)}$ signal. The need for this auxiliary information is presented very briefly: please improve it with clearer examples or references to similar approaches in the literature.\n- Sect. 3 (Methodology) is very dense and mixes architectural design and training approach: for clarity and reproducibility, it would be useful to substitute or integrate a textual description of the more algorithmic steps with pseudo code (e.g., the details in Cross-Modal Masking Strategy paragraph)", "limitations": "A point worth mentioning in the imitation is the problem of unpaired multimodal data. It would be interesting to understand if the added flexibility of OSKAR can be exploited to pretrain on a dataset with unpaired and missing modalities.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The quality of the paper is excellent. The topic is timely and highly relevant. \nThe methodology section can be improved by decoupling architectural design and training strategies, e.g., in a different subsection. The main architecture and ideas in the current presentation are quite clear, but some training details are not univocally defined, e.g.,  with the help of pseudo code. Experimental results are complete and convincing, including comparison with state of the art and ablation studies on the main components of the proposed approach.", "was_revised": true, "final_justification": "The authors engaged during the rebuttal phase, improving the manuscript and solving most of the raised concerns.", "rebuttal": "We deeply thank reviewer 5fwS for their valuable opinion and constructive feedback on our work. We are encouraged that they found our approach **“novel”**, our paper quality **“excellent”**, the topic **“timely and highly relevant”**, our architecture and ideas **“clear”**, and our experimental results **“complete and convincing”**. In the following, we address their questions:\n\n>*The encoded embeddings  are augmented with a learnable modality-specific auxiliary  signal. The need for this auxiliary information is presented very briefly: please improve it with clearer examples or references to similar approaches in the literature.* \n\nWe thank the reviewer for the opportunity to clarify the auxiliary embeddings. \n\nAs shown in our ablations (L283–286), adding learnable bounding box embeddings improves performance (e.g., ~+1% in skeleton action recognition). These embeddings serve a similar role to positional encodings in Transformers—here, encoding the **spatial location** of a person in the frame (e.g., (x, y, w, h)).\n\nThis spatial signal helps the predictor establish **pixel-to-keypoint correspondence** and disambiguate multiple people in a scene. Without it, the model may associate skeletons to the wrong subject, especially in crowded scenes (e.g., predicting joints for the wrong person when several are present).\n\nThis idea parallels techniques in multi-human trajectory forecasting [1] and pose estimation [2], where auxiliary spatial cues improve identity tracking. While we use bounding box embeddings in this work, the framework generalizes—e.g., using voice ID tags in future audio extensions.\n\nWe will revise the manuscript to make this rationale and flexibility clearer.\n\n>*Sect. 3 (Methodology) is very dense and mixes architectural design and training approach: for clarity and reproducibility, it would be useful to substitute or integrate a textual description of the more algorithmic steps with pseudo code (e.g., the details in Cross-Modal Masking Strategy paragraph)*\n\nWe appreciate the reviewer’s careful read and constructive feedback. \n\nIndeed, we recognize the opportunity to improve the representation of some of the dense technical details of the methodology section. To address this, we have included a **high-level pseudo code** outlining the core steps of OSKAR’s training pipeline. In the camera-ready version, we will further improve the organization of the architectural and training descriptions and provide a more detailed and structured pseudocode in the Appendix. \n\n\n```python\n# Inputs:\n# modalities = {\n#   'modality_1': input_1,\n#   'modality_2': input_2,\n#   ...\n# }\n# total_visible_token_budget: e.g., 128\n# total_target_token_budget:  e.g., 128\n\n# Step 1: Tokenize and embed inputs\ntokens = {}\nfor modality, input_data in modalities.items():\n    raw_tokens = Tokenizer[modality](input_data)\n    tokens[modality] = Embed(raw_tokens, modality_tag=modality)\n\n# Step 2: Sample visible and target tokens under fixed budgets\n\nmodalities_list = list(tokens.keys())\nnum_modalities = len(modalities_list)\n\nvisible_props = Dirichlet(alpha=[0.5] * num_modalities).sample()\ntarget_props  = Dirichlet(alpha=[0.5] * num_modalities).sample()\n\nvisible_counts = AllocateTokens(\n    visible_props,\n    total_visible_token_budget,\n    min_tokens=MIN_VISIBLE,\n    max_tokens=MAX_VISIBLE\n)\n\ntarget_counts = AllocateTokens(\n    target_props,\n    total_target_token_budget,\n    min_tokens=MIN_TARGET,\n    max_tokens=MAX_TARGET\n)\n\n# Sample tokens per modality based on allocated counts\nfor modality in modalities_list:\n    visible_tokens[modality], target_tokens[modality] = SampleTokens(\n        tokens[modality],\n        num_visible=visible_counts[modality],\n        num_target=target_counts[modality]\n    )\n\n# Step 3: Encode visible tokens with shared fusion encoder\nfused_latents = FusionEncoder(visible_tokens)\n\n# Step 4: Encode full tokens with modality-specific teacher encoders (for targets)\nwith no_grad():\n    teacher_outputs = {}\n    for modality in target_tokens:\n        teacher_outputs[modality] = TeacherEncoder[modality](tokens[modality])\n\n# Step 5: Predict masked target tokens using modality-specific predictors\npredictions = {}\nlosses = {}\nfor modality in target_tokens:\n    predicted = ModalityPredictor[modality](fused_latents)\n    target_indices = target_tokens[modality].indices\n    target = teacher_outputs[modality][target_indices]\n    predictions[modality] = predicted\n    losses[modality] = LossFunction(predicted, target)\n\n# Step 6: Backpropagate total loss\ntotal_loss = sum(losses.values())\ntotal_loss.backward()\n```\n\n>*A point worth mentioning in the imitation is the problem of unpaired multimodal data. It would be interesting to understand if the added flexibility of OSKAR can be exploited to pretrain on a dataset with unpaired and missing modalities.*\n\nIndeed, the design of OSKAR is very friendly to datasets with missing modalities. In fact, even in the current version trained on paired multimodal samples, the Dirichlet sampling strategy often **omits** a subset of modalities completely in some batches stochastically, effectively simulating unpaired settings. This demonstrates that OSKAR **already operates under a partially unpaired regime** and can be extended to train on unpaired or heterogeneous datasets without modification. This opens the door for training on unpaired samples from any dataset with any number of modalities.\n\n***We sincerely thank reviewer 5fwS for their time reading this rebuttal. We are committed to incorporating their comments on the auxiliary embeddings, methodology section, and training on unpaired data in the camera-ready version of our paper.***\n\n[1] Saadatnejad, S., et al. (2024). Social-transmotion: Promptable human trajectory prediction. ICLR.\n\n[2] Xu, Y., et al. (2025). DynPose: Largely Improving the Efficiency of Human Pose Estimation by a Simple Dynamic Framework. In CVPR."}]

Author rebuttal:

Meta-review: The paper proposes a training strategy for multimodal foundation models based on masked token prediction for different modalities.  The resulting model is evaluated on benchmarks for video, skeleton, and text processing. 

The paper received four reviews with mixed ratings: R (reviewer did not answer rebuttal) -  A - BA - A

The reviewers highlight the strong performance as well as the efficiency, the combination of latent feature prediction and momentum-updated modality-specific supervision. Most issues raised by the reviewers have been addressed during the rebuttal. 

While there were some concerns with regard to the performance compared to other sota models in this area, after discussion with the SAC, the decision was that there is no reason to overrule the reviewer rating in this case.

The decision is to follow the majority of the reviewer voting and to recommend accepting the paper. The AC would encourage the authors to integrate the findings of the rebuttal in the CR version of the paper.
Decision: Accept (poster)