Title: OSKAR: Omnimodal Self-supervised Knowledge Abstraction and Representation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal Self-Supervised Learning

Representation Learning

Feature Prediction
Abstract: We present OSKAR, the first multimodal foundation model based on bootstrapped latent feature prediction. Unlike generative or contrastive methods, it avoids memorizing unnecessary details (e.g., pixels), and does not require negative pairs, large memory banks, or hand-crafted augmentations. We propose a novel pretraining strategy: given masked tokens from multiple modalities, predict a subset of missing tokens per modality, supervised by momentum-updated uni-modal target encoders. This design efficiently utilizes the model capacity in learning high-level representations while retaining modality-specific information. Further, we propose a scalable design which decouples the compute cost from the number of modalities using a fixed representative token budget—in both input and target tokens—and introduces a parameter-efficient cross-attention predictor that grounds each prediction in the full multimodal context. We instantiate OSKAR on video, skeleton, and text modalities. Extensive experimental results show that OSKAR's unified pretrained encoder outperforms models with specialized architectures  of similar size in action recognition (rgb, skeleton, frozen, low-shot) and localization, video-text retrieval, and video question answering. Project website: https://multimodal-oskar.github.io


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work is built around a novel multimodal pretraining approach based on the prediction of masked high-level representations: the multimodal tokens are projected by a separate encoder and then fused into a common representation. A predictor uses the latter to reconstruct masked tokens to be compared to target tokens projected by the corresponding target encoders. These latter are updated with moving average momentum to stabilize the training chain. \nThe proposed approach's benefits are experimented with in the context of video, skeleton, and text multimodal settings. Comprehensive experimentation and ablation studies conclude the work. Supplementary material, including the software, is useful.\n\nSTRENGTHS AND WEAKNESSES:\nThe quality of the paper is excellent. The topic is timely and highly relevant. \nThe methodology section can be improved by decoupling architectural design and training strategies, e.g., in a different subsection. The main architecture and ideas in the current presentation are quite clear, but some training details are not univocally defined, e.g.,  with the help of pseudo code. Experimental results are complete and convincing, including comparison with state of the art and ablation studies on the main components of the proposed approach.\n\nQUESTIONS:\n- The encoded embeddings $\\mathbf{E}$ are augmented with a learnable modality-specific auxiliary $e_{(aux)}$ signal. The need for this auxiliary information is presented very briefly: please improve it with clearer examples or references to similar approaches in the literature.\n- Sect. 3 (Methodology) is very dense and mixes architectural design and training approach: for clarity and reproducibility, it would be useful to substitute or integrate a textual description of the more algorithmic steps with pseudo code (e.g., the details in Cross-Modal Masking Strategy paragraph)\n\nLIMITATIONS:\nA point worth mentioning in the imitation is the problem of unpaired multimodal data. It would be interesting to understand if the added flexibility of OSKAR can be exploited to pretrain on a dataset with unpaired and missing modalities.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an omni-modal self-supervised pretraining framework, named OSKAR, that simultaneously extracts representations from video, action, and text modalities, achieving competitive performance on downstream tasks such as action recognition, action localization, text-video retrieval, and video question answering (VideoQA). \nOSKAR leverages cross-modal masked prediction in latent space as its pretraining objective, consists of two branches: (1). a fusion encoder that processes visible tokens to generate fused representations; (2). modality-specific target encoders that produce target embeddings. These target encoders are updated via the exponential moving average (EMA) of the fusion encoder.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. Nice writing and presentation, the reviewer can clearly understand the framework of the method.\n2. Modality-specific target encoders can effectively balance intra-modality feature modeling and inter-modality feature alignment.\n3. Multimodal representation learning is one of the current popular topics. The method proposed in this paper can effectively integrate the features of action videos and texts.\n\n**Weaknesses:**\n1. Lack of validation on large-scale data. I think effectiveness on large-scale data is the key to current representation learning, enabling it to be better applied to VLM or VLA tasks. The paper shows competitive performance on small-scale data, but there is still a gap compared to other methods that use more data.\n2. The baseline methods used for comparison are somewhat outdated and do not fully reflect the current state-of-the-art. For example, in an action recognition task, there is no Comparison of current SOTA models(InternVideo[1], Unmasked Teacher[2]，Hiera[3]). These situations also exist for other tasks:Text-to-video retrieval(VATEX[4]), VideoQA(VAST[5]).\n\n**Ref:**\\\n[1]. InternVideo: Video Foundation Models for Multimodal Understanding. ECCV24 \\\n[2]. Unmasked Teacher: Towards Training-Efficient Video Foundation Models. ICCV23\\\n[3]: Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles.ICLR25\\\n[4]. Gramian Multimodal Representation Learning and Alignment. ICLR25\\\n[5]: VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset.NeurIPS23\\\n\nQUESTIONS:\n**Q1**: For the fusion encoder, are there special tokens used as identifiers for tokens of different modalities when they are input? \\\n**Q2**: The impact of mask rate on model training and performance. The author did not discuss the mask rate in the paper. Will using too high a mask rate cause the model to fail to converge or reduce performance? \\\n**Q3**: In the self-supervised training paradigm of mask reconstruction, asymmetric encoders and decoders[1] are usually used, such as MAR. In my understanding, the modality predictor plays a role similar to the decoder in the framework of this article. The author kept the structure of the predictor consistent with the encoder in the experiment. Why is this? Please experiment or discuss this. \\\n**Q4**: Reference Weaknesses 2. Please analyze and discuss the current sota model. \\\n\n**Ref**: \\\n[1]. Masked Autoencoders Are Scalable Vision Learners.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-28"}

comments: SUMMARY:
OSKAR introduces a scalable, self-supervised multimodal learning framework using a “fuse-then-predict” strategy in latent space. It avoids low-level reconstruction and contrastive negative pairs, leveraging modality-specific momentum encoders to predict missing features from fused video, skeleton, and text tokens efficiently. OSKAR achieves state-of-the-art results in action recognition, localization, video-text retrieval, and video QA while using fewer samples, parameters, and labels. It demonstrates effective multimodal grounding, high transferability, and scalability.

STRENGTHS AND WEAKNESSES:
### Strengths:

1. The paper is written clearly and concisely, providing rigorous and detailed experimental results and analyses, which greatly enhance the persuasiveness of the work.
2. OSKAR demonstrates flexible and extensible characteristics and shows promising generalization performance across various tasks, proving the effectiveness of the proposed method.
3. The paper focuses on specific technical details, providing detailed descriptions and experimental results on the sampling strategies, customized update rates, and other implementation details.

### Weaknesses:
N/A

QUESTIONS:
### Question:

1. In the Video QA section, only a BART-based language module was used. I am curious whether OSKAR can be effectively adapted to existing LLMs, which would further enhance the significance and effectiveness of this method.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nOSKAR is a model that takes components from other architectures and incorporates them into a unified process. These components are shared embeddings, cross-model transformers, and self-distillation.  It takes inputs from text, video, and skeleton data, fuses them together, and then fills in (predicts) the masked or missing parts in each modality. This prediction is done in a higher level, latent space, rather than at a pixel or coordinate or word level. The modality-specific predictions are checked against the feature vector calculated by the encoder for the masked/missing parts, and the encoder weights are updated.\nThe paper reasonably claims that theis method is highly scalable because of a fixed token budget that is allocated across all modalities in a way determined by the modeler. The authors present experiments on action recognition, video question answering, and retrieval. THey further present ablation tests which show that OSKAR’s architecture components all contribute to the model’s performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n* Integrates latent feature prediction, momentum-updated modality-specific supervision, and efficient cross-modal fusion.\n* Strong performance across video action recognition and question answering, and retrieval.\n* Ablation studies that are extensive which validate the architectural choices. In particular, the fusion of modalities and latent space prediction are important.\n* Practical and scalable design made possible by fixed token budget and momentum-based supervision. The fixed budget allocated across modalities makes computation manageable, and labeled data is not needed, \n\n\nWeaknesses\n* Standard errors are not reported, but performance over other, closely performing models do not seem to be statistically or significantly significant. Most of them range from 0.3 to 1.0 percentage points. I tend to look at differences under 1 or 2 points as likely not statistically significant in the absence of variance measures.\n* Ablation studies seem thorough, but they do not take into account interactions. They are also highly dependent on the order in which features were added or removed.\n* Fine-tuning of momentum may complicate generalizing to other new contexts.\n\nQUESTIONS:\n* The authors acknowledge that a limitation of the paper is that only 3 modalities were tested. How would performance change because of the fixed budget. Or is there some dynamic re-allocation of the budget across modalities as some modalities appear to be more important?\n* Can you provide cases where OSKAR did not perform as expected, or any limitations you encountered while experimenting?\n* The authors gave a reason why there are no error plots. But the results look suspiciously non-significant. It might be worth the computational cost to verify OSKAR's dominance. Or, make a statement saying even if OSKAR's gain are minor, there are other, architectural reasons for using it.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-14"}