Title: PreFM: Online Audio-Visual Event Parsing via Predictive Future Modeling
Venue: NeurIPS 2025
Accepted: yes
Keywords: Online Audio-Visual Learning

Audio-Visual Video Parsing

Audio-Visual Event Localization
Abstract: Audio-visual event parsing plays a crucial role in understanding multimodal video content, but existing methods typically rely on offline processing of entire videos with huge model sizes, limiting their real-time applicability. We introduce Online Audio-Visual Event Parsing (On-AVEP), a novel paradigm for parsing audio, visual, and audio-visual events by sequentially analyzing incoming video streams. The On-AVEP task necessitates models with two key capabilities: (1) Accurate online inference, to effectively distinguish events with unclear and limited context in online settings, and (2) Real-time efficiency, to balance high performance with computational constraints. To cultivate these, we propose the $\textbf{Pre}$dictive $\textbf{F}$uture $\textbf{M}$odeling (PreFM) framework featured by (a) predictive multimodal future modeling to infer and integrate beneficial future audio-visual cues, thereby enhancing contextual understanding and (b) modality-agnostic robust representation along with focal temporal prioritization to improve precision and generalization. Extensive experiments on the UnAV-100 and LLP datasets show PreFM significantly outperforms state-of-the-art methods by a large margin with significantly fewer parameters, offering an insightful approach for real-time multimodal video understanding.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes PreFM, a framework for online audio-visual event parsing. It introduces predictive future modeling, modality-agnostic representation, and temporal weighting to improve real-time performance. The method is evaluated on UnAV-100 and LLP, showing strong accuracy and high efficiency compared to prior work.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The paper addresses a practical and timely problem: real-time event understanding from streaming multimodal input.\n\n2. The model achieves significant performance improvements over strong baselines while using much fewer parameters.\n\n**Weaknesses:**\n\n1. The model uses future labels and supervision during training, which breaks the online assumption and creates a train-test mismatch.\n\n2. The paper does not evaluate whether the pseudo-future features are realistic or actually useful during inference.\n\n3. The distillation method depends heavily on a specific teacher model, but the paper does not analyze its effect or generalization.\n\n4. The evaluation setting constructs artificial video streams by clip concatenation, which does not reflect real-world online deployment.\n\nQUESTIONS:\n1. How does the model maintain online consistency when it uses future-supervised signals during training?\n\n2. Can you provide a quantitative analysis of the pseudo-future features’ quality?\n\n3. How sensitive is the model to the choice of teacher in MRR?\n\n4. The modality-agnostic distillation depends on OnePeace as the teacher model. Have you tried other options such as CLIP or AudioCLIP? How much does the downstream performance vary?\n\n5. How does PreFM perform in a real streaming setting with long, continuous videos?\n\nLIMITATIONS:\nKey issues like the train-test mismatch and generalization to real-world streaming settings are not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new task, Online Audio-Visual Event Parsing (On-AVEP), which aims to detect audio and visual events in a streaming setting using only past and current information. To tackle the challenges of limited context and real-time efficiency, the authors introduce PreFM, a novel framework built on three key components: (1) Predictive Multimodal Future Modeling (PMFM) to synthesize pseudo-future cues, (2) Modality-agnostic Robust Representation (MRR) via distillation from a frozen teacher, and (3) Focal Temporal Prioritization to emphasize current-time predictions. Extensive experiments on UnAV-100 and LLP datasets demonstrate that PreFM achieves state-of-the-art performance while being highly efficient in terms of parameters and computation.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- **Novel Task Definition:** The formulation of On-AVEP as a real-time, streaming version of AVEP is timely and practically relevant, filling a gap in existing literature that mainly focuses on offline processing.\n- **Strong Efficiency-Performance Trade-off:** Despite its lightweight architecture (e.g., 6.5M parameters), PreFM significantly outperforms prior methods with much larger models, making it highly suitable for deployment in real-time scenarios.\n- **Thorough Empirical Evaluation:** The paper provides comprehensive comparisons, ablations, and qualitative analyses, making the improvements clearly attributable to each design component.\n- **Good Clarity and Writing:** The paper is well-organized and easy to follow, with clear figures and sound motivation for each component.\n\n\n\n## Weaknesses\n\nGiven the aforementioned strengths, I am generally satisfied with this paper and would like to share a few questions and suggestions:\n\n- **Contradictory statements?** Lines 256–257 suggest that the PF module significantly boosts performance, whereas Lines 268–269 state that PF yields negligible gains. It would be helpful to clarify these descriptions to avoid confusion.\n\n- **Failure cases.** While Figure 5 and the supplementary materials present visualizations, the paper would benefit from a more detailed discussion of the failure cases—when and why PreFM might not perform well.\n\n- **Other questions.**\n\n  1. The On-AVVP subtask requires identifying audio-only, visual-only, and audio-visual events. Does the proposed method predict each type of event separately or simultaneously?\n  2. Could the softmax function be used to normalize the weight w(t) across frames in Eq. 9?\n  3. What are the training and inference time costs for the proposed method?\n\n- **Minor issues (do not affect my rating):**\n\n  1. The module name “current enhancement” in Line 157 is inconsistent with “current refinement” shown in Figure 2.\n\n  2. In Figure 2, the \"Focal Temporal Prioritization\" module appears disconnected from the rest of the architecture. It would be clearer to illustrate how this module interacts with others.\n\n  3. It would be helpful to include a diagram (even in the appendix) that explains the training and inference processes described in Section 3.6.\n\n  4. The paper would benefit from citing more traditional works on audiovisual event localization and video parsing, such as: Contrastive Positive Sample Propagation Along the Audio-Visual Event Line, TPAMI 2023. UWAV: Uncertainty-weighted Weakly-supervised Audio-Visual Video Parsing, CVPR 2025\n\n     \n\n####\n\nQUESTIONS:\nPlease refer to the questions mentioned in 'weaknesses' section.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Online Audio-Visual Event Parsing (On-AVEP), a novel paradigm for real-time parsing of audio, visual, and audio-visual events in streaming video content. This paper proposes the Predictive Future Modeling (PreFM) framework that addresses two key challenges: accurate online inference with limited context and real-time computational efficiency.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n\nThis paper addresses an important practical problem in real-time multimodal processing. The proposed predictive future modeling approach is novel and the idea of anticipating future multimodal cues to compensate for limited historical context in streaming scenarios is technically sound. The claimed performance gains with fewer parameters suggest good computational efficiency.\n\nWeaknesses\n\n- About predictive multimodal future modeling: This paper lacks sufficient detail about the specific mechanisms of predictive future modeling. How exactly does the model predict future cues? What is the temporal horizon?\n- Real-time Claims: The paper emphasizes real-time efficiency but doesn't provide concrete timing analysis, latency measurements. Claims about \"real-time applicability\" need quantitative support.\n- Missing Ablation Studies: There's insufficient analysis of which components contribute most to the performance gains. The interplay between predictive multimodal future modeling, modality-agnostic representation, and focal temporal prioritization needs better investigation.\n\nQUESTIONS:\n- Prediction Mechanism: Can you provide more technical details about how future audio-visual cues are predicted? What is the prediction horizon and how do you handle prediction uncertainty?\n- Baseline Fairness: How were the baseline methods adapted for online processing? Are you comparing against methods specifically designed for online settings or offline methods retrofitted for streaming?\n- Real-time Performance: Can you provide the concrete real-time performance? What are the actual processing times and computational costs compared to baseline methods in practical streaming scenarios?\n\nLIMITATIONS:\nThe paper lacks sufficient concrete analysis of its primary claimed advantages, particularly regarding the technical details of predictive modeling, component-wise contributions, and quantitative real-time performance validation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Online Audio-Visual Event Parsing (On-AVEP), a new setting for multimodal video understanding where models must parse audio, visual, and audio-visual events in a strictly online manner. To address the unique challenges of this setting—limited future context and real-time constraints—the authors propose PreFM, a lightweight framework that enhances online inference by predicting useful future multimodal cues and incorporating them during current-step inference. PreFM also introduces a modality-agnostic representation mechanism and a temporal focus strategy to improve both robustness and efficiency. The method shows strong performance with significantly fewer parameters, suggesting its practical utility for real-time applications.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths:\n- The method is intuitive and simple to understand.\n- The problem is well motivated, and empirical results support the claims.\n- The method appears to work well in online settings, with low memory footprint.\n\n### Weaknesses and questions:\n- Despite the results, the methodical contribution is quite weak. At its core, the work simply teases out \\(k\\) representations using some \"prototype\" query vectors via attention, as a proxy for the supposed future.\n- The paper uses a series of attention blocks to generate future representations. However, the **order of operations** and **inputs to the UHA blocks in TMCF** appear fairly arbitrary.  \n  Can the authors provide justification for this design? For instance:\n  - What motivates the specific sequence of UHA operations?\n  - Why were those particular inputs chosen?\n  - What happens if you simply apply another round of attention (as in the \"Pseudo future\" section)?\n  \n  It's unclear whether TMCF meaningfully contributes, or if the observed gains come primarily from increased parameter count (due to FFL) and additional attention layers like those used in the \"Pseudo future\" section.\n\n- The paper does not clearly articulate its contributions.  \n  Can the authors explicitly state the contributions and novelties in the introduction?\n\n- While the method does seem effective in online settings with low memory usage, it's important to **contextualize these improvements**.  \n  What were the bottlenecks in prior methods, and how does this work address them differently?  \n  Additionally, can the authors clarify how their parameter count compares to prior art, and justify the *huge* differences?\n\n- What are the implications of the method on training time ?\n\nOpen to increasing my score, provided my queries are addressed\n\nQUESTIONS:\nPlease refer to weaknesses\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 1, "DATE": "2025-07-05"}