PreFM: Online Audio-Visual Event Parsing via Predictive Future Modeling
Authors: Xiao Yu
Yan Fang
Yao Zhao
Yunchao Wei
Venue: NeurIPS 2025 poster ()
Keywords: Online Audio-Visual Learning
Audio-Visual Video Parsing
Audio-Visual Event Localization
Primary area: applications

Abstract: Audio-visual event parsing plays a crucial role in understanding multimodal video content, but existing methods typically rely on offline processing of entire videos with huge model sizes, limiting their real-time applicability. We introduce Online Audio-Visual Event Parsing (On-AVEP), a novel paradigm for parsing audio, visual, and audio-visual events by sequentially analyzing incoming video streams. The On-AVEP task necessitates models with two key capabilities: (1) Accurate online inference, to effectively distinguish events with unclear and limited context in online settings, and (2) Real-time efficiency, to balance high performance with computational constraints. To cultivate these, we propose the $\textbf{Pre}$dictive $\textbf{F}$uture $\textbf{M}$odeling (PreFM) framework featured by (a) predictive multimodal future modeling to infer and integrate beneficial future audio-visual cues, thereby enhancing contextual understanding and (b) modality-agnostic robust representation along with focal temporal prioritization to improve precision and generalization. Extensive experiments on the UnAV-100 and LLP datasets show PreFM significantly outperforms state-of-the-art methods by a large margin with significantly fewer parameters, offering an insightful approach for real-time multimodal video understanding.

PreFM: Online Audio-Visual Event Parsing via Predictive Future Modeling
Xiao Yu? Yan Fangl. Yao Zhao Yunchao W.,
Institute of Information Science, Beijing Jiaotong University
Visual Intelligence + X International Joint Laboratory *oorrsspondnng Author
xiaoyu@bjtu. edu. cn wychao19670gmail.com
Abstract
Audio-visual event parsing plays crucial role in understanding multimodal video content, but existing methods typically rely offline processing of entire videos with huge model sizes, limiting their real-time applicability. We introduce Online Audio- Visual Event Parsing (On-AVEP), novel paradigm for parsing audio, vi- sual, and audio-visuee events by sequentially analyzing incoming video streams. The On-AVEP task necessitates models with two key capabilities: (1) Accurate online inference, to effectively distinguish events with unclear and limited con- text in online settings, and (2) Real-time efficiency, to balance high performance with computational constraints. To cultivate these, we propose the Predictive Future Modeling (PreFM) framework featured by (a) predictive multimodal future modeling to infer and integrate beneficial future audio-visual cues, thereby en- hancing contextual understanding and (b) modality-agnostic robust representation along with focal temporal prioritization to improve precision and generalization. Extensive experiments on the UnAV- 100 and LLP datasets show PreFM signifi- cantly outperforms state-of-the art methods by a large margin with significantly fewer parameters, offering an insightful approach for real-time multimodal video understanding. Code is available at https://gthub..oo/Xiia Yu- 1123/PreFM
Introduction
Multimodal learning 5
38
79 is a significant topic in the machine learning research area.
Among various modalities, audio (62 and vision 660 are the primary ways humans perceive the world, making audio-visual learning (AVL) [19 42
35 15 essential. Among various progress [43 31 34] related to AVL, audio-visual event parsing (AVEP), i.c., understanding events in videos. becomes increasingly important with the explosive growth of video content on streaming platforms.
AVEP involves processing both modality- (audio-visual) and modality-misaligee or visual-only) events in video content. Prevailing methods [33 14 78 operate offline, analyzing entire video sequences utilize global context for accurate video events understanding. Though offering precise predictions, the necessity of whole-video processing, often coupled with large models and consequently high computational costs, makes these approaches unsuitable for real-time applications that require immediate detection and swift responses in dynamic environments such as
(audio-only
autonomous driving 655 70]. wearable devices 26 2 and human-robot interaction [50] 32
To tackle these limitations, we introduce Online Audio Visual Event Parsing (On-AVEP). new paradigm that parses audio, visual, and audio-visual events in streaming videos with online processing manner. The core characteristic On-AVEP is to perceive the environmental state and generate timely feedback using only historical and current multimodal information, while balancing model performance and efficiency particularly in resource-aware and dynamic environments.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Available
Unavailable @
Prodictive Multimodal
model
Y--r
Fature Modeling
Oaline
Straming Video
Picudo Fature
6.M
UstAV 50.3
UsAY 23.0
130,533
MRR
model
131.0
Apeaking!
Wivt is dising ot nowe 77
PreFM
(b)
(e)
Figure 1: (a) Illustration of parsing events in online scenarios: if man opens his mouth and produces a vocal sound at time T, it is unclear based solely on information from 0 to T whether this marks the beginning of musical phrase (as part of "singing") or the start of conversation (as in "speaking"). Precisely parsing events with these limited context is crucial for accurate online inference. (b) Simplified architecture of our PreFM framework, highlighting predictive future modeling and robust representation (MRR). (c) Comparison of performance and efficiency against SOTA methods on the UnAV-100 13 dataset.
Specifically, On-AVEP necessitates the model possess two key capabilities: (1) Accurate Online Inference: This requires that the model adapts to complex and dynamic scene variations and accurately predicts ongoing events by relying exclusively on past and current information without any future context. As illustrated in Figure aa), the model needs to distinguish similar events with unclear, limited context due to the lack of future information. (2) Real-time Efficiency: To meet the immediate response demands of On-AVEP applications, the model needs to achieve accurate event parsing with low computational cost, balancing performance and complexity well to satisfy the needs of online video processing.
I
().,
To cultivate these essential capabilities, we introduce the Predictive Future Modeling (PreFM) framework as illustrated in Figure [Tb). PreFM aims to predict future states through effective cuuporal-modalit, feature fusion and leverage knowledge distillation and temporal prioritization for training efficiency. To achieve (1) accurate online inference, PreFM employs predictive multimodal future modeling, using available data and fusing their features to infer beneficial future audio-visual cues. The cross-temporal and tross-modal feature interactions are utilized to effectively reduce noise within the pseudo-future context and enhance current representations. For (2) balancing real-time efficiency and overall parsing performance, PreFM integrates two designs during training: modality- agnostic robust representation distills rich, modality-agnostic knowledge from a large pre-trained teacher model for more generalized representation, and focal temporal prioritization encourages the model to focus on the most temporally critical information for online decisions, thereby boosting the model's inference accuracy while keeping high inference efficiency.
Extensive experiments on two challenging datasets, UnAV-100 [3] and LLP [54], demonstrate that the PreFM framework significantly outperforms existing otate-of-the-art (SOTA) methods in both segment-level and event-level metrics. Moreover, PreFM exhibits substantial advantages in model efficiency, striking superior balance between performance and model complexity, with the margin of +9.3 in event-level average F1 score and merely 2.7% parameters as highlighted in Figure I(c).
In summary, our main contributions are:
(I) We introduce Online Audio Visual Event Parsing (On-AVEP), a new paradigm for real- time multimodal understanding. To knowledge, this is the first work systematically address the challenge of parsing audio, visual, and audio-visual events from streaming video. We further establish that success in this paradigm requires two critical capabilities: (a) accurate online inference from limited context, and (b) real-time efficiency to balance performance with computational cost.
(II) We propose the PreFM framework, a novel and efficient architecture for On-AVEP. PreFM's core innovations include: (a) Predictive Multimodal Future Modeling mechanism to overcome the critical problem of missing future context; and (b) combination of Modality-agnostic Robust Representation and Focal Temporal Prioritization to enhance model robustness and efficiency during training, providing insightful approach to multimodal real-time vidco understanding.

---

(III) We establish new SOTA performance with unprecedented efficiency. Extensive experiments on two public datasets show that PreFM drastically outperforms previous methods (e.g.. +9.3 Avg F1-score on UnAV-100), while using a fraction of the computational resources only 2.7% of the parameters of the next best model), validating it as powerful and practical solution.
2 Related Work
Online Video Understanding encompasses online action detection for identifying actions |56 7
44), action anticipation for predicting future [6 73), and online temporal action localization 48 5
for determining action boundaries. Frameworks like JOADAA 17.. TPT [7 and MAT (58] jointly
model detection and anticipation tasks, bridging the present and future. Recent research focuses on model reliability through uncertainty quantification [18] and adaptability through open-vocabulary detection [6]]. Concurrent advancements explore leveraging large language models for complex online understanding tasks [33] 3 69]. However, these methods rely solely on the visual modality and neglect the crucial auditory perception, motivating our research into online audio-visual event parsing aiming to integrate both sensory streams for more robust and holistic real-time understanding.
Audio Visual Video Parsing (AVVP) aims to temporally classify videos within segments as audible or visible events. Early weakly-supervised methods [54] [6] use attention to infer temporal structure. Subsequent works [2 4 further address modality imbalance and interaction. A significant recent trend involves leveraging external knowledge, using language prompts ( or pre-trained models like CLIP /CLAP g to denoise or generate finer pseudo-labes labels from weak
supervision 22 77 30) Building on this, methods such as CoLeaF [49], NREP [27]. and MM-
CSE 71 focuse on sophisticated feature disentanglement and interaction for improved performance.
Audio Visual Event Localization (AVEL) is first introduced to temporally locate events that both visually and auditorily present within trimmed video clips (53) Subsequent methods [63
41
23 74) leverage cross-modal attention, background suppression, contrastive smaples and adapters to improve localization accuracy. AVE-PM (37] is developed to handle portrait- mode short videos, while OV-AVEL [75] extends the task into an open-vocabulary setting. For densely annotated, untrimmed videos featuring multiple overlapping events, UnAV [3] releases the UnAV- 100 benchmark and inspires models like UniAV [14) LOCO (44, FASTEN 39 and CCNet (78), which employ multi-temporal fusion, local correspondence correction and mross-modal consistency for dense event localization. Recent efforts [68, 15 52 40 also aim to onni-understanding using powerful large language models. However, these approaches generally rely on full-video inputs and huge model sizes, making them unsuitable for real-time parsing. Our work distinguishes itself by unifying AVEL and AVVP into a comprehensive online audio-visual event parsing framework, designed for efficient real-tin processing and capable of identifying events regardless of whether they are solely auditory, visual or audio-visual.
28
36
3 Methods
In this section. we first introduce the problem setup in Sec. and present a brief overview of our
method in Sec. [.22 with core designs: Predictive Multimodal Future Modeling (Sec. (3.). Modality-
agnostic Robust Representation (Sec. (S.c. and Focal Temporal Prioritization (Sec. 3.5) Finally, we
discuss the specifics of our approach during training and online inference in Sec. 3.6
3.1 Preliminaries
On-AVEP involves predicting events within streaming videos by sequentially processing multimodal information. This task is primarily divided into two sub-tasks: online audio-visual event localization (On-AVEL) and online audio-visua video parsing (On-AVVP). In the former, given sequence of audio-visual data pairs 4t) and the corresponding label yt=r, where T denotes the current time step, the model is required to predict the multi-label event vector ÿ1-T E {0. Ie where C... represents the total number of audio-visual event categories. While the latter task involves predicting Út=r E (0, 1) C.+ where C. and C, represent the number of audio-only and visual-only events, respectively. In both sub-tasks, models typically take the pre- pre-processed visual-audio feature vectors
3

---

Predictive Multimodal Future Modeling
4
F..
P
P.
Chasslie
Infeeeeet
Refiacucol
Streaming Video
P'
Modellng
Fi.Fi
Py.
Pacsti-Fattur Mechosien
lumpurai-Vattllly Crast Finien
8
fuls
Virad carrest
Andia carrex
Voual Sttues
Andio future
"aiba uudior visue! on mndeostto coeer of joby
Text Eandar
Lisited lyberi attuaii
Modality-
Represcntation
Pocal Tempural Prtuetiication
Figure 2: The pipeline of PreFM. It takes real-time audio-visual streams, using predictive modeling to generate multimodal future context, modality-agnossi robust representation enhance performance by transferring knowledge, and focal temporal prioritization to emphasize the current time step T.
( and (sT)== RTxD (D: feature dimension) within existing video datasets 54 for subsequent operations.
25
53
Overview
As illustrated in Figure 2 during online inference, PreFM sequentially processes incoming audio and visual features Fs F available up to the current time T. To address the challenge of missing future information, which crucial for event disambiguation, the core Predictive Multimodal Future Modeling network (Sec. (33) dynamically generates pseudo- futtre multimodal sequences. This process starts with a Pseudo-Future Mechanism that fuses current-time multimodal features and subsequently models initial pseudo-fute predictions Fy. Fy, then Temporal-Modult Cross Fusion where aseudo-future cues and current representations are mutually enhanced through comprehensive cross-temporal and cross-modal interactions. The resulting contextually augmented representations Fu F are then utilized for event parsing at time T.
To train effective and efficient PreFM model, in addition direct supervision predictions for both the current window and the pseudo- future sequences, PreFM utilizes moddalityaggostie Robust Representation (MRR, Sec. 3.4). Through MRR, event labels y, transformed into target modality-agnostic features fe using a pre-trained teacher model; PreFM's internal event representations Dt are then guided align with these target features via dedicated distillation loss term. Furthermore, Focal Temporal Prioritization (Sec. (.5) is implemented by reweighting the contributions of different relative time steps, encouraging the model to make precise predictions at current time.
3.3 Predictive Multimodal Future Modeling Network
our approach On-AVEP centers on predictively modeling multimodal pscudo-future sequences using only currently available data. To help PreFM better utilize and consolidate all available modal and temporal cues, we propose Universal Hybrid Attention (UHA) block to bridge different modalities across time. Given the target query sequence Q and the flexible list of context sets ( where each F, can represent various temporal segments of different modalities, UHA merges these features into Q follows:
Inspired by advances in online action detection 58
UHA(Q, (F))))) = FFN(LN(Q + Σ Att(Q, F())
(I)
Where Attn multi-head attention [57] IN Layer Normalization, and FFN Feed-Forward Network. UHA serves the foundational attention block for subsequent fusion operations.

---

Pseudo-Futuue Mechanism This mechanism first fuses current audio-visual information and then models an initial prediction of the future sequence. Given input features up to current time T. {(E (())(t [=)) we define current working window of length Le. This yields the initial current audio and visual features P. ( and P {fi = both RR.xD.
First, we perform an initial feature fusion between Fe and F Each sequence is processed by our UHA block with both as context. The fused current features F Fr E RlexD are produced by:
Fm UHA(PT (F2: F2)) {a,v}
Next, future modeling generates initial pseudo-future future sequences of length L.. We use learnable tokens Q", Q" x as queries. These attend to the corresponding fused current features:
FT = Attn(Q" Fem FT)), E {a. c}
(3)
This step yields the initial multimodal pseudo-future sequences Fg. Fy ERLID
Temporal-Modality Cross Fusion Having obtained the fused current features (Fo. F2) and ini- tial pssudo-future sequences (Feg, Pÿ), this stage performs further interactions to mutually refine them, reducing potential noise within the pseudo-future, while simultancously enriching the current
representations with foresight gleaned from the modeled future.
future augmentation refines the initial pseudo-future pre- dictions with UHA block:
P
F
F
mm
(H (Ff. F. E {a, f
(4)
t
4v
40
J
4Q
K
This yields augmented pseudo-future sequences FT. E RL Notably, the context list within the UHA block ables a rich combination of self-attention, as well as cross- interactions across modalities and time. For instance, the aug- mented visual pseudo-futur Fy is obtained by interacting with Fy (for self-attention). Fy (for cross-modal attention), and Ftv (for cross cross-tempone temporal attention) as shown in Figure 3
MHA
MHA
MHA
Add LN
FFN
Add & &IN
FP
Figure 3: Temporal-modality cross
Next, current refinement integrates the augmented future back into the current representations:
fusion for the pseudo-future Fy.
t UHA(F.". (Fa, F., F) {a,
(5)
This results in the final contextually-awaree current feature sequences Fo F.v E LDD
Finally, these augmented current and future features are projected by a shared classification head h(()) and Sigmoid function S()) obtain event predictions ÿe R k.C (for the current window) and y E R xC (for the future window):
Ue S(h(Concat((), F2)))
CCCCc ((b(Concat(F)) Ff)))
(6)
Here, Concat denotes feature concatenation, and C' is the event class count (either Cae or C. + C.). For online inference, the prediction ÿ. corresponding to time used. While during training, de and yf are supervised across the time steps of [T Iue 1.T + L,)., using BCE loss with annotations Ve E XC and y E R
L. BCE( M.),,, BCE(Ú,,
3.4 Modality-ggnostie Robust Representation
Learning from the rich, modality-agnostic event representations established by powerful pre-trained teacher models [46] & 16. 59 20 is efficient to obtain robust and generalizable representations
5

---

while maintaining efficiency. For each time step E, we convert the event labels yr into "a/an andio//isnalluuiiovian event of [cls]" as text prompt, which is then processed by the text encoder of frozen teacher model OnePeace 59 to obtain modality-agnostic event features St. Simultaneously, the student's s representation frr Concat(J?, ; can be easily extracted from our PreFM model, and distilled by the target representation fc. Cosine similarity used as distillation loss as follows:
T+Ls Σ
ftn (fA)
Emor
(8)
Le + Ls
t= +e+1
Where (-()) denotes the projector module implemented by linear layer to align different dimensions.
3.5 Focal Temporal Prioritization
To emphasize the predictions close to the present moment, we introduce a focal temporal prioritization scheme to the loss calculation, highlighting the significance of prediction at time step T instead of uniform weighting. Specifically, we define temporal priorities using a Gaussian function centered at
(e)²
the current time T: ggl, o) exp
20 where is the relative temporal distance from time T,
and a controls the width of the focus. We define the temporal weights w.(t) E [ + +,,]] for the current window, and (1) E + 1,7 + Ly] for pseudo-future sequences,
Ta g(t ((t--,IA Icc) welt - T)= Ek-r L. g(k T, I.c)'
I. g(t - T.p) E g(k - T, Ly)
wy((t-T) ()
(9)
Let (t) Cy(t) and (t) be the per-timestee loss in Eq. and Eq. 8
We use w(t - T) = T), wy (t- T)} to obtain the while weights sequence vector. The final loss
Concat Concat(we(t (we(t computed as:
T+L; Σ w,(t-T)-
T+L, Σ
C=
Σ
(T-t-T)
w(t (mrr(t) (10)
Tet1
The hyperparameter À balances the robust representation term (typically 1.0).
3.6 Training and Online Inference
Random Segment Sampling for Training To adapt training for the online nature of On-AVEP and enhance data utilization, we design a random segment sampling strategy. During training, for a video of total length Tau, the target prediction times Ti [1, [at] are generated by T3 = kL. + d. Here, At serves as an index for iterating across the video, and 5 E [0, L 1] is a periodically selected random integer offset. The L.-length feature sequences [ [()) - +1 act as model inputs, and nedd-nadding applied at the beginning if T: Tee 1. This strategy provides diverse training segments with fixed history length Lc, suitable for the online setting.
Online Inference During inference, the model works in a truly online manner, processing the input video stream with a sliding window of length L. and stride 1. At each step Fenf the model takes features from [Taferr Le 1, T Jerfer) generates the multimodal pseudo-future context, and gets the final event predictions for the current time step Tinfer.
4 Experiments
4.1 Experimental Setups
Dataset UnAV-100 [3] is large-scale dataset designed for dense audio-visual event localization untrimmed videos. contains 10,790 videos of varying lengths covering 100 event categories, with over 30,000 annotated audio-visual event instances. LLP (54) provides 11,49 trimmed 10-second clips across 25 categories for audio-only and visual-only event parsing. For online scenarios, we concatenate LLP clips into longer video sequences. Specifically, half of these sequences are formed

---

Table 1: Comparison with SOTA methods on On-AVEL task. Feature extractors: I. V. denotes I3D VGGGih (24] O. denotes OnePeace [59] and C.C. denotes CLIP [ [8. "PreFM+" replaces the feature extractor from CLIP/CLAP to OnePeace and the hidden dimension is expanded from 256 to 512. Methods marked with take the entire video as input, fully utilizing the complete context.
Methods
Extracters
Seqment-Level
Fl
mAP
0.1
Event-Level
0.7
Avg
Parcus,
FLOPS]
Memory.
Inference
Latency
UBAV
A.
58.3
50.9
28.7
28.6
139.4M
764.73B
10.6
34,3
UadAV
0.
47.8
66.9
50.3
30.3
130.8M
22.7G
1020
15.6
641, 1rss
CCNe
0.
54.8
62.3
58.3
46.3
37.5
27.3
15.8
37.0
238.8M
72.1G
1879.4MB
7.5
133.
PreffM (Chars)
CCC.
59.1
70.1
61.5
46.3
6.5M
56.4MH
51.9
19,
Pref'Me (Curs)
70.6
51.5
13.0
144.230B
42.0
23 Nuss
UnAV
56.1
67.8
59.3
52.7
46.7
50.6
139.4M
764.7348
10.6
94, Bus
UniAV
59.2
70.0
62.8
52.9
13085
1020 5148
15.6
64. Las
CCNec
65.0
70.6
09.0
61.0
58.3
238.8M
7.5
133.3us
Table 2: Comparison with SOTA methods on On-AVVP task. Feature extractors: R. C.C.C denotes
R3D [55IP CCLAP (8] and R.R. V. denotes R3D Ressee55 Vsh]+ VGGish (24).
"PreFM+" increases the hidden dimension from 128 to 256. Methods marked with "*gh take the entire 10-second video clips as input, fully utilizing the complete context.
Mathools
Extractors
Fl,
Fl,
Flg | MAP,
Segment-1 eeel
mAP:
mAPP
n.5
0.S.
0.5,
Exent-Leval
Ars.
Avzu
Params
FLOIS.
Mennery
Infermecce FPS:
Luseacy
VALOK
16:
37.9
35.5
0.250
LEAP
R.
09.3
453
73.3
54.
40.1
42.5
35.
386
14.5
52.05
1.065
206.700 736.MM5
51
NREP
RCC
13.7
514
45.
06.5
$2.7
41.3
18.9
45.5
34.7
18.3
82.3
13.5
9663
188K5
50 90:2M
26.4
37 27-9
MM--CSE CS2 [1
R.C.C.
53.3
56.5
43.9
74.5
70.0
57.5
30.4
50.5
38.4
37.7
46.9
36.2
6.238
3.91
33.00BB
35.1
27.7ta
(Ours)
RCC.C
90.1
513
80.3
75.7
61:1
47.1
60.9
4710
90.6
41.
8:226
7M7MR
10.rm
PreFMr (Ours)
R.C.C.
61.0
69.9
54.6
90.3
738
61.4
48.5
$1.7
43.1
51.0
12.1M
0448
$5
18:7ts
VALOR
R.C.C.
05.5
01.3
50.5
11.4
73.7
02.4
55.1
$6.9
66.7
54.2
06.10
395
0.8565
CaleaFa
R.R.E
60.5
$8.0
52.4
71.7
60.7
49.3
48.3
$3.0
42.1
48.7
51.
42.5
5.738
0.250
114.IMB
60.4
16 16.dts
TRAP 10
R.
616
61.1
56.
80.6
71.
60.3
52.5
56.4
47.9
51.3
59.0
46.7
$3.004
1.003
704 7M
19.1
51 Ses
20
RCC
67.
63-
49-0
66.2
51.0
59.9
57.5
49.8
56.7
9.8
1.893
R.C.C.
57.6
74.8
61.7
56.8
47.3
56.0
3991
27.7ts
by randomly concatenating clips to simulate the rapid scene variations often encountered in online streaming content; the other half are formed by concatenating clips from the same event category to represent longer, continuous event occurrences. Following recent works 29 77 9 49
27
segment-wise pseudo labels from CLIP 46 22
and CLAP g
are used for supervision.
Metric For model performance, follow prior work 58 using Fi-score and mean Average Precision (mAP) as segment-level metrics. For event-level evaluation, consecutive positive segments treated as complete event instance. We calculate event-level Fl-scores by setting tiou 10.1:0.1:0 1:0.9] 13 and average F1-score (Avg Fl F1-score) for overall performance. For the On-AVVP task, we adhere to the established protocol from VALOR 22)] evaluating audio-only (A), visual-only (V), and combined audio-visual (AV, denoted with subscript "av") events. Regarding model efficiency. we assess the number of trainable parameters. FLOPs per inference, peak inference memory and FPS.
For both tasks, we set 60 training epochs, with the first 10 epochs dedicated to warm-up. batch size of 128 is used, and Adam W serves as the optimizer with a weight decay of 1e-1 We set the value T. of 10 and I.y of 5 as the default setting. CL.IP [66 and CLAP (8) are used to extract visual and audio features with a temporal stride set to I second, respectively. All experiments are conducted on single RTX 3090. For the learning rate and the hidden dimension within the attention block, we use le -3 and 256 for On-AVEL, 5e -4 and 128 for On-AVVP.
Implementation details
4.2 Comparison with Existing Work
Our method is benchmarked against recent SOTA methods UnAV
UniAV 14 and CCNet 78
on UnAV-100 [13 for the On-AVEL task, while VALOR (29). CoLeaf 49
LEAP 76), NREP 27
and MM-CSE 711 on LLP (54) for the On-AVVP task. We provide two versions of our method: the
basic version "PreFM", and the improved version "PreFM+" with larger hidden size.
Performance Comparison As shown in Table PreFM clearly achieves new SOTA results for On- AVEL task, surpassing the second-best method with significant improvement of +7.8 in mAP and +9.3
7

---

in Avg Fl-score. Furthermore, our enhanced version, PreFM+, extends these gains to +8.3 in mAP and +14.5 in Avg Fi-score with only a moderate increase in parameters, highlighting the excellent scalability of the PreFM architecture for applications requiring higher precision. Similarly for On- AVVP task shown in Table 2 PreFM demonstrates consistent advantages, achieving improvements of +3.8 in mAP, av and +5.0 in Avg F1-score,v and PreFM+ further elevating performance to +3.9 in mAPav av and +6.0 in Avg Fl-scoreav over the second-best methods. Notably, we also present the original offfine results of these baseline methods (marked with "*") to show their performance under full-context conditions. Even when compared to these results, our online PreFM achieves comparable performance despite predicting with limited context.
These substantial performance gains across both tasks are largely attributed our core predictive multimodal future modeling (PMFM) design. By dynamically generating and integrating pseudo- future contextual cues from streaming data, PMFM empowers our method to effectively parse environmental states and accurately capture temporal boundaries.
Efficiency Analysis Regarding the On-AVEL task (Table I. PreFM's efficiency is remarkable. PreFM utilizes merely 2.7% parameters (6.5M vs 238.8M 8M) compared to the next best performing method, and it requires only 0.6% FLOPss (0.4G vs 72. and 4.8% peak memory (56.4MB vs 1/79.4MB for a single inference, while running at an impressive 51.9 FPS with merely a latency of 19.3ms. The compelling efficiency advantage is also evident in the On-AVVP task (Table 2 Such ability to deliver SOTA performance with drastically reduced overhead highlights that PreFM is designed with strong emphasis on practical deployability, rendering it a highly suitable and efficient solution for resource-constrainet real-time applications.
4.3 Ablation Studies
To systematically evaluate the contribution of each proposed component, we conduct comprehensive ablation studies the On-AVEL task, with results presented in Table (a). The simple prediction strategy (row 1) uses only data at time T and performs badly. Our baseline (row 2), which just extends accessible data to context Le but more improvements, achieves an Avg Fi-score of 40.8%. Introducing the pscudo future mechanism (PF, row 3) significantly boosts performance to 42.4 (+1.6 vs baseline), underscoring the importance of future context modeling over relying solely on past or current information. Further incorporating modality-agnostic robust representation (Emrr, row 5) or random segment sampling (RS, row 6) individually builds upon this, yielding Avg F1-scores of 44.2 (+3.4 vs baseline) and 44.0 (+3.2 vs baseline) respectively, demonstrating their distinct benefits. The focal temporal prioritization (wr(t) consistently improves results when applied (e.g., row 4 vs 3, and row 7 vs 5). confirming its effectiveness in focusing the model on critical information at the current moment. Finally, our full PreFM model (row 8), integrating all components, achieves a final Avg Fl-score of 46.3, marking a substantial +5.5 improvement over the baseline and validating the collective effectiveness of our design.
Main Component
Impact of future-oriented losses We investigate the impact of direct future supervision L, and future part of robust representation loss Lmer.f used in pseudo-future (PF) mechanism. Results are shown in Table Bb). A comparison of the first two rows shows that merely incorporating the extra parameters in the future module without applying any future supervision yields negligible performance gains. Conversely, the results in the subsequent three rows indicate that designing losses to explicitly guide the model in anticipating and modeling the future, whether through direct supervision or robust representation distillation, enhances model performance. These findings clearly demonstrate that the performance benefits derived from our useudo-futur mechanism are primarily attributable to the effective learning guided by these targeted future-oriented losses, rather than merely an increase in model capacity.
Impact of temporal-modalee challenging due to inherent predictive noise. Table compares our Temporal-Modality Cross Fusion (TMCF) with ablated variants that utilize only self-attention (Self), audio-visual modality fusion (M only), or temporal-only fusion (T only), focusing on their accuracy in predicting the future (the first three relative time steps) and overall event parsing performance. The inferior performance of these simplified variants underscores that uni-dimensional interactions are insufficient for producing robust future sequences, leaving its reliability and noise levels suboptimal. In contrast, our full TMCF,
cross fusion Generating reliable audio-visual pseudo-future is
8

---

Table 3: (a) Overall ablation study. PF: the pseudo future mechanism, w(t): focal temporal prioritization, Lmrr: modality-agnostee robust representation, RS: random segment sampling. (b) Ablation studies for future-oriented losses. (c) Ablation studies for temporal-modality cross fusion. Params: trainable parameters, S-L.: Segment-Level, E-L.; Event-Leevel
PF
xt())
Ener
RS
mAP
S-T.
Aog
E-I.
Tx1
S-LFI
T12
Tv3
Avg
E-L
(2)
(1)
Simple Predictions
69.1
66.6
80.8.u
29.9
L
PF
Params
S-L
E-L Avg
Self
55.4
54.6
53.7
44.6
43.0az
K
x
2.6M
69.8
44.7
only
45.3
15)
44.8.n:
(6)
x
70.5
44..
X
/
6.5M
69.9
45.2 as
M only
56.6
55.7
55.1
45.0
(7)
69.4
45.4.4
X
6.5M
69.5
45.5.ma
(81
V
/
70.1
46.3.
6.5M
70.1
46.3.1.
TMCP
(a)
(b)
(c)
Table (a) Ablation studies for different length of past and future. (c) Ablation studies for different pre-trained teacher models.
Seg-Level Fl
Event-Level
Le
l
mAP
0.5
Avg
Seg-Level F1
Event-Level
(1)
10
58.8
70.1
46.8
45.9
Models
Dimensions
mAP
0.5
Avg
(2)
59.1
70.1
46.9
46.3
(3)
10
57.3
69.9
46.4
45.4
AudioClip 20
1024
58.7
70.2
46.6
46.2
(4)
57.2
69.9
43.5
ImageBind
16
1024
58.3
70.0
47.2
45.9
(5)
48.9
65.7
38.9
38.5
ONE-PEACE S9
1536
59.1
70.1
46.9
46.3
(a)
(b)
by collaboratively leveraging both cross-modal and cross-temporal interactions from available content, generates more accurate and dependable pseudo-future sequences. This results higher-quality predictive context that more effectively mitigates noise and aids robust real-time event parsing.
Impact of context lengths L. and Ly We investigate the impact of varying lengths for the working area Le and the aseudo-future sequence Ly. with results presented in Table 4a). The optimal performance, achieving an Avg Fi-score of 46.3, is obtained with our default configuration of Le 10 and Ly 5 (row 2). Analysis of L, (rows 1, 2, 3. with L. 10 fixed) indicates that while a very short future window (L, = 1) provides insufficient predictive insight, an overly long one (L, 10) can introduce distracting noise. both degrading performance. Similarly, examining La (rows 2, 4, 5, with Ly 5 fixed) reveals that too little historical context (Le 5) offers inadequate support, whereas excessive history (I. 20) may include outdated or irrelevant information. These findings confirm the importance of appropriately sized context windows, with L. = 10 and Ly = providing the most effective balance for the immediate event parsing task.
Different teacher models in MRR We evaluate the influence of different pre-trained teacher models our mggnoicy- robust representation (MRR) module. Specifically, compare OnePeace [59], ImageBind [66,, and AudioClip [20] across multiple metrics. As the results demon- strate in Table (b). no single model consistently outperforms the others on every measure. However, OnePeace delivers better segment-level Fl-scores and average event-level performance, which lead us adopt as our default teacher.
Temporal impact of the pseudo-future Figure 4[a) illustrates how our pseudo future (PF) mechanism affects prediction accuracy across time steps relative to the current moment T. From the orange line, we observe that the model's 's peak performance occurs significantly earlier (around relative time T 6), with accuracy declining as approaches T. indicating a strong reliance on full context. In contrast, the purple line shows that incorporating the PF not only achieves generally higher accuracy but also shifts its performance peak much closer the actual target time (around T - 2). These observations underscore fundamental principle in event parsing: accurate event identification intrinsically depends on a comprehensive contextual window. Thus, the reliance on future context presents significant hurdle for online systems. Our PF mechanism effectively

---

62
Dag barking
sf PF
60
vio PF
howlinx
Shoup bleating
81
47
Frog croaking
Bull belloning,
chirping
rearing
Horse
Relative time steps
w/ MRR
w/o MRR
Figure 4: (a) The performance across different relative time steps. (b) t-SNE visualization of the pre-classifier features. We use nine animal events from UnAV-100 [3 for better illustration.
28x
GT
UnAV
MM-CSE
CCNet
PreFM
PreFM
plating
playtng trampet
plasing trasslosc
12c231 cleaner
Strging
gaiter
Iuajo
checring
Figure 5: (a) The visualization on the On-AVEL task. (b) The visualization on the On-AVVP task. GT: ground truth. The red dotted box indicates the area of mispredictions.
anticipates event trends, models and utilizes audio-vist visual future information to enhance prediction accuracy near the present moment, thereby mitigating immediate contextual limitations.
qualitatively evaluates our modality- agnostic robust representation (MRR) via t-SNE visualization of latent features from nine predefined animal events from UnAV-100 13] With MRR, event classes form more compact and well-separated clusters, unlike the more chaotic clusters from the model without MRR. This suggests that while MRR may shift the latent space, it guides the model towards more discriminative representations, enhancing event separability and overall performance.
Impact of MRR on latent feature representation Figure bb)
4.4 Qualitative Analysis
Figure 5 presents a qualitative comparison our PreFM with SOTA methods UnAV [3) CCNet 77
On-AVEL task and MM-CSE [71] On-AVVP tasks. Prior methods often exhibit limitations such as missed detections (e.g., "trombone" by UnAV, "piano" by CCNet, "cheering" by MM-CSE), fragmented predictions (e.g., "trumpet" by UnAV) depicted by red dotted box. In stark contrast, PreFM's predictions exhibit strong temporal continuity and precise event boundary localization, without the interruptions or errors in other methods. These visualizations intuitively showcase PreFM's enhanced recognition accuracy and the coherent, continuous nature of its event parsing.
Conclusions
In this work, we introduce online audio-visual event parsing enable real-time multimodal event understanding in streaming videos. We identify accurate online inference and real-time efficiency as two erucial capabilities this setting, and propose the PreFM framework, featuring a novel predictive multimodal future modeling to infer future context and modality-agnosti robust representation together with focal temporal prioritization for model's generalization. Extensive experiments on UnAV-100 100 and LLP datasets validate that PreFM significantly outperforms prior methods, achieving whateoof-the-ar performance while offering a superior balance between accuracy and computational efficiency, thus presenting viable solution for practical real-time multimodal applications.
10

---

Acknowledgments and Disclosure of Funding
This work is supported by the National Natural Science Foundation of China (No. 92470203, U23A20314), the Beijing Natural Science Foundation (No. L242022), and the Fundamental Research Funds for the Central Universities (No. 2024XKRC082).
References
[1] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-6308,
2017.
[2] Hao Chen, Yuqi Hou, Chenyuan Qu, Irene Testini, Xiaohan Hong, and Jianbo Jiao. 360+ panoptic multi-modal scene understanding dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19373-19382, 2024.
[3] Joya Chen, Zhaoyang Lx, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-online Online video large language model for streaming video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 18407-18418 2024.
[4] Yaru Chen, Ruohao Guo, Xubo Liu, Peipei Wu, Guangyao Li, Zhenbo Li, and Wenwu Wang. Cm-pie: Cross-modal perception for interactive-enhanced audio-visual video parsing. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8421-8422 IEEE, 2024.
[5] Haotian Cui, Alejandro Tejada-Lapuerta, Maria Brbić, Julio Saez-Rodrigue, Simona Cristea, Hani Goodarzi, Mohammad Lotfollahi, Fabian J Theis, and Bo Wang. Towards multimodal foundation models in molecular cell biology. Nature, 640(8059):623 633, 2025.
[6] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egacentric vision: Collection, pipeline and challenges for epic-kitchens-106 International Journal of Computer Vision, pages 1-23, 2022.
[7] Roeland De Geest, Efstratios Gavves, Amir Ghodrati, Zhenyang Li, Cees Snoek, and Tinne Tuytelaars. Online action detection. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 14. pages 269-284, Springer, 2016.
[8] Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acousticy, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[9] Yingying Fan, Yu Wu, Bo Du, and Yutian Lin. Revisit superve-d audio-visual video parsing from the language perspective. Advances in Neural Information Processing Systems, 36:40610 106622--4 2023.
[10] Jie Fu, Junyu Gao, Bing-Kun Bao, and Changsheng Xu. Multimodal imbalance-aware gradient modulation for weakly-supervised audio-visual video parsing. IEEF Transactions Circuits and Systems for Video Technology, 346))4483--4856, 2023.
[11] Junyu Gao, Mengyuan Chen, and Changsheng Xu. Collecting cross-modal -assnnceabbeece evidence for weakly-supervised audio-visual event perception. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18827-18836, 2023.
[12] Junyu Gao, Mengyuan Chen, and Changsheng Xu. Learning probabilistic presence-absence evidence for wcakly-supervised audio-visual event perception. IEEE Transactions Pattern Analysis and Machine Intelligence, 2025.
[13] Tiantian Geng, Teng Wang, Jinming Duan, Rummin Cong, and Feng Zheng. Dense-localizing audio-visual events in untrimmed videos: A large-scale benchmark and baseline. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22942-229511 2023.
[14] Tiantian Geng, Teng Wang. Yanfu Zhang, Jinming Duan, Weili Guan, and Feng Zheng. Uniav: Unified audio-visual perception for multi-task vidco localization. arXiv e-prints, pages ar arXvv-2404 2024.
[15] Tiantian Geng. Jinrui Zhang, Qingni Wang, Teng Wang. Jinming Duan, and Feng Zheng. Longvale: Vision-audio languooe--nuuggnaaa event benchmark towards time-awar aware omni-modal perception of long videos. arXiv preprint arXi::241..19772 2024.

---

[16] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin. and Ishan Misra. Imagebind: One embedding space bind them all. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15180-15190, 2023.
[17] Mohammed Guermal, Abid Ali, Rui Dai, and Francois Bremond. Joadaa: joint online action detection and action anticipation. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 1889-6898, 2024.
[18] Hongji Guo, Hanjing Wang, and Qiang Ji. Bayesian evidential deep learning for online action detection. In European Conference Computer Vision, pages 283-301. Springer, 2024.
[19] Yuxin Guo, Siyang Sun, Shuailei Ma, Kecheng Zheng, Xiaoyi Bao, Shijie Ma, Wei Zou, and Yun Zheng. Crossmae: Cross mooss-modality masked autoencoders for region- aware oudio-visual pre- pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2024.
[20] Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-20222 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 976-980. IEEE, 2022.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pallern recognition, pages 770-778, 2016.
[22] Xingjian He, Sihan Chen, Fan Ma, Zhicheng Huang, Xiaojie Jin, Zikang Liu. Dongmei Fu, Yi Yang. Jing Liu, and Jiashi Feng. Vlab: Enhancing video language pre-training by feature adapting and blending. IEEE Transactions on Multimedia, 2024.
[23] Xiang He, Xiangxi Liu, Yang Li, Dongcheng Zhao, Guobin Shen, Qingqun Kong, Xin Yang. and Yi Zeng. Cace-net: Co-guidance attention and contrastive enhancement for effective audio-visual event localization. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 985-993, 2024.
[24] Shawn Hershey, Sourish Chaudhuri. Daniel PW Ellis, Jort Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. Can architectures for large-scale- audio classification. In 2017 ieee international conference on acoustics, speech and signal processing (icassp). pages 131 IERF, 2017.
[25] Wenxuan Hou, Guangyao Li, Yapeng Tian, and Di Hu. Toward long form audio-visual video understanding. ACM Transactions on Multimedia Computing, Communications and Applications, 20(9):1-28 2024.
[26] Chao Huang, Yapeng Tian. Anurag Kumar, and Chenliang Xu. Egocentric audio-visual object localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22910- 22921, 2023.
[27] Xun Jiang. Xing Xu, Liqing Zhu, Zhe Sun, Andrzej Cichocki, and Heng Tao Shen. Resisting noise in pseudo labels: Audible video event parsing with evidential learning. IEEE Transactions on Neural Networks and Learning Systems, 2024.
[28] Yuanyuan Jiang, Jianqin Yin, and Yonghao Dang. Leveraging the video-level semantic consistency of event for audio-visual event localization. IEEE transactions on multimedia, 26:4617-4627, 2023.
[29] Yung Huug- Lai, Yen-Chun Chen, and Frank Wang. supervised audio-visual event parser. Advances Neural Information Processing systems, 36:73633- 73651, 2023.
teachers meet weakly-
[30] Yung- Hsuan Lai, Janck Ebbers, Yu- Yu-Chiang Frank Wang, François Germain, Michael Jeffrey Jones, and Moitreya Chatterjee. Uwav: Uncertainty Uncertainty-weightee veakly-supervised audio-visual video parsing. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 33561113770 2025.
[31] Guangyao Li, Henghui Du, and Di Hu. Boosting audio visual question answering via key semantic-awarc cues. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 5997- 6605, 2024.
[32] Peizhen Li, Longbing Cao, Xiao-Ming Wu, Xiaohan Yu, and Runze Yang. Ugotme: An embodied system for affective homan-robot interaction. arXiv preprint 4r 18373, 2024.
[33] Yifei Li. Junbo Niu, Ziyang Miao, Chunjiang Ge, Yuanhang Zhou, Qihao He, Xiaoyi Dong, Haodong Duan, Shuangrui Ding. Rui Qian, et al. Ovo-bench: How far is your video-Ilms from real-world online video understanding? arXiv preprint 2.01 2025.
[34] Zhangbin Li, Dan Guo, Jinxing Zhou, Jing Zhang, and Meng Wang. Object-aware craptive-positivigg learning for audio-visual question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 3306-3314, 2024.
12

---

[35] Yan-Bo Lin and Gedas Bertasius. Siamese vision transformers are scalable audio-visual learners. In European Conference Computer Vision, pages 303-321. Springer, 2024.
[36] Yan-Bo Lin, Yi-Lin Sung. Jie Lei, Mohit Bansal, and Gedas Bertasius. Vision transformers are parameter- efficient audio-visual learners. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2299-2309, 2023.
[37] Wuyang Liu, Yi Chai, Yongpeng Yan, and Yanzhen Ren. Audio-visual event localization on portrait mode short videos. arXiv preprint arXiv: 06884, 2025.
[38] Xiaohao Liu, Xiaobo Xia, See-Kion Ng, and Tat-Seng Chua. Continual multimodal contrastive learning. arXiv preprint arXiv:2503. 14963, 2025.
[39] Yin Liu, Qin Wu, Mingyong Zeng, Yahan Liu, and Yuying Pan. Fasten: Video event localization based on audio-visual feature alignment and multi-scale temporal enhancement. IEEE Signal Processing Letters, 2025.
[40] Fan Ma. Xiaojie Jin, Heng Wang, Yuchen Xian, Jiashi Feng, and Yi Yang. Vista-Ilama: Reliable video narrator via equal distance visual tokens. arXiv preprint arXiv:2312.08877 2023.
[41] Tanvir Mahmud and Diana Marculeseu. Ave-clip: Audioclip-basee multi-window temporal transformer for audio visual event localization. In Proceedings of the IEEE/CVF Winter conference on applications of computer vision, pages 5158 1188-5167,22 2023.
[42] Shentong Mo and Pedro Morgado. A unified audio-visual learning framework for localization, separation, and recognition. In International Conference on Machine Learning, pages 25006-25017. PMLR, 2023.
[43] Shentong Mo, Weiguo Pian, and Yapeng Tian. Class-incremennal grouping network for continual audio- visual learning. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7798, 2023.
[44] Zhanzhong Pang, Fadime Sener, and Angela Yao. Context-echanced memory-refined transformer for online action detection. arXiv preprint arXiv:2503. 183359 2025.
[45] Weiguo Pian, Shentong Mo, Yunhui Guo, and Yapeng Tian. Audio-visual class-incremental learning. In Proceedings of the IEFE/CVF International Conference on Computer Vision, pages 7799-7811, 2023.
[46] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PmL.R, 2021.
[47] Zhongwei Ren, Yunchao Wei, Xun Guo, Yao Zhao, Bingyi Kang, Jiashi Feng, and Xiaojie Jin. Vidcoworld: Exploring knowledge learning from unlabeled videos. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 29029 29039, 2025.
[48] Sakib Reza, Yuexi Zhang, Mohsen Moghaddam, and Octavia Camps. Hat: History-augmentee anchor transformer for online temporal action localization. In European Conference Computer Vision, pages 205-222. Springer, 2024.
[49] Faegheh Sardari, Armin Mustafa, Philip JB Jackson, and Adrian Hilton. Coleaf: A contrastiv--collaarrtive learning framework for weakly supervised audio-visual video parsing. In European Conference on Computer Vision, pages 1-17. Springer, 2024.
[50] Alvin Shek, Bo Ying Su, Rui Chen, and Changliu Liu. Learning from physical human feedback: an object-centric one-shot adaptation method. In 2023 IEEE international Conference on Robotics and automation (ICRA), pages 9910-9916. IEEE, 2023.
[51] Youngkil Song. Dongkeun Kim, Minsu Cho, and Suha Kwak. Online temporal action localization with memory-augmented transformer. In European Conference on Computer Vision, pages 74-91. Springer, 2024.
[52] Yunlong Tang, Daiki Shimada, Jing Bi, Mingqian Feng, Hang Hua, and Chenliang Xu. Empowering IIms with pseudo untrimmed videos for audio-visual temporal understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7293-7301, 2025.
[53] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In Proceedings of the European conference on computer vision (ECCV), pages 247-263 2018
13

---

[54] Yapeng Tian, Dingzeyu Li, and Chenliang Xu. Unified multisensory perception: Weakly-supervised audio-visual video parsing. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part III 16, pages 436- Springer, 2020.
[55] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look spatiotemporal convolutions for action recognition. Proceedings of the IEFE conference on Computer Vision and Pattern Recognition, pages 6450-6459, 2018.
[56] Elahe Vahdani and Yingli Tian. Deep learning based action detection in untrimmed vidcos: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4302-45 4320, 2022.
[57] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[58] Jiahao Wang, Guo Chen, Yifei Huang, Limin Wang. and Tong Lu, Memory-and-aaticipatinn transformer for online action understanding. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 13824- 13835, 2023.
[59] Peng Wang, Shijie Wang. Junyang Lin, Shuai Bai, Xiaohuan Zhou, Jingren Zhou, Xinggang Wang, and Chang Zhou. One-peace; Exploring one general representation model toward unlimited modalities. arXiv preprint arXiv: 11172, 2023.
[60] Yi Wang. Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei. Rongkun Zheng, Zun Wang, Yansong Shi, et al. Internvide02: Scaling foundation models for multimodal video understanding. In European Conference Computer Vision, pages 396-416. Springer, 2024.
[61] Yi Wang. Jilan Xu, Yinan He, Zifan Song, Limin Wang. Yu Qiao, Cairong Zhao, et al. Does video-text pretraining help open- vocabbully online action detection? Advances in Neural Information Processing Systems, 37:47908- 47930, 2024.
[62] Yihan Wu, Yifan Peng, Yichen Lu, Xuankai Chang, Ruihua Song, and Shinji Watanabe. Robust audiovisual speech recognition models with mixture-of-expers. 2024 IEEE Spoken Language Technology Workshop (SLT), pages 43-48. IEEE, 2024.
[63] Yan Xia and Zhou 7hao. Cross-modal background suppression for audio-visual event localization. In Proceedings of the IEFF/CVE conference on computer vision and pattern recognition, pages 19989-19998, 2022.
[64] Ling Xing, Hongyu Qu, Rui Yan, Xiangbo Shu, and Jinhui Tang. Locality awarc moos-modall correspon- dence learning for dense audio-visual events localization. arXiv preprint arXiv:2409.09667, 2024.
[65] Yi Xu, Yuxin Hu, Zaiwei Zhang, Gregory P Meyer, Siva Karthik Mustikovela, Siddhartha Srinivasa, Eric M Wolff, and Xin Huang. Vlm-ad: End-to-end autonomous driving through vision-language model supervision. arXiv preprint urXiv:2412.144: 144466, 2024.
[66] Jiashuo Yu, Ying Cheng, Rui-Wei Zhao, Rui Feng. and Yuejie Zhang. Mm-pyramid: Multimodal pyramid attentional network for audio-visual event localization and video parsing. In Proceedings of the 30th ACM international conference on multimedia, pages 6241-6249, 2022.
[67] Haomiao Yuan, Yi Chen, Zheyan Ji, Zhichao Zheng, Yanhui Gu, and Junsheng Zhou. Throughout procedural transformer for online action detection and anticipation. IEEE Transactions on Circuits and Systems for Video Technology, 2025.
[68] Boqiang Zhang, Kehan Li. Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng. Yuming Jiang. Hang Zhang, Xin Li, al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.1 13106. 2025.
[69] Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash-vstream: Memory-based real-time understanding for long video streams. arXiv preprint arXiv:2406.00 08085, 2024.
[70] Jimuyang Zhang, Zanming Huang, Arijit Ray, and Eshed Ohn-Bar. Feedhack-guided autonomous driving. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 15000- 15011, 2024.
[71] Pengcheng Zhao, Jinxing Zhou, Yang Zhao, Dan Guo, and Yanxiang Chen. Multimodal class-aware semantic enhancement network for audio-visuu visual video parsing. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 10448- 10456, 2025.
14

---

[72] Xianbing Zhao, Soujanya Poria. Xuejiao Li, Yixin Chen, and Buzhou Tang. Toward robust multimodal learning using multimodal foundational models. arXiv preprini 1rX1: 2024.
[73] Yue Zhao and Philipp Krähenbühl. Real-time online video detection with temporal smoothing transformers. In European Conference Computer Vision, pages 485-502. Springer, 2022.
[74] Jinxing Zhou, Dan Guo, and Meng Wang. Contrastive positive sample propagation along the audio-visual event line. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(6):7239-7257, 2022.
[75] Jinxing Zhou, Dan Guo, Ruohao Guo, Yuxin Mao, Jingjing Hu, Yiran Zhong, Xiaojun Chang, and Meng Wang. Towards open-vocabulary audio-visual event localization. arXiv preprint arXiv:2411.11278, 2024.
[76] Jinxing Zhou, Dan Guo, Yuxin Mao, Yiran Zhong, Xiaojun Chang, and Meng Wang. Label-anticipated event disentanglement for audio-visual video parsing. In European Conference on Computer Vision, pages 51. Springer, 2024.
[77] Jinxing Zhou, Dan Guo, Yiran Zhong, and Meng Wang. Advancing weakly-supervised audio-visual video parsing via segment-wise pseudo labeling. International Journal of Computer Vision, 132(11):5308-5322 2024.
[78] Ziheng Zhou, Jinxing Zhou, Wei Qian, Shengeng Tang. Xiaojun Chang, and Dan Guo. Dense audio- visual event localization under cross-moda consistency and multi-temporal granularity collaboration. In P Proceedings of the AAAI Conference Artificil Intelligence, pages 10905-10913, 2025.
[79] Yongshuo Zong, Oisin Mac Aodha, and Timothy Hospedales. Self-supervised multimodal learning: survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our contributions are outlined in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of this paper are outlined in Sec A.1
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
This paper does not involve theoretical results, assumptions or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide detailed information the experimental setup in Sec 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification: We submit our source code in the supplementary material and provide detailed information on the experimental setup in Sec 4.1
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm co/
public/guides ottp://eeeoooouuubeaesssoooooooo
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https
/twps.cccpub.l... wttee//guideeeeoeeuueeeee ssincolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the source code in the supplementary material, detail the exper- imental settings Sec. 4. outline the training and inference details of our method in Sec. [.6] and further supplemented additional online inference details in Sec. A.3
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Following previous works [29 bars statistical information.
76
13
14
78
we do not report extra eтror
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The experiments in our method can be completed on a 3090 GPU. We also provide the model parameters and computational complexity of the experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We strictly adhere to the NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The broader impacts of this paper are outlined in Sec A.1
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Our paper the open-source code assets from VALOR 29 and the public dataset UnAV-100 [3 and LLP [54).
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
21

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Our method does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy huttep//hearrppsccc//ooorrrrreee for what should or should not be described.
22

---

A Technical Appendices and Supplementary Material
A.1 Limitations and Broader Impact
Limitations While PreFM demonstrates promising results in online audio-visual event parsing, identify a couple of key avenues for future exploration and enhancement. Firstly, the current PreFM design primarily tailored for event detection and localization. Further research can extend its capabilities more complex, semantically rich tasks such video question answering detailed captioning, and enhance its capacity for long-range temporal reasoning, potentially through integration with large language models. Secondly, while PreFM's predictive modeling of pseudo-future context is a key component for enhancing online inference, the inherent nature of future prediction means that the generated cues may not always perfectly foresee subsequent events. Although our temporal- modality cross fusion (TMCF) (detailed in Sec. 3.3 is designed to refine these predictions and mitigate potential noise by leveraging cross-modal
cross-temporal interactions with its positive impact analyzed in Sec. 4.3 the noise may somewhat degrade performance. While TMCF offers an initial solution, further research can be developed to enhance the reliability and effectiveness of the future.
Broader Impact Our work on online audio-visual event parsing, using methods like PreFM, can greatly improve real-time Al systems. However, we must think carefully about serious ethical issues when using audio and video data. Important issues include protecting people's ' privacy from unwanted watching or access, reducing unfair biases that the Al might learn from its data should be considered.
A.2 Additional Results and Analysis
Quantitative Analysis of the Pseudo-Future's Quality We evaluate them from two perspectives, their semantic similarity to ground-trute truth event features, and their effectiveness in predicting future events: Top-k Similarity Accuracy: We measure if generated feature vector at relative future time step (T+1-T+5) the Top-1 or Top-5 closest match its corresponding ground-truth class feature embedding, among all 100 classes in UnAV-100. Future Prediction Fl F'-Score: We also report the standard segment-lewe Fl-score for predictions made for future time steps. The results are presented in the Table 5
This results provides two key insights: First, the pseudo-future features remarkably realistic. The Top-5 Similarity Accuracy of over 94% demonstrates that the correct event feature is almost always ranked among the top candidates. Second, these high-quality features enable strong future prediction performance, as evidenced by the solid Fi-scores.
Table 5: The quantitative results of the pseudo-future's 's quality.
Metric
T+1
T+2
T+3
T+4
T+5
Top-1 Similarity
45.4
44.6
44.0
43.3
42.6
Top-5 Similarity
95.5
95.3
95.1
94.8
94.3
Fl
57.5
56.5
55.4
54.7
54.1
Ablation study on hyperparams The ablation study on the loss weighting hyperparameter is shown in Table [6] The results indicate that performance diminishes at the tested extreme values (A = 0.1 and A 10), while the model exhibits stable and strong performance across moderate range (from 0.5 to À = 2). Therefore, we adopt as the default setting our experiments for simplicity.
23

---

Table Ablation studies for hyperparams, loss weight X.
Seg-Level F1
Event-Level
mAP
0.5
Avg
0.1
58.3
70.8
46.9
45.8
0.5
58.9
70.2
47.2
46.2
59.1
70.1
1.5
58.4
69.9
47.2
46.2
2
59.2
70.0
47.5
46.6
5
55.9
68.9
44.3
43.7
13.2
50.8
9.4
9.5
Different feature extractors Table 7r the ablation study on different feature extractors, eval- uating their impact on the performance and efficiency of On-AVEP task. The results clearly indicate that employing more powerful foundation models as feature extractors generally leads to significant improvements in parsing performance. Specifically, while the I3D VGGish 22 combination relatively lightweight, its performance is comparatively limited. In contrast, AudioClip (20] and
CLIP 4CPP 8 offer a favorable balance between performance and computational efficiency, Although OnePeace [59] achieves the best parsing results, its substantial computational requirements may hinder its practical deployment in real-world scenarios. Notably, the computational complexity of our proposed PreFM module remain relatively stable and low across all tested feature extractors. This underscores that the feature extraction stage constitutes the primary performance bottleneck and source of computational load, directly impacting the system's 's online processing capabilities.
Table 7: Ablation studies for different feature extractors. audio extractor part, visual extractor part.
Seg- I Level F1
Event-L Level
Dimensions a
HLOPS]
Methods
mAP
Avg
V
PreFM
I3D VGGish
24
30.7
48.7
23.7
24.0
128
2048
0.9G
3.5G
0.3G
AudioClip
48.0
63.6
37.7
37.0
1024
1024
2.7G
5.4G
0.1G
CLIP A5 4CLAP
59.1
70.1
46.9
46.3
768
768
23.1G
77.8G
0.5G
PEACE 59
62.4
70.6
52.2
51.5
1536
1536
78.8G
389.8G
0.4Gi
Failure cases
The quantitative findings and analysis about failure cases are shown below:
Confusion Between Similar Events We analyze the events with the lowest performance and their most common confusions in Table This results reveal that PreFM struggles to distinguish between events that are semantically acoustically similar. We hypothesize this is because current framework does not explicitly incorporate a contrastive learning design to better separate the representations of events originating from similar audio-visual sources.
Table 8: The quantitative analysis of confusion between similar events.
Event
Precision
Recall
F1
Most confused with
People slurping
0.55
0.17
0.25
People eating, man speaking Baby laughter, engine knocking
People shouting
0.42
0.19
0.27
Performance in Dense Scenes We analyze the impact of event density (the number of event classes within a video) on event-level performance in Table These results show that PreFM's performance degrade in complex videos containing a large number of distinct event classes. This suggests that while our future modeling effective, benefits are less pronounced in scenarios with very rapid scene changes and drastic context shifts.
24

---

Table The quantitative analysis of PreFM in dense scenes.
Num events
Event-Level Avg
1-3
0.54
4-6
0.23
>6
0.13
A.3 Additional Details
The detailed process of modality-agnostic representation refinement (MRR) For the MRR process, select OnePeace [59] the pre-trained teacher model. The generation of target teacher features fe at each time step f involves the following steps: First, ground-truth event labels ye are converted into textual prompts using the template "a/an aaadto//visalauuiovvvvvivoa These prompts are then processed by the OnePeace text encoder to yield the modality-agnoste event features. If multiple events active time t, the final fa is computed by averaging the features corresponding all active event classes. Concurrently, the student model's representation fe is prepared. We extract the audio features fr and visual features fr from our model at time specifically from the layer before the final classification head k(-). These extracted features are subsequently concatenated in feature dimension form the student's representation: fax Concat(
event of [cls]"
Specific values for focal temporal prioritization As detailed in Eq. and Eq. 10 our focal temporal prioritization designed to emphasize predictions closer to the current time T while maintaining the overall loss scale. This scale preservation ensures that the sum of weights for the context window, -t=T-L,+] w.((),, equals Le, and for the future window, ΣTT4/ wy((), equals Tys Table TO presents the specific numerical values of these weights for each relative time step, calculated with our default settings of Le 10 and L = 5.
Table 10: Specific values of the focal temporal prioritization for current time steps (t e [T' 9,
and future prediction time steps (t E [T' + 1, 5)).
Current
Future
Time step
T-9
T-8
T-7
T-6
T-5
T-4
T-3
T-2
T
T+1
T42
T+3
T+4
T+5
Weight value
0.76
0.83
0.89
0.95
1.01
1.06
1.09
1.12
1.14
1.14
1.12
1.10
1.03
0.94
0.81
Difference between random segment sampling and normal sampling The details of our random segment sampling strategy are described in Sec. (36] In contrast, normal sampling strategy just involves dividing a video of total length Taul into sequence of l: non-overlapping chunks, each of length L_. For such a normal approach, the target prediction time Tę for each k-th chunk is deterministically set to its final time step, specifically defined as Tr = kLe - 1. This means that predictions are consistently targeted only at the very end of these fixed chunks, unlike the more varied and diverse target prediction times generated by our random segment sampling method.
Dataset modification for online setting For the UnAV-100 dataset [3] while its original anno- tations specify continuous time segments for events (e.g.. [dss Tstart, Tend)). we convert these into frame-level discrete labels for our online task. Specifically, for any given time T and particular event within video stream, label of indicates the event is currently occurring, while 0 indicates not.
25

---

Clncea
11,849 videos in LLP
11,849 concatenated videos in LLP
Figure 6: Visualization the LLP dataset adaptation for the online setting. Various colored rectangles represent video clips of different categories. Video clips are processed by random concatenation and consistent concatenation.
The LLP dataset [44] initially provides 11,849 10-second video clips. To adapt it for our online evaluation setting, we concatenate 11,849 new, untrimmed video streams as shown in Figure 6 Each new stream is created by using the original 10-second clips as its base and concatenating additional clips. These resulting streams are specifically constructed to achieve one of six distinct target total durations: 10 seconds (representing the original clip itself), 20 seconds, 30 seconds, 40 seconds, 50 seconds, or 60 seconds. Approximately an equal number of streams are generated for each of these six target durations.
This concatenation process employs two distinct strategies: half of the 11,849 streams are formed by random concatenation, randomly combining clips from different categories. This aims to simulate the rapid and complex within-secone scene variations commonly observed in current streaming content. The other half are constructed by consistent concatenation, identifying the first event category present in the base clip and then concatenating multiple additional clips that also contain this specific event, thereby simulating longer videos with a consistent, ongoing event context. This approach allows us to assess the model's 's adaptability to complex dynamic scenes and its capability for consistent understanding and discrimination within extended event contexts.
Regarding other datasets: The case of LFAV The LFAV dataset 225 comprising 5175 untrimmed videos with diverse audio. visual, and audio-visual events, is designed for long-form visiull video parsing and thus appears initially relevant for the On-AVEP task. However, we identify two critical limitations that preclude its effective use with our PreFM framework.
Firstly, complete access to the original video data is restricted. Of the officially stated 3721 training, 486 validation, and 968 test samples, our attempts allow to retrieve only 3512, 461, and 910 samples respectively (totaling 4883 out of 5175 raw videos). The LFAV benchmark provides pre- extracted features using VGGish (24]. ResNet18 PII and R3D [55). This reliance fixed features prevents us from employing different feature extractors or leveraging pre-trained models (such as OnePeace [59] for modality -ggnostic robust representation) directly the raw video data, which is a key aspect of our method.
Secondly, LFAV is curated under weak supervision paradigm, offering only video-level annotations for its training set. The absence of readily available segment-level ground truth makes LFAV unsuitable for training critical components of our PreFM model. Specifically, mechanisms like our predictive future modeling and focal temporal prioritization require finer-grained temporal supervision than what LFAV's training annotations provide, rendering it incompatible with the training requirements for our online streaming prediction approach.
More online inference details All methods are evaluated using their officially provided checkpoints; for those without available checkpoint, reproduce the results using their official code. For any prediction at time T in online testing, only data from 0 to T is available.
pad the entire video beyond T + with zeros as input because these methods originally utilize complete video, and we use this padding to ensure uniform input length under online settings. Our method, in contrast, does not use all available
For On-AVEL tasks, SOTA methods [3 14 78
26

---

historical data up to T; instead, it processes the segment [T - Le + 1,T] as imput to derive the prediction time T.
For On-AVVP tasks, SOTA approaches [29 76 27 71 employ the segment [T - 9, 7 as input,
since these methods designed for 10-second video clips. Similarly, method utilizes the segment [T' Le + 1, L'] as input for making predictions at time T.
Re-evaluation of efficiency metries for fair comparison Table [ the On-AVEL task) and Table 12 (for the On-AVVP task) present side-by-side comparisons of efficiency metrics. These include higures from our standardized re-evaluation (denoted "Our Eval.") and those originally published in the respective papers (denoted as "Reported").
For our evaluations, adhere to strict and consistent protocol. The number of trainable param- eters for all models is calculated as the sum of elements in all parameters requiring gradients (us- ing sum (p. numel() for P in model. parameters () if p. rrequires__gg _grad)). To measure FL.OPs, we consistently employ the thop library for all methods, assessing a single forward pass (via flops,
prof ile (model, inputs= (imput, ))) All our efficiency tests conducted under identical env environmental conditions to ensure reproducibility and fair basis for comparison.
Discrepancies may be observed between our "Our Eval." figures and the "Reported" values from the original publications. Such differences can arise from variations in measurement methodologies, specific versions of libraries used, or the underlying hardware and software environments. We present both sets of values to offer a transparent perspective, respecting the data from original publications while providing benchmark that is directly comparable across methods under our unified testing framework.
Table 11: Comparison of re-evaluated ("Our Eval.") and originally reported ("Reported") efficiency metries on the On-AVEL task. ("-" indicates the metric was not provided in the original paper.
Purams
FL.Ps
Methods
Our Eval.
Reported
Our Eval.
Reported
UnAV
13
(CVPR2023)
139.4M 4M
52.4G
UniAV (Arxiv2404)
130.8M
130M
22.7G
CCNet
78
(AAA12025)
238.8M
72.1G
PrePM
12.3M
none
0.4G
none none
PreFM+
36.9M
none
0.5G
Table 12: Comparison of re-evaluated ("Our Eval.") and originally reported ("Reported")) efficiency metrics the On-AVVP task. "_" indicates the metric was not provided in the original paper.
Params
FLOPS
Methods
Eval.
Reported
Eval.
Reported
VALOR
29
(Neu1ISS2023)
4.9M
5.1M
0.45G
0.45G
Coleaf
49
(BCCV2022)
5.7M
0.25G
48.2G
LEAP
76
(ECCV2024)
52.0M
52.0M
1.09G
0.79G
NREP
S20242
9.6M
9.6M
1.69G
0.37G
MM-CSB
(AAA12025)
6.2M
4.5M
0.91G
0.80G
PreFM
3.3M
none
0.22G
none
PreFM+
12.1M
none
0.48G
none
A.4 More Qualitative Analysis
On-AVEL Figure 7 further qualitative validation our method's on the On-AVEL task through four distinct examples, comparing our results ("Ours"") against the state-of-the-art CCNet CCNet (78 model ("SOTA"). These visualizations collectively demonstrate that our approach consistently yields event localizations that more aligned with the ground truth annotations compared to CCNet.
27

---

GT
3
SOTA
8
Ours
Spreck
Ibb
GT
SOTA
Ours
Clappinza
Siagleg
Spcedh
Ansulic gilar
Vacunia dexneer
Dlgg
Spwck
Figure 8: More visualization results on the On-AVVP task.
GT
SOTA
Ours
poople dapping
sieglng
eleging
115
20
.45
GT
SOTA
Ours
spenking
singing
plaping
playlng
plasing
plasteg
playleg
Figure 7: More visualization results on the On-AVEL. task.
On-AVVP Similarly, provide further qualitative results of method's the On-AVVP task through four distinct examples, comparing our results ("Ours"" against the state-of-the-art MM- CSE 71 model ("SOTA")) The visualization results shown in Figure These visualizations highlight our method's 'ss superior performance in precisely parsing events and reducing errors compared the SOTA model.
This robust handling both unimodal and multimodal event characteristics signifies a key advantage of our approach for the online visial event parsing task.
28


Reviews:
[{"review_id": "RJZLU47Hlt", "reviewer": "Reviewer_UN6v", "summary": "This paper introduces Online Audio-Visual Event Parsing (On-AVEP), a new setting for multimodal video understanding where models must parse audio, visual, and audio-visual events in a strictly online manner. To address the unique challenges of this setting—limited future context and real-time constraints—the authors propose PreFM, a lightweight framework that enhances online inference by predicting useful future multimodal cues and incorporating them during current-step inference. PreFM also introduces a modality-agnostic representation mechanism and a temporal focus strategy to improve both robustness and efficiency. The method shows strong performance with significantly fewer parameters, suggesting its practical utility for real-time applications.", "questions": "Please refer to weaknesses", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 1, "strengths_and_weaknesses": "### Strengths:\n- The method is intuitive and simple to understand.\n- The problem is well motivated, and empirical results support the claims.\n- The method appears to work well in online settings, with low memory footprint.\n\n### Weaknesses and questions:\n- Despite the results, the methodical contribution is quite weak. At its core, the work simply teases out \\(k\\) representations using some \"prototype\" query vectors via attention, as a proxy for the supposed future.\n- The paper uses a series of attention blocks to generate future representations. However, the **order of operations** and **inputs to the UHA blocks in TMCF** appear fairly arbitrary.  \n  Can the authors provide justification for this design? For instance:\n  - What motivates the specific sequence of UHA operations?\n  - Why were those particular inputs chosen?\n  - What happens if you simply apply another round of attention (as in the \"Pseudo future\" section)?\n  \n  It's unclear whether TMCF meaningfully contributes, or if the observed gains come primarily from increased parameter count (due to FFL) and additional attention layers like those used in the \"Pseudo future\" section.\n\n- The paper does not clearly articulate its contributions.  \n  Can the authors explicitly state the contributions and novelties in the introduction?\n\n- While the method does seem effective in online settings with low memory usage, it's important to **contextualize these improvements**.  \n  What were the bottlenecks in prior methods, and how does this work address them differently?  \n  Additionally, can the authors clarify how their parameter count compares to prior art, and justify the *huge* differences?\n\n- What are the implications of the method on training time ?\n\nOpen to increasing my score, provided my queries are addressed", "was_revised": true, "final_justification": "The authors have been able to clarify and respond to the questions I had and I am wlliam to increase my score. I also read through other reviewer comments and their rebuttals to make better judgement overall, and I agree it should be accepted.", "rebuttal": "Thank you for your positive feedback and constructive comments. In regard to the weaknesses and questions raised, we provide our point-to-point responses below:\n\n---\n\n>**W1**:\nDespite the results, the methodical contribution is quite weak. At its core, the work simply teases out (k) representations using some \"prototype\" query vectors via attention, as a proxy for the supposed future.\n\n**A1**: \n\nOur core methodological contribution lies in proposing a novel framework, **PreFM**, which addresses the new and challenging task of **Online Audio-Visual Event Parsing (On-AVEP)** through the Predictive Future Modeling mechanism, not in using query vectors for future prediction.\n\nThe Predictive Future Modeling mechanism **is not simply about \"teasing out representations**\": Learnable queries are trained to actively anticipate plausible future cues based on the current context. This is followed by cross-modal and cross-temporal interactions to augment the initial, potentially noisy pseudo-future, which is then integrated back to enhance the current representations.\n\nThis simple yet effective framework achieves **SOTA performance with high efficiency**: It delivers a **+9.3** Avg F1-score improvement while using only **2.7%** of the parameters of the SOTA model.\n\n---\n>**W2**:\nWhat motivates the specific sequence of UHA operations? Why were those particular inputs chosen? What happens if you simply apply another round of attention (as in the \"Pseudo future\" section)?\n\n\n**A2**:\n\n**1. Justification for the TMCF Design**\n\nTMCF design is not arbitrary but is motivated by the core challenge of refining the noisy initial pseudo-future sequences. The key design principles are:\n\n**Parallel Operation in UHA**: The UHA block processes multiple context sets in parallel (by summing their attention outputs as Eq.1 in our manuscript), not in a rigid sequence.\n\nLogical Flow in TMCF: The whole TMCF process involves (a) Future Augmentation first, followed by (b) Current Refinement. This order is crucial: we must first refine the initial, potentially noisy pseudo-future to make them more reliable. Only then can this higher-quality future context be effectively integrated back to enhance the current representations.\n\n**Principled Input Selection**: The inputs for the UHA block in TMCF are chosen to enable three critical interactions for robust future modeling (as described in Lines 151-156 and Fig.3 in our manuscript):\n\n* Self-Attention: To model the internal temporal structure within the predicted future sequence.\n* Cross-Modal Attention: To ensure the predicted audio and visual futures are coherent with each other.\n* Cross-Temporal Attention: To ground the uncertain future prediction in the reliable, immediate past, thereby calibrating and denoising it.\n\nThis multi-faceted fusion is essential for generating a high-quality, reliable future context.\n\n**2. Contribution of TMCF vs. Another Round of Attention**\n\nTo empirically validate that the gains come from our specific design rather than merely an increased parameter count, we conduct a comprehensive ablation study, expanding upon Tab.3\\(c). We compare our full TMCF against several variants: convert all modality fusion into temporal fusion (T only), convert all temporal fusion into modality fusion (M only) and convert all fusion layer into self-attention (Self), including the reviewer's suggestion of \"simply applying another round of attention\" (Simple Self).\n\n|Methods|Params|T+1|T+2|T+3|Avg F1|\n|-|-|-|-|-|-|\n|Simple Self|4.4M|54.5|53.6|52.8|44.1|\n|Self|6.5M|55.4|54.6|53.7|44.6(+0.0)|\n|M only|6.5M|56.7|56.0|55.5|45.3(+0.7)|\n|T only|6.5M|56.6|55.7|55.1|45.0(+0.4)|\n|TMCF|6.5M|**57.5**|**56.5**|**55.4**|**46.3(+1.7)**|\n\nThe results show that simply adding a round of self-attention  performs poorly, and our full TMCF outperforms the baseline by a significant margin (+1.7 Avg F1) with the same parameter count. This strongly demonstrates that **the performance gain is attributable to the principled fusion mechanism of TMCF, not just the additional parameters.**\n\n---\n>**W3**:\nThe paper does not clearly articulate its contributions.\nCan the authors explicitly state the contributions and novelties in the introduction?\n\n**A3**:\n\nOur main contributions are three-fold:\n\n1. We introduce **Online Audio-Visual Event Parsing (On-AVEP)**, a new paradigm for real-time multimodal understanding. To our knowledge, this is the first work to systematically address the challenge of parsing audio, visual, and audio-visual events from streaming video. We further establish that success in this paradigm requires two critical capabilities: (a) accurate online inference from limited context, and (b) real-time efficiency to balance performance with computational cost.\n2. We propose the **PreFM** framework, a novel and efficient architecture for On-AVEP. PreFM's core innovations include: (a) **Predictive Multimodal Future Modeling** mechanism to overcome the critical problem of missing future context ; and (b) a combination of **Modality-agnostic Robust Representation** and **Focal Temporal Prioritization** to enhance model robustness and efficiency during training, providing an insightful approach to multimodal real-time video understanding.\n3. We establish **new SOTA performance with unprecedented efficiency**. Extensive experiments on two public datasets show that PreFM drastically outperforms previous methods (e.g., **+9.3** Avg F1-score on UnAV-100), while using a fraction of the computational resources (e.g., only **2.7%** of the parameters of the next best model), validating it as a powerful and practical solution.\n\nWe thank the reviewer for this constructive suggestion, and we will add the summary to the end of our introduction in the revised manuscript.\n\n---\n>**W4**:\nWhat were the bottlenecks in prior methods, and how does this work address them differently?\nCan the authors clarify how their parameter count compares to prior art, and justify the huge differences?\n\n**A4**:\n\n**1. Bottlenecks in Prior Methods and Our Solutions**\n\nThe bottlenecks of prior methods in an online setting are two-fold: \n\n* Fundamental Offline Design: These models are designed to process the entire video sequence to leverage global context for video understanding, which is fundamentally infeasible in a real-time streaming scenario. \n* Prohibitive Computational Cost: Their reliance on processing long sequences with heavy model architectures results in high latency and memory usage, failing to meet the demands of real-time applications.\n\nOur PreFM framework is designed to address these bottlenecks:\n\n* **Natively Online Architecture**: PreFM operates on a short, sliding window of past information, making it naturally suited for streaming data.\n* **Context Enrichment via Future Prediction**: Instead of requiring global context, PreFM's core innovation, Predictive Future Modeling, explicitly infers the necessary future cues from the local window. This enriches the limited context for robust real-time parsing.\n* **Training for Efficiency**: This is complemented by Modality-agnostic Robust Representation and Focal Temporal Prioritization, which boost inference accuracy while maintaining a lightweight training process.\n\n**2. Parameter Count and Justification**\n\nWe calculate trainable parameters for all models using a consistent protocol (`sum(p.numel() for p in model.parameters() if p.requires_grad)`) for a fair and transparent comparison. We show the parameter results in Tab.1 and Tab.2 in our manuscript. Below is a summary for the reviewer's convenience:\n\n|On-AVEL|Params↓|\n|-|-|\n|UnAV|139.4M|\n|UniAV|130.8M|\n|CCNet|238.8M|\n|PreFM (Ours)|6.5M|\n|PreFM+ (Ours)|13.8M|\n\n|On-AVVP|Params↓|\n|-|-|\n|VALOR|4.9M|\n|CoLeaF|5.7M|\n|LEAP|52.0M|\n|NERP|9.6M|\n|MM-CSE|6.2M|\n|PreFM (Ours)|3.3M|\n|PreFM+ (Ours)|12.1M|\n\n\nThe significant difference in parameter counts stems from: Prior Methods (e.g., UnAV, CCNet) **rely on a large transformer backbone** to process a very long sequence of video features and capture long-range dependencies across different time spans. PreFM, in contrast, **is built upon a custom-designed, lightweight UHA block**. The purpose of the UHA block is to efficiently perform two targeted tasks on a short history window: fusing current temporal-modal features and intelligently modeling the future. \n\n\n---\n>**W5**:\nWhat are the implications of the method on training time ?\n\n**A5**:\n\n**PreFM offers a compelling advantage in training time.**\n\nWe use the official code to reproduce the second-best baseline methods CCNet on the On-AVEL and MM-CSE on the On-AVVP tasks. These results, including the number of epochs, total training time required to converge, and performance, are summarized in the table below. All tests are conducted on a single RTX 3090 GPU:\n\n|On-AVEL|Epoch|Training time|Avg F1|\n|-|-|-|-|\n|CCNet|45|11h 4mins|37.0|\n|PreFM|60|1h 1min|46.3|\n|PreFM+|60|1h 25mins|51.5|\n\n|On-AVVP|Epoch|Training time|Avg $_{av}$ F1|\n|-|-|-|-|\n|MM-CSE|60|53mins|36.2|\n|PreFM|60|45mins|41.2|\n|PreFM+|60|48mins|42.2|\n\nDue to its lightweight and efficient design philosophy, Our PreFM framework demonstrates a significant advantage in training efficiency."}, {"review_id": "K0qTcQ2Ukn", "reviewer": "Reviewer_VEsL", "summary": "This paper introduces Online Audio-Visual Event Parsing (On-AVEP), a novel paradigm for real-time parsing of audio, visual, and audio-visual events in streaming video content. This paper proposes the Predictive Future Modeling (PreFM) framework that addresses two key challenges: accurate online inference with limited context and real-time computational efficiency.", "questions": "- Prediction Mechanism: Can you provide more technical details about how future audio-visual cues are predicted? What is the prediction horizon and how do you handle prediction uncertainty?\n- Baseline Fairness: How were the baseline methods adapted for online processing? Are you comparing against methods specifically designed for online settings or offline methods retrofitted for streaming?\n- Real-time Performance: Can you provide the concrete real-time performance? What are the actual processing times and computational costs compared to baseline methods in practical streaming scenarios?", "limitations": "The paper lacks sufficient concrete analysis of its primary claimed advantages, particularly regarding the technical details of predictive modeling, component-wise contributions, and quantitative real-time performance validation.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n\nThis paper addresses an important practical problem in real-time multimodal processing. The proposed predictive future modeling approach is novel and the idea of anticipating future multimodal cues to compensate for limited historical context in streaming scenarios is technically sound. The claimed performance gains with fewer parameters suggest good computational efficiency.\n\nWeaknesses\n\n- About predictive multimodal future modeling: This paper lacks sufficient detail about the specific mechanisms of predictive future modeling. How exactly does the model predict future cues? What is the temporal horizon?\n- Real-time Claims: The paper emphasizes real-time efficiency but doesn't provide concrete timing analysis, latency measurements. Claims about \"real-time applicability\" need quantitative support.\n- Missing Ablation Studies: There's insufficient analysis of which components contribute most to the performance gains. The interplay between predictive multimodal future modeling, modality-agnostic representation, and focal temporal prioritization needs better investigation.", "was_revised": true, "final_justification": "I sincerely thank the authors for their patient response and the effort put into the rebuttal. My concerns have been addressed. and I will increase my score. I hope the authors can include the additional experimental results and analysis mentioned above in the paper to further improve its quality.", "rebuttal": "Thank you for your positive feedback and valuable suggestions. In regard to the weaknesses and questions raised, we provide our point-to-point responses below:\n\n---\n>**W1 and Q1**:\nPrediction Mechanism: How exactly does the model predict future cues? What is the temporal horizon? Can you provide more technical details about how future audio-visual cues are predicted? What is the prediction horizon and how do you handle prediction uncertainty?\n\n**A1**:\n\nHere we provide a clearer explanation of the predictive future modeling mechanism, temporal horizon and uncertainty handling:\n\n**1. Prediction Mechanism:**\n\nInitial Future Modeling: We first use a set of learnable query tokens ($Q_a , Q_v$) for future time steps. These queries attend to the current audio-visual features via an attention mechanism to generate an initial, context-aware pseudo-future ($\\tilde{F}_f^a, \\tilde{F}_f^v$ in Eq.3).\n\nRefinement via TMCF: This initial, and potentially noisy, pseudo-future is then refined within our Temporal-Modality Cross Fusion (TMCF) stage. Using unified hybrid attention (UHA) block, this step produces an augmented and more reliable future representation ($\\hat{F}^a_f,\\hat{F}^v_f$ in Eq.4 and Fig.3).\n\nSupervision: The entire model is learned end-to-end, supervised by losses on these future predictions, specifically the classification loss $\\mathcal{L}\\_f$ and the modality-agnostic representation loss $\\mathcal{L}_{mrr}$. This process effectively teaches the model to map the current context to reliable future events.\n\n\n**2. Temporal Horizon**\n\nThe prediction horizon (the length of the pseudo-future window) is a hyperparameter $L_f$ and our default setting is $L_f=5$ seconds. This choice is empirically validated by our ablation study in Tab.4 (in the appendix), which shows that $L_f=5$ achieves the best balance between providing sufficient future context and avoiding excessive predictive noise.\n\n**3. Handling Prediction Uncertainty**\n\nOur framework handles this inherent uncertainty through the **Temporal-Modality Cross Fusion (TMCF)** stage. TMCF refines the initial, potentially noisy pseudo-future by enforcing further cross-modal and cross-temporal interactions (as described in Lines 151-156 and Fig.3 in our manuscript), which can be broken down as follows:\n* Self-Attention: Models the internal temporal structure and consistency within the predicted future sequence.\n* Cross-Modal Attention: Ensures the predicted audio and visual future streams are coherent with each other.\n* Cross-Temporal Attention: Grounds the uncertain future prediction in the more reliable current context, effectively calibrating and denoising it.\n\nThe effectiveness of this design is quantitatively demonstrated in Tab.3(c). The full TMCF achieves higher F1-scores on future predictions (e.g., **57.5** at T+1) compared to ablated versions (e.g., 55.4 with \"Self\" attention only), proving that our fusion mechanism successfully handles the uncertainty and improves the reliability of the pseudo-future context.\n\n---\n\n>**W2 and Q3**:\nClaims about \"real-time applicability\" need quantitative support.: Can you provide the concrete real-time performance? What are the actual processing times and computational costs compared to baseline methods in practical streaming scenarios?\n\n**A2**:\n\nWe conduct a detailed timing analysis of processing latency to further substantiate our real-time claims. For the reviewer's convenience, we present a comprehensive comparison below, including the performance, latency and other efficiency metrics in Tab.1 and Tab.2. All tests are conducted under a single RTX 3090 GPU.\n\n|On-AVEL|Avg F1↑|Params↓|FLOPs↓|Peak Memory↓|FPS↑|Latency↓|\n|-|-|-|-|-|-|-|\n|UnAV|28.6|139.4M|52.4G|764.7MB|10.6|94.3ms|\n|UniAV|30.3|130.8M|22.7G|1020.5MB|15.6|64.1ms|\n|CCNet|37.0|238.8M|72.1G|1179.4MB|7.5|133.3ms|\n|PreFM (Ours)|46.3|**6.5M**|**0.4G**|**56.4MB**|**51.9**|**19.3ms**|\n|PreFM+ (Ours)|**51.5**|13.8M|0.5G|144.2MB|42.0|23.8ms|\n\n|On-AVVP|Avg $_{av}$ F1 ↑|Params↓|FLOPs↓|Peak Memory↓|FPS↑|Latency↓|\n|-|-|-|-|-|-|-|\n|VALOR|33.0|4.9M|0.45G|**20.1MB**|62.2|16.1ms|\n|CoLeaF|29.7|5.7M|0.25G|114.1MB|60.4|16.6ms|\n|LEAP|34.3|52.0M|1.09G|204.7MB|19.3|51.8ms|\n|NERP|33.5|9.6M|1.69G|90.2MB|26.4|37.9ms|\n|MM-CSE|36.2|6.2M|0.91G|33.0MB|36.1|27.7ms|\n|PreFM (Ours)|41.2|**3.3M**|**0.22G**|20.7MB|**94.4**|**10.6ms**|\n|PreFM+ (Ours)|**42.2**|12.1M|0.48G|55.9MB|53.5|18.7ms|\n\nTaking the On-AVEL task as a representative example, we observe that: \n\n* **Low Latency**: PreFM has a latency of only **19.3 ms** per inference. This is approximately 7 times faster than the next best performing method, CCNet, which has a latency of 133.3 ms.\n* **Superior Efficiency**: This real-time performance is achieved while using fewer resources: **+9.3** Avg F1-score with only **2.7%** of the parameters and **0.6%** of the FLOPs compared to CCNet.\n\nIn the revised manuscript, we will add these latency results and a concise discussion to fully substantiate our “real-time applicability” claims.\n\n---\n\n>**W3**:\nMissing Ablation Studies: There's insufficient analysis of which components contribute most to the performance gains. The interplay between predictive multimodal future modeling, modality-agnostic representation, and focal temporal prioritization needs better investigation.\n\n**A3**:\n\n**1. Individual Component Contributions**\n\nTo better isolate which component contributes most, we conduct additional ablation studies by adding each component individually to our baseline model (which uses a context window of $L_c$ but no other improvements). The results are summarized below:\n\n|Components|mAP|Avg F1|\n|-|-|-|\n|Baseline|69.1|40.8 (+0.0)|\n|+Pseudo Future Mechanism ($PF$)|69.7|42.4 (+1.6)|\n|+Focal Temporal Prioritization ($w(t)$)|69.3|41.1 (+0.3)|\n|+Modality-agnostic Robust Representation ($\\mathcal{L}_{mrr}$)|69.2|41.7 (+0.9)|\n|+Random Sampling ($RS$)|69.8|41.8 (+1.0)|\n|+All|70.1|46.3 (+5.5)|\n\nThis analysis clearly shows that **Pseudo Future Mechanism ($PF$) provides the most significant performance gain** when applied individually, underscoring its central role in our framework. Other components like $w(t)$ and $\\mathcal{L}_{mrr}$ also provide substantial improvements.\n\n**2. The Interplay Among Components**\n\nOur original ablation study in Tab.3(a) is designed to reveal the interplay between components. Based on that table and the new results above, we provide a more focused analysis of their synergy:\n\n* **Synergy between $PF$ and $\\mathcal{L}\\_{mrr}$**: $PF$ alone improves the Avg F1-score by +1.6 and $\\mathcal{L}\\_{mrr}$ alone improves by +0.9. When combined $PF$ with $\\mathcal{L}\\_{mrr}$, it yields an improvement of +3.4 (row 5 in Tab.3(a) ). This strong synergistic effect suggests that while the $PF$ mechanism is the primary driver for performance by providing future context, the $\\mathcal{L}_{mrr}$ acts as a powerful regularizer and guide. It enriches the pseudo-future predictions by forcing them to align with the robust, semantically-rich feature space of the teacher model, leading to a more effective and generalizable representation than either component could achieve alone.\n* **$w(t)$ as a Consistent Enhancer**: Adding $w(t)$ to the $PF$ model (row 4 vs. row 3 in Tab.3(a) ) boosts the gain from +1.6 to +2.2. Similarly, adding it to a model with $PF$ and $\\mathcal{L}_{mrr}$ (row 7 vs. row 5 in Tab.3(a) ) elevates the gain from +3.4 to +4.6. This consistent improvement highlights the unique value of $w(t)$: by applying a Gaussian-weighted focus to the loss, it compels the model to prioritize making a precise and robust prediction at the most critical moment—the present $T$—thereby sharpening the final online decision-making process.\n\n---\n>**Q2**:\nBaseline Fairness: How were the baseline methods adapted for online processing? Are you comparing against methods specifically designed for online settings or offline methods retrofitted for streaming?\n\n**A2**:\n\n**1. Justification: Establishing Baselines in a Clear Research Gap**\n\nWe compare PreFM against SOTA offline methods that we adapt for the online setting. This is a deliberate and necessary choice due to a clear research gap in the field because: \n* Existing Audio-Visual Event Parsing methods (like CCNet for On-AVEL, or MM-CSE for On-AVVP) are exclusively offline and designed to process entire videos at once, possessing no native online capabilities.\n* Existing Online Video Understanding methods, conversely, are almost entirely unimodal (vision-only). They focus on tasks like action detection and are not equipped to handle the multimodal audio-visual parsing required by On-AVEP.\n\nGiven this landscape, the most rigorous and scientifically valid approach is to adapt the strongest SOTA methods from the relevant offline domain to create the first powerful set of baselines for this new online task.\n\n**2. Fair Adaptation Protocol for Online Processing**\n\nWe detail the fair adaptation protocol in the appendix (Lines 637-647) in our manuscript. For convenience, we summarize the process here:\n\n* For On-AVEL baselines, which require full-video inputs, we feed the model the available stream up to the current time $T$ and mask all future frames beyond $T$ with zero-padding.\n* For On-AVVP baselines, which operate on fixed-length clips, we feed the model the most recent segment ending at $T$ (e.g., from $T-9$ to $T$).\n\nThis protocol guarantees that all methods, including our PreFM, **operate under the exact same online constraints, creating a fair and valid comparison.**"}, {"review_id": "7C2n2TR3Zx", "reviewer": "Reviewer_No1a", "summary": "This paper proposes a new task, Online Audio-Visual Event Parsing (On-AVEP), which aims to detect audio and visual events in a streaming setting using only past and current information. To tackle the challenges of limited context and real-time efficiency, the authors introduce PreFM, a novel framework built on three key components: (1) Predictive Multimodal Future Modeling (PMFM) to synthesize pseudo-future cues, (2) Modality-agnostic Robust Representation (MRR) via distillation from a frozen teacher, and (3) Focal Temporal Prioritization to emphasize current-time predictions. Extensive experiments on UnAV-100 and LLP datasets demonstrate that PreFM achieves state-of-the-art performance while being highly efficient in terms of parameters and computation.", "questions": "Please refer to the questions mentioned in 'weaknesses' section.", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- **Novel Task Definition:** The formulation of On-AVEP as a real-time, streaming version of AVEP is timely and practically relevant, filling a gap in existing literature that mainly focuses on offline processing.\n- **Strong Efficiency-Performance Trade-off:** Despite its lightweight architecture (e.g., 6.5M parameters), PreFM significantly outperforms prior methods with much larger models, making it highly suitable for deployment in real-time scenarios.\n- **Thorough Empirical Evaluation:** The paper provides comprehensive comparisons, ablations, and qualitative analyses, making the improvements clearly attributable to each design component.\n- **Good Clarity and Writing:** The paper is well-organized and easy to follow, with clear figures and sound motivation for each component.\n\n\n\n## Weaknesses\n\nGiven the aforementioned strengths, I am generally satisfied with this paper and would like to share a few questions and suggestions:\n\n- **Contradictory statements?** Lines 256–257 suggest that the PF module significantly boosts performance, whereas Lines 268–269 state that PF yields negligible gains. It would be helpful to clarify these descriptions to avoid confusion.\n\n- **Failure cases.** While Figure 5 and the supplementary materials present visualizations, the paper would benefit from a more detailed discussion of the failure cases—when and why PreFM might not perform well.\n\n- **Other questions.**\n\n  1. The On-AVVP subtask requires identifying audio-only, visual-only, and audio-visual events. Does the proposed method predict each type of event separately or simultaneously?\n  2. Could the softmax function be used to normalize the weight w(t) across frames in Eq. 9?\n  3. What are the training and inference time costs for the proposed method?\n\n- **Minor issues (do not affect my rating):**\n\n  1. The module name “current enhancement” in Line 157 is inconsistent with “current refinement” shown in Figure 2.\n\n  2. In Figure 2, the \"Focal Temporal Prioritization\" module appears disconnected from the rest of the architecture. It would be clearer to illustrate how this module interacts with others.\n\n  3. It would be helpful to include a diagram (even in the appendix) that explains the training and inference processes described in Section 3.6.\n\n  4. The paper would benefit from citing more traditional works on audiovisual event localization and video parsing, such as: Contrastive Positive Sample Propagation Along the Audio-Visual Event Line, TPAMI 2023. UWAV: Uncertainty-weighted Weakly-supervised Audio-Visual Video Parsing, CVPR 2025\n\n     \n\n####", "was_revised": true, "final_justification": "The authors have addressed my questions thoroughly in the rebuttal. I have also considered the insightful comments from other knowledgeable reviewers. The additional experiments and detailed explanations provided in the response adequately resolve the major concerns. This paper proposes a novel and timely task, presents an effective and efficient baseline, and includes extensive experiments accompanied by comprehensive discussions. Overall, I believe this is a strong submission and deserving of acceptance. I maintain my original positive rating.", "rebuttal": "Thank you for recognizing our work and for your constructive comment. In regard to the weaknesses and questions raised, we provide our point-to-point responses below:\n\n>**W1**:\nContradictory statements? Lines 256–257 suggest that the PF module significantly boosts performance, whereas Lines 268–269 state that PF yields negligible gains. It would be helpful to clarify these descriptions to avoid confusion.\n\n**A1**:\n\nThank you for pointing out the potentially confusing phrasing in Lines 268–269. \n\nIn Lines 268–269, we wish to clarify that the performance benefits derived from our pseudo-future mechanism are attributable to the effective learning guided by these targeted future-oriented losses, rather than merely an increase in model capacity. The phrase “negligible gains” refers to the variant in which we **merely add the  $PF$  architecture’s parameters without applying any future supervision** (Row 2 in Tab.3(b) of our manuscript). \n\nWe will revise the manuscript to express these points more clearly.\n\n---\n\n>**W2**:\nFailure cases. While Figure 5 and the supplementary materials present visualizations, the paper would benefit from a more detailed discussion of the failure cases—when and why PreFM might not perform well.\n\n\n**A2**:\n\nThank you for this excellent suggestion. \n\nWe conduct a more detailed analysis on the On-AVEL test set to identify specific failure cases. While the rebuttal format prevents us from including new figures and PDF, we present our quantitative findings and analysis below:\n\n1. **Confusion Between Similar Events**: We analyze the events with the lowest performance and their most common confusions.\n\n| Event | Precision | Recall | F1 | Most confused with |\n| -------- | -------- | -------- |-|-|\n|people slurping|0.55|0.17|0.25|people eating, man speaking|\n|people shouting|0.42|0.19|0.27|baby laughter, engine knocking|\n\nThis results reveal that PreFM struggles to distinguish between events that are semantically or acoustically similar. We hypothesize this is because our current framework does not explicitly incorporate a contrastive learning design to better separate the representations of events originating from similar audio-visual sources.\n\n2. **Performance in Dense Scenes**: We analyze the impact of event density (the number of event classes within a video) on event-level performance.\n\n| num_events | Avg F1 |\n| -------- | -------- |\n| 1-3     | 0.54     |\n|4-6|0.23|\n|>6|0.13|\n\nThese results show that PreFM's performance degrade in complex videos containing a large number of distinct event classes. This suggests that while our future modeling is effective, its benefits are less pronounced in scenarios with very rapid scene changes and drastic context shifts.\n\nWe will add this analysis and qualitative visualizations of these failure cases to the appendix of our revised manuscript.\n\n---\n>**W3.1**:\nThe On-AVVP subtask requires identifying audio-only, visual-only, and audio-visual events. Does the proposed method predict each type of event separately or simultaneously?\n\n**A3.1**:\n\n**Our method predicts all event types simultaneously.**\n\nWe follow the standard paradigm for Audio-Visual Video Parsing (AVVP) tasks by treating it as a multi-label classification problem over a unified class set. Specifically, our model's final classifier produces a prediction vector of size $C_a+C_v$ to predict audio-only, visual-only and audio-visual event (an event is considered \"audio-visual\" when its auditory and visual components are both present) results, where $C_a$ and $C_v$ represent the number of audio and visual event classes, respectively. \n\n\n\n---\n>**W3.2**:\nCould the softmax function be used to normalize the weight w(t) across frames in Eq. 9?\n\n**A3.2**:\n\nThe softmax function can be used to normalize the weight $w(t)$, but we do not recommend it for our specific purpose. Our reasoning is as follows:\n\n* Our current normalization approach, after obtaining the Gaussian distribution weights, can be expressed as: $w(t) = \\frac{g(t)}{\\sum_{i} g(t_i)}$. This direct normalization **preserves the proportional relationship of the Gaussian distribution**, creating a smooth focus.\n* In contrast, the softmax function is expressed as: $w(t) = \\frac{e^{g(t)}}{\\sum_{i} e^{g(t_i)}}$. The exponential mapping in softmax would exaggerate the difference between high and low weight values. This would cause the model to **overly ignore the edge information** (i.e., the earliest historical context and the most distant future cues), which is not what we intend.\n\nFurthermore, we conduct an ablation study on the On-AVEL task where we replace our normalization method with softmax:\n\n|  | mAP | Avg F1|\n| -------- | -------- | -------- |\n| Ours     | 70.1     | 46.3     |\n| Ours w/ softmax | 70.0| 45.9 |\n\nAs the results show, model performance slightly decreases after switching to softmax. This empirically supports our choice of using direct normalization.\n\n\n---\n>**W3.3**:\nWhat are the training and inference time costs for the proposed method?\n\n**A3.3**:\n\nThe total training time required to converge, inference processing time costs (latency) and model performance of our method are shown below. We use the official code to reproduce the second-best baseline methods CCNet on the On-AVEL and MM-CSE on the On-AVVP tasks to provide further context:\n\n|On-AVEL|Epoch|Training time|Latency↓|Avg F1↑|\n|-|-|-|-|-|\n|CCNet|45|11h 4mins|133.3ms|37.0|\n|PreFM|60|1h 1min|19.3ms|46.3|\n|PreFM+|60|1h 25mins|23.8ms|51.5|\n\n|On-AVVP|Epoch|Training time|Latency↓|Avg $_{av}$ F1↑|\n|-|-|-|-|-|\n|MM-CSE|60|53mins|27.7ms|36.2|\n|PreFM|60|45mins|10.6ms|41.2|\n|PreFM+|60|48mins|18.7ms|42.2|\n\nAs the results demonstrate, our PreFM framework offers substantial improvements in training and inference time costs.\n\n---\n>**W4**:\nFour minor issues.\n\n**A4**:\n\nThank you very much for your careful reading and these valuable suggestions to improve our paper's clarity and completeness. We will address all of them in the revised manuscript:\n\n* **Module Name Inconsistency**: We will revise the module name in Line 157 from \"current enhancement\" to \"current refinement\" to ensure it is consistent with Fig.2.\n* **Clarity of Fig.2**: We will revise Fig.2 to visually connect the \"Focal Temporal Prioritization\" module to the final loss computation. This will better illustrate its role in weighting the training objective.\n* **Training/Inference Visualizations**: As suggested, we will add a new figure to the appendix that visualizes the training and inference processes as described in Sec 3.6.\n* **Additional Citations**: Thank you for pointing out these highly relevant papers. We will add and discuss the suggested works (CPSP and UWAV), along with other recent related papers, in our Related Work section to provide a more comprehensive background.\n\nWe appreciate this detailed feedback, which will certainly help us improve the quality of our paper."}, {"review_id": "FG0c3UT26F", "reviewer": "Reviewer_jUXg", "summary": "This paper proposes PreFM, a framework for online audio-visual event parsing. It introduces predictive future modeling, modality-agnostic representation, and temporal weighting to improve real-time performance. The method is evaluated on UnAV-100 and LLP, showing strong accuracy and high efficiency compared to prior work.", "questions": "1. How does the model maintain online consistency when it uses future-supervised signals during training?\n\n2. Can you provide a quantitative analysis of the pseudo-future features’ quality?\n\n3. How sensitive is the model to the choice of teacher in MRR?\n\n4. The modality-agnostic distillation depends on OnePeace as the teacher model. Have you tried other options such as CLIP or AudioCLIP? How much does the downstream performance vary?\n\n5. How does PreFM perform in a real streaming setting with long, continuous videos?", "limitations": "Key issues like the train-test mismatch and generalization to real-world streaming settings are not discussed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper addresses a practical and timely problem: real-time event understanding from streaming multimodal input.\n\n2. The model achieves significant performance improvements over strong baselines while using much fewer parameters.\n\n**Weaknesses:**\n\n1. The model uses future labels and supervision during training, which breaks the online assumption and creates a train-test mismatch.\n\n2. The paper does not evaluate whether the pseudo-future features are realistic or actually useful during inference.\n\n3. The distillation method depends heavily on a specific teacher model, but the paper does not analyze its effect or generalization.\n\n4. The evaluation setting constructs artificial video streams by clip concatenation, which does not reflect real-world online deployment.", "was_revised": true, "final_justification": "I believe the paper has merit in its current form, and while there may be areas for improvement, I maintain my score based on its overall contribution.", "rebuttal": "Thank you for your positive feedback and constructive comments. In regard to the weaknesses and questions raised, we provide our point-to-point responses below:\n\n>**W1 and Q1**:\nOnline assumption and train-test mismatch: How does the model maintain online consistency when it uses future-supervised signals during training?\n\n**A1**:\n\nOur framework strictly adheres to the online setting during inference. The use of future labels is a deliberate design choice confined to the training phase, and it does not create a train-test mismatch.\n\n1. **Strict Adherence to Online Assumptions and Consistency During Reasoning**: As detailed in Sec 3.6, at any given time step $T$, the model's prediction relies solely on historical and current features available up to $T$. No future ground truth or features are accessible, ensuring true online consistency at test time.\n2. **A Deliberate and Principled Training Strategy**: The use of future labels during training is an explicitly designed strategy to equip the model with the ability to anticipate the future, which is the core of our Predictive Future Modeling (PreFM) framework. Our goal is to have the model internalize the predictive capability.\n\nIn summary, this design choice **does not introduce a train-test mismatch and serves as a necessary supervision mechanism**. It effectively bridges the gap by teaching the model to generate its own useful context at inference time, addressing the inherent challenge of missing future information in an online setting.\n\n---\n\n>**W2 and Q2**:\nThe quality and usefulness of pseudo-future features: Can you provide a quantitative analysis of the pseudo-future features’ quality?\n\n**A2**:\n\nHere we provide the analysis of the quality and usefulness of pseudo-future features:\n\n**1. Quantitative Analysis of the Pseudo-Future's Quality**\n\nWe evaluate them from two perspectives, their semantic similarity to ground-truth event features, and their effectiveness in predicting future events:\n\n* Top-k Similarity Accuracy: We measure if a generated feature vector at a relative future time step (T+1~T+5) is the Top-1 or Top-5 closest match to its corresponding ground-truth class feature embedding, among all 100 classes in UnAV-100.\n* Future Prediction F1-Score: We also report the standard segment-level F1-score for predictions made for future time steps.\n\nThe results are presented in the table below:\n\n| Metric| T+1  | T+2  | T+3 | T+4 | T+5 |\n|-|---|---------|-----|-----|-----|\n|Top-1 Similarity | 45.4 | 44.6 | 44.0  | 43.3| 42.6|\n|Top-5 Similarity | 95.5 | 95.3 | 95.1| 94.8| 94.3|\n|F1 Score|57.5|56.5|55.4|54.7|54.1|\n\nThis results provides two key insights:\n* First, the pseudo-future features are remarkably realistic. The Top-5 Similarity Accuracy of over 94% demonstrates that the **correct event feature is almost always ranked among the top candidates**.\n* Second, these **high-quality features enable strong future prediction performance**, as evidenced by the solid F1-scores.\n\n**2. Quantitative Analysis of the Pseudo-Future's Usefulness for Inference**\n\nWe now demonstrate their usefulness for the main task—improving prediction at the current time $T$. Our ablation study on the Predictive Future ($PF$) mechanism in Tab.3(b) in our manuscript directly addresses this. For the reviewer's convenience, we highlight the key comparison below:\n\n| $\\mathcal L_f$ | $\\mathcal L_{mrr,f}$ | $PF$ | mAP | Avg F1|\n| -------- | -------- | -------- |-|-|\n| ×     | ×     | ×     |69.8|44.7|\n|√|√|√|**70.1**|**46.3 (+1.6)**|\n\nThe results clearly show that incorporating the $PF$ mechanism, which **generates and utilizes these high-quality future features, leads to a significant performance improvement** (e.g., +1.6 in Avg F1 Score). This quantitatively proves that the pseudo-future context is actually useful during inference for enhancing the model's understanding of the present moment.\n\nIn conclusion, our comprehensive analyses confirm that the pseudo-future features are both realistic in their semantic representation and highly useful for boosting the model's final online parsing performance.\n\n---\n\n>**W3, Q3 and Q4**:\nThe choice of teacher model: How sensitive is the model to the choice of teacher in MRR? Have you tried other options such as CLIP or AudioCLIP? How much does the downstream performance vary?\n\n**A3**:\n\n**PreFM's performance is robust and not overly sensitive to the choice of the teacher model.**\n\nWe include this analysis in our original submission in Tab.5 in the appendix. For the reviewer's convenience, we present the key results below:\n\n| Teacher Models       | Dimensions  | mAP | Avg F1 |\n|--|--|---|-|\n| AudioClip    | 1024 | 70.2  | 46.2  |\n| ImageBind    | 1024  | 70.0    | 45.9    |\n| ONE-PEACE    | 1536   | 70.1       | 46.3      |\n\nAs the table shows, our PreFM framework demonstrates robust and stable performance across different powerful, pre-trained teacher models. The variation in the final Event-Level Avg F1-score is minor (ranging from 45.9 to 46.3). \n\nThis indicates that while the choice of teacher has a slight impact, our MRR approach **is not overly sensitive to a specific teacher and generalizes well to other strong multimodal teacher models**.\n\nRegarding the reviewer's suggestion to try CLIP: we specifically chose teacher models like AudioClip, ImageBind, and OnePeace because they provide a unified embedding space that includes the audio modality. Standard CLIP models are primarily designed for vision-language tasks and **do not process audio**, making it unsuitable for providing modality-agnostic supervision for our audio-visual parsing task.\n\n---\n\n>**W4 and Q5**:\nEvaluation with clip concatenation: How does PreFM perform in a real streaming setting with long, continuous videos?\n\n**A4**:\n\nOur primary evaluation is conducted on the UnAV-100 dataset, which consists of long, untrimmed, in-the-wild videos and directly addresses this concern. The LLP dataset acts as a complementary benchmark.\n\n**1. SOTA Performance on a Real-World Continuous Video Dataset (UnAV-100)**\n\nThe UnAV-100 dataset contains 10,790 videos of varying length and it is specifically designed for dense event localization in **long, untrimmed, real-world videos**. Our SOTA results on UnAV-100, detailed in Tab.1 of our manuscript, provide strong evidence of PreFM's effectiveness. For convenience, we show the key results below and include the latency results:\n\n|On-AVEL|mAP↑|Avg F1↑|Params↓|FLOPs↓|Peak Memory↓|FPS↑|Latency↓|\n|-|-|-|-|-|-|-|-|\n|UnAV|58.3|28.6|139.4M|52.4G|764.7MB|10.6|94.3ms|\n|UniAV|66.9|30.3|130.8M|22.7G|1020.5MB|15.6|64.1ms|\n|CCNet|62.3|37.0|238.8M|72.1G|1179.4MB|7.5|133.3ms|\n|PreFM (Ours)|70.1|46.3|**6.5M**|**0.4G**|**56.4MB**|**51.9**|**19.3ms**|\n|PreFM+ (Ours)|**70.6**|**51.5**|13.8M|0.5G|144.2MB|42.0|23.8ms|\n\nAs the results show, PreFM significantly outperforms all SOTA methods on this challenging dataset (e.g., achieving a **+9.3** gain in event-level Avg F1-score over the next best method) while using significantly fewer parameters and lower computational complexity. This demonstrates its strong performance and robustness in realistic, continuous video streams.\n    \n**2. The Role of the LLP Dataset as a Complementary Benchmark**\n\nWe designe the modified LLP dataset  not as a substitute for real-world data, but as a controlled, complementary benchmark to analyze model capabilities under two distinct and challenging online scenarios:\n\n* Random Concatenation: This setup simulates **the rapid and unpredictable scene changes** common in online streaming content, testing the model's adaptability. \n* Consistent Concatenation: This setup simulates **longer, continuous event occurrences**, testing the model's ability for consistent understanding and discrimination over an extended context. \n\nThis approach allows us to perform a more nuanced assessment of PreFM's robustness, complementing the evaluation on the in-the-wild UnAV-100 dataset.\n\n**3. Qualitative Visualization on Continuous Streams**\n\nWe provide extensive qualitative results in Fig.5 (main paper) and Fig.7 & 8 (appendix). These figures visualize PreFM's superior ability to generate accurate and temporally coherent event boundaries on long video timelines from both datasets.\n\nIn summary, PreFM's effectiveness in realistic streaming settings has been thoroughly validated through its SOTA quantitative performance on the large-scale UnAV-100 dataset, supported by targeted analyses on our complementary LLP benchmark and extensive qualitative evidence."}]

Author rebuttal:

Meta-review: This paper introduces a new task, Online Audio-Visual Event Parsing, which requires detecting audio, visual, and audio-visual events in videos under online constraints (i.e., using only past and present context). To address the lack of future information and the need for efficiency, the authors propose PreFM (Predictive Future Modeling).

Overall, this work defines a novel and important problem, proposes a solution that is both effective and efficient, and supports its claims with extensive experiments and analyses. While reviewers initially raised concerns about contribution clarity, baseline fairness, and practical evaluation, the authors provided convincing rebuttals with new results that addressed these issues. The combination of task novelty, methodological soundness, and strong empirical results makes this paper a solid contribution. All reviewers were satisfied after the rebuttal, and no unresolved concerns remain.  The AC concurs with the reviewers and recommends acceptance. The authors should incorporate the new results and discussions into the final version of the paper.
Decision: Accept (poster)