MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks
Authors: Sanjoy Chowdhury
Mohamed Elmoghany
Yohan Abeysinghe
Junjie Fei
Sayan Nag
Salman Khan
Mohamed Elhoseiny
Dinesh Manocha
Venue: NeurIPS 2025 poster ()
Keywords: Audio-visual learning
Audio-Visual RAG
Multi-Video Linkage
Primary area: deep_learning

Abstract: Large multimodal models (LMMs) have shown remarkable progress in audiovisual understanding, yet they struggle with real-world scenarios that require complex reasoning across extensive video collections. Existing benchmarks for video question answering remain limited in scope, typically involving one clip per query, which falls short of representing the challenges of large-scale, audiovisual retrieval and reasoning encountered in practical applications. To bridge this gap, we introduce a novel task named AVHaystacksQA, where the goal is to identify salient segments across different videos in response to a query and link them together to generate the most informative answer. To this end, we present AVHaystacks, an audio-visual benchmark comprising 3100 annotated QA pairs designed to assess the capabilities of LMMs in multi-video retrieval and temporal grounding task. Additionally, we propose a model-agnostic, multi-agent framework MAGNET to address this challenge, achieving up to 89% and 65% relative improvements over baseline methods on BLEU@4 and GPT evaluation scores in QA task on our proposed AVHaystacks. To enable robust evaluation of multi-video retrieval and temporal grounding for optimal response generation, we introduce two new metrics, STEM, which captures alignment errors between a ground truth and a predicted step sequence and MTGS, to facilitate balanced and interpretable evaluation of segment-level grounding performance.

MAGNET: A Multi-agent Framework for Finding Audio Visual Needles by Reasoning over Multi-Vite Video Haystacks
Sanjoy Chowdhury', Mohamed Mohamee Elmoghany?, Yohan Abeysinghe', Junjie Fei", Sayan Nag",
Salman Khan", Mohamed Elhoseiny². Dinesh Manocha'
University of Maryland, College Park KKAUST *MBZUAI UUniversity of Toronto
(sanjoyc, dmanocha y@und. edu sayan nag@mail......mmmmmmmmm........a www.uttorooooommmm
mohamed, www.eeeeeesssennsssts..ooom
(yohan. abeysingee,,,,,,, kalmann.kann)@iee @ww.uiii.i....iiaaaa ac. ae m. www.sam..eemoee..... ssssmaaellooghaan@@mal........ com
https:/sscoowdhury671... .ww.gggthhhhbb www.ub..o/mgggnnn io/magnet_project..
Abstract
Large multimodal models (LMMs) have shown remarkable progress in audio- visual understanding, yet they struggle with real-world scenarios that require complex reasoning across extensive video collections. Existing benchmarks for video question answering remain limited scope, typically involving one clip per query, which falls short of representing the challenges of large-scale, audio- visual retrieval and reasoning encountered in practical applications. To bridge this gap, we introduce novel task named AVHaystacksQA, where the goal is to identify salient segments across different videos in response to a query and link them together to generate the most informative answer. To this end, present AV Haystaakss, an audio-visual benchmark comprising 3100 annotated QA pairs designed assess the capabilities of LMMs in multi-video retrieval and temporal grounding task. Additionally, we propose a model-agnostic, multi-agent framework MAGNET to address this challenge, achieving up 89% and 65% relative improvements over baseline methods on @ and GPT evaluation scores in QA task our proposed AVHaystacks. To enable robust evaluation of multi-vidco retrieval and temporal grounding for optimal response generation, we introduce two new metrics, STEM, which captures alignment errors between a ground truth and a predicted step sequence and MTGS, to facilitate balanced and interpretable evaluation of segment- grounding performance.
1 Introduction
Large Multimodal Models (LMMs) [166] have achieved remarkable progress in audio-visual un- derstanding. However, they continue face significant challenges [ when comes retrieving and reasoning over large-scale multimedia collections particularly in tasks such as audio- visual retrieval-augee auggeeee generation (RAG) and vidti-video temporal grounding. These limitations hinder their effectiveness in real-world applications, such querying personal video archives, how-to repositories, or educational video libraries, where complex queries often require joint processing of both audio and visual modalities across multiple video segments.
Despite their growing capabilities, the best of knowledge, no existing task benchmark systematically evaluates LMMs' ability to identify and integrate salient segments from multiple videos to construct informative, grounded responses. As result, there remains gap in properly assessing their performance on large-scale audio-visual retrieval and reasoning tasks. Existing benchmarks g are generally limited in scope typically associating each question with only single short clip, as shown in Tab. I However, real-world information-sedd seekiing scenarios often
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Conseiring the varles songs tecksiques presartst-iran book chard elreeriigg hes off vocation, how larner effectvely their pracfiie acsioon grossolly iecepprste and wuster riess skub?
M/- Haystooks
MAGNET
8076
9716
Thael
APM
De 2277
207s
735e
Atflegetecka
06
540s
6415
7856
Sign Baugnngs
702s
780s Bits
955g 982s
(1) Begin mastering
(t.g. using delbbrate
prachis courting (lid in (817-96% 96%) (2; Use loping exerites to relforce mack menory for specifi pessages gradasly iecressng tempa. (vid (277 28/5]) (3) Infograte
techniges (such kod off and advanced perfected before cordtwing (Vid 153 (570-644 () (+) Eegsberly resew chilleying
stronrsgg) in small legrenirs amarin is full long seclieee and boarseule practlex delke and
mosiel each
(Vd U3 (200)) Vid (702-766s 81- 4555))
Figure I: A new task and a benchmark. We introduce novel task AVHaystacksQA, which requires multi-video linkage and reasoning to generate the most informative answer. To this end, we curate a new benchmark AVHaystacks comprising 3100 QA pairs obtained through careful inspection.
demand searching through hundreds or thousands of video segments, identifying the most relevant audio-visual snippets, and synthesizing coherent, badence-backed answers from them.
To address this gap, introduce a new benchmark AVHaysttcks facilitate novel task AVHaystack- sQA. This task is designed to comprehensively evaluate the LMMs ability to perform large-scale, multi-video audio-visual retrieval and reasoning. Each query in our benchmark is grounded in massive collection of up to 500 video clips, requiring models to localise relevant segments both temporally and across sources and reason over their combined audio and visual signals. This setup reflects the complexity of real-world information needs more accurately than previous single-video setups.
central challenge in building such a benchmark is designing specific, unambiguous questions that genuinely require both audio and visual understanding and cannot be accurately answered using single modality or video clip alone. To address this, we implement a robust data filtering pipeline that combines the strengths of large language models (LLMMs) and human annotators remove generic. redundant, or overly broad queries. For instance, questions such as "How do I improve my strumming while playing guitar?" or "How do you adjust for tightness in your lips to hinder a clear operatic tone?" "hee?"ee are carefully selected and validated to ensure they demand cross-modal reasoning and temporal grounding within specific video segments.
To enable LMMs to tackle this challenging task, we introduce MAGNET novel retrieval- -ugmenntd, multi-agent framework designed to find audio-visual needles within multi-video haystacks. Our approach integrates multiple specialized audio and video encoders to capture rich multi-modal semantics, and incorporates multi-agent framework that scores the relevance of retrieved video segments with respect to the input query. By tightly coupling retrieval with reasoning, MAGNET facilitates efficient and scalable exploration of large-scale video corpora. Extensive experiments demonstrate that our framework significantly enhances both retrieval accuracy and answer generation performance.
Main Contributions:
(1) We propose novel task, AVHaystacksQA, and introduce AVHaystacks new benchmark consisting of 3100 audio-visual QA pairs drawn from videos across diverse domains (Fig. This benchmark pushes the boundaries of video retrieval and reasoning by requiring models to navigate and reason over large-scale video collections. To the best of our knowledge, no existing benchmark systematically evaluates multi-video keypoint detection and reasoning capabilities.

---

Dataset
Train
Test
TA
MVL
AVR
AVD
RQA
AVQA
LC
Avg. Dur. (s)
Video Datasets
ShareGPT4Viddo [
Cinepile [1:]
X
X
160
NEXT-QA [4]
X
48
Video
LongVideoBench
10
X
480
MovieChat
13
X
420
Audio- Visual Datasets
UnAV-100
16
K
X
X
42
VAST-27M
17
X
AVQA 18
AVInstruct
19
x
X
115
Music-AVQA
20
X
VGGSound 21
AVBench-SAVEnn Vid
X
182
AV Haysttacks (Ourg)
738
Table 1: Comparison with prior iideo/audio-visuall benchmarks. MS: Model-Assisted; TA:
Temporal Annotation; MVL: Multi-Video Linkage; AVR: Audio-Visuu fine-grained Reasoning; AVD: Audio Visual Deseription; RQA: Retrieval-base QA Answering: AVQA: Audio-Visual QA;
LC: Long Context, where QA context spans over 5 mins.
(2) We conduct extensive evaluations of state-of-the-art audio-visual models AVHaystacks, analyzing their performance across range of multi-video retrieval and reasoning setups. Our findings reveal that current models perform suboptimally in retrieving relevant videos from large corpora and struggle to reason effectively across multiple clips to identify the key segments needed to answer complex queries.
(3) To enable robust evaluation of audio-visual retrieval and grounded temporal reasoning, we introduce two novel metrics: STEM, which quantifies alignment errors between the ground-truth and predicted step sequences in multi-video audio-visual answer generation; and MTGS, which provides a balanced and interpretable assessment of segment-level grounding performance.
(4) Finally, we propose model-agnostic, multi-agent training strategy, MAGNET, designed to enhance model performance identifying key segments across multi-video haystacks. Experimental results show that our framework achieves up 89% and 65% relative improvements over baseline methods on BLEU@4 and GPT-based evaluation scores, respectively, on AVHaystacksQA.
2
AVHaystacks: Audio-Visual Benchmark for Multi- Temporal Grounding and Reasoning
The benchmark curation pipeline consists of five stages: (1) Video curation: Following careful manual inspection, we collect 500 videos from YouTube spanning 27 diverse categories, including how-to guides, cooking. travel, musical instrument tutorials, language instruction. and vocal lessons. Each vidco is selected to ensure its suitability for audio-visual QA tasks specifically, queries where answering correctly requires strong understanding of both audio and visual modalities, with complementary cues essential for accurate reasoning. (2) Blind question generation: Using OpenAl 03-MINI with custom prompts (details in the supplementary), per topic. This promotes comprehensive evaluation by introducing diversity in reasoning types, modality dependence, and task complexity. (3) Transcript cleaning and segmentation: Subtitle overlaps are resolved, and transcripts are segmented into coherent instructional subtopics to support fine-grained QA generation. (4) Segment-aware QA prompting: Segment-specific questions automatically generated to require multimodal comprehension leveraging audio, visual, and textual cues. (5) Answer grounding: Answers are constructed in step-wise manner, referencing at least two distinct video segments. Additional details are provided in the supplementary material.
generate 50 topic-agnostic questions
Dataset Selection. We apply four filtering criteria for video curation: (1) synchronized audio, on-screen text, and visual changes; (2) clear procedural step sequences; (3) duration between 5-25 minutes; and (4) availability of English captions.
Grounded Audio Visual Question Answering. Each QA item includes: (i) a free-form question, (ii) a step-by-step answer, and (iii) list of (videolD, start, end) references. Unlike prior single-clip datasets, 82% of our QA pairs require evidence from at least two distinct videos, making them well-suited for LMMs. Examples of extended answers shown in supplementary.

---

Answer
!) Ramember that the kit's drumheods and cymbals offer different rebound lerels compared to the is's bounciness (Video #8 [739- 129 :)) grip tonsoon slighlly accommodate the drum's cynbol's give (Video 129 =]). 2) Mainloir the some wrist-ied stroke fundamenlais; only very the angle or force as needed (Video #8 [739- - 929 s). Vidro #12 (48 103 I). Stort the
Sarpleg
Salient Frome Selector Module
Question
quicie, aprrontaoivs
and cymeals (Vides 41/2 [48 :03 +) v) Keep the same counting rudment exercises preserve muscle memory (Video it8 [239 sj. Videa #11 [a 173s])
(closest feel to pod). then expond
Prggct back fo vide
Audio-Visual LLM
Questian
Meta Agent
AV-Agent
AV-Agent
Audio-Visual LLM
Answer
Vidko #t
Video this
Top Vids
S(c.av)
*Average and Sort
AV-RAG
Ques-AV Sim
Sig, pap) Vae(V
Top K Vieas
Eg(s)
Ecap(oap)
Ques- Cap Sim
Question
How adopt my technique when moving from a pracfice pad a full drums Xit to moistain consiteen strole qualiy and rebound?
AV-DB
Ques. Erc.
AV Enc.
AV Cap
Capton
Cap. Enc
Supersised Fins luring using Low Rank Adeptation Module
Sollent Taken Locafor
Sollnt Toéens
Reciundont Tokens
Audio-Vaual Tokens
Figure 2: Overview of MAGNET. Given a user query, AV-RAG retrieves the top-K relevant videos (with audio), which are then processed by dynamically spawned Audio- agents and a meta-agent aggregator generate temporally grounded, step-wise responses. An adaptive, modality-agnostic frame selection module improves performance over baselines (see Tab. 2).
Multiple Audio Visual Entity Linkage. To enable fine- grained, cross- entity grounding without manual bounding boxes, we generated 3100 QA queries (train/test split 2k/1k) drawing evidence from a 500-video pool. To facilitate experiments with baseline approaches we curate small subset AVHaystacks-50. Instead of box-level labels, each entity is grounded via a sequence of multimodal spans: (i) textual segments from cleaned transcripts, (ii) audio intervals capturing distinctive sounds, and (iii) visual frame intervals showing the entity onscreen. These spans often span multiple videos in a defined order (c.g., steps 1-3 of a recipe across two clips), capturing both temporal and cross-video dependencies (details in supplementary).
3 Method
3.1 Task Definition: AVHaystacksQA
Given a question and a collection of N videos V = (Vi....,Vy), our framework aims to retrieve the top-k most relevant videos to support AVILMS understanding and answering the question q- MAGNET accomplishes this through two-step retrieval process designed to effectively identify and reason over relevant videos for each question demonstrated in Fig 2
3.2 Audio Visual Preprocessing
AV RAG. We compute similarity scores between the averaged query representation and (i) fused audio-visual features (using Hadamard fusion) (-) and (ii) encoded audio-visual captions Ecap (-)
for each video V, E V using cosine similarity, as defined in Eq. [
S(q. V) = E
for V V,
(1)
Here, S denotes the similarity score between query g and video set V; cos cosine similarity; E, is the text encoder; and E E (Eav)
We use IMAGEBIND 23 to encode both the query and audio-visual features. Captions generated using Gemini 1.5 Pro 24 and encoded with IMAGEBIND to obtain E-ap(:); all embeddings are cached in our retrieval database enable fast similarity computation without runtime re-encoding. To compute final relevance, we average S(av, q) and S(cap, q) to obtain Sim then rank videos in descending order and select the top-k most relevant videos.
Salient Frame Selector Module (SFS). Long videos often contain sparse but crucial moments relevant to query. To efficiently localize these key events, introduce salient frame selection

---

Algorithm SFS
Algorithm 2 STEM: Step-wise Error Metric
Input: m total frames, target count
Input: Ground Truth Steps: (Gi,..., C... Predicted Steps:
matrix Q
(P.....; P.)m, J. Text Similarity Threshold:
0.5.
Output: Selected frame indices
Output: Missing Step: SMr. Hallucinated Step: Su, Wrong Step Order: So. Step wisc Video ID False Positives and Negatives:
I: Initialize: C/O.... m mm (0...kt
S Step-wise IoU on time intervals: Stø Similarity
oo, c
Matrix: Msm. Step Similarity Function: Sim(-). Hungarian
2: Initialize:
Matching Algorithm: Hung(-), Matched Steps: GT, P D- Compute similarity matrix
backtrack(0 (0... mtarrrcckk............... m O0..
1: Mam Sim(G, P)
3: for j (1, k] do
4: for i [j m.)
[ do
2:
G, P Hung( (aam: G, P)
b Obtain matched pairs
i do 5: for p E [j - ....i
3: for matched pairs P)) do
4: if + then
6:
< CbD Cppl
1] +
So f SS for groundings (Epred, toed coad)) in P, do
Wrong Step Order
Q[pl[
then
7:
CMb - Clplli
if Upred & {Un Gi) then SFr + SFr +1
1] [
Video Mismatch
8:
backtrack [1|b]] t
9:
else
p
10:
Stu < loU (Iiim tm [ gamd
IoU
9: Initialize: result [].
k 4- m
11:
for groundings (Ugs, Samn (and) in G, do
12:
if Uy {Upred P3} then SEN SEN for unmatched (G G): do
while j > do
13:
Video ID Mismatch
11:
result.append (2)
14:
i backtrack +
15:
SM < 5M +
b. Missing Step
16:
for unmatched (P P)5 do
13: return resull.
17: Su t Sy +
> Hallucinated Step
module that focuses on visually and semantically important content. This ensures attention directed to both the right content and the right time, facilitating efficient reasoning.
We begin by uniformly sampling m candidate frames without any prior. The objective is select representative frames that are both visually diverse and temporally dispersed.
Let 4, denote the t-th sampled frame, and E Rd its (Hadamard) fused audio-visual embedding from ImageBind. We compute the pairwise cosine similarity between all frame pairs:
2g 26
T ab
Ya, b E (1....,m)
2 22 201122
To discourage temporally adjacent selections, we apply temporal separation penalty to frame pairs, where y is the separation penalty factor:
bl) +1 -1)
Aeb
sin
The total affinity matrix is defined as Qab = T ab + Aas- We then select a sequence of k frame indices .a....,,4.] (x} such that < 11 <... la m and the total pairwise similarity is minimized (process detailed in Algorithm [ using the following equation:
T (2,,,,,...,
T arg
min
Quaann
TC
1=1
3.3 Grounded Question-Answering with Audio- Visual agents
Once the potentially relevant videos are shortlisted (i.e., those likely to be helpful in answering the given query), we deploy dynamic agentic setup based on AVLLM backbone. Due to its recent success in fine-gre fine-grained audio-visua, comprehension, we utilise Qwen 2.5 Omni [25] as the backbone for our AVLLM agents. For each shortlisted video, dedicated instance of Qwen 2.5 Omni is spawned to process that video independently. Each agent analyses assigned video and predicts the most relevant temporal segments along with response that may contain answer to the query. The outputs from all individual AVLLM agents, the identified time windows and corresponding partial responses are then aggregated by meta-agent. In our setup, GPT-4o|26 (26 acts as the meta-agent,
5

---

which ingests the agent responses and synthesises a coherent, contextually grounded final answer for the input query.
Experiments
4.1 Metrics
We utilize a set of 4 metrics to evaluate the outcomes of our approach which are follows:
Response Alignment Scores, We evaluate the semantic alignment between the summarized outputs, comprising step-wise responses to given query, and the corresponding ground truth using bination automated and human tuman-centric metrics. For automated evaluation, we employ standard text-based similarity metrics such BLEU @4 and CIDEr. Additionally, we utilize the GTE-L model (27) to extract sentence embeddings for both the predicted response and the ground truth, and compute their cosine similarity. Beyond these, we adopt the GPT-as-a-Judge framework to score the predicted responses against the ground truth on a 10-point scale, subsequently normalizing these scores for consistency. Finally, we include human evaluation scores (on scale 1-5 with being lowest) as complementary metric (averaged across 20 evaluators) to assess the alignment and overall quality of the predicted answers with respect to the ground truth.
To evaluate how well our system retrieves relevant videos from the haystacks, we present recall values using R@ 1, R@3, 3, and @@5. These metrics help us assess the video retrieval accuracy by examining the presence of relevant videos within the top few ranks.
Retrieval Evaluation Scores.
Matched Temporal Grounding Score. To evaluate the alignment between predicted and ground truth temporal segments (along with video IDs), for each query, we propose the Matched Temporal Grounding Score (MTGS). This metric measures the average temporal overlap (IoU) between predicted and ground truth time intervals, but only for video instances where the video IDs match. video ID is not present in both prediction and ground truth, it is excluded from the computation. Furthermore, for each matched video ID, we compute the temporal IoU over the union of all corresponding intervals. This ensures that the metric reflects segment-wine grounding accuracy at the video level. Formally, let Va and Vp denote the sets of video IDs present in the ground truth and predicted outputs, respectively. Let VMr VG n Vp be the set of matched video IDs. For each E VM, let G. {( ( be the set of ground truth intervals and P. [( (gna)) } be the set of predicted intervals. We define the temporal IoU for video v as: IOU,- = Dnrraiion[ntersscttion P..)P.))
Duration (uuration(Union[CC (G.. P.))
where the intersection and union are computed by merging overlapping intervals across G, and Pa. and Duration(-) computes the total length of the resulting intervals. The final MTGS then computed as the mean IoU across all matched video IDs: MTGS- | Evev,e IoU,. In cases where there is no matched video ID between prediction and ground truth (i.e., [VM| = O), we define MTGS- 0. This metric provides balanced and interpretable evaluation of segment-level grounding performance, with sensitivity to both partial and full overlaps, while ensuring fairness by averaging over matched video contexts. We report the average value of this score MTGS in Tab. 3
STep-wise Error Metric. The Step-wise Error Metric (STEM) quantifies alignment errors between a ground truth step sequence and a predicted step sequence in instructional or procedural data (detailed in Algorithm 2). Given Ground truth steps: (G1, G2..... G. Predicted steps: (PI, Pz.... Pm } Text similarity threshold: т, E [0, 11, typically set to 0.5, we begin by computing similarity matrix Mam E R where each entry is given by a cosine similarity function Sim(Grexr p )P computed between the step-wise text embeddings. We obtain a set of valid matched pairs of predicted and ground truth steps via Hungarian matching (28) which minimizes the total dissimilarity. If i + j. the prediction is out of order, contributing to the wrong step order count. Subsequently, to assess grounding mismatch, for each matched step, we compare video IDs and compute corresponding IoUs between prediction and ground truth. Finally, unmatched ground truth steps add to the missing step count, and unmatched predicted steps are considered as hallucinated steps.
4.2 Baselines
In our experiment, we have evaluated several open and closed-sourced Audio Visual models on the retrieval and AVHaystacksQA performance. We extensively evaluate AVHaystacks on VideoRAG g Video-RAG (29], Qwen 2.5 Omni (25). Unified 102 [4, Video-SALMONN (0). We suitably adopt Video-RACG VideoR AG for our task. To accommodate videos in 2.5mn-2.,-mmni, Unified- 102, Video- SALMON for AVHaystacks-55 we sparsely sample frames and downsize them to low resolution and also compress audio using [31].

---

AoHaystucks-S0
FVHayytucks-Fll
Method
Cr
OPTEval
H 1
B@41
Text Sim
GPT
Eval
VideoRAGi
1:31
1.22
41.59
5.5
6.1
3.32
42.64
117.56
5.23
3.37
5.97
25
10.84
1,90
1,07
Unilied
2.15
1.02
ViderSAI.MONN
30
11.90
32.32
2007
2.39
0.91
3.93
2.59
4.30
MAGIET +03it
24.78
81.79
3.85
4.52
2:54
27.95
76.1
3,69
4.35
2.45
MAGNET .Ques 2. Oueal
30.54
85,56
4.01
4.73
2.64
28.49
81.74
3.85
4.57
2.54
MAGINET- 45506541200000.0
$2.30
144.40
6:20
7.46
3.96
49.24
136,86
5.96
7.19
3,81
MAGNET 4b3iis 300.1T
53.66
146. 18
6.28
7.58
4.00
51.45
142.56
6.12
7.34
3.91
MAGNET +fpee Chanker
55.82
153,98
6.53
7.84
4.15
53.69
146.30
6.28
7,56
4.01
MAGNET 75
57.67
157.72
6.69
8.03
4.25
55.80
153.95
6,53
7,80
4.15
Table 2: Response Alignment Scores. Our proposed MAGNET offers significant gains over baseline approaches (first section) and our adapted baselines (second section) across multiple objective and subjective metrics on two dataset splits. 3: BLEU@4, Cr: CIDEr, H Eval: Human Evaluation. Closed source model: reference for upperbound.
Methel
MTGikey
1
[S
SE
50,
SFP
SPX
MTGG
T
[M
SRR
S01
(FFP)
SIN
AVHaysattoo5
AVHlastack-Pull
6:48
6:55
0.31
235
B.11
532
D-5
B-4
8.33
B-3
0.30
0.35
R-W51
MAGSET
coateeiEE 20.21
6:4
033
0.55
1683
0.30
0,55
0121
Tea
ImageBinn-RMM RAGU
$2.84
7W.23
$2.57
34.57
66.54
60.41
06.13
71.60
Vidn-KAG
69.96
69:33
31.93
MAGSET aeannr seanurrsay
6/19
014
015
5.11
116
014
102
811
8.30
1031
118
VileoRMG
55.57
89.79
73.15
79.30
032
6.32
6.17 0.12
Table 3: Grounding evaluation and Step-wise error results on AVHaystack-50 and AVHaystack-Full datasets using MTGS and STEM (SM, SH, SO, SFP, SFN) metries respectively.
Table 4: Retrieval Evaluation Scores AVHaystack-50 and
AVHaystack-Fin datasets.
4.3 Main Results
Audio Visual QA. Experimental results in Tab. 2 demonstrate that among FT models MAGNET
-wwen 2.5 Omni-FF. achieves best performance across all automatic and human evaluation metries. On both AVHaystacks splits, it achieves the highest scores outperforming both ZS and fine-tuned variants of MAGNET combined with other AVLLMS (e.g.. VideoSALMONN, Unified IO2).
Grounding Evaluation and Step-wise Error Assessment. Tab. 3 indicates that our approach substantially improves the grounding capabilities and reduces step- wise rates of the open-source AVLLMs, as reflected from the MTGS and STEM values, respectively. To robustly validate our proposed STEM, conduct a human evaluation and observe strong correlation between human judgments and the metric. Refer to supplementary material for more details.
Notably, in both Tabs. 2 to provide an upper bound, we incorporate a strong closed-source model with powerful generative capabilities such as Gemini- -Pro [32] within multi-agent framework MAGNET, coupled with Salient Frame Selector module and report its performance. We observe that our best model MAGNET -Qwen 2.5 Omni: FT almost reaches the upper bound values for
Audio Visual Retrieval. Our method sets a new benchmark on AV retrieval task across both the dataset splits (Tab. 4.. Compared to existing approaches, it achieves the largest margin of improvement in R@3 3 and R @5,, particularly on the more challenging AV AVHaystacks-Full, where gains over strong baselines like VideoRAG and Text-RAG are 2.7 points in R@3 and 7.6 points in @@5 These improvements indicate that our approach retrieves more relevant samples consistently and scales effectively to larger retrieval spaces. The results also highlight the benefit of leveraging richer modality integration and adaptive reasoning in our model over static retrieval pipelines.
The above results (Tabs. 4 demonstrate that employing multi-modal RAG pipeline not only improves retrieval quality but also aligns better with human judgments.
4.4 Ablations
Importance of modalities. The results in Tab. show that performance is generally best when both audio and visual modalities are used, highlighting the benefit of multi-modal information. Gemini- 1.5-Pro consistently outperforms Dwen-2.5-Omel across all retrieval and response alignment scores metrics, indicating the benefits of MAGNET in formulating coherent and information responses. Qualitative assessment indicates a strong correlation across tasks, underlining the utility of our RAG pipeline.
Sampling strategy. For both backbones, using SFS significantly improves performance across all metrics. For Swen2.5-OOmiiFF, switching from Uniform to SFS increases BLEU @4 score by 0.17
7

---

Woal
BLaire11
Meined
Unsions SB5
BLEUED
SestSin: 1
GPE G
Homan Eval
MACAET
7
5:41
5.1
2.3
Ala
6.3
5.3
3.0
A938
MAGNET
678
5/4
Table 5: Performance of MAGNET
Table 6: Effect of sampling strategy. We systemati- cally analyse our design choice replacing the sampling algorithm with 3 models on AVHaystacks-
under different modality settings on Fayytaaccs-Full.
4
Test 5im
GPT
Eval
Iuman Eval
6:24
50
40-
5.23
3.00
2:75
Figure 3: Effect of y eval metries for MAGNET +Quem-25500mi-FT and MAGNET +Gemini-1.5-Pre
and Human Eval score by 1.03. A similar trend is observed with Gemini 1.5 Pro where the best results are obtained with our proposed sampling strategy as seen in Tab. 6 These results underscore the advantage of semantically guided sampling over uniform strategies, as SFS more effectively captures informative segments, leading to better grounding, coherence, and human preference.
demonstrates a steady rise in performance across all metrics as y increases, although a slight dip in performance is observed at 7 25, notably in BLEU @ 4 and Human Eval for MAGNET + 2wwe-225-0mmi-FF7 potentially indicate the onset of overlitting or increased parameter sensitivity in that region. The varying magnitudes of the dip across metrics indicate that the effect of y is not uniform across different aspects of model performance.
Penalty hyperparameter. Fig.
4.5 Qualitative Results
Fig showcases our system's 's ability to retrieve the most relevant videos and accurately localize temporal segments needed to answer query using audio-visual cues. The meta-agent effectively highlights key instructional moments e.g., forming the fulerum (Video 1), handshake shaping (Video 8), and diagonal pivoting (Video 1) in alignment with expert references. It also merges comple- mentary segments from Videos and 18 to describe finger placement, demonstrating robustness to redundancy and variation. Overall, MAGNET excels at retrieving, grounding, and synthesising evidence across videos into coherent, temporally aligned responses compared to a recent baseline which fails to retrieve suitable videos and subsequently fails to temporally ground the salient regions.
Additional qualitative and quantitative results are provided in the supplementary material.
Related Works
Video QA Benchmarks. Video Question Answering (VidQA) involves answering natural language queries using visual content alone or in combination with other modalities like audio (33/36). Early benchmarks such as MovicQA (37 relied heavily on subtitles, with minimal visual grounding
38
Datasets like ActivityNet-QU QA
39 and How2QA 40 target visual understanding
daily and
instructional contexts. More recent efforts, including NeXT-QA (11, Perception Test 42 STAR
43
and AGQA 44 emphasize
and causal reasoning. EgoSchema
45] extends
VidQA to long-form egocentric videos using LLM-generated questions. Other benchmarks address longer video reasoning [46] 36 and specialized domains like instructional [47 48) and egocentric
content 52). Despite progress, most VidQA datasets are constrained by limited modalities or fixed time windows. Our work enables large-scale, cross-video retrieval and multimodal reasoning, bridging VidQA and broader audio-visual understanding.
MLLMS for Video Understanding. Open-source LLMs 55.55 have enabled Video MLLMS that connect visual encoders to LLMs via projection bridges short clips, they struggle with long videos due to context limits and temporal complexity [.]. To
56
34
57
58). While effective on
address this, long- context LLMs [59 62] and token compression 63
15
64 scale input capacity
and support agent-based decomposition and retrieval 665|67]. MovieChat 15], for example, uses hierarchical memory for frame-level summarization. However, these models lack the ability to reason over audio-visual content across multiple videos and generate coherent responses. Our multi-agent framework addresses this gap.
8

---

What are the essential knife skills for different vegetables to ensure uniform cuts and safety?
Divide the stick into thirds and create a fulenuma between your thamed and inder finger (thumbprise against the stick, not the nai)- [Video 2. (005 (0077)-0446)] 2) Form relaxed hardshake shape, laging the stick so it rests naturally above the pad dram. [Video (004s--00ess) @) Lightly whap the remaining fingers avound the stick, leaving [Vides (coe (ocesy-0277) 72755) video 2. (03125 (05i22--133s)) 013365] ( Keep the grip loose enough to allow natural rebound enough maintain contral [Viden (apees- 02755)] Maintain. sight angle across the hard for proper piret and allow the back control the rebound [000s 75-01466)
Video
Hiw held mnrr drom stick the
teach Studleds hold stick tiikk
timee Take the stick, divide into aqual
the stick betweaes thame and buder
finger the bottow thind the stick. This called the falorion Use the custer-tatter
this where the stick will phoo the hand, very, very importaant that the student places their thundpprise on the stick...
secure
fingers
3 split the stick into three equal sections and position your thume index
hand first thing going de
out lhke I'm givnn shabe conallody's hand, Ohay, goone malt with
stick down, Ohing going take
form pivot (endure the thumaprind -nat nall-cowtacts the stick). [Video (oos 0100001
point it yon't
it my lift. New I'm goons lay the gun. barrel down just abovt the pad. Okay, new I'm going to take stick
loaded. Okey, from wrist, i'm
New,, in the first joint of my mingermman, need samd hand cream, these fingers pretty dry
your initiating relesed handshakk, allowing the stick naturally above the
That's dkay, It's cole...
[Video (00425-000 33 Gently making fingers
large gaps [Viden (0105s-0275 (0115) 001200] 4 Maintain enough perwit natural rebound, yet firm enough to retain control. [Videp (0101s-027es)) @ Held n slight dingonal across your palm ensure smooth pivoting, with the back fingers helping to guide and regulete the reboard. [Video (00is)--01500)
grip
Gorman Vie bruncing a baslothall the hands are positioned over the sticks and you simple maee them up and dasen uting the natural kings st your wrist. French grip, the other hand, 8s like twisting a doorkeod -ou bring your hards inmards and under the sticks and simply rotate your hands back and forth, great way to practice each of thest grips to play simple patterns and gradually move them back and forth from the German protion to the French. This aloos you feel out how each position...
Break the stick into few parts and use your fingers to make a turning point. 2 Close your other fingers arownd the drum stick foosely (3) The grip should not be tight but alio too loose.
Figure 4: Qualitative results. Powered by efficient video retrieval pipeline and multi-agent configu- rations of MAGNET -Qwen-2.5-0mmi- demonstrates strong reasoning abilities by first identifying the key videos followed by audio-visual temporal grounding to localise the salient regions across multiple videos when subjected to a how-to question.
Retrieval Augmented Generation (RAG). RAG improves generative models by integrating retrieval to inject external knowledge (3). While well-studied in text domains [9), recent efforts have extended RAG to vision-language tasks (5) MuRAG [8]] uses non-parametric multimodal memory, and MIRAGE [82] employs CL.IP-based retrieval. Other methods convert images to text via OCR, captioning, and detection before dense retrieval [86] 79). Multimodal RAG has also been applied in domains like healthcare [87 [88], leveraging images as contextual input. While prior models focus on text or vision alone, MAGNET integrates off-the-shelf models with custom retrieval-fusion pipeline for scalable audio-visual-languaa retrieval and generation.
Conclusions and Future Work
We introduced novel benchmark and framework for evaluating LMMs in audio-visue retrieval and reasoning an area less explored than image or single-video based settings. Our task targets the challenging problem of retrieving relevant audio-visual segments from large video corpora, requiring joint temporal, auditory, and visual reasoning, akin to real-world multimedia search. To tackle this, we proposed MAGNET, scalable -ugmeeete generation system that identifies key moments across multiple videos and synthesizes grounded responses. It combines a sampling strategy, off- the-shelf models, and multi-agent relevance scoring mechanism extract and fuse salient content. Experiments show substantial gains over baselines, underscoring the promise of methods in audio-visual reasoning. We hope this work inspires richer benchmarks that push LMMs toward dynamic, temporally grounded multimodal understanding.
While MAGNET and AVHaystacks advance AV multi-video reasoning, several future directions remain. Replacing off-the-shelf components with end-to-end trainable modules could improve retrieval and frame selection. Enhancing agentic reasoning with collaborative mechanisms (e.g., planning or voting) may boost interpretability and performance. Lastly, integrating personalisation (e.g., user-driven retrieval) would support real-world applications like education and assistive tools.

---

References
[1] Henghui Du, Guangyao Li, Chang Zhou, Chunjie Zhang, Alan Zhao, and Di Hu. Crab: A unified audio-visual scene understanding model with explicit cooperation. arXiv preprint 3rXiv:2503.3306 2025.
[2] Sanjoy Chowdhury, Sayan Nag, Subhrajyoti Dasgupta, Jun Chen, Mohamed Elhoseiny, Ruohan Gao, and Dinesh Manocha. Meerkat: Audio-visual large language model for grounding in space and time. In European Conference on Computer Vision, pages 52-70. Springer, 2024.
[3] Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, Yifan Zhang, Shaoqi Dong, Xiong Wang. Di Yin, Long Ma, et al. Vita: Towards open-source interactive omni multimodal Ilm. arXiv preprint wwXiv:2408.05222
2024.
[4] Jiasen Lu, Christopher Clark, Sangho Lec, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitton, pages 26439-26455,, 2024.
[5] Shoubin Yu, Jaehong Yoon, and Mohit Bansal. Crema: Generalizable and efficient video-language reasoning via multimodal modular fusion. arXiv preprint UrXiv:2402.05889, 2024.
[6] Yunlong Tang, Daiki Shimada, Jing Bi, and Chenliang Xu. Avicuna: Audio visual Ilm with interleaver and context boutddrrb alignment for temporal referential dialogue. arXiv e-prints, pages arXiv-2403, 2024.
[7] Tsung-Han Han Wu, Giscard Biamby, Jerome Quenum, Ritwik Gupta, Joseph E Gonzalez, Trevor Darrell, and David M Chan. Visual haystacks: A vision-centric needle-in-a--hayttcc benchmark. arXiv preprint arXiv:2407. 2002244777766,,. 2024.
[8] Xubin Ren, Lingrui Xu, Long Xia, Shuaiqiang Wang, Dawei Yin, and Chao Huang. Videorag: Retrieval- augmented generation with extreme long-context videos. arXiv preprint arXiv:2502.01549 2025.
[9] Chaoyou Fu. Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang. Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. rideo-mme: The first-ever comprehensive evaluation benchmark multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075. 2024.
[10) Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: benchmark for long-context interleaved video-language understanding. Advances in Neural Information Processing Systems, 37: 28828-28857, 2024.
[11] Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04266 2024.
[12] Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang. Zehui Chen, Haodong Duan, Zhenyu Tang, Li Yuan, et al. Sharegptávideo: Improving video understanding and generation with better captions. Advances in Neural Information Processing Systems, 37:19472-19499 2024.
[13] Ruchit Rawal, Khalid Saifullah, Miquel Farré, Ronen Basri, David Jacobs, Gowthami Somepalli, and Tom Goldstein. Cinepile: A long video question answering dataset and henchmark. arXiv preprint mrXiv:2405.088/33 2024.
[14] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of guestion- explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777-9786, 2021.
[15] Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long vidco understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
pages
[16] Tiantian Geng, Teng Wang, Jinming Duan, Runmin Cong, and Feng Zheng. Dense-localizing audio-visual events in untrimmed videos: A large-scale benchmark and baseline. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 22942-22951, 2023.
[17] Sihan Chen, Handong Li. Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. Vast: A vision aadio--udii-see audii-uuuuilee subtide-textt omni- modality foundation model and dataset. Advances in Neural Information Processing Systems, 36:72842-7228 72866, 2023.
10

---

[18] Pinci Yang, Xin Wang, Xuguang Duan. Hong Chen. Runze Hou. Cong Jin, and Wenwu Zhu. Avqa: A dataset for audio-visual question answering on videos. In Proceedings of the 30th ACM international conference on multimedia, pages 3480-3491, 2022.
[19] Qilang Ye. Zitong Yu, Rui Shao, Xinyu Xie, Philip Torr. and Xiaochun Cao. Cat: Enhancing multimodal large language model answer questions in dynamic audio-visual scenarios. In European Conference on Computer Vision, pages 146-164. Springer, 2024.
[20] Guangyao Li. Yake Wei. Yapeng Tian, Chenliang Xu, Ji- Ji-Rong Wen, and Di Hu. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 19108-19111 2022.
[21] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio- visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP). pages 721-725. JEEE, 2020.
[22] Jungang Li, Sicheng Tao, Yibo Yan, Xiaojic Gu, Haodong Xu, Xu Zheng, Yuanhuiyi Lyu, Linfeng Zhang. and Xuming Hu. Saven-vid: Synergistic audio-visual integration for enhanced understanding in long video context. arXiv preprint arXiv:2411. 2024.
[23] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin. and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15180-151990 2023.
[24] Google AL. Gemini: Google's multimodal ai model. Google Al Research, 2024. https: wwtbllogggemiiivvs.e
[25] An Yang. Baosong Yang. Beichen Zhang. Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprini 2rXiv:2412.15115, 2024.
[26] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
[27] Zehan Li, Xin Zhang, Yanzhao Zhang. Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.0322 2023.
[28] Harold W Kuhn. Variants of the hungarian method for assignment problems. Naval research logistics quarterly, 3(4):253-258, 1956.
[29] Yongdong Luo, Xiawu Zheng, Xiao Yang, Guilin Li, Haojia Lin, Jinfa Huang. Jiayi Ji, Fei Chao, Jiebo Luo, and Rongrong Ji. Video-rag: Visually- reureeeal-augmeetedd long video comprehension. arXiv preprint 2rXiv:2411.13099 2024.
[30] Guangzhi Sun. Wenyi Yu, Changli Tang, Xianzhao Chen, Tian Tan. Wei Li, Lu Lu, Zejun Ma, Yuxuan Wang, and Chao Zhang. video-salmonn: Speech-enhanced audio-visual large language models. arXiv preprint ww..i:2240..57000000 2024.
[31] Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. figh-fidelity audio compression with improved rvggan. Advances in Neural Information Processing Systems, 36: 27980-27999 2023.
[32] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang. et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint 30Xiv:2403.05530, 2024.
[33] Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Vidco-mme: The first-ever comprchensive evaluation benchmark of multi-moda llms video analysis. arXiv preprint arXiv:2405. 21075 2024.
[34] Kunchang Li. Yali Wang, Yinan He, Yizhuo Li, Yi Wang. Yi Liu, Zun Wang. Jilan Xu, Guo Chen, Ping Luo, al. Mybench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages -22260,, 2024.
[35] Yaoyao Zhong, Junbin Xiao, Wei Ji, Yicong Li, Weihong Deng, and Tat-Seng Chua. Video question answering:
Datasets, algorithms and challenges. arXiv preprint arXi::2003.0/22. 2022.
[36] Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi. Xiaotao Gu, Shiyu Huang. Bin Xu, Yuxiao Dong et al. Lvbench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406. 08035, 2024.
11

---

[37] Makarand Tapaswi, Yukun Zhu, Rainer Stiffelhagen, Antonio Torralba. Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through question-ansswerigg In Proceedings the IEEE conference on computer vision and pattern recognition, pages 4631-4640, 2016.
[38] Bhavan Jasani, Rohit Girdhar, and Deva Ramanan. Are we asking the right questions in movieqa? In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0-0, 2019.
[39] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao, Activitynet-qa: dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9127-9134, 2019.
[40] Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Elliott, Loic Barrault, Lucia Specia, and Florian Metze. How2: large-scale dataset for multimodal language understanding. arXiv preprint arXiv: 1. 1.00447, 2018.
[41] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question answering explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777-9786, 2021.
[42] Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang. Carl Doersch, et al. Perception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing Systems, 36: 42748-42761, 2023.
[43] Bo Wu, Shoubin Yu. Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning in real-world vidcos. arXiv preprint 2405.2440 2024.
[44) Madeleine GrundeMMcLaughe aughlin, Ranjay Krishna, and Maneesh Agrawala. Agqa: A benchmark for compositional ppatio-tempore reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11287-11297 2021.
[45] Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik, Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36:46212-4611 2023.
[46] Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideohench: benchmark for long-context interleaved video-language understanding. Advances in Neural Information Processing Systems, 37: 28828-28857, 2024.
[47] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learning to answer questions from millions of narrated videos. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1686-1697, 2021.
[48] Haiwan Wei, Yitian Yuan, Xiaohan Lan, Wei Kc, and Lin Ma. Instructionbench: An instructional vidco understanding benchmark. arXiv preprint 250450250. 2025.
[49] Toby Perrett, Ahmad Darkhalil, Saptarshi Sinha, Omar Emara, Sam Pollard, Kranti Parida, Kaiting Liu, Prajwal Gatti, Siddhant Bansal, Kevin Flanagan, et al. Hd-epic: A hithll-detailed egocentric video dataset. arXiv preprint 2rxiv22502.04 144, 2025.
[50] Kristen Grauman, Andrew Westbury, Eugene Byrne Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18995-19012, 2022.
[51] Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-cxo4d: Understand- ing skilled human activity from first-and third person perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19383-19401 2024.
[52] Dima Damen, Hazel Doughty. Giovanni Maria Farinella, Sanja Fidler. Antonino Furnari. Evangelos Kazakos. Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens datasct. In Proceedings of the European conference on computer vision (ECCV), pages 720-736, 2018.
[53] Wei- Chiang. Zhuohan Li. Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open- sour-source chatbot impressing gpt-4 with 90%* * chatgpt quality. See https://vicuaa. Imsys. org (accessed 14 April 2023), 2(3)66 2023.
12

---

[54] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397/ 2023.
[55] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnicr, al. Mistral 7b. arXiv preprint arXiv:2310.0682. 2023.
[56] KunChang Li, Yinan He, Yi Wang. Yizhuo Li, Wenhai Wang. Ping Luo, Yali Wang. Limin Wang, and Yu Qiao. Videochat: Chat-centric videa understanding. arXiv preprint arXiv:2305.063333 2023.
[57] Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction- tuned audio-visudl language model for video understanding, arXiv preprint (ii.::::300.0285566 2023.
[58] Bo Li. Yuanhan Zhang. Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint 1rXiv:2408.03322 2024.
[59] Hao Liu, Wilson Yan, Matei Zaharia, and Picter Abbeel. World model on million -enhthh video and language with ringattention. arXiv preprint arXiv:2402.08260 2024.
[60] Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwci Liu, Long context transfer from language to vision. arXiv preprint arXiv:2406. 16852, 2024.
[61] Fuzhao Xue, Yukang Chen, Dacheng Li. Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408. 10188, 2024.
[62] Xidong Wang. Dingjie Song, Shunian Chen, Chen Zhang, and Benyou Wang. Longllava: Scaling multi-modal ilms to 1000 images efficiently via hybrid architecture. arXiv preprint arXiv:2409.02889, 2024.
[63] Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:2311. 170433, 2023.
[64] Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojic Jin. Flash- vstream: Memory-based real-time understanding for long video streams. arXiv preprint 240.024406.08883 2024.
[65] Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory- augmented multimodal agent for video understanding. arXiv preprint arXiv:2403.1 11481, 2024.
[66] Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Screna Yeung-Levy. Vidcoagent: Long-form video understanding with large language model as agent. arXiv preprint arXiv:2403. 10517, 2024.
[67] Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin Eskin, Jachong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Vidcotree: Adaptive tree-based video representation for 1m reasoning on long vidcos. arXiv preprint arXiv:2405. 19209, 2024.
[68] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmmnted generation for knowledge-intensiv alp tasks, Advances in neural information processing systems. 33:9459-9493 2020.
[69] Zhengbao Jiang. Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yre Yu, Yiming Yang. Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages
2023.
[70] Yang Bai, Xinxing Xu, Yong Liu, Salman Khan, Fahad Khan, Wangmeng Zuo, Rick Siow Mong Goh, and Chun-Mei Feng. Sentence-level prompts benefit composed image retrieval. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. Spollight Presentation.
[71] Chun-Mei Feng, Yang Bai, Tao Luo, Zhen Li, Salman Khan, Wangmeng Zuo, Xinxing Xu, Rick Siow Mong Goh, and Yong Liu. Vqa4cir: Boosting composed image retrieval with visual question answering. arXiv preprint 2221v:2312.12273, 2023. URL //axxi wwtp://wwww.gg.a////..... 12273
[72] Xiaohua Wang. Zhenghua Wang. Xuan Gao, Feiran Zhang, Yixin Wu, Zhibo Xu, Tianyuan Shi, Zhengyuan Wang, Shizheng Li. Qi Qian, et al. Searching for best practices in retrieval- augmentted generation. arXiv preprint arXiv:2407.0m 012119 2024.
13

---

[73] Shangyu Wu, Ying Xiong, Yufei Cui, Haolun Wu. Can Chen, Ye Yuan, Lianming Huang, Xue Liu, generation for natural language processing: A survey.
Tei-Wei Kuo, Nan Guan, al. arXiv preprint arXiv:2407.13190 2024.
[74] Yizheng Huang and Jimmy Huang. A survey on retrieval-augmennte text generation for large language models. arXiv preprint arXiv:2404, 10981, 2024.
[75] Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhengren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang. Wentao Zhang. Jie Jiang, and Bin Cui. Retrieval-augmentee generation for ai-generated content: A survey. arXiv preprint arXiv:2402.1 19473, 2024.
[76] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannanch Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310. 2023.
[77] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-War Wei Chang. Realm: Retrieval- augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning (ICML), pages 3929-3938. PMLR, 2020.
[78] Hongyin Luo. Yung-Sung Chuang, Yuan Gong, Tianhua Zhang. Yoon Kim, Xixin Wu, Danny Fox, Helen Meng, and James Glass. SAIL: Search- augmented instruction learning. arXiv preprint arXiv:2305. 15225, 2023.
[79] Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781. Association for Computational Linguistics, 2020.
[80] Jun Chen, Dannong Xu, Junjie Fei, Chun-Met Feng, and Mohamed Elhoseiny. Document haystacks: Vision-language reasoning over piles of 1000+ documents. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 24817- 24826, 2025.
[81] Wenhu Chen, Hcxiang Hu, Xi Chen, Pat Verga, and William W. Cohen. MuRAG: Multimodal retrieval- augmented generator for open question answering over images and text. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5558-5570. Associ- ation for Computational Linguistics, 2022.
[82] Tsung-Han Han Wu, Giscard Biamby, Jerome Quenum, Ritwik Gupta, Joseph E. Gonzalez, Trevor Darrell, and David M. Chan. Visual haystacks: A vision-centric needle-in-s-haystack henchmark. arXiv preprint arXiv:2407. 13766, 2024.
[83] Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, et al. Visrag: Vision- retrieval- -ugmentee generation on multi- modality documents. arXiv preprint arXiv:2410.1054, 2024.
[84] Yin Wu, Quanyu Long, Jing Li, Jianfei Yu, and Wenya Wang. Visual-rag: Benehmarking text-to-image retricval augmented generation for visual knowledge intensive queries. arXiv preprint arXiv:2502. 2025.
[85] Ryota Tanaka, Taichi Iki, Taku Hasegawa, Kyosuke Nishida, Kuniko Saito, and Jun Suzuki. Vdocrag: Reureeval-augmenttd generation over visually-rich documents. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 24827-24837, 2025.
[86] Weizhe Lin and Bill Byrne Retrieval augmented visual question answering with outside knowledge. arXiv preprint arXiv:22 10.03300 2022.
[87] Liwen Sun, James Zhao, Megan Han, and Chenyan Xiong. Fact-aware multimodal retrieval augmentation for accurate medical radiology report generation. arXiv preprint 2rX. 2024.
[88] Peng Xia, Kangyu Zhu, Haoran Li. Hongtu Zhu, Yun Li, Gang Li, Linjun Zhang, and Huaxiu Yao. Rule: Reliable multimodal rag for factuality in medical vision language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1081-1093, 2024.
[89] Maxime Zanella and Ismail Ben Ayed. Low-rank few-shot adaptation vasion-language models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 15593-6033, 2024.
[90] Nader Asadi, Mahdi Beitollahi, Yasser Khalil, Yinchuan Li, Guojun Zhang, and Xi Chen. Does combining parameter efficieen modules improve few-shot transfer accuracy? Artificial Intelligence for Engineering, 2024.
14

---

[91] Xuan Zhang, Navid Rajabi, Kevin Duh, and Philipp Koehn. Machine translation with large language models: Prompting, few-shot lew-shot learning. and fine-tuning with glora. In Proceedings of the Lighth Conference on Machine Translation, pages 468 468-481, 2023.
[92] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world any resolution. arXiv preprint or 24v: 12191, 2024.
[93] OpenAI. Gpt-4o: Enhanced multimodal language model. OpenAl Research, 2024, https://opent com/
[94] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In International Conference on Learning Representations (ICLR), 2024. URL https:////// wwepppenreeieeee eww.w.
www.eeewwwwwwweeeeeeeeeeeeeeeeee nwt/teeuurumeeee 1d=1 ( tZbq88f27
[95] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.0891 2022.
[96] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al, Mmmu: A massive coulti-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv: J. 16502, 2024.
[97] Yash Goyal, Tejas Khot, Douglas Summers-Star Stay, Dhruv Batra, and Devi Parikh. Making the in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[98] Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu. Kai-Wei Chang, Peng Gao, and Hongsheng Li. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? arXiv preprint 2tt.22003.44244 2024.
[99] Pan Lu, Hritik Bansal, Tony Xia. Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng. Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint 20.00233 2023.
[100] Xubin Ren, Lingrui Xu, Long Xia, Shuaiqiang Wang. Dawei Yin, and Chao Huang. Videorag: Retrieval- augmented generation with extreme long-context videos. arXiv preprint arXiv:2502.01549, 2025.
[101] Sanjoy Chowdhury. Sreyan Ghosh, Subhrajyoti Dasgupta, Anton Ratnarajah, Utkarsh Tyagi. and Dinesh Manocha. Adverb: Visually guided audio dereverberation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7884-7896, 2023.
[102] Sanjoy Chowdhury, Sayan Nag. KJ Joseph, Balaji Vasan Srinivasan, and Dinesh Manocha. Melfusion: Synthesizing music from image and language cues using diffusion models. Proceedings of the IEFF//VV Conference on Computer Vision and Pattern Recognition, pages 26826-26835, 2024.
[103] Xinhao Mei. Varun Nagaraja, Gael Le Lan. Zhaoheng Ni, Ernie Chang, Yangyang Shi, and Vikas Chandra. Foleygen: Visually-guided audio generation. In 2024 IEEE 34th International Workshop on Machine Learning for Signal Processing (MISP), pages 1 1-6. IEEE, 2024.
[104] Zineng Tang, Ziyi Yang, Mahmoud Khademi. Yang Liu, Chenguang Zhu, and Mohit Bansal, Codi-2: In-context interleaved and interactive any-to-any generation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 27425 27434,, 2024.
[105] Sanjoy Chowdhury, Subhrajyoti Dasgupta, Sudip Das. and Uijwal Bhattacharya. Listen to the pixels. In 202 IEEE International Conference on Image Processing (ICIP), pages 2568-2574 IEEE, 2021.
[106] Sanjoy Chowdhury, Aditya Patra, Subhrajyoti Dasgupta, and Ujjwal Bhattacharya. Audvisum: Self- supervised deep reinforcement learning for diverse audio-visual summary generation. In BMVC, page 315, 2021.
[107] Junyu Gao, Hao Yang, Maoguo Gong, and Xuelong Li. Audio-visual representation learning for anomaly events detection in crowds. Nearocomputing, 582:127489, 2024.
[108] Parthasaarathy Sudarsanam, Irene Martin-Morató, and Tuomas Virtanen. Representation learning for semantic alignment of language, audio, and visual modalities. arXiv preprint arXiv:2505. 2025.
[109] Sanjoy Chowdhury, Hanan Gani, Nishit Anand, Sayan Nag. Ruohan Gao, Mohamed Elhoseiny, Salman Khan, and Dinesh Manocha. Aurelia: time-time reasoning distillation in audio-visual llss. arXiv preprint arXiv:2503.23219. 2025.
15

---

[110] Sanjoy Chowdhury. Sayan Nag, Subhrajyoti Dasgupta, Yaoting Wang. Mohamed Elhoseiny, Ruohan Gao. and Dinesh Manocha. Avtrustbench: Assessing and enhancing reliability and robustness in audio-visual Ilms. Xrviv preprint urXiv:2501, .02135. 2025.
[111] Bin Lin, Bin Zhu, Yang Ye. Munan Ning. Peng Jin, and Li Yuan. Video-Ilava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.1012 2023.
[112] Akash Ghosh, Arkadeep Acharya, Sriparna Saha, Vinija Jain, and Aman Chadha. Exploring the frontier of languae models: A survey of current methodologies and future directions. arXiv preprint arXiv:2404.07214, 2024.
[113] Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu. Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408, 0204408.22222 /2528, 2024.
[114] Shengqiong Wu, Hao Fei, Leigang Qu. Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal Ilm. In Forty-first International Conference on Machine Learning. 2024.
[115] Changan Chen, Ziad Al-Halah, and Kristen Grauman. Semantic audio-visual navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15516 15525, 2021.
[116] Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. Video- bench: A comprehensive benchmark and toolkit for evaluating video-based large language models. arXiv preprint 2rXhv:2311.16100 2023.
[117] Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-vided A long form multi-shot benchmark for holistic video understanding. Advances in Neural Information Processing Systems, 37:89098 89124, 2024.
[118] Kaixiong Gong, Kaituo Feng, Bohao Li, Yibing Wang, Mofan Cheng. Shijia Yang. Jiaming Han, Benyou Wang, Yutong Bai, Zhuoran Yang, Av-odyssey bench: Can your multimodal Ilms really understand audio-viste visual information? arXiv preprint axXiv:2412.0266 2024.
[119] Yizhi Li, Zhang, Yinghao Ma. Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang. Jiaheng Liu, Zekun Wang, Jian Yang. et al. Omnibench: Towards the future universal omni-language models. arXiv preprint arXiv:2409. 15272, 2024.
[120] Tiantian Geng. Jinrui Zhang, Qingni Wang, Teng Wang, Jinming Duan, and Feng Zheng. Longvale: Vision-audio-llnguuge-eewee benchmark towards time-aware omni-modal perception of long videos. arXiv preprint 2rXiv:2411.19777 2024.
[121] Kim Sung-Bin. Oh Hyun-Big Byun-Bin, JungMok Lee, Arda Senocak, Joon Son Chung, and Tae-Hyu Oh. Avhbench: A cross modal hallucination benehmark for audio-visue large language models. arXiv preprint 2410. 18325, 2024.
16

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant
information is Not Available.
Please provide a short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked "[Yes] is generally preferable to "[No] ". is perfectly acceptable to answer "[No] provided a proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset used"). In general, answering "[No] oo "[NA] is not phrased in binary way. acknowledge that the answer
the checklist as of the factors in their evaluation. While
grounds for rejection. While the questions often more nuanced, so please just use your best judgment and write a justification elaborate. evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
supporting
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We have tried to be thorough while presenting the primary contributions of this work. We will provide more details in the supplementary
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can he expected to generalize other setrings.
It is fine to include aspirational goals motivation long it is clear that these goals attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We add limitations and future work in the conclusions section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors encouraged create separate "Limitations" section in their paper.
17

---

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model sell-sppciiicatton asympiotic approximations only holding locally). The authors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect the scope of the claims made, c.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance the approach. example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-ter text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If l'applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve integrity the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: We don't provide any theoretical results. Our method and benchmark explained in detail, and we back every claim with empirical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should he numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have tried to be thorough while explaining our approach, and we will publicly release the code and dataset upon acceptance
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of model checkpoint, other means that appropriate to the research performed.
18

---

While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. . a large language model), then there should either be a way to access this model for reproducing the results a way to reproduce the model (e.g.. with open-source datasct instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of bosed-source models, it may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other rescarchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does paper provide open access to the data and code. with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will publicly release the code and dataset to ensure reproducibility
Guidelines:
The answer means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines httpss:///ips.cc/pupppp
guides/CodeSuumiss ssinoloccy for more details.
While we encourage the release of code and data, we understand that this might not be possihle, so "No" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (e.g.. for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: https::///ps..oo cacippscccppuie cctpubl Quides//CoeeSuiie siiiissssooolooooooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer. etc.) necessary to understand results?
Answer: [Yes]
Justification: We have tried add all such details
Guidelincs:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [NA]
Justification: Not applicable for
Guidelines:
The answer NA means that the paper does not include experiments.
19

---

The authors should answer "Yes" if the results are accompanied by enoor bars, confidence intervals, statistical significance tests, least for the experiments that support the main claims the paper.
factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions madc should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is report I-sigma error bars, one should it. The authors should preferably report a 2-sigma error than state they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric enror bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question: For each experiment, docs the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We will add these details in the supplementary
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide amount compute required for each the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't I make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https::/neurt: http://wwwpp..ccooooooll//c/csss
Answer:
[Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require deviation from the Code Ethics.
authors should make sure preserve anonymity (e.g., if there special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts the work performed?
Answer: [NA]
Justification: We don't 't deal with any sensitive information. The collected samples are under appropriate licence and free to be used
Guidelines:
The answer NA means that there is no socictal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
20

---

Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g., deploy- ment lechnologies that could make decisions that unfairly impact specific groups), privacy considcrations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible relcase of data or models that have a high risk for misuse (e.g. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work doesn't 't contain such risk
Guidelines:
The answer means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of model, for example by requiring users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We duly cite and refer to parts that has been used as a part of our study, We will further acknowledge all the baselines in the final version of the paper
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version the asset is used and, possible, include a URL..
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwaeersw thcode. haaeeee.ccm/aaaaaaaaaeeeeeeeeeae danaetss has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
this information not available online, the authors are encouraged reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
21

---

Justification:
We will release the data and code publicly and will add the documentation
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/codc/model as part of their sub- missions via structured templates. This includes details about training, license, limitations,
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We will add this in the supplementary
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least minimum wage country data collector.
15. Institutional review board (TRB) approvals or equivalent for research with human subjects
Does paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [Yes]
Justification: Our work doesn't 'I involve any risk
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review,
Declaration LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non ata-standard methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
component of the
Answer: [No]
Justification: We don't involve the usage of LLM other than experimental and data preparation purposes
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, or non-standard components.
Please refer to our LLM policy (ttt:::://eerriisssccc//oreeeees should or should not be deseribed.
for what
22

---

MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Vider Haystacks
Supplementary Material
The supplementary is organised as follows:
Supplementary Video
Dataset Statistics
Qualitative Results
More Ablation Results
Human Evaluation on STEM
F
More Details on Benchmark Construction
G
SFS Prompt
Failure Cases
Implementation Details
More Related Works
Human Study Details
A Supplementary Video
In the supplementary video, we elaborate demonstrate the intricacies involved in multi-video linked QA setting. The video shows how the given question involves referring to multiple videos obtain comprehensive answer. followed by the meta agent summarising the responses to come up with the final answer. We also highlight the salient components of MAGNET and discuss the end-to-end flow. The use of headphones is recommended for better audio-visual QA experience.
proposed task AVHaystacksQA with illustrative examples and
B Dataset Statistics
In this section, provide additional details about AVHasstacks. Tab. summariizs the topics from which the samples are collected, along with the number of questions per category and the corresponding video-to-question ratio. The benchmark comprises 103 hours of video content from 500 video samples across 27 diverse categories, accompanied by carefully annotated QA pairs that temporally ground salient segments within the videos. To the best of our knowledge, this is the first benchmark of its kind, as no prior work provides multi-video linked audio-visual QA pairs.
Fig. depict the distribution of total video hours across the various categories included in our benchmark. As shown, the samples well distributed and sourced from wide range of scenarios. A significant portion drawn from Travel Destinations, Education and I Language Learning, Music and Performing Arts, and DIY and Creative Hobbies domiinss that demand strong audio-visual comprehension for effective temporally grounded AV QA.
Fig. illlsttrates the distribution of sample durations (in minutes) within AVHaystacks, Most video samples fall within the 6-15 minute range, posing substantial challenges for temporal grounding tasks. Evaluation models must handle long context windows and effectively manage the increased complexity processing extended multimodal sequences (vision and audio).
Additionally, Fig. [ shows the distribution of the number of videos associated with each question. As observed, the majority of questions require referencing two or more videos to determine the answer, further emphasizing the complexity and richness of AVHaystacks.
Finally, it is to be noted that: (i) Our dataset annotation process requires manual inspection to ensure complete correctness. Although the sample curation process is automated, ensure strict sanity we manually validate each sample, which both tedious and tnne-consumnng (ii) The models evaluated our study are already hidiolvisually informed, and we fine-tune them efficiently using LoRA adapters. Our salient frame selection module enhances context by focusing on key frames. With LoRA, we require fewer samples to fine-ture our audio-visual agents, they heavily pre- pre-trained on audio-visual data. Importantly, we only fine-tuning
23

---

Total Hours by Tople Cadegory
30
Time in Hours
20
3/1
10-
Thoud Deslinations
Ederation & Langrags Leasiind
Malic is Purforninn Arts
per Craatie Habblp
Pasooall Deviopound Sells
Sports Outler Artivitae
Noo Correct Evouts
Figure 5: Number of hours of videos per topic category in AVHaystacks benchmark.
Topic Chitegory and Number of Videos
125
Number of Videos
110
3.00
100
90
50
40
25
20
Treed Dasticatin
Education & Langrage Leaniing
Malk Perfornind Arts
Drr Creatire Hublie
Personal Developennt & Lift Skilt
Sports Oubdiar Astvitti
Curnast Enands
Figure 6: Number of videos per topic category AVHaystacks benchmark.
the agents, not training the entire retrieval pipeline. Notably, the SFS module can be used with a glosed-source
model where finetuning was not done. As seen in Tables and these methods demonstrate strong performance
without any fine-tuning. Similar observations made in recent literature [89 91 demonstrating that employing LoRA enables these models perform well with limited samples.
Distriutioon
Videos Acrosa Davations
200
Namber of Videos
150
137
100
688
46
50
min
11-47 mid
16-20 min
24-25 mide
10-40 min
40-40 min
6-10
45-30
Figure 7: Distribution of videos based on their duration Aayasstacks benchmark.

---

Ditrilition of Queations Asross the Nummber of Videos Used to Anowerr
1400
1308
Numxber of Questions
943
200
11
4
2
Number Videos Used
Figure 8: Number of videos referred to in each question AV Haystacks benchmark.
Topic
Videos
# Questions
Ratio Videos:Question:
General Cooking Tutorials
20
117
1/6
Learn Vocals
20
51
2/5
Learn English
20
127
1/6
Learn Arabic
174
1/9
Learn Chinese
20
115
1/6
Learn Urdu
247
1/9
Travel Turkey
10
78
1/8
Travel Brazil
78
1/4
Travel UAE
20
105
1/5
Travel Hawaii
20
105
1/5
Travel USA
20
106
1/5
Travel Italy
20
106
1/5
Sing Beatbox
20
129
1/6
Learn Opera
20
123
1/6
DIY
20
187
1/9
Learn Sketching
20
89
1/4
3D Printing
20
189
1/9
Public Speaking
20
97
1/5
First-Aid
20
125
1/6
Self Defense
20
126
1/6
Soccer Analysis
20
139
1/7
News Tornado
20
112
1/5
Sign Language
20
107
1/5
Hiking/Backpacking
20
151
1/7
Playing Music Drums
20
80
1/4
Playing Music String Instrument
34
1/7
Playing Music Piano
50
1/10
Total
500
3147
1/6
Table Distribution videos and QA pairs across different topics.
25

---

What the essential knni's skills for different vegetables to ensure uniform cuts and safety?
Smuud trot
3 Keep lovves sharp to minimise risk and preserve ingredient quiliy. jo (01365- (1s35) 2 for onions, leave the rost intact to raduce tearing: posiiion thres fiegtrs on top, nit the knife sightly and use quik slising motion. [3 (00021-004 mool--- (3) For mushrooms, apply "one fieger forward, two bchind" guided mation. sliting steadll in het pan if needed. tat (co53s-coEas) & aeese (01365- 03---- ( for leeks, seli lengthonise into quarters, rinse off dirt, and slice evenly [4-tet (0053s-00) (Osts--0SRs) confident grip and keep fingers behind the knife spint for safety [4.trt (0053s-0088S]
Video
Eip have sharp knives Therd's commin mnlcnicaptin that shary knives are dangerous, when in fact dall' bnives are even more dangerous. You have to put more presourt: dall kedkf, and gesrre more tiidy to dij and cut gearsil your buuft in'tt sharp, you Try
and that bringi to
chat's confortaile your should super shorp you struggle while tucting."
feem how sharpen it
videe this
(S. Maintain
(3) sharp lnives are essential for safety and maintaining ingredient integrity. [a 231244-05-m 20224-00-00)-22 2. when cutting onions, keep the root attached to avoid initation adopt a claw grip and make switt, angled cuts. [1 [itbt
then, grip the onion tenvis to hold Firmly in place Use the weightt the throough
(0004s-00k110) ( For mushrooms, use guited sicing method- advance one finger while keeping two bakind for control, especially
mation continor shopping what war'r left with just the really wise, fiodly chappeed onion new, bativali rice -ine king of all vites-light light, fullly and deficine
And look that
[4.tet (00575-000 000300 & autet (01301-02850) (4) To clean and cut looks, siice them lengthuris into quarters, rinse
heated
I'm going to show you how to cook it perfectly
0.39
125
Meixtain control halking the lodie securely and tucking fingers from the blade [4.tt (0048s-008 925))
tharsughly to remove dirt, and ckop evenly [4 (00805-0082s]:
video
$L
Sharp loiiees ensure safety and preserve ingredients when catting oviond. keep the rose incact to reduce irritation, use a claw grip, and cut quickly at an angle. For mushhrooms, guide the knifs with ane finger forward and two behind for control, especially in a hot pan To propare leeks, quarter them lengthwise, rinse well, then chop evenly. Always hold the knile firmly and keep firgers tucked away from the blede
bol, start the soace thingg the muderaans First off, renezmbrr: fingers ooe froee, two lethie mind thin
up-and idoi mation to sice calilys and ovtely that's al" the water coming out of the mashresms Nest, findly shop fst clove of gurlic Then propart your looks toke your koifs and down through center, turn over, and slice again into quarter Theyll up
up little quarters Add all that ieok the mushesemi -baatiill
taps
any
Figure 9: Performance comparison of MAGNET and Hideo-SALMONT on cooking video tutorial.
C Qualitative Results
The qualitative examples in Fig. D to Fig. 13 showcase a variety of scenarios, including cooking demonstrations, language-se -peauggggg tutorials, and 3D printing lessons. In these examples, we compare the responses generated by baseline models with those produced by MAGNET. As illustrated, AVLLMs enhanced with our retrieval and multi- agent audio-visual reasoning modules consistently outperform the baselines on AVHaystacksQA. While the baseline models often struggle to identify and retrieve the most informative video segments, our method accurately selects the most relevant videos for each question, exhibiting strong audio-visual comprehension and reasoning capabilities. Furthermore, the temporal windows identified by our approach generally precise and focused, reflecting nuanced cross-modal understanding and effective temporal grounding across videos.
D More Ablation Results
Top-k Video Selection. The top-k selection experiment on AVHaystacks-Full shows (Tab. 8) that both Qwmn-2.5-Ommi-FF and Gemini-1.5 1.5-Po achieve peak performance at l = 6, where BLFU4, Text Similarity, GPT Eval, and STEM-Missing metrics all indicate optimal retrieval and generation quality. Gemini-1.5-PP consistently outperforms Qwen across all l values, demonstrating stronger contextual understanding and output coherence. While increasing & generally improves performance by reducing missed content, values beyond A = 6 offer slightly diminishing returns and may introduce noise. Further, it leads to increase in compute (since more AVLLM agents are required). These findings highlight the importance of selecting an appropriate and leveraging more capable language models like Gemini for effective multimodal retrieval.
Meta Agents. Tab D analyses the impact of different meta agents within the MAGNET framework on the AVHaystacks-Fuu Full benchmark. Across both base models: Qwen-2.5-0mmin
and Gemini-1. 1.5-Pro performance consistently improves as stronger meta agents are used, with Gemini achieving the best results in all metrics (BLEU4, Text Sim, GPT Eval, and SrEMMOrder). Notably, Gemini, both the meta agent and core model, yields the highest overall performance, highlighting superior capability in segment selection, coherence, and multimodal reasoning. These results underscore the critical role of the meta agent in guiding high-quality content synthesis.
Frame Sampling Function. Tab. I0 compares the effect of different frame sampling functions (SFS) on the HaHastackssFFul Full benchmark using base models: MAGNET +.wm Om and MAGNET +Gemini- Pro.
26

---

How can conscious jaw movement improve Englishh vowel clarity?
Loosen the jaw each
Video
vowel (eg. "sh," "er")
resonates fully [10.tt (0235-493) 95] 2 Practice
opening slightly wider for vewels
"father,"
what Americans do take smaller words and pronounce them ways that easier daily speech. For example. the article 'the often provounced "tha," like in "Whatt' the weather today? becomes "whar's weather today? because it's simply casier: Similarly instead clearly saging 'Do you for walk? wwhcch like started learning English-they might say.
aligning mouth corners
outward [19.tet (01135-
01778]] Sustain vowel in words WHee "doctor" or "water, focusing jaw drop. [37.txt (0115:-038*5) Record and compare with native samples, adjusting jaw tension for clarity [2 txt (0520s- os 01445)
'yycu go walk? That sounds native. Another thing you might notice.....
0.44
Video
making these sounds As say all my videos, order to matter these American sounds, we need practice them over and over and over again- like going to the gym. Think in as your personal trainer. There's no way around
Our bodies are just not
and anghody tells you atherwise, they've lging it takes practice And on this video, and on my other videos on this channel that's exactly what we're going to do. you haven't get. subseribe to my channel so you can get access these videos to get you one step closer mastering
Relax the jaw to enkance sound,
each vowel
such
"er." [30.txt (025s-
oss3J. Open the mouth slightly
for low
Video
ACCENTS
NET
vowels like "father."
ensuring horizontal mouth alignment:
[x9 [01105-01795] $ Focus
on jaw movement when
explores koy traits of three American accents The New York accent is knawn for dropping the "R"
sustaining vowels in words
at the and or in the middle words (4g. "water" becomes "wata"" "parkc" becomes "pabk," "New York" becomes "New Yauk") and changing the "AL" and "OR" sounds to "aw" (4g. "talk" becomes "tawk," "call" becomes "caw"), Reductionn are also common- - "you"" often becomes "ye"" The Boston accent shares these features, especially dropping "R's (eg. "car" becomes "cak') and changing "OR" sounds to "aw" (e "force" becomes "Fawce"), it also features an upward inflection at the and of sentences Meanuchile, the Southern drowl is slower and stretches out vewels by turning short vowel sounds into diphthongs. often adding schula (egg "have" becomes "ha -w""....
"doctor" "water."
[37.txtt (01201-0348s) +
Record your speech and refine articulation by
adjusting jaw tension match
native
pronunciations. (01175-01976)
[1tt
Video
CAN
Loosen your jaw to improve vowel sounds. Open your mouth for certain maintaining vowels. Notice jaw aligument movement when holding sounds. Use recordings fine -tund your articulation.
explains the glottal stop, feature commmnn in casual or regional American English. Instead of pronouncing the "T" sound in words like button or mountain, speakers often replace the "pp with a quick paise or break. This different from the flep T, where the "Tr sounds like a soft "D" (as in butter budder)) With the glottal stop, the "T" disappears entirely- eevely-button -utton sounds like bulon, and mountain like mouniain. This feature is especially naticeable in fast, informal speech.
Figure 10: Performance comparison of MAGNET and Unified 102 on english pronunciation tutorial.
Across both models, we observe that the proposed SFS function (highlighted), which dynamically scales dissimilarity using inverse sine function, consistently outperforms cosine and exponential alternatives. For instance, with Owen-2.5-0mmiiFFr, the proposed SFS achieves the highest scores across all metries. Similarly, when paired with -eminni-1.5-Pro the same function yields the best performance, achieving high BI BLEU4 and a notable MTGS These results suggest that the iatuusidddliivees--baaaa.. SFS is more effective at capturing temporal importance for reasoning tasks, likely due to its sharper penalisation of semantically redundant or temporally proximal frames. This indicates that thoughtful frame selection plays critical role in enhancing downstream multimodal generation quality.
Number of Video Frames Selection. Tab Tab anallyzs the impact of varying the number of uniformly sampled frames m on AVHaasstakss-Hi using two base models. Across both models, performance consistently improves as the number of frames increases from 15 to 75. For instance, with Owen-2.5-0mml 45.89 to 53.61, and MTGS,sg improves substantially from 0.43 to 0.83, indicating that denser frame sampling leads more accurate and temporally grounded responses. Similarly, Gemini-1.5-Pa shows a consistent upward trend, reaching its peak performance at m = 75. These results suggest that richer frame coverage provides stronger contextual grounding for both models, reinforcing the need for higher temporal granularity in nudio-visual reasoning tasks.
BLEU @4 rises from
Text Similarity Threshold.
Tab. 12 presents the effect of varying the text similarity threshold r. used in frame filtering on Haysttacks-Full For both methods, performance peaks = 0.5, with BLEU@4, Text Sim, and GPT Eval achieving the
27

---

When reading a Quranic phrasse, how do shift from the "light" alif a "heavy" letter immediately after?
Gonad Tram
Let
nextral
keep the vecal tract lightly open. [12.ttt (062 -o446] 2) Prepare for the heavy letter (eg. J) by tending back your (075-0e1s) g tongue root upward just before producing the heavy sound (20.tt (075-09IS) & 17.xt (045-0875) 4. Practice short recitation segments out the transition. [30.0xt (075-
Video
segment introdaces concepts taficheom (henrises) and tarqeea (lightnes) Arabic pronanciation, Tefkhccm targooq soft, light light-swnnding
pronounced with heany, white
light say -ffy- with spesker gives csamples, letter produce
require rounding lips
1:02
2:44
(91s)
Video 12
[muusens nano
3) keep the alf nestral with gently open vocal tract. [32.txt [057-0836] ( engage
back tongue prepare heary letters like d. [30 (070-08Rs) (S) slightly raise the before
segmeent explains how several Avabic lounds art produced from the throat, an arta communiyy
challenging for nan -native speckers who may confuse these sounds. To clarify their pronanciation, the
throat is drvided into three main regions: the upper threet (deepest point, near the crag). the widdle
throat (oightly alove that), and the lower throst (at the start the throst), The lesson focuses on
helping learnors distingush between these siwisr- sounding letters by understanding their eoact points of articulation...
articulating the heavy
sott (073 (075-0810 es) & 17.60 (041 0441-600]] 4 rehearse short
recitation passages to ensure smoother transitions. [30,00 (073-9905)
Video 17
learning Arabic, speciiiallly forsning set of Arabic letters: (KAF), J (Ldm), (M0m)) (Nan). (H48), (Wilbu)), and if ('d)). The first letter discased (KAP), whish likened to the "i" soond English, such as the & in "cat"" The teacher emghasizes focusing on the sound, not the English letter used to spell the word (since "cat" starts with a C. but sounds tike it starts with a K). The explanation also kines that different forms of the Arbbic letter (solated and postiooal) will shown and practiced....
Keep the vocal tract neutral. Use the back of the tongue for certain heavy sounds. slightly lift the tongue before these consonants. Practice short passages smoother flow.
Figure 11: Performance comparison of MAGNET and Qwen-2.5-Ommi Urdu pronunciation tutorial.
What are the steps to correctly articulate consonants and vowels during vocal exercise?
p
light
movemonts (eg. gentle gawns, stretched lip (0505;-05561] 4nxt0+66s-05760) X
Video
articulators. focused articulation drils, shaping (Ah, (eg. 'Buh!" Tuk))
reciting consonant
separately [40xt (0005s-0039:)
Larticulation routine begins with something natural guuning but challnging putterre tongse twisters and repeated phraee lik "pago bard goy play gay" Though the -. the iuportance of accuracy our breath intral-tistion the lip. jesr, tangue thoold avidad, and braathing will felow naturnlly artioulation priorizzed ericcintory mustls, making speech sharper confident. professionll...
both
offetive
keep minimal jaw and tangue movement
open space smooth transitioes between vawels short
Practiss simple pèrases
emghaiising crisp contonants and resonant.
encourages daily repetiion, whether for
unttraked vowels. 130xtt0446s-04550 @
speed gradually
tangue twisters repeated ?
patterns. [5 [5.00(0355s-0577))
lips. jaw, and tongue, releasing detected. [3.ue(po4ss--eaaa & Integrate these
C:03
0.5
740
refined sounds into songs or loeger pieaaees. (4 (+-00(04466-55765))
Conseet nane
Disgii with light facial and mouth movements such as gostle youns or Uip stretakes to activets the articalators [Sttst (o2nts-osstsl] (2) perform lip or tangue trilss to further lossen the mostles [4.b (0649)-055 soj 3 use articulation drill- -recit consonants like "Bak"" and "Tuk," and shape voneli (Al, Es. Gh. Oo) distinetly. [4.btt (0000:-20435)
heles dovelop and expressive -happing indivilual concontnts and buids the
practite only
finndation for faord speesh. minimal strain, and dear facus, As become more consittied.
theie frandational dells can be used to build into more advarced spealing or singing condects with
sase...
4) minimite jaw and toegut movemont to moistaln
openness and fluid vourl transitions 3 rehearse
short porases or scales, fecusing on clear consonants
and relaxed, resemant vowels. [3 tat (04315-04700 7661]
6, gradually increase speed using tongue twitters
Start gentle
movements warn up. Lossen moscles with lip or
repented phovatic patteres [3.btt (05225-03 (0022--03700) vosl
tongue trills Practice clear condonards distinct vowels. keep jaw and tongue imooth transitions. Use short phrases gradualyy incrense speed with exercises. Rekaas Jnyy tention, and apply these techniques when singing pirases.
7. monitor for and release any tension in the fips,
jaw, tongue [btttt (04345-0475) 7-03) (8) apply
these refined techniques while singing full phrases.
[4.ut (0458--05445)
Figure 12: Performance comparison of MAGNET and Video-RAG on vocal exercise tutorial,
28

---

In first layer calibration, how does proper bed leveling and nozzle height adjustment influence overall print success?
18:56
first layer crucial strong athetion the build surface (saee(ssea tut (0064)=01sé60)
Video ø
Using feeler gruge or paper test endures the nazzle the correct distance- cisse couses squished layers (elephantss foot). too leads to poor adhessin (sciionnaccc (so. (00st:-01sed))
starting skirt prine the nazti, folowed by perseaters and infil. satiffed. the fir saved and traneforred to the pridder Nust comsss bed levelng vital step that trips up many beginners The Hed is minsally aijusted using corner scrted and a piace paper to tnturt cotn quacing betweon the Proiesting
how
will
Manual adjistment through corner scrows achioves an even. gap across the bed. [o (0857)-1094s]] ]]] ( proper first loger provents sulstquent laytring limes. enturing consittnnt (ager bonding (16-tet (0069)-01650)]
homing pridter
this aqjatment...
2346
3) tesure the initial layer well-formod, es provides essential adhasion to print
[36 (00765-05749) 2) use a paper or feeler
gaige tast to set the nauzls height-tto -oo low results
compressed base ("elephant's foot"), while too high weakess adhesion. [3 (G0715 006685])
it causes squilhing,
The first layer critical for encoring sold athesion and overal print quality the noute too cise.
damage, 'laphand's foot" the flammnt athere
5. marualll level the bed by adjusting corner
screws to maintain on even gop across the surface
gauge confenus koming heating this property. the right wazale to- succesful right also prevents laytring delects and improves dlmeniional occuricy final print.
to (004s-100114] (6) well well-exxccuted first
layer supports conaistent banding in inter loyers.
minimiting print defects. [56 txt (00e6s--15-9))
layer good
height carefully
evenly by adjuting
strong layer helps endure conostent, defect-free pristing
Figure 13: Performance comparison of MAGNET and VideoRAG on 3D printing tutorial.
highest scores, while the TYEM-Missing and STEM-Order emrors are minimized. Specifically. Qwen-2.5-
Omni-FT improvements in BLEU @4 from 51.32 to 53.61 and drop in SrEM-Missing from 0.15 to 0.13 as T, increases from 0.3 0.5. Similarly, Gemini-1.5-PPe reaches optimal performance at T, 0.5, with the best overall textual coherence and minimal temporal grounding crror.. However, setting T, 0.7 degrades performance across metries, suggesting that overly strict filtering removes useful context. These findings highlight the importance of balancing informativeness and precision in frame selection by carefully tuning the similarity threshold.
E
Human Evaluation on STEM:
Tab. 13 presents comparative evaluation of various MAGNET model configurations using both automated STEM and human evaluation metrics averaged across 20 raters (Cohen's 0.82). In particular, the fine-tuned MAGNET Qwen Ommi-FT model achieves strong overall performance, outperforming other models in most categories. Specifically, it ties for the lowest SM, achieves the lowest SH, and second-lowest SO scores under STEM, indicating improved semantic and syntactic alignment. It also performs competitively human evaluations, with HM, HH, and HO scores close hetter than all other non-proprictary models. Although MAGNET Gemini 1.5 Pho demonstrates slightly superior performance across several metrics. These results underscore the efficacy of fine-tuning with Qwen 2.5 Omni, particularly in aligning model outputs with human judgments.
Method
K
BLEU@41
Text Sim
GPT Eval +
STEM BeEMMMMsssingg
1
46.67
4.67
5.37
0.29
3
50.44
5.93
6.88
0.20
MAGNET +Qwen-2.5-0mmll
6
53.61
6.28
7.53
0.13
10
53.38
6.18
7.36
0.14
1
49.80
4.84
5.04
0.27
3 52.09
5.22
6.31
0.19
MAGNET +Gemini-
6 55.87
6.53
7.81
0.12
10
55.63
6.41
7.64
0.12
Table 8: Top-k selection on AVHaystacks-Fun
F More Details on Benchmark Construction
In this section, we provide further details on AVHaystacks construction. We outline the steps involved in data preparation Fig. 14 The complete benchmark creation pipeline details.
29

---

Method
Meta Agent
BLEU@4
Text Sim
GPT Eval
STEM-Order
Reka
51.38
5.91
6.88
0.25
Qwen-2.5-Omnic
51.98
5.98
6.96
0.24
MAGNET =Qmen-25-0mmiPF
Claude
52.29
6.05
7.34
0.22
GPT
52.93
6.11
7.37
0.20
Gemini
53.61
6.28
7.53
0.19
Reka
54.89
6.02
6.80
0.22
Claude
55.02
6.13
7.09
0.20
MAGNET +Gemini-1.5-P 1-PPoo
GPT
55.32
6,20
7.48
0.19
Gemini
55.87
6.53
7.81
0.17
Table Effect meta agents on AVHaystacks-Full
Method
SFS Function
BLEU@4
Text Sim
GPT Eval
MTGS$p
aab 7 (oos (3 - bl) - 1) 10
$2.85
5.93
7.24
0.79
MAGNET 5-0mnFF
Aub (eNla-M 1): 10,
53.18
6.03
7.47
0.81
Aat 21o 81)11 -1 20
53.61
6.28
7.53
Aab (oos (gla bl) 1); 7 10
55.20
6.01
7.33
0.80
MAGNET
A y (e 1): 10, 55.79
6.15
7.62
Aa =
20
55.87
6.53
7.81
0.85
ain ii(G(ooAl)++
Table 10: Effect different frame sampling functions on AVHaystacks-Full
Method
BLEU@4
Text Sim
GPT Eval
MTGS
15
45.89
4.46
5.10
0.43
MAGNET +Qwen-2 OmmiiiFT
50
49.27
5.19
6.03
0.62
75
53.61
6.28
7.53
0.83
47.36
4.92
5.44
0.51
MAGNET +Gemini-1.54
51.56
5.81
6.62
0.69
75
55.87
6.53
7.81
0.85
Table 1I: Effect of number of frames selection on AVHaystacks-Fin Full.
Method
T,
@ f
Text Sim.
GPT Eval
STEM-
STEM-Order
0.3
51.32
5.75
6.68
0.15
0.24
MAGNET +Qwen22.5 Omni PT
0.5
53,61
6.28
7.53
0.13
0.19
0.7
50.67
5.52
6.43
0.19
0.22
54.13
6.31
7.56
0.13
0.22
MAGNET -iiemini-..-PPoo
55.87
7.81
0.17
53.95
7.51
0.17
0.20
Table 12: Text similarity threshold on AVHaystacks-Fur
F.1 Dataset Examples
Please find example videos with QAs from different categories shared in the supplementary zip (refer within the supplementary
the 'AVHoystacks-datass samples' folder). The video files are compressed material size limit. Samples are collected from different areas (how-to, musical lessons, news etc) making our benchmark extremely diverse and considerably challenging for the models. The purpose of curating samples from such diverse sources is to robustly evaluate every model their generalization capabilities.
30

---

STEM
Human Eval.
Method
SM 4
SH 4
SO 4
HM 4
HH
HO
MAGNET -VdooS5LLMNN--S
0,41
0.33
0.43
0.39
0.37
0,41
MAGNET +Unilied IO2-ZS
0.49
0.39
0.37
0.46
0.35
0.36
MAGNET +Qwen 2.5 Omni
0.43
0.34
0.39
0.45
0.37
0.42
MAGNET -VddooSALMONN-FT
0.13
0.18
0.23
0.15
0.17
0.21
MAGNET +Unified IO2-FT
0.15
0.18
0.20
0.19
0.23
0.22
MAGNET +Qwen 2.5 Omni-FT
0.13
0.16
0.19
0.17
0.18
0.16
MAGNET +Gomini 1.5 Pro
0.12
0.14
0.17
0.14
0.18
0.13
Table 13: STEMvs Human on AVHaystack- dataset. SM: STEM- Missing, SH: STEM- Hallucination, SO: STEM- Order, HM: Human Eval Missing, HH: Human Eval. HO Human Eval. Order
Halllcciation,
Transonjet Sonipts
Blind Questions Gévération
Transcripts Segmentation
Segmertss Quessiins Generation
Question Answer Pair
You Tube
Transcripts
are different so questions that be answered by video and auáio and car. be awswered by <TOPI'
Transcripts
generate precise, instructional, and comprrseensvv answerss for eack question by waticuloudyy extrrccingg and synthesizing information solely from the provided transcripts
Transtripts
Go over eack file one by and question. each instructional focus segment
saftstor
Crt mahili
topics
Sope
Ping for
Anablor Heaget
on miai
wini
In Siacial
Figure 14: Steps involved benchmark data collection.
F.2 Pipeline overview
Our multi-step data collection strategy is as follows in sequence: (1) download and curate 500 videos that satisfy the modr-modality filter (Dataset Selection); (2) issue 50 blind prompts per topic (Listing 1 to GPT without revealing transcripts to encourage cross-video reasoning; (3) repair caption timing (Alg. 1) and convert HH:MM:SS -scconds for uniform indexing; (4) segment transcripts into sub-topics (Listing(4) and create one segment-awee aware question each (Listing and (5) assemble QA pairs whose answers cite least two videos (Listing[))
F.3 Question-pipeline specifics
(i) Blind phase: prompt GPT with Listing discard any question whose answer is general knowledge information answer that does depend audio and visuals; (ii) Segment phuse: To expand the number of questions, we used segmented-phal questions. each sub-topic, prompt GPT with Listing ensure the generated question references audio, visual, and caption tokens.
Listing I: Blind questions generation prompt
I am collecting data about how to <TOPIC>.
What are different 50 questions that must be answer by video, audio and can not be answered by text only?
Listing 2: Segments- based questions generation
Go over each file one by one and create question for each segments
31

---

F.4 Caption repair and segmentation
(1) Overlap fix: downloaded subtides had mis-matched intervals which to fixed (example Listing (3):
(2) Normalization: map all times to seconds and drop duplicate caption lines; (3) Segmentation: apply the template in Listing 4 and retain segments whose duration lies in [15, 120]
Listing 3: Transcripts timing mismatches
00:00:00 --> KM030* hi everyone welcome to youtube channel 00:00:**0100 ** --> 00:00:05
about parrots. am david. i'm here with
Listing 4: Transcripts segmentation output format
Segment [Segment Number]
Time: [Start Seconds] [End Seconds]
Title: [Concise Title]
Details:
Instructional Focus: Brief description
Key Steps and details:
[Step Detail 1]
[Step Detail 2]
[Step / Detail 3]
[Step / Detail 4]
[Step Detail 5]
Audio Cues: Audio elements description
F5
QA-pair generation
After transcripts are segmented, GPT is prompted to produce a triple consisting of (i) the question itself, (ii) step-by-step answer, and (iii) list of (videolD, start, end) references that support each step. Crucially, every answer must draw evidence from multiple videos, making cross-clip reasoning requirement of AVHasstack (see Listing 5).
Listing Question- pair output format
Question
Answer:
1) step 1
2) step 2
3) step 3
4) step
5) step 5
References:
1.txt 0817s > 0074s, 8. txt 8045s 0270s,
txt 0850s 0100s, txt 0110s 0150s
G
SFS Prompt
Below, we add the prompt used to select the key frames using the SFS algorithm. We provide step-by-step. clear instructions about the task, the reasoning process, and the expected output. Among various other prompts employed, this produces the best results.
32

---

How blend violin with cello for balanced and harmonic emotion?
Agtnt Oxtput
Violin leads with
Video
melody; supports with soft
cello
arpeggios. Clear
The violin Read with the indledy, white the cell plagi sift arpeygios underaesth. There's clear ieparation regiider between the instruannss The counderpaint between theme contributes balanced timsbra fextart...
register separation and create balance.
counterpoint
timbral
010
1Lbx(0112-01365)
Vidco
② Call-and cand-response
adds dynamic interplay.
phrasing
vesponne happening hare-that pirasing creates dgnasnic interping
Matching articulation unifies tone while EQ keeps them distinct. [2.txt(2-2 105]]
betwrden The orticulation. really matched, helps anify same time, daing work to keep sack part distinct mik...
0:4
(
Close harmanies in similar range use bowing contrast for clarity. Natural revers adds spatial separation. [3.txt(1- 146)]
Video
reail helps keep things giving owns place.
range, the Eawingg-haat seporation. betweon parts,
space
Figure 15: Failure case of MAGNET.
Salient Frame Selection Prompt
Task: Given question and of key frames extracted from video, identify the most relevant frames that best support answering the question.
Step 1:
Reasoning Process Explain your selection by considering the following factors (in order of importance):
1) Presence of objects or actions explicitly mentioned the question
2) Scenes that clearly align with the question's context
3) Visual elements directly related to the question details
4) Location background context, even if the main object/action is visible
5) Semantically related or typically co-occurring objects
6) Human motion or activity suggesting relevant events
Step 2: Output List the selected image indices using the format: [idx], idx2, idx3, ...]
The objective is select visual evidence useful for answering the question, not
to answer the question itself.
H Failure case
Fig. [5 illustrates failure case MAGNET. Owing to the visual and auditory similarity between violin and cello, the retrieval module fails to accurately identify segments where both instruments are played simultancously. As shown, the selected segment from the first vidco fcatures only the cello, omitting the presence of the violin. The other two retrieved videos result from incorrect retrieval, leading erroncous temporal grounding and highlighting limitation of the system distinguishing between acoustically and visually similar sources.
I Implementation Details
Training done epochs on GPUs and checkpoint selected for evaluation. Following the success of Low-Ran Rank Adaptation (LoRA). we apply it with a rank 8 and an alpha value of 32 for fine-tuning AdamW is used as optimiser with a learning rate of le-4. We use per-device batch size of I and gradient accumulation step of 16. A cosine learning rate scheduler is employed with warmup ratio of 0.05.
33

---

Dataset
MS
TA
MVL
AVR
AVD
ROA
AVQA
LC
Video Datasets
Mideo-Bench 116
X
X
X
X
X
X
EgoSchema 45
x
x
X
X
XXX
X
AXX
MVBench [34
X
X
X
X
MMBench- Video
117
X
X
X
X
X
Audio- Visual Datasets
AVOdesscyBBench
118
x
X
X
X
EXX
OmniBench 119
x
x
X
x
LongVALE 200
/
x
X
AVHBench [11
X
X
AVHaystacks (Ours)
Table 14: Comparison with prior video/audio-visua benchmarks. MS: Model-Assisted: TA:
Temporal Annotation: MVL: Multi-Video Linkage; AVR: Audio Visual ane-grained Reasoning; AVD: Audio Audio-Visual Description; RQA: Retrieval-based QA Answering; AVQA: Audio-Visual QA;
LC: Long Context, where QA context spans over 5 mins.
J More related works
Large Multimodal Models. Large Multimodal Models (LMMs) have advanced significantly in understanding reasoning over single and multiple images 93 24
95], expanding vision- langual capabilities across diverse tasks and domains 99] Their strength lies in large-scale cross-modal alignment and powerful language modeling. However, LMMs still struggle with scaling to large image or vidco collections
(82) [00] duc computational and representational challenges. cetrieval-basee approaches address this by enabling efficient access, processing. and reasoning over extensive multimedia content, including vidco and audio.
(Audio) Benchmarks. compare AVaaystacks with recent audio-visual and vidco QA benchmarks Tab. [4] shown, most existing benchmarks do not offer multi-video linked QA annotations, making AaHasstacks the first of its kind and inherently more challenging. Our data collection framework introduces a scalable, semi-automated, and richly annotated pipeline in real-world settings, providing significant advantages for future research. We hope this benchmark will inspire the community explore and advance work in this promising direction.
Audio- Visual Learning. Powered by improved models and high-quality annotated data, audio- learning has advanced significantly in areas such as cross- cross-modal generation [101] representation learning
multimodal large language models [109 110 30
57 6 11]. and cross-modal integration 1124 Recent
works have contributed to cross-modal generation by leveraging visual and/or language context generate coherent, complex audio [101 [02). The work active audio-visual separation and embodicd agents highlights the role of motion and egocentric perception learning robust representations. These ideas extend naturally to
audio-visual LLMs 115 2) where perceptually grounded models interact with dynamic environments.
K Human Study Details
We conducted human study involving participants evaluate the following: (i) the correctness and reliability the samples collected AVHaystacks, (ii) the quality responses generated by MAGNET, (iii) correlation between the proposed metric STEMand human evaluation. Application details are presented in Fig. 16 and user consent information is provided in Fig. 17
Each participant received detailed instructions outlining the goals of the study and their specific tasks. They shown several samples obtained through semi-automated data collection strategy and asked to rate the quality each sample on a scale from
The aggregated ratings indicate high relevance and correctness, with an average score of 4.6/5 for the collected samples. Participants also evaluated the responses generated by MAGNET on the proposed AV HaystacksQA,, as discussed in the main paper.
The user study protocol was approved by the Institutional Review Board (IRB). No personal information was collected, stored, or shared any stage of the study.
34

---

USER USER STUDY TUDYAAAA APPLICATION
Abstrast
This study locuses on ecalustion oithe quality collected sampiee for benchmurh dessaes, results Audio Visua Noadles Peesorinng Mulh Video Haysladhu assess qualitg uma on done asting humman hww.gngtttnrrrreroorrooo partopenta the qullity sample mocel reaporaans
Subject Selection:
anerymity
Eligibiity Criteria Anyone w normal and vedon, atleaet W and
portclure Eingleh elgite Partiparts with carrecee nornal hearing alsobe
screening questions Our screeting question "Do you have normal partiipart answers both quesiiona alowed condinue with
survey.
Rationale: Evalvation
hearing
Partcparts aloo need to be lexet years old need proficentt English
undersard speach contents being played.
Enroliment Numbers: people
Numbers: number givea starisiooll results
Procedures
Using headest recorn recormmnnddd mendedddddddd possibly accomm to eidecs and giiing your prefenrrces cerersssso
werwil nit os surwey dataa fvn indighhle parloperte We serata thy gelllyyy sevomm
involee
saiplles in the presette thenchinee carcchhkkk. Each campie hei queetion ca lectiin vidiest
1-10 lhinng the lovest The namples divene flom coanariis involving travelvogge
Secondly you needit ower model resporsss a and saco them based their accunady.
proposed aspects
hal Illlccnatioon order and milding stepe.
Makk:: There known rika.
There direct beneltu pertiipents This provide researchers from audio waual communit beller undentaading how good mulii mull--ideo linked ppaliine dhaillnggng
Confidentialty:
collocted
store on a password protectdd comp der and will be securely wiped in years from the day coeston. invecipetors study access
Consent Process:
online study, information particpenta, they head and click button agror" to indicate their consent and continu questions request walver consent online stady
present scrwning quections before consending to ensure
audjects
Subjeeds can choose not condruse they
their comsent. The walvor
wrtdin consent not adversdly offect rights wallore of subjecss.
research oslect pradticably
walver
reglons
and will not be sble to collect signatures from each sudject
(4)
will provide our contect information during the study and ancoumae
additional pertinent information after partidpation
(9)
Partspant can save their signed connent form. For companiion, any deception.
Contilit Intersst
Pape
Novied OOE 20E1203
35

---

HIPAA Complaanc::
Not Applicable
Research Outaide the United Statea::
Applicable
Research Involingg Prisoners:
Applicable
12. SUPPORTING DOCUMENTS
Document) the information required nama 1-11 abowe, and nélevant supporingg documants, induding: consert forms, lotors sont roouut partoports. questonnazes partippants.
The consent forna in your approved (Bithet PACKAGE botton, consent form. version the form.
creating "uptodate"
your package
PROJECT OVERVIEW IRBNen
sumber will lsstd at the next your project lite.
Figure 16: User study guidelines.
CONSENT TO PARTIOPATE
Reescning over MVD Vitto Asyithole
Purpose of the Study
This conducted resarch
lnvtiig you to-eema reonin viuual reilhes
The rocmal
datsiet resposse conechata propoled
Procedures
repeoee emertcy oie Anall cal the
acre
Potential Risk and
There Ao known rote from partopating this research study.
benelto partiipating rasearch
hope fukue, other people benalld from though improved undersinding how to penerate more reailoee
Confidentatly
i pssovord proteced computer No ldvnttable inforaatio will
scoms to the dack.
dtendly protected ond Your infimaatio may ahared with reprosentavee
requined
IRBNe 1004040-111441111
Intlal:
36

---

Compenaation
Right to Withdrow and
Your particaation redearch is componey volentayy You may ressarch, you may stop perlopating any kimw. you decicl not you not de ponalized or benolts which efterwid quality
concams complninee lf you need report an segay relstd to please inveslgstoo:
Placeholder anorymsit
Participart Rights
your report resesrch realted legur, pleare contect
Rlaroved for anorymitt
infermation
Diricend for ancmacity
prooedires for ressarch Inveling hamen suljects.
Page of
Pettert Parkage 0986481-1
ape you have nomual conected to nomual heving and you are proficenn English you read this connent form and you voluntally downlded or save oopy form your records.
pertopatte the research Pleas
Yoo agree partiipute,
agree?
Figure 17: User consent application.
37


Reviews:
[{"review_id": "2vWZQYQTJA", "reviewer": "Reviewer_HRuz", "summary": "This Manuscript introduces AVHaystacks, a new large-scale audio-visual benchmark specifically designed to evaluate multi-video retrieval and temporal grounding capabilities of LMMs. The benchmark AVHaystacks comprises 3,100 QA pairs across a broad range of domains, where answering each query requires finding and integrating evidence from multiple videos and modalities. In conjunction, the authors propose MAGNET, a model-agnostic, multi-agent framework leveraging RAG as well as specialized audio-visual agents for improved segment localization and answer synthesis. The work also introduces two new evaluation metrics, STEM and MTGS, tailored to balanced and interpretable assessment of grounding and multi-step QA accuracy. Experimental results highlight substantial gains over strong baselines in both automatic and human assessments.", "questions": "Please refer to the *Strengths and Weaknesses* section.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "1. The submitted manuscript is clearly written, with a well-defined motivation, making it easy to understand the overall idea of the paper.\n2. The proposed new task, method, dataset, and evaluation metrics are all valuable contributions that can help advance the audio-visual research community.\n3. Has the applicability of the proposed method been considered on other audio-visual temporal localization or QA datasets (eg. AVVP, LFAV, MUSIC-AVQA, etc.)? Or perhaps on some relevant visual datasets?\n4. There are noticeable traces of GPT-generated writing throughout the manuscript.\n5. It is recommended to include proper citations for the baseline methods listed in the experimental tables.\n6. In the Related Work section, it is suggested to add a dedicated subsection on audio-visual learning or audio-visual scene understanding.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback. We are glad that the reviewer found that: our manuscript is clearly written, with a well-defined motivation, easy to understand; the proposed task, method, dataset, and evaluation metrics are all valuable contributions, that can help the community. \n\n\n> Has the applicability of the proposed method been considered on other audio-visual temporal localization or QA datasets (eg. AVVP, LFAV, MUSIC-AVQA, etc.)? Or perhaps on some relevant visual datasets?\n\n\nPer suggestion, we are sharing the results of our proposed approach on the LLP, LongVALE, AVQA datasets (without fine-tuning on the respective training splits):\n\n### Segment level Audio-visual event parsing\n| Method   | LLP (Acc.) |\n| -------- | ---------- |\n| AVVP     | 55.4       |\n| LSLD     | 62.2       |\n| VALOR    | 66.8       |\n| MM-CSE   | 68.9       |\n| **MAGNET** | **70.1**   |\n\n### Omni modal temporal video grounding\n| Method        | LongVALE (mIOU) |\n|---------------|-----------------|\n| PandaGPT      | 2.2             |\n| NextGPT       | 4.0             |\n| VTimeLLM      | 6.4             |\n| LongVALE-LLM  | 11.0            |\n| **MAGNET**      | **14.6**        |\n\n### Audio-Visual Question Answering\n| Method   | Music AVQA (Acc.) |\n|----------|-------------------|\n| AVLLM    | 45.2              |\n| OneLLM   | 47.6              |\n| AVicuna  | 49.6              |\n| **MAGNET** | **50.3**          |\n\nPlease note that since these datasets do not contain multi-video linkage annotations, the experiments are carried out on single video instances, hence the retrieval module was not employed. As can be seen, our approach demonstrates strong performance on these datasets as well, underlining its generalizability.\n\n\n> It is recommended to include proper citations for the baseline methods listed in the experimental tables.\n\nThank you for pointing this out. We have updated the tables accordingly.\n\n\n> In the Related Work section, it is suggested to add a dedicated subsection on audio-visual learning or audio-visual scene understanding.\n\nThank you for the suggestion. We are adding a subsection in related works on Audio-Visual Learning."}, {"review_id": "ItT09lfR4x", "reviewer": "Reviewer_A5XP", "summary": "The authors propose a new task, benchmark dataset, and training strategy to address challenging QA and reasoning tasks on questions that require audio-visual understanding across multiple different videos. Their proposed method is the first of its kind that does this fine-grained audio-visual reasoning over multiple independent videos, and they show significant performance improvements over simple video-based RAG and single-video reasoning baselines.", "questions": "1. 3100 QA pairs in the proposed dataset is not very large. Some brief justification of why this scale of QA pairs is sufficient for fine-tuning and evaluating these models on this complex task should be included in the text.\n\n2. How does the method perform on traditional single-video AV QA benchmarks? This evaluation would help confirm that the model maintains its advantage even for questions that can be answered entirely within one video. Or if this is not applicable, justification in the text for this would be important to include.\n\n3. It is surprising that audio + visual leads to an improvement in performance in these video settings - although the questions are automatically curated to force reliance on audio comprehension, the video filtering process seems reliant on videos with spoken transcripts, which will interfere/overlap with any semantically important audio signal useful for answering the question. Was this aspect of the videos considered in the QA pair curation process?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Quality:**\nThe paper is high -quality. The motivation for reasoning over multiple videos and the related work sections are comprehensive, and the method is explained formally and clearly in detail. The evaluation metrics are also well-motivated in the text.\n\n**Clarity:**\nThe paper is well-written and easy to understand. The figures are clear and effective, and introduction of the task and the method in formal notation are at the appropriate level of detail.\n\n**Significance:**\nThe proposed method outperforms existing baselines on several metrics by large margins, including human evaluation, LLM-as-a-judge, and their own metrics designed for this task.\n\n**Originality:**\nThe work is original - they are the first to propose a method for AV QA and reasoning over collections of videos, instead of within a single video.", "was_revised": true, "final_justification": "The authors sufficiently address my concerns/questions, including convincing empirical results that address my concern about single-view AV datasets. The 'Accept' rating is maintained.", "rebuttal": "We thank the reviewer for the constructive feedback. We are glad that the reviewer found that we propose a new task, benchmark dataset, and training strategy; the proposed method is the first of its kind, our approach shows significant performance improvements; the paper is high quality, comprehensive, the method is explained well and clearly, the evaluation metrics are well motivated; the paper is well written, easy to understand, with clear figures and appropriate details. The work is original and first of its kind. \n\n\n> 3100 QA pairs in the proposed dataset is not very large. Some brief justification of why this scale of QA pairs is sufficient for fine-tuning and evaluating these models on this complex task should be included in the text.\n\n\nThanks for the question. The factors contributing to the size of our benchmark are: \n\n(i) The dataset annotation process requires manual inspection to ensure complete correctness. Although the sample curation process is automated, to ensure strict sanity we manually validate each sample, which is both tedious and time-consuming.\n\n(ii) The models evaluated in our study are already audio-visually informed, and we fine-tune them efficiently using LoRA adapters. Our salient frame selection module enhances context by focusing on key frames. With LoRA, we require fewer samples to fine-tune our audio-visual agents, as they are heavily pre-trained on audio-visual data. Importantly, we are only fine-tuning the agents, not training the entire retrieval pipeline.\n\nPlease note that the SFS module can be used with a closed-source model where finetuning was not done. As seen in Tables 2 and 3 of the main paper, these methods demonstrate strong performance even without any fine-tuning. \n\nWe find many works in the literature that support our claims:\n\n[a] Low-Rank Few-Shot Adaptation of Vision-Language Models - Zanella et al.\n\n[b] Does Combining Parameter-efficient Modules Improve Few-shot Transfer Accuracy? - Asadi et al.\n\n[c] Machine Translation with Large Language Models: Prompting, Few-shot Learning, and Fine-tuning with QLoRA - Zhang et al.\n\nThis body of work demonstrates that employing LoRA enables these models to perform well even with limited samples. As suggested, we will add these details to the final draft.\n\n\n> How does the method perform on traditional single-video AV QA benchmarks? This evaluation would help confirm that the model maintains its advantage even for questions that can be answered entirely within one video. Or if this is not applicable, justification in the text for this would be important to include.\n\n\nPer suggestion, we are sharing the results of our proposed approach on the single video datasets LLP, LongVALE, MusicAVQA:\n\n### Segment-level Audio-visual event parsing\n| Method   | LLP (Acc.) |\n| -------- | ---------- |\n| AVVP     | 55.4       |\n| LSLD     | 62.2       |\n| VALOR    | 66.8       |\n| MM-CSE   | 68.9       |\n| **MAGNET** | **70.1**   |\n\n### Omni-modal temporal video grounding\n| Method        | LongVALE (mIOU) |\n|---------------|-----------------|\n| PandaGPT      | 2.2             |\n| NextGPT       | 4.0             |\n| VTimeLLM      | 6.4             |\n| LongVALE-LLM  | 11.0            |\n| **MAGNET**      | **14.6**        |\n\n### Audio-Visual Question Answering\n| Method   | Music AVQA (Acc.) |\n|----------|-------------------|\n| AVLLM    | 45.2              |\n| OneLLM   | 47.6              |\n| AVicuna  | 49.6              |\n| **MAGNET** | **50.3**          |\n\nPlease note that since these datasets do not contain multi-video linkage annotations, the experiments are carried out on single video instances; hence, the retrieval module was not employed. As can be seen, our approach demonstrates strong performance on these datasets as well, underlining its generalizability.\n\nZero-shot performance on the above datasets indicates the generalizability of our approach. \n\n\n> It is surprising that audio + visual leads to an improvement in performance in these video settings - although the questions are automatically curated to force reliance on audio comprehension, the video filtering process seems reliant on videos with spoken transcripts, which will interfere/overlap with any semantically important audio signal useful for answering the question. Was this aspect of the videos considered in the QA pair curation process?\n\n\nThank you for the question. We carefully select samples that rely on both visual and audio modalities when forming the QA pairs. For a detailed explanation, please refer to the demo video at the 30s mark (in the supplementary material), which addresses a key question necessary for cooking, such as “How to grip the knife.” This demonstrates the importance for a novice chef to not only **visualise** an expert's demonstration but also **hear** and understand the steps for a comprehensive learning experience, highlighting the significance of both visual and audio modalities.\n\nWe request the reviewer to refer to other dataset examples in our supplementary materials, which further underscore the utility of both modalities, thereby supporting the enhanced performance of audio-visual models compared to models using only one modality."}, {"review_id": "WATlIVyi1h", "reviewer": "Reviewer_8Nrk", "summary": "The paper introduces AVHaystacksQA, a new audio-visual question-answering task that requires retrieving and temporally grounding evidence scattered across multiple videos. The authors also propose a benchmark of QA pairs drawn from 500 YouTube videos spanning different topics, with each answer linked to at least two video segments. The paper also proposes its own model, MAGNET, which is based on retrieval-augmented multi-agent framework. Additionally, the paper defines two new metrics (STEM and MTGS) on their task.", "questions": "1. Section 3.2 simply averages the similarity scores from audio-visual features and caption features to rank videos. Did the authors explore learnable weighting schemes or late-fusion strategies, and if so, how did those affect recall and downstream QA quality?\n\n2. The 500-video corpus is dominated by instructional YouTube content such as cooking and musical tutorials. Have authours tested MAGNET on domains that feature faster scene changes or less structured narration. How can authors maintain the generalization ability of their methods?", "limitations": "1. The benchmark draws on only about 500 instructional video clips.  Such a narrowly focused and modest-sized collection makes it hard to judge whether the retrieval-plus-reasoning pipeline would scale or stay robust in more diverse settings.\n\n2. Most QA pairs and step-by-step rationales are generated by GPT prompts with little human post-editing.  This can introduce the language patterns and biases of the source LLM into the benchmark, which may potentially inflate system performance for models that share training data or stylistic cues with GPT.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1. The paper proposes a novel task, which jointly tests multi-video linkage, audio-visual reasoning and long-context temporal grounding. This is very interesting.\n\n2. The authors did thorough experiments, including enough comparisons, ablation studies and human evaluations.\n\n3. The paper is well structured, flows logically, and makes complex ideas easy to follow.\n\n\nWeaknesses\n1. The size of dataset is relatively small. It only has  500 videos, while it contains some long videos, the small sample size may not be very helpful in this new task.\n\n2. STEM and MTGS still hinge on fragile text-similarity thresholds, leaving them ill-equipped to accurately capture multimodal reasoning or grounding quality; as a result, the reported performance gains are hard to interpret with confidence.\n\n3. The MAGNET's core reasoning is off-loaded to proprietary pre-trained models that are simply prompted, not modified.  Without an open-weight substitute or an algorithmic alternative, it is hard to tell whether the reported improvements come from the authors’ architectural ideas or just from the raw power of those closed models.", "was_revised": true, "final_justification": "Thanks for the author's feedback, I still maintain my original decision.", "rebuttal": "We thank the reviewer for the constructive feedback. We are glad that the reviewer found that the paper proposes a novel and interesting task, the experiments are thorough, with enough ablations and human evaluations, and the paper is well structured. \n\n\n> The size of dataset is relatively small. It only has 500 videos, ... may not be very helpful in this new task.\n\nOur benchmark includes samples from **27 diverse categories**, totalling **103 hours of video**, with **69.58%** of questions involving >=2 video linkages, each verified by human annotators involving meticulous manual inspection (Cohen's $\\kappa$ value of 0.87 when validating our samples with subject matter experts, indicating strong alignment with expert annotations). Please note that our benchmark includes approximately **20 videos per subject**, making it challenging.\nWe manually validate each sample to ensure strict quality control, which is both tedious and time-consuming. Notably, ours is the first benchmark to offer:\n\n(i) Multi-video linkage\n\n(ii) Fine-grained AV reasoning\n\n(iii) Long-context annotation\n\n(iv) AV retrieval-based QA\nWe believe our dataset construction pipeline lays a foundation for future work in this direction, which can extend the benchmark size.\n\n\n\n> STEM and MTGS still hinge on fragile text-similarity thresholds, ... performance gains are hard to interpret with confidence.\n\n\n\nWe would like to emphasise the following:\n\nSpecifically, STEM uses text similarity only as a filtering mechanism (with a tunable threshold, typically $\\tau_{s}$ = 0.5) to determine plausible matches between steps, after which it evaluates errors in a structured manner, including step order violations, grounding mismatches (via **IoU** on video segments), and unmatched steps (both missing and hallucinated). Importantly, matched steps must not only pass the similarity threshold but also satisfy alignment via **Hungarian matching**, which encourages globally optimal correspondences rather than spurious local matches. Thus, the metric is **robust to partial paraphrasing and places more emphasis on alignment accuracy than lexical similarity**.\n\nSimilarly, please note MTGS does not depend on any textual similarity at all. It purely evaluates grounding quality by measuring the temporal **intersection-over-union (IoU)** between predicted and ground truth intervals, only for video IDs that match. The merging of intervals before computing IoU ensures the metric captures both coverage and precision at the video level. In doing so, MTGS fairly evaluates grounding fidelity even in the presence of partial overlaps, thereby offering interpretable and grounded performance insights.\n\nThe combination of these metrics provides a complementary and robust evaluation of both structural and temporal alignment in multimodal settings. Moreover, our performance gains under these metrics reflect meaningful improvements in temporal and procedural grounding, not just textual matches.\n\n\n\n\n> The MAGNET's core reasoning is off-loaded to proprietary pre-trained models ... it is hard to tell whether the reported improvements come from the authors’ architectural ideas ...\n\n\nWe would like to clarify that MAGNET does not solely rely on proprietary models, and several key contributions go beyond the capabilities of any particular foundation model.\n\n(i) MAGNET incorporates explicit model modifications through LoRA-based fine-tuning of audio-visual (AV) agents. These agents are adapted within our framework to better handle multi-modal, temporally grounded reasoning tasks.\n\n(ii) MAGNET introduces novel algorithmic contributions, including **Salient Frame Selection**, ensuring attention is focused on both the right content and the right temporal window, facilitating efficient grounded question-answering. Crucially, these contributions are model-agnostic and can be integrated into a variety of AV models.\n\nTo demonstrate this generality and to isolate the impact of our architectural innovations, we extensively evaluate MAGNET using open-source/non-proprietary models such as Video-SALMONN, Unified-IO2, and Qwen-2.5-Omni, under both zero-shot and LoRA fine-tuned settings. Our experiments show consistent improvements over their vanilla baselines (refer to Table 2), underscoring the effectiveness of MAGNET's design beyond the use of any proprietary models.\n\nAdditionally, our Audio-Visual Retrieval-Augmented Generation (AV-RAG) module uses **open-source encoders** such as ImageBind for grounding external knowledge, further emphasising that the pipeline is built to support transparency and reproducibility.\n\nIn summary, the reported performance gains are not solely attributable to the use of powerful proprietary models. Rather, they stem from general-purpose, open-source-compatible architectural innovations that demonstrably enhance AV understanding across non-proprietary settings.\n\n\n\n> ... Did the authors explore learnable weighting schemes or late-fusion strategies, and if so, how did those affect recall and downstream QA quality?\n\n\n\nWe employed the following schemes for the retrieval scoring:\n\n(i) **Scheme 1**: We employ a learnable weighting parameter $\\alpha$ wherein weighted average Sim_avg = $\\alpha$ * S(av, q) + (1 - $\\alpha$) * S(cap, q). We use the training set to learn this $\\alpha$ parameter.\n\n(ii) **Scheme 2**: We compute Hadamard fusion between S(av, q) and S(cap, q) and obtain final similarity scores.\n\n(iii) **Scheme 3**: We concatenate the S(av, q) and S(cap, q) and train a linear layer (with output dimension same as S(av, q)) to obtain the final scores.\n\nAs shown in the table below, even though we observe some improvements for Schemes 1 and 3, they are not substantial and involve training. On the contrary, with simple averaging, we had obtained comparable performances without involving any training. With Scheme 2, however, we have observed a drop in the retrieval and QA performances. \n\n| Retrieval | Scheme reported in paper (avg) | Scheme 1  | Scheme 2  | Scheme 3  |\n|--------|--------------------------------|-----------|-----------|-----------|\n| R@3                              | 73.15                    | 73.18 | 70.28 | 73.68 |\n| R@5                              | 79.20                    | 79.51 | 77.85 | 79.92 |\n\n\n| QA (w/ $\\textbf{MAGNET}_{\\text{Qwen-2.5-Omni-FT}}$ | Scheme reported in paper (avg) | Scheme 1  | Scheme 2  | Scheme 3  |\n|--------|--------------------------------|-----------|-----------|-----------|\n| B@4         | 53.69           | 53.73 | 50.36 | 54.12 |\n| GPT-Eval         | 7.56           | 7.58 | 6.81 | 7.69 |\n| H-Eval         | 4.01           | 4.01 | 3.76 | 4.08 |\n| MTGS-avg         | 0.79           | 0.80 | 0.72 | 0.82 |\n\n\n\n\n> ... Have authours tested MAGNET on domains that feature faster scene changes or less structured narration ... generalisation ability of their methods?\n\n\nOur benchmark indeed contains samples from cases that involve rapid scene changes measured with Average Scene Duration (ASD) (Cutting & Candan (2015)). Below, we report category-wise values for these scenarios:\n\n| Example category       | B@4  | Text Sim | GPT Eval | ASD (in sec) |\n|------------------------|------|----------|----------|-|\n| DIY                    | 52.17| 5.88     | 6.97     | 9.58 |\n| 3D printing            | 51.90| 4.92     | 7.12     | 7.24 |\n| Self-Defense           | 54.72| 5.23     | 8.75     | 8.19 |\n| Playing string instrument | 54.05| 5.67     | 8.17     | 12.05 |\n\nWe will add these results to the final draft.\n\n\n> ... modest-sized collection makes it hard to judge whether the retrieval-plus-reasoning pipeline would scale or stay robust ... \n\n\nPlease note the following practical challenges in the data curation involved in the audio-visual QA setting:\n\n(i) **Multi-Video Linkage Complexity**: Creating connections between multiple videos is inherently challenging due to the need to accurately align and synchronise content across different media. This requires sophisticated techniques to ensure that related segments are appropriately linked.\n\n(ii) **Curating Multi-Video Questions**: Formulating questions that involve multiple videos is complex because it demands a deeper understanding of the content and the relationships between the different video segments. This requires careful consideration to create meaningful and contextually relevant questions.\n\n(iii) **Precise Temporal Grounding**: Our dataset necessitates accurate temporal grounding of events across various videos, which is both resource-intensive and requires human verification to guarantee precision. This manual effort is crucial to ensure the accuracy and reliability of the linked events.\n\nAdditionally, we provide a smaller subset of AVHaystacks-50 and compare performance against AVHaystacks-Full to show the scalability of our pipeline for both retrievals as well as reasoning.\n\n\nPlease note, this work is the first of its kind in this direction, laying a foundation for future research in fine-grained AV retrieval and setting a precedent for similar endeavours. Future attempts can be made to expand the benchmark size and make it more comprehensive.\n\n\n> Most QA pairs and step-by-step rationales are generated by GPT prompts ... which may potentially inflate system performance ...\n\n(i) We use AV LLMs, and as shown in Tab 1, the direct application of these models does not yield meaningful performance. However, by implementing our pipeline, which includes **AV RAG**, **salient frame selection**, and **redundant token removal**, we improve over the baseline. Incorporating LoRa further enhances performance, highlighting the effectiveness of our approach.\n\nIt’s important to note that models like VideoSALMONN and UnifiedIO2 are trained on public AV datasets, unlike GPT, which lacks this AV training. If stylistic biases existed, these models would perform better in a zero-shot setting without MAGNET.\n\n(ii) GPT-4 is a text-based model that only summarises text transcripts and does not directly process AV inputs."}, {"review_id": "JerEE9Aexc", "reviewer": "Reviewer_3r1a", "summary": "This work proposes a new benchmark and a corresponding framework for evaluating large multimodal models in audio-visual retrieval. This is useful when retrieving relevant audio-visual segments from large video corpora where the search is to be performed on multiple video sequences. The retrieval mechanism is based on retrieval-augmented generation system that identifies key moments across multiple videos. The results are impressive and they demonstrate substantial gains over baselines.", "questions": "Ablations study considers the importance of each of two modalities: audio and visual. However, you also obtain transcript in the form of encoded audio-visual captions. Transcripts are easier and smaller to work with, I would be interested to see the impact of using the transcript as the third modality. \n\nHow generalizable this work is to a different type of questions? For example, instead of asking to find videos showing how to do something, can you ask to identify all frames in which a particular behavior is present, e.g., standing and talking?", "limitations": "I do not think limitations are discussed.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This appears to be first-of-its-kind benchmark consisting of 3100 audio-visual QA pairs of videos across diverse domains for searching across multiple video sequences. The task that is proposed to be solved is referred to as AVHaystacksQA - finding precise videos in a large collection of videos. \n\nTwo metrics are proposed to evaluate the performance: STEM - quantifies alignment errors between the ground-truth and predicted step sequences in multi-video audio-visual answer generation and MTGS - provides a balanced and interpretable assessment of segment-level grounding performance.\n\nThere is also a new model-agnostic, multi-agent training strategy designed to enhance model performance in identifying key segments across multiple videos. \n\nExtensive evaluation, including evaluation by a human, indicates the usefulness and performance of the proposed work.", "was_revised": true, "final_justification": "The issues that I was interested in were addressed in the author's comments. I hope they will include these additional results into the final manuscript as well. I found this work insightful and meaningful, and I my recommendation is to accept it.", "rebuttal": "We thank the reviewer for the constructive feedback. We are glad that the reviewer found that our proposed task is useful, with impressive results and first-of-its-kind benchmark; our merics provide a balanced and interpretable assessment with extensive evaluation.\n\n> Ablations study considers the importance of each of two modalities: audio and visual. However, you also obtain transcript in the form of encoded audio-visual captions. Transcripts are easier and smaller to work with, I would be interested to see the impact of using the transcript as the third modality.\n\nBelow, we compare the performance of various input configurations that underline the importance of the audio-visual setting:\n\n\n### AVHaystacks-Full results under different configurations\n\n|      Method                            | B@4              | MTGS-avg |\n|----------------------------------|------------------|----------|\n| $\\textbf{MAGNET}_{\\text{Qwen-2.5-Omni-ZS}}$ (V + A) | **28.49**       | **0.49** |\n| $\\textbf{MAGNET}_{\\text{Qwen-2.5-Omni-ZS}}$ (V + T) | 26.98           |     0.42     |\n| $\\textbf{MAGNET}_{\\text{Qwen-2.5-Omni-ZS}}$ (T)    | 20.46           |      0.21    |\n| $\\textbf{MAGNET}_{\\text{Qwen-2.5-Omni-ZS}}$ (V + A + T) | 27.90       |   0.48       |\n| $\\textbf{MAGNET}_{\\text{Qwen-2.5-Omni-FT}}$ (V + A) | **53.69**       | **0.79** |\n| $\\textbf{MAGNET}_{\\text{Qwen-2.5-Omni-FT}}$ (V + T) | 48.63           |     0.65     |\n| $\\textbf{MAGNET}_{\\text{Qwen-2.5-Omni-FT}}$ (T)    | 35.02           |      0.28    |\n| $\\textbf{MAGNET}_{\\text{Qwen-2.5-Omni-FT}}$ (V + A + T) | 52.91        |   0.76       |\n\nExperimental results demonstrate that adding text as the third modality over audio and visual does not result in improved performance (rather results in a marginal performance drop possibly because of increased input token size). However, using only the text modality results in a significant dip in performance across metrics, particularly on MTGS-avg (even with fine-tuning). This validates the importance of having audio and visual modalities for improved temporal grounding.\n\n\n\n> How generalizable this work is to a different type of questions? For example, instead of asking to find videos showing how to do something, can you ask to identify all frames in which a particular behavior is present, e.g., standing and talking?\n\nBelow, we report the performance on two new prompts for $\\text{MAGNET}_{\\text{Qwen-2.5-Omni-FT}}$. Experimental results demonstrate the generalizability of our approach.\n\n| Prompt                    | B@4  | MTGS-avg |\n|---------------------------|------|----------|\n| \"Standing and talking\"    | 51.28| 0.72     |\n| \"Sitting and playing guitar\" | 52.76| 0.75     |\n\n\n\n\n> I do not think limitations are discussed.\n\nWe discuss the limitations of our approach in the future works section and elaborate on the failure cases in Failure Cases (**Section H of appendix**). Per suggestion, we will make a separate section for limitations in the final draft and revise accordingly."}]

Author rebuttal:

Meta-review: This work introduces a new task:  named AVHaystacksQA aimed at localizing AV clips relevant to a text query and synthesizing a response from these. To this end, this work proposes:
(i) a benchmark AVHaystacks, composed of 3100 QA pairs based on 500 videos comprising 27 categories.
(ii) a RAG based framework (MAGENTA) to first localize the relevant clips from top-k promising candidate videos, and, then to synthesize the response.
(iii) two metrics: to assess the localization (using IoU), and another to assess the text response (i.e., paraphrasing mismatches, multi-step response alignment, and temporal grounding).

The proposed method is also demonstrated to work well for existing single-clip AV benchmarks  (event parsing, qa, temporal video grounding), where it performs well without fine-tuning.

The reviewers unanimously appreciated the novel AV task, specifically, the multi-clip linkage and response synthesis. The new task has the potential to spur further development of benchmarks and AV methods. The strong performance of the proposed method, combined with the development of a new task and benchmark make this a valuable contribution for the community. Hence, I am inclined to recommend this paper for acceptance. I strongly urge the authors to incorporate the feedback and the experiments presented in their response in the final version.
Decision: Accept (poster)