Query-Guided Spatial–Temporal–Frequency Interaction for Music Audio–Visual Question Answering
Authors: Kun Li
Michael Ying Yang
Sami Sebastian Brandt
Venue: ICLR 2026 Poster (2026)
Keywords: Audio–visual question answering
Multimodal
Music scene understanding
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Novel Query-guided Spatial–Temporal–Frequency interaction method to enhance audio–visual understanding

Abstract: Audio–Visual Question Answering (AVQA) is a challenging multimodal task that requires jointly reasoning over audio, visual, and textual information in a given video to answer natural language questions. Inspired by recent advances in Video QA, many existing AVQA approaches primarily focus on visual information processing, leveraging pre-trained models to extract object-level and motion-level representations. However, in those methods, the audio input is primarily treated as complementary to video analysis, and the textual question information contributes minimally to audio–visual understanding, as it is typically integrated only in the final stages of reasoning. To address these limitations, we propose a novel Query-guided Spatial–Temporal–Frequency (QSTar) interaction method, which effectively incorporates question-guided clues and exploits the distinctive frequency-domain characteristics of audio signals, alongside spatial and temporal perception, to enhance audio–visual understanding. Furthermore, we introduce a Query Context Reasoning (QCR) block inspired by prompting, which guides the model to focus more precisely on semantically relevant audio and visual features. Extensive experiments conducted on two AVQA benchmarks demonstrate the effectiveness of our proposed method, achieving significant performance improvements over existing Audio QA, Visual QA, Video QA, and AVQA approaches. The code is released under https://github.com/lik1996/QSTar.

Published as a conference paper at ICLR 2026
QUERY-GUIDED SPATIA--TEPPRAA--REEEEEE INTERACTION FOR MUSIC AUDIO- QUESTION
ANSWERING
Kun Lit.3" Michael Ying Yang" Sami Sebastian Brandt
University of Twente, NI. University of Bath, UK IT University of Copenhagen, DK
k. 2.11 @utwente.nl, myy35@bath.ac.ukk s tu.dk
ABSTRACT
Audio-Visual Question Answering (AVQA) is challenging multimodal task that requires jointly reasoning over audio, visual, and textual information in a given video to answer natural language questions. Inspired by recent advances in Video QA, many existing AVQA approaches primarily focus on visual information pro- cessing, leveraging pre-trained models to extract object-level and motion-level representations. However, in those methods, the audio input is primarily treated as complementary to video analysis, and the textual question information contributes minimally to audio-visual understanding, as it is typically integrated only in the final stages of reasoning. To address these limitations, we propose novel Query- guided
(QSTar) interaction method, which effec- tively incorporates question- guoiodd clues and exploits the distinctive frequency- domain characteristics of audio signals, alongside spatial and temporal perception, to enhance audio-visus understanding. Furthermore, we introduce a Query Con- text Reasoning (QCR) block inspired by prompting, which guides the model to fo- cus more precisely on semantically relevant audio and visual features. Extensive experiments conducted two AVQA benchmarks demonstrate the effectiveness our proposed method, achieving significant performance improvements over existing Audio QA, Visual QA. Video QA, and AVQA approaches. The code is released under
com www/lokikkkoooooooo..oooo 996/0STar
INTRODUCTION
Understanding audio-visual scenes in videos crucial for a wide range of real-world applications, including autonomous driving (Wang et al.] 2024), human-computer interaction (Li et al.) 2023c) and event localization (Crumiaux et al.) 2022). Querying specific information from videos provides an intuitive and interactive interface that enables humans to engage with machines and gain deeper insights into the physical world. As a representative task in multimodal video understanding. Audio- Visual Question Answering (AVQA) (Yun et al. 202T Yang al. 2022 requires models to jointly interpret and reason over both aural and visual modalities to answer natural language questions about video content. Unlike traditional Visual or Video QA tasks YYu et al.) 2019 Le et al.) (2020) that rely primarily on visual cues, AVQA demands deeper integration of sound cues and visual content to capture the full semantics of scene.
In many real-world scenarios, audio conveys critical information that visuals alone may fail to cap- ture. Tasks such as identifying speaker in a conversation or distinguishing between visually simi- lar events often rely heavily on auditory cues (He et al., 1999 Ji et al. (021). Therefore, effective AVQA systems must not only interpret visual content but also recognize sound events, understand their temporal dynamics, and model their interactions with visual signals. To facilitate research in this direction, [i et al. (2022) proposed a large-scale dataset for music scene understanding, called MUSIC-AVQA, which has become standard benchmark for questioning audio-visual content. Similarly, in music AVQA, audio is especially helpful when visual cues are limited (e.g., subtle flute
Work done while at IT University of Copenhagen 1 "oorrsponndig author

---

Published as a conference paper at ICLR 2026
Question:
Is the flute the video always playing?
o
(a) Audio visual question answering (AVQA) task.
object-leve: flute & violin
spatial, temporal, frequency
Answer: Yes
Answer:
(b) Prior Methods.
(c) Our QSTar.
Figure 1: Illustration of AVQA task and comparison between prior methods and QSTar. (a) Input sample (audio, video, and question) for an AVQA task. (b) Prior works rely on object-level cues, struggling with subtle motions (e.g., inactive flute player). (c) QSTar enhances patial-temporal frequency interaction. Green patches highlight spatial focus. The red box shows diminishing high- frequency bands as the flute stops, while the violin remains active.
performances where motion is minimal despite continuous presence), as illustrated in Fig. This emphasizes the value of audio cues in improving question answering accuracy.
Most existing AVQA methods primarily focus on visual information processing. For instance, PSTP perception module to select Top-K frames for align- and audio-object matching scheme that leverages pre-trained object detectors to enhance visual recognition. However, these approaches primarily rely on visual cues, treating audio as secondary modality for temporal alignment. For instance, in PSOT (Li et al. 2025). audio was used only to select sound-driven patches that were then aggregated into a largely vision-centric pipeline. As a result, the distinctive properties of the aural modality are underutilized. Furthermore, question information is often incorporated only at the final stage via simple operations (e.g., multiplication), limiting the semantic alignment between the query and the multimodal content. For instance, TSPM (Li et al., 2024a) incorporated the question only at the final fusion and prediction stage, and its auxiliary prompt was used merely to identify key temporal segments. We believe that the late fusion leads redundant audio-visual representations and hinders model performance for AVQA.
(Li et al. 2023a) introduced a
ment. APL (Li al.] (202b)) proposed
In this paper, we jointly consider query guidance for both audio and visual feature learning and enhance cross-modal interactions across multiple dimensions. This design is motivated by the fol- lowing observations: questions typically target one a few instruments, requiring focused rather than holistic audio-visual representation; 2) wind instruments (e.g., clarinet and saxophone) often exhibit subtle visual cues but distinctive spectral characteristics, making frequency-domait analy- sis more effective, as shown in Fig- [ 3) instruments of the same category still differ in temporal patterns during performance. Based on these insights, our method emphasizes the role of linguistic throughout the pipeline to support fine-grained cross-modal reasoning. We emphasize spatial, temporal, and requenncydomain interactions. These are especially important in polyphonic scenar- ios where multiple instruments play simultaneously and subtle timbral harmonic (Agostini et al. 2003) cannot be captured by visual or temporal features alone.
To address these limitations, propose Query-guided
Interaction (QSTar) method for the music AVQA task. Specifically, we design a query-guidee multimodal cor- relation module that refines audio and visual features conditioned on the question from the early
2

---

Published as a conference paper at ICLR 2026
stage. This module consists of three components: modality-speciffe self-enhancing, cross-modal guidance capturing. and information propagation. Next, we introduce a teepoeal--eeuuueeeeeeee interaction module that further enriches the query-guided features by emphasizing semantic rele- vance across spatial, temporal, and frequency dimensions, particularly beneficial for distinguishing sound patterns over time and frequency. Finally, a query context reasoning block is employed to fine and fuse features before prediction. It uses prompting to incorporate linguistic context and task- specific constraints for more precise reasoning. In summary, our method improves query-guided alignment throughout the entire pipeline and effectively integrates audio and visual cues across spa- tial, temporal, and frequency domains. We validate our method on the benchmark MUSIC-AVQA and other AVQA datasets and show that our approach significantly outperforms previous Audio QA, Visual QA, Video QA, and AVQA methods.
Our main contributions can be summarized as:
We propose QSTar, a novel framework that integrates query guidance throughout the entire pipeline to refine modality-specht features. By embedding linguistic context early, QSTar enhances both audio and visual representations in a question-aware manner, enabling more precise cross-modal reasoning.
We introduce fine-grained interaction module that emphasizes semantic cues across spa- tial, temporal, and frequency dimensions. This design enhances discriminative understand- ing, especially in polyphonic scenarios where subtle audio or visual cues are critical.
We design a reasoning block that injects task-aware linguistic context through prompting guide final predictions, improving semantic alignment between the question and multi- modal features.
Extensive experiments demonstrate that QSTar achieves new ttae-offthh-an the-art performance the MUSIC-AVQA benchmark.
2 RELATED WORK
2.1 AUDIO-VISUAL SCENE UNDERSTANDING
Over the past decade, significant progress has been made in multimodal learning, particularly in tasks involving audio-visual scene understanding (Duan al. 2023). These tasks focus on the aural and visual modalities, which are fundamental components of human perception, given their critical roles in interpreting daily life with semantic, spatial, and temporal coherence. This field encom- passes a variety of challenges. including sound source localization (Grumiaux et al., 2022)), motion recognition (Kuchne al. 2011), video parsing (Zhang et al. 1995), semantic segmentation (Li et al. 2023b), and video description (Xu et al. (016). Recent advances exploit the complementary nature of different modalities to enhance information alignment, facilitating more robust scene rea- soning. While unified multimodal models can capture global spatiotemporal representations, further research is needed to enhance fine-grained, task-specific focuses.
2.2 AUDIO-VISUAL QUESTION ANSWERING
Audio- Visual Question Answering (AVQA) leverages multimodal information from video content to answer user-posed questions. Compared to other QA tasks (e.g., audio QA, visual QA, and video AVQA is more challenging as it requires integration of multiple modalities to address linguistic queries effectively. In particular, inadequate aural or visual perception can lead to suboptimal or even incorrect predictions. To advance research in this domain, several AVQA datasets have been introduced, including Pano-AVQA (Yun et al.) 202I). MUSIC-AVQA (Li et al.) (022). and AVQA (Yang et al. 2022). Recent approaches such as LAVISH (Lin al. 2023) aim to enhance audio- visual association and improve model training efficiency on large-scale data. APL. (Li et al. 2024b) introduces a question-conditione
clue discovery module that uses visual or audio embeddings as queries and attends to the question at single stage. TSPM (Li et al. 2024a) introduces prompt used in the sequential temporal perception module to identify key temporal audio and visual cues and incorporates the question only at the final fusion stage. PSOT (Li et al. (025) emphasizes object-level and motion-level visual changes to boost performance. In contrast to prior approaches that primarily focus visual processing and incorporate question information only at later stages,
auxiliary
3

---

Published as a conference paper at ICLR 2026
Query-GUlded Multimodal Correlation
(QGMC)
Visual Encoder
Audio
Prompt
FPN
Video
Attention
Aitertion
piano playing longer than the violin?
Text
Encoder
XP
Spatia- Temporal-
Query Context Reasoning (QCR)
Question
Muit-heed
Mall-rasd
Frequency Interaction (STFI)
Yes Answer
Audio
Encoder
Audio
Interationn
Mut-heed
Nutihhan
Adtention
Alantiin
ComBlook
Afvertion
Figure 2: Overall framework of the proposed QSTar method. We use pre-trained encoders to ex- tract audio, visual, and linguistic features, Fa, F.,, F.. respectively. The Query-Guided Multimodal Correlation module (QGMC in yellow area) refines Fa and F. using query information, result- ing in Fag and F" These features are further enhanced by the Spatial-Temporal Frequunnyy In- teraction module (STFI in purple area), which integrates
Interaction (STI) and features from AST (Gong
Temporal- Frequency Interaction (TFI), using additional frequency-awan et al. 2021). The Query Context Reasoning block (QCR in green area) incorporates prompt- based context promptp) and sentence-level linguistic features (Fsentence) to guide multimodal fusion for answer prediction. For brevity, we remove self-attention units.
method emphasizes richer, query-relevant refinement of both audio and visual features while enabling the model to attend to linguistic cues throughout the entire processing pipeline.
METHOD
To address the challenges of music AVQA, we propose QSTar, a novel network that enables query- guided extraction of visual and audio cues and facilitates effective spatial-temporal-ffequeees in- teraction and query context reasoning, thereby enhancing answer prediction performance. Fig. illustrates the overall framework of the proposed method.
3.1 INPUT REPRESENTATION
The input video sequence is first divided into T' non-overlapping audio and visual segments, each with a duration of one second. These segments, along with the input question, are then encoded using moddlitty-speciii models.
Visual Representation. Each visual segment is divided into M patches and appended with a special [CLS] token at the beginning. To effectively represent visual content from the video, we employ pretrained CLIP (Radford al. (20T)) model with frozen parameters to extract two types of features: frame-level and leteh-level representations. The patch level features are further compressed using Token Merging (ToMe) (Bolya et al. 2023), reducing them to M' tokens that preserve spatially sensitive information. Finally, the frame-level and patch-level features across all T' video segments denoted F. [ TTXD and F, ( T'''XXX respectively.
Audio Representation. Following previous works, each audio segment is processed using 2D CNN, VGGish (Gemmeke et al. (017), which is pretrained on a large-scale AudioSet dataset. The resulting audio teatures are represented as F. (fi)T E RTxD
Text Representation. The input question is first tokenized into individual words. We then utilize a pretrained CLIP text encoder to extract both sentence-level and word-level linguistic features. The sentence-level features capture the overall semantic context of the question and are denoted
4

---

Published as a conference paper at ICLR 2026
as Fsentence RD. The word-level features represented as fixed-length sequence of token embeddings, denoted as Fw E RNXD, where N is the number of tokens.
3.2 QUERY-GUIDED MULTIMODAL CORRELATION MODULE
In this subsection, we propose an effective module, Query-Guided Multimodal Correlation (QGMC), to enhance unified interaction among visual, audio, and linguistic features. In contrast to the prior two-stage fusion methods that first integrate audio and visual features before incorporating textual information, our approach jointly emphasizes modality-specieeeoo specific features that are semantically aligned with the question. This enables more focused and context-awarr multimodal reasoning.
QGMC consists of three stages: self-enhancing, capturing, and propagating. First, internal rela- tionships are strengthened by applying multi-head self-attention (SA) (Vaswani et al.] (017) units independently to the visual, audio, and linguistic features. Next, the self-enhanced word-level lin- guistic features serve as query in multi-head cross-attention (CA) (Vaswani et al.] 2017) units to capture shared semantics from the frame-level visual features and audio features (used as keys and values, respectively). For simplicity, the visual capturing step is formulated as (with a similar oper- ation for audio obtain Fqa):
Fgu =
To fuse the captured information, aggregate the cross-modal outputs with residual linguistic features,
Fgg = + Fgo + FA(Fw)
where Fag represents the query query-guided semantic context. Subsequently, this guidance is propagated back to the visual and audio streams via cross- -oosstttt with F, as query and Fag as keys and values (or use Fa to obtain Fag). This step is denoted
Feg = CA Fag, Fag).
(3)
Finally, the outputs are refined by incorporating the original modality features via residual connec- tion, followed by a Feed-Forward Network (FFNN):
rg FFN(F, F.); Flg FFN Fa).
where Flg and Feg denote the query-guided visual and audio representations.
3.3
TEMPORAL--RRQURESS INTERACTION MODULE
To more effectively exploit the audio information in videos, we introduce two dedicated submodules to localize performing instruments across spatial, temporal, and frequency dimensions.
Spatial-Temporal Interaction. Since videos contain both spatial and temporal dimensions that may cover redundant information, we design a previously obtained query- guided features. Specifically, Fp to better align fine-grainee spatial details with the query-guided audio context. First, Fp is refined using SA unit, and then used the query in CA unit, where the query- query-guided audio features F serve as keys and values:
submodule to selectively refine the enhance the patch-level visual features
F.: Ca
(5)
This operation encourages the model to focus on sounding regions that correspond to the question. In parallel, to capture global temporal dependencies, we perform a temporal interaction between the query-guided audio and visual features. Specifically, we compute the dot product between F and F. followed by softmax function:
Fs F'g softmax (FAT Frgg);
where T is the transpose operation. We integrate the spatial and temporal outputs by concatenating the corresponding features after reshape, followed by freedFoowward Network:
Fu: FFN
Fu)).
where Concat denotes the concatenation layer. This fusion enables the model to yield refined spatial- temporal visual features conditioned on the query.
5

---

Published as a conference paper at ICLR 2026
Femporal-Frequency Interaction. In some cases, spatial-tempon perception alone is insufficient for comprehensive music scene understanding. For instance, a flutist typically keeps the instrument near his/her mouth, with minimal subtle motion that is difficult to detect visually, potentially leading to misinterpretation. In such cases, frequency-hasee analysis plays crucial role in distin- guishing instruments by capturing their unique spectral signatures. These spectral "fingerprints" often more informative than time- or spatial-domain features, particularly in polyphonic scenarios or acoustically complex environments for cross-model reasoning.
To better exploit these discriminative clues, we design temporal-frequency submodule to refine the audio representation. Specifically, we utilize a pretrained Audio Spectrogram Transformer (AST) Gong et al. (022)) to extract rich frequency-
features from the audio waveform, denoted as Past E 1 X XFXD Notably, while two instruments may generate similar time-domain waveforms or pitches, their timbral characteristics often differ significantly due to the unique distribution of over- tones and harmonics across frequency bands. These subtle spectral distinctions are more effectively captured by AST compared to models like VGGish (Gemmeke et al. 2017). This is largely due to architectural differences: AST's transformers model long-range dependencies across frequency bins, enabling it to represent harmonic structures and overtone patterns that define timbre (Agostini eet al. 2003] Zhao et al., 2018 Koutini et al.] 2021] CCenn et al., 2022). To further highlight informa- attention over the AST features, enabling the model to explicitly emphasize frequency bands that are most relevant for query-guided instrument recognition. In detail, we first aggregate the AST-based audio features over the temporal dimension to obtain a condensed frequency representation. Using this representation and the question em- beddings, we compute frequency attention weights that highlight question-relevanne spectral bands. These weights are then broadcast and applied across the original AST features to emphasize salient frequency regions:
tive frequency
T regions,
introduce frequency-
T Σ
Imnan
T Fasltt,;, :.
(8)
af softmax (W1 Fwr W2 ReLU(Ws SThean)),
(9)
F Fast= aa Fasts
(10)
where W1. W2. and W3 are learnable projection matrices. Finally, align and integrate Fast and F'gg we concatenate them and apply a convolutional fusion block composed of two convolutional layers with batch normalization and ReLU activation:
Fa. Conv (ConlalckCCotaFFF.... Fe Fdg));
(11)
This fusion allows the model to effectively associate discriminative spectral patterns with the corre- sponding instruments, producing more informative and query query-guided audio features.
3.4 QUERY CONTEXT REASONING BLOCK AND PREDICTION
To enhance final answer prediction using the refined visual and audio features, we introduce Query Context Reasoning (QCR) block inspired by prompt-based conditioning (Khattak et al. 2023) Prompts can provide focused guidance for interpreting visual and audio content under task-specific constraints. In our music scene understanding task, the relevant cues often involve musical instru- ments, specifically their type. performance duration, location, temporal sequence, and loudness. These aspects, though derived from the dataset's 's question types, capture general audio-visual rea- soning dimensions. They are used to construct the query context, which guides the refinement of the updated modality-specific features. More details about the prompts can be found in Appendix Sec. A.2.3 We first encode these context-related keywords using the same CLIP text encoder, yield- ing prompt embeddings denoted as Fprompt. Since not all aspects are equally relevant for each question, we incorporate the sentence-level question embedding F L'sentence- They are concatenated and passed through multi-head self-attention unit to produce the query context feature Fge:
Fgr SASSAACCCaaa prompt; F Esentence)),
(12)
Next, we apply cross-attention using Fge as the query to guide the refinement of visual and audio features Fus and Fas, respectively:
Fjo CA Fui, Foi); Fja- CA(Fges Fai, Fai),
(13)

---

Published as a conference paper at ICLR 2026
This process enables the model to extract the most informative audio and visual cues under the linguistic query context. For final answer prediction, we perform a simple multimodal fusion and classification. The fused representation is obtained by
(14) where Fyu and Faa are combined with a concatenation layer and a linear layer followed by a tanh activation function. Then we adopt an element-wis multiplication operation o to integrate the ques- tion features and final visual audio representations as: e = Fsentence Fao. The resulting feature is then used to predict the final answer from predefined vocabulary set.
Fae = FC((oncat(Froo Ffa));
4
EXPERIMENTS
4.1 DATASETS AND EVALUATION METRIC
The experiments were primarily conducted on the widely-used MUSIC-AVQA (Li et al, 2022) dataset. MUSIC-AVQA contains more than 40K QA pairs distributed across 9,288 videos. These pairs, centered around music-related scenarios, are categorized into three types based on the modalities required to answer them: Audio QA, Visual QA, and Audio-Visur QA. The questions were generated using set of predefined templates. We also evaluated models on AVQA (Yang al. 2022), which comprises over 57K QA pairs derived from real-world videos, and provided discussion on the compared results in Appendix Sec.A.1 A.
Dataset.
Evaluation Metric. Following the standard protocol in previous AVQA studies, we report answer accuracy (Antol et al. (2015) for each question type (as classified in the benchmark) along with the overall average to evaluate model performance in different audio-visual scenarios.
4.2 IMPLEMENTATION DETAILS
Videos were segmented into I-second clips from 60-second recordings. We used a CLIP-ViT-L/I4
(Radford et al, 202I) model for visual and textual feature extraction. Audio features were extracted using VGGish (Gemmeke et al., 2017) pre-trained AudioSet, while AST (Gong al. 2021) was employed to capture time time-frequency representations. All features were projected into 512- dimensional vectors for consistency. We trained all models using the AdamW (Loshchilov et al.) 2017) optimizer with an initial learning rate of le-4, decayed by a factor of 0. every 10 epochs. The batch size and number of training epochs were set 64 and 30, respectively. All experiments were conducted using Py Torch on a single NVIDIA H100 GPU.
4.3 COMPARISON WITH STATE-OF-THE-TT METHODS
Compared Methods. To verify the effectiveness of the proposed method, we compared it with pre- vious ofate-of the-art art multimodal QA approaches. For audio QA, adopted FCN-LSTM (Fayek) & Johnson] 2020). For visual QA, we considered MCAN (Yu et al.] 2019). For video QA, we
evaluated PSAC (Li et al.) 2019). HME (Fan et al. 2019). and HCRN (Le al., 2020). Finally, we compared against a comprehensive set of recent AVQA methods, including AVSD (Schwartz) al. (2019), PanaAVQA (Yun al.) (021), AVST (Li al. (2022), COCA (Lao al. (023), PSTP (Li et al. (022a)), LAVISH (Lin et al. 2023). APL (Li et al. (024b). TSPM (Li et al. (224a)),
MCCD (Ma et al. 2024). PSOT (Li et (025). and QA-TIGER (Kim et al. (025). All methods were trained and evaluated using the same dataset split for fair comparison. We further compared our method with recent large multimodal models (MLLMs). including GPT-4o (Hurst et al., 2024).
Lad222 (Cheng et al. 2024), Qwen2.5-Ommi (Xu et al. 2025) and Ming-Omni (Al et al. 2025).
Quantitative Results. The quantitative results are reported in Table I Our proposed QSTar achieved state-of-the-art results across most question types, outperforming TSPM and QA-TIGER by 19% and 1.36% in overall accuracy, respectively. Compared with earlier models designed for dingle-modalit reasoning with simple multimodal fusion (e.g.. MCAN for visual QA and HCRN for video QA), QSTar showed substantial gains across all metrics, which consistent with trends observed in other AVQA models. These improvements underscore the advantage of deep multi- modal alignment among audio, visual, and textual features for complex video scene understanding.
7

---

Published as a conference paper at ICLR 2026
Table 1: Comparison with existing methods on the MUSIC-AVQA (Li et al. 2022) test set, re- porting accuracy (%) across different question types. The short names are the abbreviations for question types "Counting", "Comparative", "Location", "Existential", and "Temporal", respectively. For space, FCN-LSTM, VideoLLaMA2, and Qwen2.5-Omin are denoted as FCN, VLLaMa2, and Owen-Omni, respectively. The best results are bold while the second best are underlined.
Audio QA
Audio- Vistal QA Count. Local, Avg Exist. Count. Local, Comp. Temp. Avg
Visual QA
Method
Avg
Count.
Comp. Avg
RCN
Payek
Johnson
70.80
65.66 68.90) 64.58 48.08 82.22 59.92 46,20 62.94 47.45
MCAN
2019
78.07
57.74 70.58 71.76
71.76 71.76 80.77 65.22 54.57
56.77 46.84 61.52
PSAC
2019
75.02
66.84 72.00
68.00
70.78 69.41 79.76 61.66 55.22
61.13
59.85 63.60
HME
73.65
63.74 69.89
67.42
70.20 68.83 80.87 63.64 54,89
63.03 60.58 64.78666755
Schwarz
2019
72.47
62.46 68.78
66.00
74.53 70.31 80.77 64.03 57.93
62.85 61.07 67.44
HCRN
Le et al.
2020
71.29
50.67 63.69
65.33
64.98 65.15 53.28 41.74
51.04 46.72 59.32
LAVIT
Yun et a1
202
75.71
65.99 72.13
70.51
75.76
73.16 65.38 61.30 63.67 62.04 69.53
AVST
20:2
77.78
67.17 73.87
73.52
75.27
82.49
69.88 64.24
64.67 65.82 69.5371.59
79.35
67.68 75.42
75.10
75.43
83.50 66.63
69.72
64.12 65.57 72.33
20230
73.97
65.59 70.91
77.15
77.36
77.26 76.18 72.23
71.80
71.79 69.00 72.57
LAVISH
Liaet ct al 2023
82.09
65.56 75.97
78.98
81.43
81.71 75.51 66.13 63.77 67.96 71.26 74.46
API.
ml. 50244
82.40
70.71 78.09
76.52
82.74 79.69 82.99 73.29
66.68
64.76 65.95 70.96
Ma2
Chengee ctall
2024
79.65
52.69 69.71
81.20
83.02
77.43 63.48
69,67
62.67 68.13 71.98
GPT-40
al
65.68
37.04
55.12
72.77
62.20
55.87 54.94
59.57
38.24 42.58
50.35
TSPM
all
2025a
84.07
64.65
76.91
82.29
84.90
83.61 82.19 76.21
71.85
65.76
71.17 73.51 76.79
MCCD Ma et
83.87
79.14 71.04
79.78
76.73
78.24 80.87
71.46
51.63
64.67
64.60
67.13
PSOT
2005
78.22
80.07
72.61 75.29
Qwan-Dmni
62.93
39.56
54.31
67.92
59.02
63.42
55.10
60.98
36.42
40.27
53.82
Ming- Omni
62.44
40.74
54.44
67.50
58.29
52.73
53.91
62.17
37.24
41.85
43.98
QA-TIGER
Kim ct ml 2025
84.86
67.85
78.58
83.96
86.29
85.14
83.10
78.58
72.50
63.94
69.59 77.74
QSTar (ours)
85.64
72.05
80.63
83.46
84.90
84.17 83.81 79.76
72.72
70.03
72.38
78.98
Notably, QSTar surpassed existing AVQA approaches that primarily rely visual processing tech- niques such as frame selection or object-level perception. For example, QSTar achieved 78.98% average accuracy, compared to 76.79% by TSPM and 74.53% by APL. It also outperformed the previous SOTA QA-TIGER by 2.05% on Audio QA and 2.24% on Audio- example, in Audio QA. our method achieved 4.2% gain over QA-TIGER on comparative ques- tions. In Audio-Visual QA, comparative and temporal questions improved by 6.09% and 2.79%, respectively, compared with QA-TIGER, These gains arise because frequency energy patterns pro- vide precise indicators of instrument activity, especially for polyphonic scenes. Unlike visual cues, which may be subtle, occluded, or ambiguous, frequency features clearly reflect when an instrument starts, stops, or changes intensity. These results demonstrate the strength of our frequency-awar modeling design. Despite not relying on pre-trained object detectors or specially designed visual perception modules, QSTar still performed competitively on Visual QA type, trailing QA-TIGER by only 0.97%, while significantly outperforming other vision-centric methods such as APL (+4.48%) and PSOT (+4.1%). This demonstrates that our query-guided multimodal design preserves strong visual reasoning. The comparison highlights the effectiveness of our model and motivates future enhancements in the spatial localization of performing instruments.
QA types. For
In addition, we conducted experiments and report comparisons with GPT-4o, Qwen2.5 5-mmmi and Ming-Omni. Following common practice for proprietary models, these MLLMs were evaluated in zero-shot setting due to their closed-source nature and large-scale pre-training benefits. The highly capable GPT-4o achieved only 55.72% average accuracy while Qwen2.5- (7B) and Ming-Lite- Omni achieved similar modest results at 53.83% and 53.98%, respectively. These models strug- gled particularly with question types requiring fine-grained temporal and comparative reasoning in polyphonic scenes, exhibiting drops of nearly 30% on these categories. In contrast, QSTar signifi- cantly outperformed all evaluated zero-shot MLLMS across every question type. We further assessed MLLM performance under fine-tuning setting by comparing against VideoLLaMA2, a represen- tative open- source MLLM. Even with aask-specific fine-tuning, VideoLLaMA2 still trailed QSTar substantially particularly comparative questions, where QSTar exceeded it by 19.36% (Audio QA) and 7.36% (Audio- Visual QA). This performance gain highlights the specialized strength of QSTa on cross-modal reasoning within complex polyphonic scenes.
8

---

Published as a conference paper at ICLR 2026
Table 2: Ablation study on the proposed main modules.
Table Ablation study on the query guid- ance. The notations r/m in B, M. and F rep- resent the exclusion of query guidance dur- ing the beginning. middle, and final stages, respectively.
Method
Audio QAJVisual QA QA
Avg
w/o all
73.87
79.15
70.33
73.29
w/o QGMC
79.08
83.44
72.92
76.80
w/o QCR
79.33
83.24
75.43
78.19
Method
Audio QA Visual QAJA-V QA|
Avg
w/o STT
79.21
82.62
75.06
77.80
r/m in B
78.96
83.61
74.90
77.93
w/o TFI
78.21
83.24
74.39
77.41
r/n in M
80.07
83.65
75.65
78.55
w/o STFI
77.09
81.79
74.02
76.62
r/m
79.39
83.32
75.47
78.25
QSTa
80.63
84.17
75.98
78.98
QSTar
80.63
84.17
75.98
78.98
Table Efficiency analysis and comparison on MUSIC-AVQA test set.
Method
Trainable Parameters (M) FLOPs (G) Avg Accuracy (%)
AVST (Li al. 2022
18.48
3.19
71.59
LAVISH (Lin et al. 2202
21.09
74.46
TSPM (Li al. 2024a
6,22
1.42
76,79
QA-TIGER (Kim cl al.
2025)
14.51
2.70
77.62
Qa (w/o TFI)
11.95
2.15
77.41
QSTar (ours)
13.20
2.43
78.98
4.4 ABLATION STUDIES
Ablation Study on Main Modules. To verify the effectiveness of the main modules (i.e., QGMC, STI, TFI, QCR, etc.), we conducted ablation studies by removing each module. Table reports the corresponding results. Removing all modules and retaining only simple multimodal fusion led to performance drop of over 5%, highlighting the overall importance of our design. Specifically, removing QGMC and QCR reduced the overall accuracy to 76.80% and 19.,,%, respectively. Per- formance dropped across all question types as well, underscoring the importance of multimodal feature correlation and context-aware reasoning tailored to each query. The removal of the spatial- sypporal-frequeeee interaction module also significantly impacted performance. Without STI, over- all accuracy dropped by 1.18%, and notably, Visual QA accuracy declined by 1.55%, emphasizing the value of our spatial perception and temporal alignment. Eliminating the TFI module caused a sharp decrease in Audio QA and Audio Visual QA performance (2.42% and 1.59%, respectively), indicating that questions requiring recognition of which instrument is sounding or when becomes active rely heavily frequency-domain cues. The improvement is particularly notable for compar- ative and temporal questions, where reasoning depends on detecting onset/offset transitions that are often imperceptible in RGB frames. These results demonstrate the necessity of frequency-domaie reasoning for auditory understanding. In summary, each module plays a critical role in enhancing model performance. When integrated, they work synergistically, enabling QSTar to achieve the best results on the MUSIC-AVQA benchmark. More ablation studies on these modules and their alternatives can be found in Appendix Sec. A.2
Effect of Query Guidance throughout the Pipeline. To assess the impact of query guidance throughout the QSTar pipeline, we performed ablation studies by selectively removing its corre- sponding components. Query guidance is integrated at three stages: beginning (via query-guided semantic context in QGMC), middle (through question embeddings in TFI), and final (using prompt- ing in QCR). The resulting ablated models are denoted M, Mm and M,. respectively. Note that the final-stage multiplication, as commonly used in prior works, was retained in all ablations. The results, shown Ta indicate consistent performance drops when any stage of query guid- ance is removed. Removing carly-stage guidance (M,) resulted in 1.05% drop, indicating that early query-guided refinement helps identify representative audio and visual features. the middle stage, M, which omits question embeddings in TFI, led to only slight drop (0.43%), suggesting that prior query-guided features already carried useful linguistic information. Final-stage prompting also proved beneficial, M, underperformed QSTar (by 0.73%), confirming the value of incor- porating query context during reasoning. These findings underscore the importance of integrating query guidance throughout the pipeline to enhance audio-visual scene understanding. Additional studies on the prompting mechanism provided in Appendix Sec. A.2.3

---

Published as a conference paper at ICLR 2026
musiral indnumenne swund vateo?
Iethy inttrmment the rght deythenir thon the intrummntion
@o Ditk Tloee
Ques Four
GT: Foss
@4 TIGER No
Oun Yo.
Gii Vo
(a) Result comparison against QA- TIGER and ground truth.
than right!
(b) example of QSTar prediction (Left: input sample; Right: prediction and question-related sounding area and timestamps).
2022) dataset. (a) compares answer pre- dictions between QSTar and QA-TIGER (Kim et al. 2025 on two examples. (b) visualizes spatial, temporal, and frequency focuses from QSTar's STFI module. Green boxes highlight the patch-level visual attention at key timestamps selected via audio-basee temporal focus. A 2D spectrogram pro- vides an overview of frequency dynamics for better interpretabiiity.
Figure 3:
Qualitative results in the MUSIC-AVQA (Liet al.
4.5
COMPUTATIONAL COSTS
Table 4 reports the computational costs of QSTa compared with AVST, LAVISH, TSPM, and QA- TIGER. QSTar used fewer trainable parameters, lower FLOPs, and achieved higher accuracy than AVST and LAVISH. Compared to QA-TIGER, QSTar delivered higher performance while maintain- ing comparable computational costs. Although TSPM is the most lightweight, its accuracy lagged behind QSTar by 2.19%, indicating that QSTar provides substantially better accuracy-efficiee trade-off. Despite introducing an additional AST-based audio branch, QSTar remains efficient be- cause it employs lightweight ceqquency-interactiin an increase of 1. 1.25M trainable parameters and 0.28G FLOPs compared to QSTar without TFI, while still keeping the overall complexity comparable lower than other models. The AST branch introduced only a modest computation overhead for clear accuracy gain (1.57%).
module (TFI). As shown in the table, this led to
4.6 QUALITATIVE RESULTS
In Fig. we present qualitative results from the MUSIC-AVQA
Subfigure (a) demonstrates QSTar's advantage over the previous SOTA method, QA-TIGER, in complex multi-instrument sce- narios. For instance, even when the cello is consistently visible, QSTar correctly predicted the audio cues. Similarly, our method succeeded in
answer Four by leveraging frequency-enhancee distinguishing two guzhengs in a performance scene. Subfigure (b) visualizes temporal visual tentions and frequency-relevant audio information. The STFI module captures dynamic changes in audio intensity, correctly identifying that the clarinet continues playing while the bassoon stops at the middle timestamp. These examples highlight QSTar's ability to localize query relevant instruments across space, time, and frequency, enabling accurate reasoning in complex musical scenes.
CONCLUSION
In this work, we propose a Query-Guided method for the music AVQA task. We design query-guided multimodal correlation module that incorporates the query guidance into audio-visual feature learning to enhance relevant representa- tions. We also propose spatial, temporal, and frequency interaction module to effectively align au- dio, visual, and textual modalities across these dimensions. Furthermore, a prompting-bad based query reasoning block is employed to incorporate linguistic context before final prediction. Extensive experiments on the standard AVQA datasets demonstrate that QSTar consistently outperforms state- of-the-art methods, while ab ablation studies confirm the effectiveness of its components.
Interaction (QSTar)
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This work is partially funded by the European Union within the Horizon Europe research and inno- vation programme under grant agreement No. 101136006 XTREME project, coordinated by S.S. Brandt/IT University of Copenhagen, Denmark. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union. European Union can not be held responsible for them. The work is also supported in part by the Pioneer Centre for AI (DNRF grant number Pl).
REFERENCES
Giulio Agostini, Maurizio Longari, and Emanuele Pollastri. Musical instrument timbres classifica- tion with spectral features. EURASIP Journal on Advances in Signal Processing, 2003((): 2003.
Inclusion AI, Biao Gong, Cheng Zou, Chuanyang Zheng, Chunluan Zhou, Canxiang Yan, Chunx- iang Jin, Chunjie Shen, Dandan Zheng, Fudong Wang, et al. Ming-omnn -ang- unified multimodal model for perception and generation. arXiv preprint arXiv:2506.00 2500:22500.0934,, 2025.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE International Conference Computer Vision, pp. 2425-2433, 2015.
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In International Conference on Learning Repre- sentations, 2023.
Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg-Kirkpatrrck,, and Shlomo Dubnov. Hts-at: A hierarchical teken-semantic audio transformer for sound classification and detection. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 646-650, 2022.
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.074766 2024.
Haoyi Duan, Yan Xia, Zhou Mingze, Li Tang, Jieming Zhu, and Zhou Zhao. Cross-modal prompts: Adapting large pre-trained models for audio-visual downstream tasks. Advances in Neural Infor- mation Processing Systems, 36:56075-56094, 2023.
Chenyou Fan, Xiaofan Zhang, Shu Zhang. Wensheng Wang, Chi Zhang, and Heng Huang. Hetero- geneous memory enhanced multimodal attention model for video question answering. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1999-2007, 2019.
Haytham M Fayek and Justin Johnson. Temporal reasoning via audio question answering. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2283-222 2020.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In JEEE International Conference on Acoustics. Speech and Signal Processing, pp- 776-780, 2017.
Yuan Gong, Yu-An Chung, and James Glass. Ast: Audio spectrogram transformer. In Proc. Inter- speech 2021, pp. 571-575, 2021.
Pierre-Amaury Grumiaux, Srdan Kitié, Laurent Girin, and Alexandre Guérin. A survey of sound source localization with deep learning methods. The Journal the Acoustical Society of America, 157 2022.
Liwei He, Elizabeth Sanocki, Anoop Gupta, and Jonathan Grudin. Auto-summarization of audio- video presentations. In Proceedings of the seventh ACM International Conference on Multimedia, pp. 489-498 1999.
11

---

Published as a conference paper at ICLR 2026
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Xinya Ji, Hang Zhou, Kaisiyuan Wang, Wayne Wu, Chen Change Loy, Xun Cao, and Feng Xu. Audio-driven emotional video portraits. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pullern Recognition, pp. 14080-14089, 2021.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shah- baz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19113-19122, 2023.
Hongyeob Kim, Inyoung Jung, Dayoon Suh, Youjia Zhang, Sangmin Lee, and Sungeun Hong. Question-aware gaussian experts for audio- question answering. In Proceedings the Com- puler Vision and Pattern Recognition Conference, pp. 13681-13690, 2025.
Khaled Koutini, Jan Schlüter, Hamid Eghbal-Zadeh, and Gerhard Widmer. Efficient training of audio transformers with patchout. arXiv preprint arXiv:21 2021.
Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: large video database for human motion recognition. In International Conference on Computer Vision, pp. 2556 2563, 2011.
Mingrui Lao, Nan Pu, Yu Liu, Kai He, Erwin M Bakker, and Michael S Lew. Coca: Collaborative causal regularization for audio-visual question answering. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 12995-13003, 2023.
Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran. Hierarchical conditional relation net- works for video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9972-9981, 2020.
Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu, Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 19108-19118, 2022.
Guangyao Li, Wenxuan Hou, and Di Hu. Progressive spatio-temporal perception for audio-visual question answering. In Proceedings of the ACM International Conference on Multimedia, pp- 7808-7816 16, 2023a.
Guangyao Li, Heng Du, hui, and Di Hu. Boosting audio visual question answering via key semantic- aware cues. In Proceedings of the ACM International Conference on Multimedia, pp. 5997-6005, 2024a.
Kexin Li, Zongxin Yang, Lei Chen, Yi Yang, and Jun Xiao. Catr: Combinatorialldeeeedennce audio- queried transformer for audio-visual visual video segmentation. In Proceedings of the ACM Interna- tional Conference on Multimedia, pp. 1485-1494, 2023b.
Kun Li, George Vosselman, and Michael Ying Yang- Interactive image segmentation with cross- modality vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp. 762-772, 2023c.
Xiangpeng Li, Jingkuan Song, Lianli Gao, Xianglong Liu, Wenbing Huang, Xiangnan He, and Chuang Gan. Beyond mns: Positional self-attention with co-attention for video question answer- ing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 8658-8664 2019.
Zhangbin Li, Dan Guo, Jinxing Zhou, Jing Zhang, and Meng Wang. Object-aware adaptive- positivity learning for visual question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 3306-3314, 2024b.
Zhangbin Li, Jinxing Zhou, Jing Zhang, Shengeng Tang, Kun Li, and Dan Guo. Patch-level sounding object tracking for audio-visual question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. pp. 5075-5083, 2025.
12

---

Published as a conference paper at ICLR 2026
Yan-Bo Lin, Yi-Lin Sung, Jie Lei. Mohit Bansal, and Gedas Bertasius. Vision transformers parameter-effician audio-visual learners. In Proceedings of the IEEE/CVE Conference on Com- puter Vision and Pattern Recognition, pp. 2299 2009, 2023.
lya Loshchilov, Frank Hutter, al. Fixing weight decay regularization in adam. arXiv preprint arXiv: 170000001, 2017.
Jie Ma, Min Hu, Pinghui Wang, Wangchun Sun, Lingyun Song, Hongbin Pei, Jun Liu, and Youtian Du Look, listen, and answer: Overcoming biases for audio-visual question answering. In The eighth Annual Conference on Neural Information Processing Systems, 2024.
Baoqi Pei, Yifei Huang, Guo Chen, Jilan Xu, Yali Wang. Limin Wang. Tong Lu, Yu Qiao, and Fei
Wu. Guiding audio-visual question answering with collective question reasoning. International Journal of Computer Vision, pp. 1-18, 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-87888 0003887633222 2021.
Idan Schwartz, Alexander Schwing, G, and Tamir Hazan. A simple baseline for audio-visudl scene- aware dialog. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- tion Processing Systems, 30, 2017.
Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, and Jiwen Lu. Drivedreamer: Towards real-werld-drive world models for autonomous driving. In European Conference on Computer Vision, pp. 55-72, 2024.
Jin Xu, Zhifang Guo, Jinzheng He. Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang. Yang Fan, Kai Dang, al. Qwen2. 5-omni technical report. arXiv preprint Stti::2503222//1.. 2025.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Mst-vtt: large video description dataset for bridging video and language. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5288-528 02965,296772 2016.
Pinci Yang. Xin Wang, Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu, Avqa: A dataset for audio-visual question answering on videos. In Proceedings of the ACM International Conference on Multimedia, pp. 3480-3499 2022.
Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6281-6290, 2019.
Heeseung Yun, Youngjae Yu, Wonsuk Yang, Kangil Lee, and Gunhee Kim. Pano-avqa: Grounded -isual question answering on 360deg videos. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 2031-2041, 2021.
HongJiang Zhang, Chien Yong Low, Stephen W Smoliar, and Jian Hua Wu. Video parsing, retrieval and browsing: an integrated and content- based solution. In Proceedings of the ACM International Conference Multimedia, pp. 15 24, 1995.
Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick. Josh McDermott, and Antonio Torralba. The sound of pixels. In Proceedings of the European conference on computer vision (ECCV), pp. 570-586, 2018.
Xingyi Zhou, Anurag Arnab, Shyamal Buch, Shen Yan, Austin Myers, Xuehan Xiong, Arsha Na- grani, and Cordelia Schmid. Streaming dense video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18243-18252, 2024.
13

---

Published as a conference paper at ICLR 2026
A
APPENDIX
This supplementary document includes detailed discussion the results on AVQA (Yang et al. 2022) dataset, additional ablation studies on main modules, and more qualitative results across dif- ferent question types. We also provide a discussion for future work.
A.1 RESULTS ON AVQA DATASET
To further assess the robustness of our proposed method, we evaluated it on the AVQA dataset, which includes more diverse scenarios beyond music-related events. In detail, AVQA covers diverse everyday scenes (e.g., alarms, dog barking, speech, water flow, and vehicle sounds). These signals typically broadband, less structured, and often non-harmonic. The corresponding questions generally focus on salient, dominant audio events (e.g. "Is the alarm ringing?") rather than subtle harmonic differences, compared MUSIC-AVQA. To ensure direct comparability and mitigate potential implementation discrepancies, we thus follow the exact reporting format established by prior methods (Li et al. 2023a 2024a) in Table 5 We also report our model's performance with and without query prompting for fair comparison. Our method outperformed previous PSTP and TSPM in terms of average accuracy, demonstrating that the proposed modules are not restricted to music-specific settings.
A.2 MORE ABLATION STUDIES
In this section, we further report several groups of ablation studies with respect to the designs of query-guided multimodal correlation (QGMC) module, peatialtemporal-frequu interaction (STFI) module, and query context reasoning (QCR) block.
A.2.1 ABLATION ON QUESTION-GUIDER) FEATURE PROCESSING
To evaluate the impact of different strategies for early-stage multimodal feature processing, we conducted ablation study focused question-guided representations. We compared three al- ternatives: (a) direct fusion of audio and visual features, as in AVST (Li et al. 2022); (b) sepa- rate cross-attention between linguistic features and each modality (audio/visual); and (c) sequential cross-attention with audio and linguistic cues for visual update (similar for audio), as in QA- QA-TIGER (Kim al. )025). Table presents the results. All methods incorporating early question guidance outperformed (a). with at least 0.8% gain in average accuracy. Method (b) confirms the utility of basic cross- attention for enhancing
-specifi representations. While (c) achieved slight edge over our method (d) on Audio Visual QA type (0.08%), it did not surpass our approach in overall performance. These findings highlight the effectiveness of question-guides processing and suggest that early fusion of audio and visual features offers limited additional benefit.
A.2.2 ABLATION ON FEATURE USAGE IN STFI
To assess the contribution of the additional features introduced beyond the query-guided represen- tations from QGMC, we conducted an ablation study by removing patch-level visual features (Fp) and AST-based (Gong et al.) (021) audio features (Fast) from the pratial-tempor interaction (STI) and semporal-frequencce interaction (TFI) modules, respectively. The results are reported in Table 7 Excluding Fp led to 1.43% and 0.9% drop in Visual QA and Audio-Visual QA perfor- mance, confirming the finding that patch-level features are critical for spatial perception. Removing Fast caused 1.37% decrease in average accuracy, notably degrading performance on audio-related questions. This aligns with recent work (Pei et al. 2025) showing that AST provides richer audio representations than VGGish (Gemmeke et al. 2017), especially in complex audio-video scenes. We attribute the improved performance to our rrquuenc-wwis audio modeling, which enhances the model's ability distinguish instruments both within and across categories.
A.2.3 ABLATION ON PROMPTING IN QCR
Before introducing the prompts used in our method, first examine the MUSIC-AVQA (Li al. 2022) benchmark to motivate the design of our query context. Specifically, we identify five key aspects (type, duration, location, sequence, and loudness) fundamental to understanding music
14

---

Published as a conference paper at ICLR 2026
Table 5: Comparison with existing methods on the AVQA
Yang al. 2022 test set.
Method
Prompting Total Accuracy (%)
PSAC (Li et al.
2019)
87.4
HCRN Le
2020
89.0
PSTP (Li
90.2
TSPM [L ct al.
2024a
90.8
QSTar (ours)
90.9
QSTar (ours)
91.2
Table 6: Ablation study on early-stage crestion--uidded feature processing. (a) integrates audio
and visual features to form sounding visual features (AVST (Li et al. 2022 (b) applies separate
cross-attention between linguistic and each modality; (c) uses dual-branch sequential cross-attention
(QA-TIGER (Kim et al., 2025)); (d) our proposed query-guided multimodal correlation (QGMC) module.
Method
Audio QA
Visual QA
Audio Visual QA
Avg
78.03
83.69
74.43
77.52
80.20
83.86
75.06
78.30
80.07
84.06
76.06
78.89
d (ours)
80.63
84.17
75.98
78.98
scenes and guiding multimodal reasoning. The MUSIC-AVQA dataset features questions involving 22 instruments, categorized into four types: String (i.e., violin, cello, guitar, ukulele, erhu, guzheng, pipa, bass, banjo), Wind (i.e., tuba, trumpet, suona, bassoon, clarinet, bagpipe, flute, saxophone), Percussion (i.e., drum, xylophone, congas), and Keyboard (i.e., accordion, piano). Each instru- ment class involves distinct visual and auditory cues, and depending on the user's question, different modalities and temporal spans may become more relevant for accurate reasoning. We present all 33 sample questions from MUSIC-AVQA in Table g along with summary of the key analytical as- pects they target. Most questions focus on one to three core elements of music scene understanding. Accordingly, our prompt-based query context construction uses set of guiding keywords (type, per- formance duration, location, temporal sequence, and loudness) to capture these aspects effectively. these reasoning dimensions: identifying what is sounding
These prompts explicitly correspond or appearing, when an event occurs, where it occurs, what order, and with what intensity. For example, under the "type" aspect, the underlying focus may concern instrument category audio identity. Note that we do not craft question-specttn unified prompt formulation for two reasons: (1) Avoiding prompt-answer leakage. Using tailored prompts per question risks encoding partial answers or shorteuts. Unified prompts ensure the model must rely on audio, visual, and textual A consistent, domain-agnostic prompt set scales naturally to new question types and datasets, while also providing stable semantic anchors for multimodal alignment.
prompts or explanations. We intentionally use
during reasoning. (2) Ensuring scalability and stability.
To evaluate the effectiveness of our proposed prompts, we conducted an ablation study on MUSIC- AVQA. As baseline, we removed the prompts entirely, following the M7 variant described in the main text. We then compared three alternative prompting strategies: (1) the declarative prompt formulation used in TSPM (Li et al., 2024a),, which converts questions into related statements. For instance, a given question "Where the first sounding instrument?" is converted "The instruments in the video do not sound at the same time."; "; (2) prompts generated using a pre-trained video cap- tioning model, Scenic (Zhou al. 2024), to describe overall video dynamics as longer narrative
Table 7: Ablation study on the feature usage in the spatial-eemporal-feqqueeee interaction module.
Method
Audio QA Visual QA|
Audio Visual QA
Avg
w/o Fp
79.27
82.74
75.08
77.88
w/o Fost
78.90
83.24
74.53
77.61
STFI (ours)
80.63
84.17
75.98
78.98
15

---

Published as a conference paper at ICLR 2026
Table 8: Overview of the five question types and 33 sample questions from MUSIC-AVQA Li et al. 2022), along with their summarized focus areas (i.e., type, duration, sequence, location, and loudness) for the answering task.
Question Types
Sample Questions
Focuses
Is there clarinct sound?
type & loudness
How many musical instruments were heard throughout the video?
type & loudness
How many types musical instruments Is there a violin in the entire video!
heard throughout the vidco?
type & londness
type type
there drum and piano instruments
videu?
Counting
How many types of musical instruments appeared in the entire video?
type
How many guzhengs are in entire video?
type
How many instruments
sounding in the video?
type & Joudness
How many types of musical instruments sound in the video?
type & loudnesss
How many instruments the vidco did sound from beginning cad?
type duration
How many sounding sounas in the video?
type & loudness
Is the guitar more fhythmic than the collo?
type & sequence
the clarinet louder than the bussoon?
type & loudness
the saxophone playing longer thun the hanjo?
type & duration
Comparative
Is the instrument the left rhythmic than the instrument the right?
lozation sequence
Is the instrument on the right louder thun the instrument on the left?
location & loudness
the pipa left more rhythmic than the erhu
right?
type & location & sequence
Is the ukulele on the right louder than the bass on the left?
type & location & loudness
Where is the performance?
location
What the instrument the left of xylophone?
type location
What kind of musical instrument is it?
type
Location
What kind of instrument the leftmost instrument?
type location
Where the loudest instrument?
location & loudnees
Is the first sound coming from the right instrument!?
location & sequence
Which is the musical instrument that sounds at the time the congus?
type & sequence
What is the right instrument of the last sounding instrument?
type & location & sequence
Is this sound Is the accordion in the video always playing?
instrmmen
video?
loudness
Existential
type & duration
Is there a vaiceover?
type & Joudness
Where fiss sounding instrument?
sequence
Temporal
Which pipa makes the sound last?
type & sequence
Which instrument makes sounds before the trumpet?
type & sequence
Table 9: Ablation study on alternative prompting strategies for query-context construction. Trans-
lation refers to question-to-statemenn conversions used in TSPM (Li et al. 2024a); Caption denotes
video-level descriptions generated by a pretrained captioning model (Zhou et al.
2024); Generative
Prompts correspond to constrained attribute-oriented texts produced by GPT-4o
Hurst al.
2024).
Method
Audio QA Visual QA| Audio-Visual QA
Avg
w/o Prompts
79.39
83.32
75.47
78.25
Translation
79.45
83.77
75.59
78.44
w Caption
79.33
84.35
75.06
78.28
w Generative Prompts
79.21
84.56
75.16
78.37
QCR (ours)
80.63
84.17
75.98
78.98
paragraph (e.g., "A man is strumming a guitar while a girl plays the bass beside him in room.";; (3) attribute-expansion prompts, commonly adopted in text-to-image generation. We use the following generative-style prompt template: "A detailed description of instruments, their location, appearance, and actions that help answer the <question>." 'Tee produced long-form natural- natural-language descrip- tions by GPT-4o (Hurst et al., 2024) are encoded using the same pre-trained text encoder as in the the results. Models using prompting strategies consistently outper-
other basclines. Table(y|peseet formed the baseline. Variant (1), which uses temporal translations, underperformed due to its limited focus on temporal cues, failing to capture location-based or comparative queries. Variant (2), aided by global visual summaries, performed better on Visual QA type but struggled with audio-related questions. Variant (3), although attribute-expanded prompts provide more constrained descriptions than free-form captions, they still introduce substantial irrelevant details and dilute query specifcc information. Consequently, they did not outperform unified prompts. Even worse, they may risk prompt-answer leakage. These findings highlight the effectiveness of our tailored prompting approach in capturing diverse query contexts.
16

---

Published as a conference paper at ICLR 2026
Table 10: Ablation study on the temporal audio pathway.
Method
Audio QA
Visual QA
Audio- Visual QA|
Avg
W Fast
78.96
83,94
75.12
78.14
W Fa (ours)
80.63
84.17
75.98
78.98
W Fast & Fo
79.83
84.19
75.65
78.65
A.2.4 ABLATION ON FEATURE USAGE FOR AUDIO PATHIWAY
In QSTar, the temporal audio pathway is designed to capture coarse temporal dynamics for input au- dios. For this purpose, VGGish- style convolutional features are sufficient and, importantly, provide stable frame-level embeddings aligned to video timestamps. In contrast, AST provides fine- ane-grained spectral modeling with long-range attention over frequency bins, which is essential for distinguish- ing harmonic structures, overtone distributions, and instrument timbre. This is why AST features are specifically used inside the TFI module, where frequency reasoning is required. To directly as- sess this, include additional ablation where replaced the temporal VGGish pathway with AST-only features. As shown Table 10 this substitution did not improve performance and fact slightly reduced accuracy due poorer temporal alignment and feature redundancy. We extended this ablation by adding variant that combined VGGish and AST features as the feature extraction. The results show that simply merging the two feature did not lead performance improvement. The overall accuracy (78, (88) remained very close to the our version (78.9) 98%), and the Audio QA accuracy even dropped slightly (-0.8%). The observations further support our design choice of assigning distinct roles to the two backbones: VGGish for temporal audio pathway and AST for fine-grained frequency reasoning.
A.3 QUALITATIVE RESULTS
In this section, present the predictive results of our method each question type of the MUSIC- AVQA et al. 2022) dataset in Fig. aad Fig.5) For each example, we show video frames, audio waveforms, the question, our model's top-3 predictions, and the ground truth. These results highlight our model's ability to interpret diverse audio-visual cues and generate accurate answers. Notably, even in challenging cases involving rare instruments (e.g., guzheng, erhu), our method demonstrates strong reasoning and robust performance. Overall, these examples showcase the model's general- ization ability across question types and confirm its effectiveness in music scene understanding.
A.4 DISCUSSION AND FUTURE WORK
Although our proposed method has achieved strong performance in music-scene AVQA, it still has some directions that can be explored in the future. The current model operates on Exed-length video segments and may struggle with longer videos that require temporal abstraction memory mecha- nisms. In this work, our frequency-doman modeling is applied only to the audio modality, as audio frequency patterns (timbre, harmonics, onset energy) provide discriminative cues that are unavail- able in visual frames. In turn, explicit visual frequency-domain processing (e.g., high-pass/low-paan filtering or DCT-based feature extraction) may help capture micro-motions that are hard detect in the RGB and audio domains, which is an interesting future direction. Another promising direction is integrating dynamic prompting or large language model-based reasoning, which could enable more flexible and interpretable context modeling. We plan to extend the model to handle open- pneedended questions beyond the current predefined setup, allowing for richer natural language understanding. For Visual QA, especially location-relate questions, our proposed method still slightly lags behind the state-of-the-art. We believe there is substantial room for improvement by integrating object- level cues (e.g., object detectors, motion-based visual features, segmentation cues) and richer relational reasoning mechanisms such as scene graphs, which could strengthen fine- mne-grained spa- tial understanding. Last but not least, expanding the approach to other multimodal QA benchmarks (e.g., .. movie scenes and open-world scenes with complex sound compositions) beyond music-relate datasets will further validate its generalizability.
17

---

Published as a conference paper at ICLR 2026
Questions Are Bereaoosti yetker and vades swuull
Oues Yeu New Zlece
OT/Ya
(a) Audio QA: Counting
Queator: sichamaae wall plyteg looger thoo Bust
Shacc Ney Iadiner
(b) Audio QA: Comparative
Coustas saazy vollas
(c) Visual QA: Counting
Curleng
(d) Visual QA: Location
Figure 4: Prediction results of our method for different question types of Audio QA (counting, comparative) and Visual QA (counting, location) MUSIC-AVQA (Li et al. 2022), with video ids: "00007961", "00002646", "00002454", and "00004109",, respectively. We provide the top answers predicted by our method in sequence.
A.5 REPRODUCIBILITY STATEMENT
We have made every effort to ensure that the results presented in this paper are reproducible. The code will made publicly available to facilitate replication and verification. The experimental setup, including training steps, model configurations, and hardware details, is described in detail in paper. Additionally, the datasets used in this paper are publicly available, ensuring consistent and reproducible evaluation results.
A.6
LLM USAGE
Large Language Models (LLMs) were used to aid in the polishing of the manuscript. Specifically, used LLM assist in refining the language, improving readability, and ensuring clarity in various sections of the paper. The model helped with tasks such as sentence rephrasing, grammar checking, and enhancing the overall flow of the text. not involved in the ideation, research methodology, or experimental design. All research concepts, ideas, and analyses were developed and conducted by the authors.
important to note that the LLM was
18

---

Published as a conference paper at ICLR 2026
Coustion laae rotsonar
Oute big Yoe
Audio Visual QA: Existential
Cuostian Horw Iutrazaata dae ridee ald avezna Eveas begiesing tosed??
One Boue:
Dise
Audio- Visual QA: Counting
Queatier Where & Welooeet tnstreness?
Ousi Rights Left Osnbor
om Right
Audio- Visual QA: Location
peer
Questio: the plas on the digst der than acouttic galtar oa the
Shant Yeu Acccecic
Audio- Visual QA: Comparative
Queator: Which make
Chw.:Sn.llovwwwaly Right
Audio- Vistal QA: Temporal
Figure 5: Prediction results of method for different question types of Audio- Visual QA (existen- tial, counting, location, comparative, temporal) in MUSIC-AVQA (Li et al., 2022), with video ids: "00003803". "00004995", "00008437". "00005464", and "00004026" respectively. We provide the top 3 answers predicted by our method in sequence.
19


Reviews:
[{"review_id": "kGOnV9Bgnp", "reviewer": "Reviewer_92oA", "summary": "The submitted manuscript addresses the problem that previous studies in AVQA task often overlooked the importance of the audio modality. To tackle this issue, the authors propose a Query-guided Spatial-Temporal-Frequency framework (QSTar), which effectively integrates question-guided cues with the distinctive frequency-domain characteristics and spatio-temporal perception of audio signals to enhance audio-visual understanding. Experiments conducted on relevant datasets demonstrate the effectiveness of the proposed method. Overall, the framework exhibits a clear novelty.", "questions": "My main questions are reflected in the Weaknesses Section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem addressed in the manuscript is clearly defined and well-motivated. By focusing on frequency-domain information from the audio perspective, the work presents a distinctive approach that draws attention.\n2. The proposed QSTar framework shows a degree of originality and has been extensively validated on multiple datasets, confirming its effectiveness.\n3. The writing is clear and well-structured, making the paper easy to read and understand.", "weaknesses": "1. Figure 1 effectively illustrates how frequency-domain cues assist in detecting instrument activities that purely spatial or temporal reasoning may miss. However, can similar improvements be observed for fine-grained visual understanding?\n2. How does the model handle off-screen audio sources, such as when the sound of an instrument exists in the audio but the corresponding instrument does not appear in the video?\n3. Considering the limited spatial supervision for audio-visual correspondence, how does the method effectively associate audio and visual cues, especially for rare instruments such as the suona or erhu?\n4. Although the motivation for QCR originates from the analysis of the MUSIC-AVQA dataset, the paper does not discuss in depth the specific prompt forms or their performance compared with more dynamic or generative alternatives, raising some concerns about the scalability of the approach.\n5. Some writing suggestions include avoiding widowed words at the end of paragraphs and adding references for the comparative methods listed in the tables.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mCxZwZtI6o", "reviewer": "Reviewer_VpPp", "summary": "This paper proposes QSTar, a Query-guided Spatial–Temporal–Frequency Interaction framework for Audio–Visual Question Answering task, with a focus on music-related datasets. The key idea is to enhance multimodal reasoning by introducing frequency-domain analysis alongside spatial and temporal interactions. The method introduces three main components: Query-Guided Multimodal Correlation (QGMC) module for early query-conditioned feature alignment; Spatial–Temporal–Frequency Interaction (STFI) module for fine-grained multimodal fusion; and Query Context Reasoning (QCR) block inspired by prompt-based context modeling.\nExperiments on MUSIC-AVQA show that QSTar achieves competitive results.", "questions": "Please check the above section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- It is interesting and meaningful to introduce frequency-level interaction in AVQA tasks. This idea is natural given the acoustic characteristics of music-related videos and provides a new perspective besides conventional spatial–temporal fusion.\n- The overall framework is well structured, clearly integrating linguistic guidance across multiple stages of the model.\n- The implementation details and experimental setup are well described, and the ablation studies comprehensively verify the contribution of individual modules.", "weaknesses": "- The experimental analysis is rather shallow. Most evaluations focus only on accuracy numbers. More in-depth discussion on how frequency-level cues contribute to specific question types or modalities would strengthen the claim. For example, what kinds of questions benefit most from frequency reasoning and why?\n- The improvements over recent strong baselines (e.g., QA-TIGER) are relatively modest—about 1–2% overall—and mainly appear on Audio QA and temporal AVQA subsets. There are no gains in Visual QA. This raises questions about the generality of the proposed design beyond audio-dominant scenarios.\n- The ablation results show only slight differences between “with QCR” and “without QCR,” suggesting that the reasoning block contributes limited additional value.\n- The paper would benefit from more qualitative or case-level studies, such as visualizing which frequency bands or audio/video segments are attended to under different query types. This would make the contribution of “frequency-level interaction” more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oKWwJfRykR", "reviewer": "Reviewer_BoFL", "summary": "The paper introduces QSTar, a novel method for Audio-Visual Question Answering (AVQA) with a strong focus on complex musical scenes. The core contribution is a multi-stage, query-guided architecture that injects textual guidance throughout the pipeline: early with a Query-Guided Multimodal Correlation (QGMC) module, in the middle via a Spatial-Temporal-Frequency Interaction (STFI) module, and finally with a Query Context Reasoning (QCR) block that uses task-aware prompts. The model explicitly enhances audio processing by incorporating frequency-domain analysis to capture timbral characteristics crucial for instrument identification. The method achieves new state-of-the-art performance on the MUSIC-AVQA benchmark, reporting an average accuracy of 78.98%. A smaller-scale evaluation on the general AVQA dataset further demonstrates the model's robustness.", "questions": "Please see the weakness part.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Strong Empirical Performance: The method achieves state-of-the-art results on MUSIC-AVQA, with consistent improvements across Audio, Visual, and Audio-Visual question types.\n- Thorough Ablation Studies: The paper provides extensive ablations that validate the contribution of each key module (QGMC, STI, TFI, QCR) and design choice, strengthened by further controls in the supplementary material.\n- The Query Context Reasoning (QCR) block implements a lightweight and reproducible prompting mechanism. It uses a fixed set of task-relevant keywords (e.g., instrument type, duration, location) derived from the dataset's question types, which are encoded and used to guide the final fusion of audio-visual features.", "weaknesses": "At the top: I have reviewed an earlier version of this paper. The current submission remains largely unchanged, except for minor language polishing. All major methodological and empirical concerns I previously raised remain unaddressed.\n- First of all, the motivation: The paper makes broad claims that prior work treats audio as secondary and uses text late, really? How the author provide some results/examples to demonstrate it? Reader could NOT buy the motivation just by plain sentences. To summarize, the motivation does not substantiate these claims with targeted analysis or experiments on specific baselines.\n- The author say AST is superior to VGGish for capturing timbral information, why? this claim is untested.\n- I think the dataset scope is too narrow: Strong focus on MUSIC-AVQA; AVQA appears only as a small table; no Pano-AVQA. Limits generality claims beyond music scenes.\n- My another concern drop into 'Prompting fairness': The fixed keyword prompts mirror the dataset’s question taxonomy. In the wild, question types are not given and may differ (long-tail intents, non-music scenes). The models' performance might hinges on dataset priors rather than generalizable reasoning.\n- Also, given the recent progress of MLLMs (e.g., GPT-4o, Phi series) in vision-language tasks, I suggest that the authors either: (1) include a comparison between QSTar and one or more MLLM baselines, or (2) clearly explain why such comparison is not applicable and discuss the corresponding limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xmASIybhJr", "reviewer": "Reviewer_ttss", "summary": "To address the audio-visual question answering task, this paper proposes a Query-guided Spatial-Temporal-Frequency (QSTar) interaction method. QStar primarily consists of the query-guided multimodal correlation module, spatial-temporal-frequency interaction module, and query context reasoning block. Experiments conducted on the MUSIC-AVQA and AVQA datasets verify the effectiveness of the proposed method.", "questions": "- In the Introduction, the paper highlights the advantages of frequency in music scenarios, which may be suitable for the MUSIC-AVQA dataset. But the AVQA dataset contains more diverse scenarios. Would the frequency be less effective in AVQA, or would the writing in the introduction make it inconsistent for different datasets (scenarios)? \n- The paper highlights the frequency by employing the AST backbone to extract features. Similar to VGGish, AST is also a general model for audio feature extraction. Why not directly use the identical AST in the temporal audio extraction? \n- The query-context reasoning block utilizes the 'context-related keywords' (Line 307). How these context-related keywords are obtained?\n\n- Figure 2 can be improved. For example, the abbreviation of several modules should be added at the end of the corresponding full module names.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The discussion on the audio frequency and query context is good.\n- The proposed method achieves new state-of-the-art performance on the relevant datasets.\n- In general, the proposed method is described clearly.", "weaknesses": "- Throughout the method section, the proposed network heavily relies on prevalent self-attention and cross-attention mechanisms. The query-guided multimodal correlation module is similar to the prior method, APL. Prior work, like TSPM, etc, has already explored the utilization of question modality for early fusion. Overall, the proposed method still falls into the convention of multimodal fusion, not providing sufficient advances or improvements.\n\n- Although the introduction of frequency makes sounds, it incorporates an additional AST backbone for feature extraction, which may make the proposed framework more complex. An analysis of the efficiency is required to justify this.\n\n- The paper lacks a discussion or comparison of MLLMs. For the studied audio-visual question answering problem, the current omni multimodal large language models, such as Qwen2.5-Omni, Ming-Omni, and video-SALMONN2, can be used. What performance can be achieved by such omni models?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)