Target-Aware Spatio-Temporal Reasoning via Answering Questions in Dynamic Audio-Visual Scenarios
Authors: Yuanyuan Jiang
Jianqin Yin
Venue: EMNLP 2023 Findings (2023)
Keywords: audio-visual question answering
spatio-temporal reasoning
multimodal learning
video question answering
TL;DR: This paper presents a novel single-stream target-aware joint spatio-temporal grounding network for AVQA, effectively integrating audio-visual fusion and question-aware temporal grounding, while focusing on the specific target of the inquiry.

Abstract: Audio-visual question answering (AVQA) is a challenging task that requires multistep spatio-temporal reasoning over multimodal contexts. Recent works rely on elaborate target-agnostic parsing of audio-visual scenes for spatial grounding while mistreating audio and video as separate entities for temporal grounding. This paper proposes a new target-aware joint spatio-temporal grounding network for AVQA. It consists of two key components: the target-aware spatial grounding module (TSG) and the single-stream joint audio-visual temporal grounding module (JTG). The TSG can focus on audio-visual cues relevant to the query subject by utilizing explicit semantics from the question. Unlike previous two-stream temporal grounding modules that required an additional audio-visual fusion module, JTG incorporates audio-visual fusion and question-aware temporal grounding into one module with a simpler single-stream architecture. The temporal synchronization between audio and video in the JTG is facilitated by our proposed cross-modal synchrony loss (CSL). Extensive experiments verified the effectiveness of our proposed method over existing state-of-the-art methods.

Target-Aware Spatio-Temporal Reasoning via Answering Questions in Dynamic Audio-Visua Scenarios
Yuanyuan Jiang and Jianqin Yin"
School of Artificial Intelligence Beijing University of Posts and Telecommunications
(iw.jayiin)ebupt. edu. cn
Abstract
Qeellww many dnstrmeenn the videoidid not suundfrem beginning to end?
Audio-visual question answering (AVQA) is challenging task that requires multistep spatio- temporal reasoning over multimodal contexts. Recent works rely on elaborate target-agnostic parsing of audio-visual scenes for spatial grounding while mistreating audio and video separate entities for temporal grounding. This paper proposes new target-aware joint spatio-temporal grounding network for AVQA, It consists of two key components: the target- aware spatial grounding module (TSG) and the single-stream joint audio-visual temporal grounding module (JTG). The TSG can focus on audio-visual cues relevant to the query sub- ject by utilizing explicit semantics from question. Unlike previous two-stream temporal grounding modules that required an additional audio-visual fusion module, JTG incorporates audio-visual fusion and question-aware tem- poral grounding into one module with a sim- pler single-stream architecture. The temporal synchronization between audio and video in the JTG facilitated by our proposed cross- modal synchrony loss (CSL.). Extensive experi- ments verified the effectiveness of our proposed method over existing state-of-the-art methods.'
++ p
AVQQ
Thrre
Figure An illustration of the Audio-Visual Question Answering task. Concretely, the question is centered around "instruments" (i.e., target) and is broken down into "how many", "did not sound", and "from beginning to end" in terms visual space, audio, and temporality, respectively. Identifying the three instru- ments that did not produce sound throughout the video may entail a significant time investment for a human viewer. Nonetheless, for AI system with effective spatio-temporal reasoning capabilities, the task can be accomplished much more efficiently.
ing audio and visual information to obtain the corre- lation of the two modalities, especially when there are multiple sounding sources, such as ambient noise, or similar categories in either audio or visual feature space, such as guitars and ukuleles. Sec- ondly, it requires capturing the question-relevar audiovisual features while maintaining their tem- poral synchronization
1 Introduction
Audio-visual question answering (AVQA) has re- ceived considerable attention due to its potential applications many real-world scenarios. pro- vides avenues integrate multimodal information to achieve scene understanding ability as humans.
multimedia video.
Although there have been number of promis- ing works (Zhou al., 2021; Tian al., 2020; Lin and Wang. 2020) in the audio-visual scene understanding community that attempted to solve the first challenge, they are primarily a targetless parsing of the entire audio-visual scenes. Most of them (Xuan et al., 2020; Wu et al., 2019; Mercea et al., 2022) obtain untargeted sound-related visual regions by designing attention schemes perform- ing on audio-to-visual while ignoring the question- oriented information from the text modality. How- ever, the understanding of audio-visual scenes AVQA tasks is often target-oriented. For exam-
As shown in Figure 1, the AVQA model aims to answer questions regarding visual objects, sound patterns, and their spatio-temporal associations. Compared to traditional video question answering, the AVQA task presents specific challenges in the following areas. Firstly, involves effectively fus-
"Corresponding author: Jianqin Yin
'The code and data are available at https: http:/ggtttub.ccc comm Bravo5542/TJSTG.

---

Asaliscuaal
modality in the question and introduce it during audio-visual alignment. In this way, there will be a noticeable distinction between concepts such as the ukulele and the guitar. Accordingly, we propose an attention- based target-aware (TA) module to rec- ognize the query subject in the question sentence first and then focus on the interesting sounding area through spatial grounding.
Anadio Strvar
Teakare
Osuetin Usery
Fusiea
Image Servar
Andliccional
Audio Strcan
fosturr
Queitioo Ueery
Fasice
Irage Straam
Secondly, we propose a cross-modal synchrony loss (CSL) and corresponding joint audio-visual temporal grounding (JTG) module. In contrast to the existing prevalent two-stream frameworks that treat audio and video as separate entities (Figure 2.a), the CSL enforces the question to have syn- chronized attention weights on visual and audio modalities during question-aware temporal ground- ing (Figure 2.b) via the JS divergence. Furthermore, presents avenues to incorporate question-aware temporal grounding and andio-visual fusion into a more straightforward annle-stream architecture (Figure 2.c), instead of the conventional approach of performing temporal grounding first and fusion later. In this way, the network is forced to jointly capture and fuse audio and visual features that are supposed to be united and temporally synchronized. This simpler architecture facilitates comparable or even better performance.
Jeer attectio cnai Lilee. Deecanpted fesiee
Andiooniinal
feutars
Queviion Query
Mtatiandaa Stream
Figure 2: Comparison of different question-awand -ware tem- poral grounding. (a.) The traditional approach usually adopts a dual-stream network that treats audio and video as separate entities. (b.) Our proposed cross-modal syn- chrony loss ensures the interaction between audio and visual modalities.
(c.) Our proposed single-stream ar- chitecture is able to treat audio and video as a whole, thus incorporating temporal grounding and fusion.
ple, as illustrated in Figure 1, our focus lies solely on the subject of inquiry, i.e., instruments, disre- garding the singing person or ambient sound. Tra- ditional AVQA approaches (Li et al., 2022: Yun al., 2021; Yang et al., 2022), inherited from the audio-visual scene understanding community, rely on aligning all audio-visual elements in the video to answer a question. This results much irrelevant information and difficulties in identifying the rele- vant objects in complex scenes. As for the second challenge, most existing methods (Yun et al., 2021; Yang al., 2022: Lin et al., 2023) employ typical attention- based two-stream framework. As shown in Figure 2.a. such a two-stream architecture pro- cesses audio and video in each stream separately while overlooking the unity of audio and visual modalitics. In particular, the temporal grounding and audio-visual fusion are isolated, with fusion occurring through an additional module.
The main contributions of this paper are summa- rized as follows:
We propose novel single-stream framework, the joint audio-visual temporal grounding (JTG) module, which treats audio and video as unified entity and seamlessly integrates fusion and tempo- ral grounding within single module.
We propose a novel target-aweer aware spatial grounding (TSG) module to introduce the explicit semantics of the question during audio-visual spa- tial grounding for capturing the visual features of interesting sounding areas. An attention-based target-aware (TA) module is proposed to recognize the target of interest from the question.
We propose a cross-modal synchrony loss (CSL) to facilitate the temporal synchronization
To effectively address these two challenges. we propose target-aware joint spatio-temporal grounding (TJSTG) network for AVQA. Our pro- posed approach has two key components.
between audio and video during question-aware temporal grounding.
2 Related Works
Firstly, we introduce the target-aware spatial grounding (TSG) module, which enables the model to focus on audio-visual cues relevant to the query subject, i.c., target, instead of all audio-visual el- ements. We exploit the explicit semantics of text
2.1 Audio-Visual-Lann Learning
By integrating information from multiple modal- itics, it is expected to explore a sufficient under- standing of the scene and reciprocally nurture

---

the development of specific tasks within single modality. AVLNet (Rouditchenko al., 2020) and MCN (Chen et al., a)) utilize audio to enhance text-to-video retrieval. AVCA (Mercea et al., 2022) proposes to learn multi-modal representations from audio-visual data and exploit textual label embed- dings for transferring knowledge from seen classes of videos to unseen classes. Compared to previ- works in audio-visual learning, such as sound- ing object localization (Afouras et al., 2020; Hu al., 2020, 2022), and audio-visual event local- ization (Liu et al., 2022: Lin et al., 2023), these works (Mercea et al., 2022; Zhu et al., 2020; Tan et al., 2023) have made great progress in integrat- ing the naturally aligned visual and auditory prop- erties of objects and enriching scenes with explicit semantic information by further introducing tex- tual modalities. Besides, there are many works (Akbari et al., 2021: Zellers et al., 2022; Gabeur et al., 2020) propose to learn multimodal represen- tations from audio, visual and text modalities that can be directly exploited for multiple downstream tasks. Unlike previous works focused on learning single or multi-modal representations, this work delves into the fundamental yet challenging task of spatio-temporal reasoning in scene understanding. Building upon MUSIC-AVQA (Li al., 2022), our approach leverages textual explicit semantics to in- tegrate audio-visual cues to enhance the study of dynamic and long-term audio-visual scenes.
posed a hierarchical audio-visual fusing module explore the impact of different fusion orders be- tween the three modalities on performance. LAV- ISH (Lin et al., 2023) introduced a novel parameter- efficient framework to encode audio-visual scenes, which fuses the audio and visual modalities in the shallow layers of the feature extraction stage and thus achieves SOTA. Although LAVISH proposes robust audio-visual backbone network, it still ne- cessitates the spatio-temporal grounding network proposed in (Li et al., 2022), as MUSCI-AVQA contains dynamic and long-duration scenarios re- quiring a significant capability of meatio-temporal reasoning. Unlike previous works, we propose a TSG module to leverage the explicit semantic of inquiry target and JTG to leverage the temporal syn- chronization between audio and video in novel single-stream framework, thus improving the mul- timodal learning of audio-visualllange languugee
3 Methodology
To solve the AVQA problem, propose target- aware joint spatio-temporal grounding network and ensure the integration between the audio-visual modalities by observing the natural integrity of the audio-visual cues. The aim is to achieve better audio-visual scene understanding and intentional apatio-tempone reasoning. An overview of the pro- posed framework is illustrated in Figure 3.
3.1 Audio-visual-language Input Embeddings
2.2 Audio Visual Question Answering
Given input video sequence containing both visual and audio tracks, it is first divided into T non-overlapping visual and audio segment pairs (Vi, A.) The question sentence Q consists of a maximum length of N words. To demonstrate the effectiveness of our proposed method, we followed the MUSIC-AVQA (Li et al., 2022) approach and used the same feature extraction backbone network. Audio Embedding. Each audio segment A, is encoded into fa E Rdd by the pretrained on Au- dioSet (Gemmeke et al., 2017a) VGGish (Gem- meke et al.. 2017b) model, which is VGG-like 2D CNN network, employing over transformed audio spectrograms.
The demand for multimodal cognitive abilitics in AI has grown alongside the advancements in deep learning techniques. swering (AVQA), unlike previous question answer- ing (Lei et al.. 2018; You et al., 2021: Chen et al., 202 lb; Wang et al., 2021), which exploits the nat- ural multimodal medium of video, is attracting in- creasing attention from researchers (Zhuang et al., 2020; Miyanishi and Kawanabe, 2021; Schwartz et al., 2019; Zhu et al., 2020). Pano-AVQA (Yun et al., 2021) introduces audio-visual question an- swering in panoramic video and the correspond- ing ceasffomer-bbased encoder-decoded approach. MUSIC-AVQA (Li et al., 2022) offers a strong baseline by decomposing AVQA into audio-visual fusion through spatial correlation of audio-visual elements and question-aware temporal grounding through text-audio cross cross-attention and tixt-visual cross-attention. AVQA (Yang et al., 2022) pro-
Andio-Visual Question An-
Visual Embedding. A fixed number of frames are sampled from all video segments. Each sam- pled frame is encoded into visual feature map ft.m Ax by the pretrained on ImageNet (Russakovsky et al., 2015) ResNet18 (He et al., 2016) for each segment V,, where and w are the

---

Avg()
Sounding
Input video
Visual fam
Encoder
Interesting sounding
fez
fu
fa
Input audio
Audio Encoder
Interesting
Target-aware Spatial Grounding
MHAC)
idxx
Target-aware module
Avg(-)
fegz
Classifier
fatt
fau
fgg
MHA()
How many instruments the
Three
video did not sound
fe
from beginning end?
Text
Cross-modal Synchrony Loss
Encoder
Ioput text
Joint Audio-visual Temporal Grounding.
Answer Prediction
score
Visual attention weights time
Audio attention weights over time
Dot product
wise product
Figure 3: The proposed target-aware joint spatio-temporal grounding network. We introduce text modality with explicit semantics into the audio-visual spatial grounding to associate specific sound-related visual features with the subject of interest, i.c., the target. We exploit the proposed cross-modal synchrony loss incorporate audio- visual fusion and question-aware temporal grounding within single-stream architecture. Finally, simple fusion is employed to integrate audiovisual and question information for predicting the answer.
height and width of the feature maps, respectively. Question Embedding. The question sentence Q is tokenized into N individual words {gn) 1 n=1 by the wrod2vec (Mikolov et al., 2013). Next, a learn- able LSTM is used to process word embeddings obtaining the word-level output fg E IR- Nd, and the last state vector [hN; cN] R2xd,, [hvv eN] is then transformed using MLP yield hg E IR Ixxd, the encoded sentence-level question feature. Noted the used pretrained models are all frozen.
target according to the question-comtributte scores. To compute question- -enestion-oontrbbuteed scores, use sentence-level question feature hy, query vec- tor and word-level question feature f, as key and value vector to perform multi-head self-attention (MHA), computed as:
8 = a haf
(1)
where f, ......... f and a scaling fac- tor with the same size as the feature dimension. E 1 represents the weight of each word's contribution to the final question feature. Next, we index the feature of the target, which will be enhanced in the subsequent spatial grounding, as:
3.2 Target-aware Spatial Grounding Module
While sound source localization in visual scenes reflects the spatial association between audio and visual modality, it is cumbersome to elaborately align all audio-visual elements during question an- swering due to the high complexity of audio-visual scenes. Therefore, the target-aware spatial ground- ing module (TSG) is proposed to encourage the model to focus on the truly interested query object by introducing text modality from the question.
idæ
arg max {s(n)} 1=1,2....N fegt pide
(2)
where figt has the highest contribution weight the question feature.
Target-aware (TA) module. For the word-level question feature f, C R. N.d,,, we aim to locate the target subject, represented fegt IRd, which owns the explicit semantic associated with the audio-visual scenes. Specifically, we index the
Interesting spatial grounding module. One way of mapping man-made concepts to the natural environment is to incorporate explicit semantics into the understanding of audio-visual scenarios.

---

For each video segment, the visual feature map fr the audio feature fa and the interesting target feature Sege compose a matched triplet. Firstly, We reshape the dimension of the fr.m from h x w x de to hw x d, For each triplet, we can compute the interesting sound-related visual features fo, as:
implicitly modeling the synchronization between the audio and video. Concretely, given a hy and audio-visual features {fa fe) we first compute the weight of association between the given ques- tion and the input sequence, based on how closely each timestamp is related to the question, as:
f v.m OSSa (g)
(4)
fufa Vd
Ag a
(8)
8/ ((f)) fmmm
(5)
V, o Kig Vd
8q a(ffigt)
fumm (6)
(9)
Bq s9l(s9-11
(7)
where f
(f.:... fo and fo
where Sa, Sq E IR 1xkw fi.s E Rlxd, is the softmax function, and (+) represents the transpose operator. In particular, we adopt a simple threshold- ing operation to better integrate the text modality. Specifically, T the hyper-parameter selecting the visual areas that are highly relevant to the query subject. I(-) is an indicator function, which out- puts when the input is greater than or equal to O, and otherwise outputs O. By computing text-visual attention maps, it encourages the previous TA mod- ule to capture the visual-related entity among the question. Next, we perform the Hadamard product on the audio-visual attention map and text-visual attention map to obtain the target-aware visual at- tention map. In this way, the TSG module will focus on the interesting sounding area instead of all sounding areas. To prevent possible visual infor- mation loss, we averagely pool the visual feature map fi obtaining the global visual feature ft. The two visual feature is fused as the visual repre- sentation: f+ = FC(tanh fo Fu. where FC represents fully-connected layers, and f RIxd,
[ .. [:]: hg E Riidd So E RTxd, fo e R''x,, d is scaling factor with the same size the feature dimension. In this way, obtain the question-awee aware weights Ag, Vg IR I of audio and video sequence, respectively.
Next, we employ the Jensen- ShasennShannon (JS) diver- gence as a constraint. Specifically, the JS diver- gence measures the similarity between the proba- bility distributions of two sets of temporal vectors, corresponding to the audio and visual question- aware weights, respectively. By minimizing the JS divergence, we aim to encourage the temporal distributions of the two modalities to be as close as possible, thus promoting their question-contributed consistency in the JTG process. The CSL can be formulated as:
Lest = 2 DKL(A, + 1 DVL(VV
(10)
M = 7 (A, + Vg)
(11)
2
P(t)
DKI(PI Q) Σ P(t) log
(12)
Q(t)
3.3 Joint Audio-visual Temporal Grounding
Note that JS divergence is symmetric, i.e.. ([((P]QQ=-S(Q)) Joint audio-visual temporal grounding (JTG) module. Previous approaches to joint audio-visual learning have typically used a dual-strean struc- ture with a decoupled cross-modal fusion module. However, the proposed CSL makes single-stream networks for audio-visual learning possible and can naturally integrate audio-visual fusion and tempo- ral grounding into one module. Specifically, we first interleave the LSTM encoded video feature tensor and audio feature tensor along rows, i.e., the temporal dimension, as:
In the natural environment, visual and audio infor- mation different attributes of the same thing, ie.. the two are inseparable. Therefore. we propose the joint audio-visual temporal grounding (JTG) module and cross-modal synchrony (CSL) loss to treat the visual modality and audio modality as a whole instead of separate entities as before.
Cross-modal synchrony (CSL) loss. Temporal synchronization is characteristic of the audio and visual modalities that are united, but in multimedia videos, they do not strictly adhere to simple syn- chronization. We use the question feature as the intermediary to constrain the temporal distribution consistency of the audio and visual modalities, thus
f - IL (ff.ffa))) fa) = [f:: :f: (13)

---

where IL denotes that the features of two modal- ities are InterLeaved in segments. far E R22xd represents the multimedia video features, and d = de dg- Next, we perform MHA to aggregate crit- ical question-aware audio-visual features among the dynamic audio-visun scenes as:
probability vector and the corresponding ground- truth label y, we optimize it using cross-entropy loss: Lg = - 1 Me log( During testing, the predicted answer would be č = arg max(p).
Experiments
21 Sav) Σ
This section presents the setup details and experi- mental results on the MUSIC-AVQA dataset. We also diseuss the model's performance and specify the effectiveness of each sub-module in our model through ablation studies and qualitative results.
f = (HA(hg: f f
wi Jan (14)
[=1
w00 Sootmmxx (ecimmax((hgh)) (h (x)
(15)
flvv fat + +M
(16)
4.1 Experiments Setting
where fäv E Ixld represents the ques- tion grounded audiovisual contextual embedding, which is more capable of predicting correct an- swers. The model will assign higher weights to segments that are more relevant the asked ques- tion. Then, we can retrieve the temporal distribu- tion weights specific to each modality from the output of multi-head attention MHA and apply our proposed CSL as follows:
Dataset. We conduct experiments on the MUSIC- AVQA dataset (Li et al., 2022), which contains 45,867 posstoon-answer pairs distributed in 9,288 videos for over 150 hours. It was divided into sets with 32,087/4,595/9, ing//alidatiiontessingg The MUSIC-AVQA dataset is well-suited for studying spatio-tempore reason- ing for dynamic and long-term audio-visual scenes. Metrie. Answer prediction accuracy. We also evaluate the model's performance in answering dif- ferent questions.
QA pairs for train-
Lest JS( (S w.))
(17)
27 W. { 21 1:11,-.7
(18)
Implementation details. The sampling rates of sounds and frames 16 kdlz and fps.. We divide the video into non-overlapping segments of the same length with Is-long. For each video segment, we use frame to generate the visual features of size 14x14x512. For each audio segment, we use a linear layer to process the extracted 128-D VGGish feature into 512-D feature vector. The dimension of the word embedding is 512. In experiments, we used the same settings in (Li et al., 2022) and sampled the videos by taking 1s every 6s. Batch size and number of epochs are 64 and 30, respec- tively. The initial Icarning rate 2c-4 and will drop by multiplying 0.1 every 10 epochs. Our network is trained with the Adam optimizer. We use the torchsummary library in PyTTrrch to calculate the model's parameters. Our model is trained on NVIDIA GeForce GTX 1080 and implemented in PyTorch
Wa 21- i=,,".
2T
(19)
where Way Wy C RIXT are question-aware tem-
poral distribution weights of audio and video, re- spectively. By leveraging the CSL, the proposed JTG module can effectively perform both temporal grounding and audio-visual fusion while consid- ering the synchronization between the audio and visual modalities. The resulting single-stream ar- chitecture simplifies the overall system and treats audio and video as whole.
3.4 Answer Prediction
In order verify the audio-visual fusion of our proposed joint audio-visual temporal grounding module, we employ simple element-wise mul- tiplicaiion operation to integrate the question fea- tures hy and the previously obtained audiovisual features f Concretely, it can be formulated as:
Training Strategy. As previous methods (Li et al., 2022) use a two-stage training strategy, train- ing the spatial grounding module first by designing coarse -rarseegriinddd audio-visual pair matching task, formulated as:
e = f4 O hy
(20)
Next, we aim to choose one correct answer from pre-defined answer vocabulary. We utilize linear layer and softmax function to output probabilities p E C for candidate answers. With the predicted
L. Loe maaly
match g')
(21)
O' = a (ML
(22)

---

Audio Question
Visual Question Location
Audio Disuo-Visual Questio
All
Methed
Couning
Canguraivv
Avg.
Countng
Esistemial
Couning
Location
Conpunsive
Tempunil
AV Sehrcann 2019)
72.41
18.78
66.00
74.33
70.31
80.77
57.93
61.07
Pano-AV Cann-AVQQAAannnccomm 2021)
75.71
66. 99
72.15
70.51
75.76
73.16
82.09
65.38
61.30
63.67
62.04
66.97
69.53
AVSTUL 2002)
73.87
74.40
69.88
64:34
64.67
68.82
TISTCKOur))
80.38
69.87
76.19
77.55
76.88
82.59
71.54
64.24
66.21
Table I: Comparisons with state-of-the-ar methods on the MUSIC-AVQA dataset. The top-2 results are highlighted.
Audio Question Comparaiive
Visual Question Location
Audio Visual Question
Method
Coanting
Counting
Avg.
Existential
Counting
Location
Comparative
Temporal
Avg.
Avg.
wlo T-A
79.35
68.01
75.17
75.94
76.16
76.05
82.79
71.70
64.24
65.12
64.11
69.85
72.44
Cust
29.94
68.52
25.73
75.19
22.06
26.14
82.09
71.07
63.80
64.94
63.35 63.50
69.35
72.28
wlo foo
79:06
68.52
75.17
74.35
75.31
74.94
83.10
70:36
63.80
63.03
65.09
69.21
71:78
TSSTG((urr)
80.38
76.47
76.19
77.55
76.88
82.59
64.24
64.84
70.13
73.04
Table Ablation studies of different modules on MUSIC-AVQA dataset. The top-2 results are highlighted.
Method
A Avg.
V Avg.
AV Avg.
All
questions require a high conceptual understand- ing and reasoning ability. The considerable im- provement achieved by TISTG can be attributed to our proposed TSG module, which introduces textual modalities with explicit semantics into the audio-visual spatial grounding process. Although we were slightly behind AVST the audio-visue temporal question, we still achieved the highest ac- curacy of 70.13% on the total audio-visual question with a simpler single-stream architecture, outper- forming AVST by 0.6%t. Benefiting from our proposed JTG leveraging the natural audio-visue integration, our full model has achieved the highest accuracy of 73.04% with more straightforward architecture, which outperforms AVST by 1.45% f.
AVST(Li al. 2022)
73.87
74.40
69.53
71.59
AVST T-A
75.17
76.34
69.70
72.43
AVST w/ Cest
76.10
77.04
69.15
72.47
Table 3: Ablation studies of different modules against baseline model. The top-2 results are highlighted.
Method
Avg.
Avg.
Avg.
TSG w/ Avg
75,42
76.34
70.02
72.65
TSG w/ Max
76.02
76.75
69.92
72.70
TSG w/ hidden
75.85
76.34
70.04
72.74
TSG w/ TA (ours)
76.47
76.88
70.13
73.04
Table Effect of the Target-aware module on the accu- racy(%). The top results are highlighted.
We utilize the pretrained stage module in (Li et al., 2022) directly without retraining for certain layers that overlap with our approach. We our pro- posed L Lge + Lest + XL, to train for AVQA task, where À is 0.5 following previous setting (Li et al., 2022).
4.3 Ablation studies
The effectiveness of the different modules in our model. To verify the effectiveness of the pro- posed components, we remove them from the pri- mary model and re-evaluate the new model on the MUSIC-AVQA dataset. Table 2 shows that after removing single component, the overall model's ss performance decreases, and different modules have different performance effects. Firstly, when we re- move the TA module and the target-aware process during spatial grounding (denoted as "w/o T-A") and use traditional audio-visual spatial grounding, the accuracy decreases by 1.24%. 0.83% and 0.27% under audio, visual and audio-visua questions, re- spectively. This shows that it is essential to have a targeting process before feature aggregation instead of attending all the audio-visual cues. Secondly, remove the proposed CSL (denoted as "w/o Lest") and the overall accuracy drops to 72.28% (0.76% below our full model). Lastly, we remove two modules and employ a vanilla single-stream
4.2 Comparisons with SOTA Methods
We challenge our method against current SOTA methods on AVQA lask. For a fair compari- son, we choose the same audio and visual fea- tures as the current methods. As shown in Ta- ble 1, We compare our TISTG approach with the AVSD (Schwartz et al., 2019), PanoAVQA (Yun et al., 2021), and AVST (Li et al., 2022) meth- ods. Our method achieves significant improve- ment on all audio and visual questions compared to the second-best method AVST (average of 2.60% t and 2.48% respectively). In particular, our method shows clear superiority when answering counting (average 2.31% and comparative (av- erage of 2. 2.12%)) questions. These two types of

---

Method
Avg.
Avg.
AV Avg.
All
Method
Trainable Parum. (M))
Accuracy (%)
7 0.000
75.98
76.14
69.19
72.23
Dual-stream Net
14.6
72.89
0.002
75.61
76.80
69.74
72.65
Single-strent streamm Net
11.1
73.04
0.005
76.47
76.88
70.13
73.04
0.010
75.92
76.63
69.84
72.71
Table 7: Comparison of dtre-stream structure and singe-
stream structure.
Table 5: Impact of various values of on the system accuracy. The top-2 results are highlighted.
as "TSG w/ TA") achieves the highest accuracy of 73.04%. The experimental results not only prove the superiority of our proposed target-aware mod-
Method
Avg.
V Avg.
AV Avg.
All
Va w/ Lest
75.92
76.67
69,49
72.53
Cat(V.) w/ Cesl
75.85
75.93
70.23
72.74
IL(AN)) Lest
76.78
76.88
70.04
73.04
ule but also further demonstrate the effectiveness of our introduction of textual modalities carrying ex- plicit semantics in the audio- learning stage. In addition, we explore the impact of hyperpa- rameter T on model performance. As shown in Table 5, while plays a role in selecting relevant visual areas, our experiments revealed that it does not significantly impact performance within the context of the MUSIC-AVQA dataset (Li et al., 2022). The highest accuracy of 73.04% is achieved when T = 0.005. However, the removal of the thresholding operation (r = 0.000) causes a de- crease of 0.81% in accuracy. This may be caused by information redundancy, and we believe that this phenomenon can be improved by utilizing pre- trained model with priori information image- text pairs in future work.
IL(VA) w/ Los (ours)
76.47
76.88
70.13
73.04
Table 6: Effect of the Cross-modal synchrony loss on the accuracy(?6). "IL" denotes that audio and visual features are interleaved in segments. "Cat" denotes that audio and visual features are concatenated. The top 2 results are highlighted.
structured network (denoted as "w/o Las the overall accuracy severely drops by 1.36%, from 73.04% to 71.78%. These results show that every component in our system plays an essential role in the AVQA.
Effect of introducing text during audio-visual learning. As Table 2 shows, removing the TA mod- ule and target-aware process resulted in a lower accuracy 77) for audio questions consisting of counting and comparative questions compared to the "w/o Lest" (75.73%) and our full model (76.47%). In Table 3, we utilize AVST (Li et al., 2022) as baseline model to further validate the robustness and effectiveness of our proposed targel- aware approach. We implement AVST with our proposed TA module and the corresponding target- aware process denoted as "AVST w/ T-A", which surpasses AVST by 0.84% in overall accuracy (from 71.59% to 72.43%). These results demon- strate that the explicit semantics in the audiovisual spatial grounding can facilitate audio-visual ques- tion answering.
Effect of singe-stream structure. We validate the effectiveness of our designed specialized audio- visual interleaved pattern, i.e., IL(A;V), which maintains both the integrity of the audio-visual con- tent at the segment level and the relative indepen- dence between the audio and visual content at the vidco level. As shown in Table 6, we explore dif- ferent ways of arranging visual and audio features, and our interlcaved-by- by-segments pattern is 0,41 % higher on average than the concatenated-by-mon pattern. we also conduct comprehensive compar- ison between single-stream and dual-stream net- works. During the temporal grounding, we switch to the prevalent two-stream network structure like in (Li et al., 2022), but still with our proposed TSG module and cross-synchrony loss, which is denoted as "Dual-stream Net" in Table 7. As shown in Table 7, the "Single-stream Net" that omits the additional fusion module yields 0.15% higher accuracy with 3.5M fewer parameters than the "Dual-stream Net". This indicates the superiority of single-stream net- works over two-stream networks, which utilize the integration of the audio and visual modalities to si- multancously accomplish question- question-aware aware temporal grounding and audio-visual fusion.
Effect of Target-aware module. As shown in Table 4, we adopt different ways to introduce ques- tion information into the spatial grounding module, thus verifying the effectiveness of our proposed Target-aware module during the target-aware pro- cess. Specifically, we conduct average-poolin pooling (de- noted "TSG w/ Avg") and max-pooling (denoted "TSG w/ Max") on the LSTM-encoded question embedding fg to represent the target feature, respec- tively. We also adopt the question feature vector hg as the target feature during spatial grounding. Compared to these methods, our approach (denoted

---

accurate spatial grounding compared AVST (Li et al., 2022). When there no target of interest in the video, as shown in Figure 4.(c), i.e., the ukulele, can be seen that our method presents an irregular distribution of spatial grounding in the background region instead of the undistinguished sounding area of the guitar and bass presented by AVST (Li et al.. 2022). Furthermore, the JTG module aggregates the information of all timestamps based the ques- tion. These results demonstrate that our proposed method can focus on the most question-relevan au- dio and visual elements, leading to more accurate question answers.
STG
TASTG
L0 stbee la rider ridersteog plarina A Yox.
STG
YASPG
TISTG
Figure 4: Visualized target-aware -empoal grounding results. Based on the grounding results of our method, the sounding area of interest are accordingly highlighted in spatial perspectives in different cases (a- c), respectively, which indicates that our method can fo- cus the query subject, facilitating the target-oriented scene understanding and reasoning.
5 Conclusions
This paper proposes a target-aware spatial ground- ing and joint audio-visual temporal grounding to better solve the target-oriented audio-visual scene understanding within the AVQA task. The target- aware spatial grounding module exploits the ex- plicit semantics of the question, enabling the model to focus on the query subjects when parsing the audio visual scenes. Also, the joint audio-visual temporal grounding module treats audio and video as a whole through single-stream structure and encourages the temporal association between au- dio and video with the proposed cross-modal syn- chrony loss. Extensive experiments have verified the superiority and robustness of the proposed mod- ule. Our work offers an inspiring new direction for audio-visual scene understanding and spatio- temporal reasoning in question answering.
Effeet of cross-modal synchrony loss. Simi- larly, as shown in Table 3, we verified the validity of our proposed eest on AVST (denoted as "AVST w/ Lest"). "AVST w/ Lest" achieved an accuracy of 72.47%, exceeding the baseline model by 0.88%. We further consider the impact of the combination order of multimedia video features f on perfor- mances, as shown in Table Specifically, we compose f av by interleaving the audio and visual features but putting the audio modality front (de- noted as "II(A:V) w/ Lest"). Compared to our full model (denoted as "IL(V:A) w/ ")st"), the overall accuracy is the same (both are 73.04%). The con- catenate operation also has similar results. That is, the order in which the audio-visual features combined does not have significant impact on the performance of our entire system. These re- sults validate the robustness and effectiveness of our proposed CSL.
Limitations
The inadequate modeling of audio-visual dynamic scenes potentially impacts the performance of ques- tion answering. Specifically, although our pro- posed TSG module enables the model to focus on question-related scene information, sometimes information not directly related the question can also contribute to answering the question. Experi- mental results demonstrate that adding the targer- aware spatial grounding module to the basic model resulted in marginal improvement in the accuracy of answering audio-visual questions compared to incorporating the cross-modal synchrony loss into the basic model. We believe this limits the overall performance of our approach, showing an incre- mental improvement for the audio-visua, question type (0.6% on average) compared to a significant improvement for the uni-modal question type (2.5%
4.4 Qualitative analysis
Figure 4, we provide several visualized larget- aware spatial grounding results. The heatmap in- dicates the location of the interesting-sounddii source. Through the results, the sounding targets are visually captured, which can facilitate spatial reasoning. For example, in the case of Figure 4.(a), compared to AVST (Li et al., 2022), our proposed TSTJG method can focus on the target, i.c., the flute, during spatial grounding. The TSG module offers information about the interesting-soundin object in each timestamp. In the case of Figure (b(b) with multiple sound sources related to the target, Le., instruments, our method also indicates a more

---

on average). In the future, we will explore better ways integrate natural language into audio-visual scene parsing and mine scene information that not only explicitly but also implicitly related to the
Di Hu, Rui Qian, Minyue Jiang, Xiao Tan, Shilei Wen, Errii Ding, Weiyao Lin, and Dejing Dou. 2020. Discriminative sounding objects localization via self-supe audiovisual matching. NeurIPS, 13:10077-10087.
question.
Xixi Hu, Ziyang Chen, and Andrew Owens. 2022. Mix and localize: Localizing sound sources in mixtures. In CVPR, pages 10483-10492.
Acknowledgements
This work was supported partly by the National Natural Science Foundation of China (Grant No. 62173045, 62273054), partly by the Fundamen- tal Research Funds for the Central Universities (Grant No. 2020XD-A04-3) and the Natural Sci- ence Foundation of Hainan Province (Grant No. 622RC675).
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg. 2018. Tvqa: Localized, compositional video ques- tion answering. In EMNLP. pages 1369-1379.
Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. 2022. Learning to an- swer questions dynamic audio-visual scenarios. CVPR, pages 19108-19118,
Yan-Bo Lin and Yu-Chiang Frank Wang. 2020. Audio- visual transformer with instance attention for audio- visual event localization. In ACCV.
References
Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman. 2020. Self-supervised leam- ing of audio-visual objects from video. In ECCV, pages 208-224.
Yi-Lin Lin, Yan-Bo Sung, Jie Lei, Mohit Bansal, and Gedas Bertasius. 2023. Vision transformers are parameter-efficient audio-visual learners. In CVPR.
Hassan Akbari. Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang. Yin Cui, and Boqing Gong- 2021. Vatt: Transformers for multimodal self- supervised learning from raw video, audio and text. NeurlPS, 34:24206-244 24221.
Shuo Liu, Weize Quan, Chaoqun Wang, Yuan Liu, Bin Liu, and Dong-Ming Yan. 2022. Dense modality in- teraction network for audio-visual event localization. TMM, pages 1-1.
Otniel-Boge Bogidn Mercea, Lukas Riesch, A Koepke, and Zeynep Akata. 2022. Audio-visual generalised zero- shot learning with cross-modal attention and lan- guage. In CVPR, pages 10553-10563.
Brian Chen, Andrew Rouditchenko, Kevin Duarte, Hilde Kuehne, Samuel Thomas, Angie Boggust, Rameswar Panda, Brian Kingsbury, Rogerio Feris, David Harwath, et al. 2021a. Multimodal clustering networks for self-supervised learning from unlabeled videos. ICCV, pages 8012-8021.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jef- frey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint 1rX.v:1301.378I.
Feilong Chen, Xiuyi Chen, Can Xu, and Daxin Jiang. 202 lb. Learning to ground visual objects for visual dialog. In Findings of the Association for Computa- tional Linguistics: EMNLP 2021, pages 1081-1091.
Taiki Miyanishi and Motoaki Kawanabe. 2021. Watch, listen, and answer: Open-ended videoqa with modu- lated multi-streat stream 3d convnets. In EUSIPCO, pages 706-710.
Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. 2020. Multi-modal transformer for video retrieval. In ECCV, pages 214-229.
Andrew Rouditchenko, Angie Boggust, David Harwath, Brian Chen, Dhiraj Joshi, Samuel Thomas, Kartik Audhkhasi, Hilde Kuehne, Rameswar Panda, Rogerio Feris, al. 2020. Avlnet: Learning audio-visual language representations from instructional videos. arXiv preprint arXiv:2006.09199
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. 70. Audio set: An ontology and human- dataset for audio events. In ICASSP, pages 776-780.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. 2017b. Audio set: dataset for audio
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015. Imagenet large scale visual recognition chal- lenge. IJC,, 115:211-252.
An ontology and events. In ICASSP, pages 776-780.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning image recogni- tion. In CVPR, pages 770-778.
Idan Schwartz, Alexander G Schwing, and Tamir Hazan. 2019. A simple baseline for audio-visual scene- aware dialog. In CVPR, pages 12548-12558.

---

Reuben Tan, Arijit Ray, Andrea Burns, Bryan Plum- mer, Justin Salamon, Oriol Nieto, Bryan Russell, and Kate Saenko. 2023. Language-guided audiolvisual source separation via trimodal consistency. In CVPR.
Yapeng Tian, Dingzeyu Li, and Chenliang Xu. 2020. Unified multisensory perception: Weaely- audio-visual video parsing. In ECCV, pages 436-
454.
Junjie Wang. Yatai Ji, Jiaqi Sun, Yujuu Yang. and Tet- suya Sakai. 2021. Mirtt: Learning multimodal in- teraction representations from trilinear transformers for visual question answering. In Findings the Association for Computational Linguistics: EMNLP 2021, pages 2280-2292.
Yu Wu, Linchao Zhu, Yan Yan, and Yi Yang. 2019. Dual attention matching for audio-visual event localization. In ICCV, pages 6292-6300.
Hanyu Xuan, Zhenyu Zhang, Shuo Chen, Jian Yang, and Yan Yan. 2020. Cross-moda attention network for temporal inconsistent audio-visual event localization. In AAAI, volume 34, pages 279-286.
Pinci Yang, Xin Wang, Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu. 2022. Avqa: dataset audio-visual question answering on videos. In ACM MM, pages 3480-3491.
Chenyu You, Nuo Chen, and Yuexian Zou. 2021. Self- supervised contrastive cross-modality representation learning for spoken question answering. In Find- ings of the Association for Computational Linguis- tics: EMNLP 2021, pages 28-39.
Heeseung Yun, Youngjae Yu, Wonsuk Yang, Kangil Lee, and Gunhee Kim. 2021. Pano-avqa: Grounded audio-visual question answering on 360deg videos. In ICCV, pages 2031-2041.
Rowan Zellers. Jiasen Lu, Ximing Lu, Youngjae Yu. Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusu- pati, Jack Hessel, Ali Farhadi, and Yejin Choi. 2022. Merlot reserve: Neural script knowledge through vision and language and sound. In CVPR, pages 16375-16387.
Jinxing Zhou, Liang Zheng, Yiran Zhong, Shijie Hao, and Meng Wang. 2021. Positive sample propagation along the audio-visual event line. In CVPR, pages 8436-8444.
Ye Zhu, Yu Wu, Yi Yang, and Yan Yan. 2020. Describ- ing unseen videos via multi-modal cooperative dialog agents. In ECCV, pages 153-169.
Yueting Zhuang, Dejing Xu, Xin Yan, Wenzhuo Cheng. Zhou Zhao, Shiliang Pu, and Jun Xiao. 2020. Mul- tichannel attention refinement for video question an- swering. TOMM, 16(1s):1-23.


Reviews:
[{"review_id": "W51ObAv5lw", "reviewer": "Reviewer_gs1d", "summary": "", "questions": "- What is difference between question sentence feature f_q and the last hidden state feature vector h_q, i.e., the question feature? It was not clear to me.\n\n- For the proposed cross-modal synchrony (CSL) loss, it was not clear why using text as intermediary is better than directly using video and audio.\n\n- The authors use the total loss L = Lqa + Lcsl + \\lambda * Ls and how about adding another regularization parameter to Lcsl and tuning them? Does it impact the performance?\n\n- In Figure. 4(c), it was still not clear why the model attends on mostly on background, not instruments, as the whole image might need attentions to answer the question.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer gs1d:\n\n*Paper Topic And Main Contributions:*\n\n*This paper tackles the problem of audio-visual question answering and proposes three components: 1) target-aware spatial grounding (TSG) module by using the question to ground interesting sounding areas, 2) single-stream framework of the joint audio-visual temporal grounding (JTG), and 3) cross-modal synchrony loss (CSL) to promote the temporal synchronization between audio and video by using text. The experimental results show that the proposed approach outperforms the prior art and ablation study demonstrates the effectiveness of individual proposed components.*\n\n**Response**: Thank you for your conclusive comments! We appreciate the time and effort that you have dedicated to evaluating and improving our work. As detailed below, we have addressed each of your comments carefully.\n\n*Reasons To Accept:* \n\n1. *The proposed components seem sound and novel*\n2. *Experimental results show that the proposed approach outperforms the prior art*\n3. *Authors provide thorough analysis including ablation study for each proposed component and with prior art, STG, and qualitative analysis*\n\n**Response**: Thanks for your positive comments and your recognition! We hope that the following responses can address your concerns and further improve the article as well as facilitate successful publication.\n\n*Reasons To Reject:*\n\n1. *The proposed approach was applied to only one dataset and this might limit to show its effectiveness.*\n2. *Some improvement in performance seems tiny, e.g., 0.15% between STG w/ TSG+Lcsl and TJSTG and thus it might not be significant.*\n\n**Response**: Thank you for raising these valid concerns, and please be assured that we are dedicated to addressing them in the best possible manner within the current constraints. We will respond to your comments point by point.\n\n1. **Insufficiency of One Dataset.** \n   \n      We understand your concern regarding the limited dataset available for this relatively new task introduced in 2022. **In our case, the only publicly available dataset that aligns with the AVQA task requiring spatio-temporal reasoning ability is the one we have employed for experimentation, i.e., MUSIC-AVQA[1].** MUSIC-AVQA aligns with common practices within the field, where it serves as a standard benchmark for evaluating methods. While we acknowledge the limited scope, it's worth noting that the field's consensus often revolves around this dataset [1]\\[5].    We focus on conducting thorough experiments to verify our motivation and draw meaningful insights from the existing dataset. \n   \n      Nonetheless, we try to validate our method on a related dataset proposed very recently in [4]. Unfortunately, we encountered challenges in accessing the complete dataset as it was only provided via YouTube URLs, encompassing a substantial collection of 57.0K+ video data. The accessibility limitations have posed difficulties in obtaining the complete dataset for comprehensive experimentation. Despite our best efforts, the nature of this arrangement has constrained our ability to fully incorporate this dataset into our analysis in time.\n   \n   However, we want to assure you that we are actively monitoring developments in dataset creation within the community. Thank you for your understanding of the challenges we are facing in this regard. We are dedicated to addressing this limitation as effectively as possible and appreciate your consideration.\n2. **Consideration of Performance Improvements.** \n   \n   We appreciate your detailed examination of the performance gains in our study. It's important to highlight that our TJSTG significantly outperforms the current SOTA STG [1] by 1.45% (from 71.59% to 73.04%).  The 0.15% improvement you observe is the result of our proposed module under different settings. Specifically, \"STG w/ TSG+Lcsl\" denotes the integration of our method into the previous two-stream network. This experimental result shows that the integration of **our proposed TSG and Lcsl in our designed single-stream network** exceeds the integration in the previous two-stream network by 0.15%, which demonstrates the effectiveness of our novel single-stream structure.\n   \n   Moreover, compared to “STG w/ TSG+Lcsl”, TJSTG eliminates the need for additional fusion modules. As shown in the following table, we obtain higher performance with fewer parameters and a simpler architecture, which proves the superiority of TJSTG. It is significant to underline that the introduction of TJSTG represents a novel direction, emphasizing the potential improvements of our proposed single-stream methodology in terms of efficiency and integration. \n\n|Method|Accuracy|Trainable Param.|\n|:--:|:--:|:--:|\n|dual-stream  (STG w/ TSG+$\\mathcal{L}_{csl}$)|72.89|15.3 M|\n|single-stream  (TJSTG)|**73.04**|**11.6 M**|\n\nTable R1. Comparison of dual-steam structure and singe-stream structure. TJSTG exceeds \"STG w/TSG+Lcsl\" by 0.15% with 3.7M less training parameters.\n\n*Questions For The Authors:* \n\n*Q1：What is difference between question sentence feature f_q and the last hidden state feature vector h_q, i.e., the question feature? It was not clear to me.*\n\n**A1**: For the question sentence feature $\\mathbf{f}_q \\in \\mathbb{R}^{N\\times d_q}$, it contains the features of N words in a sentence. For the question feature $h_q \\in \\mathbb{R}^{1\\times d_q}$, it compresses the entire sequence's context into a fixed-size representation. $d_q=512$ refers to the dimensionality of the query features. For a fair comparison, we use exactly the same question encoding as [1]. For the input question sentence Q, its maximum number of words is set to N, and the insufficient one will be padded. Here we give the detailed procedure as shown below:\n\n$$E_q=word2vec(Q)$$\n$$\\mathbf{f}_q, (c_t,h_t)=LSTM(E_q,(c_0,h_0))$$\n$$h=Reshape(Cat[c_t;h_t]), h\\in \\mathbb{R}^{1\\times2d_q}$$\n$$h_q=MLP(h)$$\n\nwhere $E_q \\in \\mathbb{R}^{N\\times d_e}$ is the word embedding of input question text Q, $c_0, h_0$ are the initial cell and hidden state of LSTM, $\\mathbf{f}_q \\in \\mathbb{R}^{N\\times d_q}$ is the output of LSTM,  $c_t, h_t \\in \\mathbb{R}^{1\\times d_q}$ are the last cell and hidden state of LSTM, and $h_q \\in\\mathbb{R}^{1\\times d_q}$ is the query feature we adopted, projected from the last state vectors of the LSTM. *Cat* denotes the concatenation. *Reshape* transforms the tensor $Cat[c_t;h_t]$ from a 2-dimensional shape of size $2\\times d_q$ into $h$, a 1-dimensional shape of size $1\\times2d_q$.\n\nWe understand your query regarding the distinction between $f_q$ and $h_q$ because we omit some details of the question encoding due to page limit. We apologize for any confusion caused by insufficient explanations in the paper, and we appreciate this opportunity to address this point. \n\n*Q2：For the proposed cross-modal synchrony (CSL) loss, it was not clear why using text as intermediary is better than directly using video and audio.* \n\n**A2**: The rationale behind utilizing text as an intermediary in the proposed cross-modal synchrony loss (CSL) is rooted in its ability to effectively bridge the gap between audio and visual modalities. By employing the Jensen-Shannon (JS) divergence, the CSL loss quantifies the alignment of text-audio and text-video temporal correlations, offering a unified measure that aids in capturing the intricate cross-modal relationships. In contrast, if audio and video are constrained directly at the feature level, not only will it lose the information specific to the modality, but it is also impractical since video and audio are heterogeneous in nature. Our approach leverages the inherent semantics of text to facilitate a more comprehensive and flexible alignment, contributing to the effectiveness of the proposed query temporal grounding.\n\n*Q3：The authors use the total loss L = Lqa + Lcsl + \\lambda  Ls and how about adding another regularization parameter to Lcsl and tuning them? Does it impact the performance?*\n\n**A3**: We appreciate your suggestion regarding adding a regularization parameter to Lcsl. However, our experimentation with L2 regularization resulted in significantly longer training times (from 5mins/epoch to 29mins/epoch) and a slight decrease in performance from 73.04% to 72.51%, which prompts us to prioritize the balance between computational efficiency and performance gains. It is worth noting that after adding L2 regularization, the performance on both the validate set and the test set decreases. We think that both Batch Normalization and Dropout provide regularization effects that effectively alleviate model overfitting in our approach. Experimental results show that introducing an extra regularization loss function does not lead to significant changes in the model's effectiveness. Therefore, we do not consider adding an additional regularization loss function.\n\n*Q4：In Figure. 4(c), it was still not clear why the model attends on mostly on background, not instruments, as the whole image might need attentions to answer the question.*\n\n**A4**: Figure 4 shows the intermediate output of our model, i.e., the enhanced visual features obtained by the TSG module. The question is “Is there a \\<ukulele\\> in the entire video?”, and there is no ukulele in the video, thus our method presents an irregular spatial grounding result in the background region instead of the incorrect sounding area of guitar and bass as in other methods. Enhanced background visual features can more accurately answer that there is no ukulele in the video than false visual features of other instruments, because the background’s feature is significantly different from the ukulele’s feature compared to other instruments’ features, so it does not bring similar misinformation to influence judgment.\n\n*Typos Grammar Style And Presentation Improvements:*\n\n1. *TSG and TA are used to refer to the same entity, which may be confusing.*\n2. *Some implementation details are missing, e.g., what are d_q, d_a, and d_v?*\n\n**Response**: Thank you for your valuable comments. We will respond to your comments point by point.\n\n1. **TSG module and TA module.**\n\nWe would like to clarify that TSG and TA are indeed two distinct modules in our approach. TA refers to the \"Target-aware\" module, which aims to extract the object of interest in the question sentence, i.e., the target. The TA module focuses on processing textual modality. TSG refers to the \"Target-aware Spatial Grounding\" module, which focuses on enhancing features of audio-visual cues of interest through the introduction of “target” obtained by the TA module.\n\n2. **Implementation details.**\n\nWe sincerely appreciate your feedback highlighting the missing implementation details, specifically regarding the variables $d_q$, $d_a$, and $d_v$. \nTo address this concern, we will provide comprehensive explanations for these variables in the final version of our paper. Specifically, $d_q=512$ refers to the dimensionality of the query features, $d_a=128$ refers to the dimensionality of the audio features, and $d_v=512$ refers to the dimensionality of the visual features. By including these explanations, we aim to ensure that our readers have a clear understanding of the critical variables involved in our approach.\n\nBesides, we will open-source our code to ensure reproducibility so all implementation details will be found in the project.\n\n[1]    Learning to Answer Questions in Dynamic Audio-Visual Scenarios. Li et al., In CVPR 2022. \n\n[2]    Audio-Visual Event Localization in Unconstrained Videos. Tian et al., In ECCV 2018.\n\n[3]    Cross-modal Background Suppression for Audio-Visual Event Localization. Xia et al., In CVPR 2022. \n\n[4]    AVQA: A Dataset for Audio-Visual Question Answering on Videos. Yang et al., In ACM MM 2022.\n\n[5]    COCA: COllaborative CAusal Regularization for Audio-Visual Question Answering. Lao et al., 2023. In AAAI.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed components seem sound and novel \n- Experimental results show that the proposed approach outperforms the prior art\n- Authors provide thorough analysis including ablation study for each proposed component and with prior art, STG, and qaualitative analysis", "reasons_to_reject": "- The proposed approach was applied to only one dataset and this might limit to show its effectiveness. \n- Some improvement in performance seems tiny, e.g., 0.15% between STG w/ TSG+Lcsl and TJSTG and thus it might not be significant.", "questions_to_authors": "- What is difference between question sentence feature f_q and the last hidden state feature vector h_q, i.e., the question feature? It was not clear to me.\n\n- For the proposed cross-modal synchrony (CSL) loss, it was not clear why using text as intermediary is better than directly using video and audio.\n\n- The authors use the total loss L = Lqa + Lcsl + \\lambda * Ls and how about adding another regularization parameter to Lcsl and tuning them? Does it impact the performance?\n\n- In Figure. 4(c), it was still not clear why the model attends on mostly on background, not instruments, as the whole image might need attentions to answer the question.", "paper_topic_and_main_contributions": "This paper tackles the problem of audio-visual question answering and proposes three components: 1)  target-aware spatial grounding (TSG) module by using the question to ground interesting sounding areas, 2) single-stream framework of the joint audio-visual temporal grounding (JTG), and 3) cross-modal synchrony loss (CSL) to promote the temporal synchronization between audio and video by using text. The experimental results show that the proposed approach outperforms the prior art and ablation study demonstrates the effectiveness of individual proposed components.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- TSG and TA are used to refer to the same entity, which may be confusing. \n- Some implementation details are missing, e.g., what are d_q, d_a, and d_v?"}, "extra_text": {}}, {"review_id": "gbWyrPmRgl", "reviewer": "Reviewer_jPEZ", "summary": "", "questions": "please revise the weaknesses carefully before paper submission.", "limitations": "", "rating": 1, "confidence": 4, "soundness": 1, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer jPEZ:\n\n*Paper Topic And Main Contributions:*\n\n*This paper presents a new target-aware joint spatio-temporal grounding network for audio-visual question answering. It consists of two key components: the target-aware spatial grounding module and the single-stream joint audio-visual temporal grounding module. The first module can target on audio-visual cues relevant to the query subject by utilizing explicit semantics from the question. The second module incorporates audio-visual fusion and question-aware temporal grounding into one module with a simpler single-stream architecture.*\n\n**Response**: Thank you for your comments! We appreciate the time and patience that you have dedicated to evaluating and improving our work. We would like to express our sincere apologies for the oversight highlighted in your review. As detailed below, we have addressed each of your comments carefully. \n\n*Reasons To Accept:*\n\n*Results are promising. Figures are clear.*\n\n**Response**: We extend our sincere gratitude for your positive feedback on our paper. Your recognition of our efforts in presenting clear figures and promising results is greatly appreciated. **Thank you for recognizing the superiority of our methodology and the clarity of our presentation of the proposed methodology in our article.** We sincerely hope that the following responses can address your concerns and further improve the article as well as facilitate successful publication.\n\n*Reasons To Reject:*\n\n*The paper is difficult to understand since many symbols are not defined, e.g.,*\n\n**Response**: Thank you for your time and patience. **Due to the page limit, We omit some definitions that are fundamental in the field of CV, which may not be common to the field of NLP.** We will address your questions point by point.\n\n1. *in line 224,* $f_a^t \\in \\mathbb{R}^{d_a}$. *What is* $d_a$ *?* $f_v^t$, *I think, is visual feature?*\n   \n   **Response**: As referred to line 224 of our manuscript:\n   \n   “***Audio Embedding.** Each audio segment* $A_t$ *is encoded into* $f_a^t\\in \\mathbb{R}^{d_a}$ *by the pretrained on AudioSet VGGish model.*”  \n   \n   This is in explaining the audio feature extractor we use, i.e., VGGish. After passing through the audio feature extractor, the audio signal of *t*-th segment is encoded as an audio feature $f_a^t$ with feature dimension $d_a$. **The term** $d_a$ **corresponds to the dimensionality of the extracted audio features.** We set $d_a$ to 128 as previous methods. $f_v^t$ does not appear in line 224. However, **as presented in line 227-231,** $f_v^t$ **is** **the extracted visual feature of *t*-th video segment.** \n2. *in line 229, what are h, w,* $d_v$*?*\n   \n   **Response**: In line 229, **the variables** h, w, **and** $d_v$ **correspond to the height, width, and dimensionality of the visual features, respectively.** These dimensions play a crucial role in the visual feature extraction. We set h=w=14 and $d_v$=512 as previous methods.\n   \n   To ensure clarity for our readers, we will provide explicit definitions and explanations for these variables in the final version of our paper.\n3. *in ablation study, in line 488, the authors state that they remove the proposed CSL (denoted as “w/o L_{csl}\") in Table 2. However, I cannot find the ablation model (“w/o L_{csl}\"). Besides, in line 492, the authors introduce an ablation model (“w/o TA+L_{csl}). I cannot find it in Table 2.*\n   \n   **Response**: We genuinely appreciate your detailed review of our paper and your keen observations regarding the ablation study. We apologize for any confusion caused by the incorrect labeling. \n   \n   The incorrect labeling of ablation models in Table 2 as “w/o $L_{tri}$” and “w/o TA+$L_{tri}$” instead of  “w/o $L_{csl}$” and “w/o TA+$L_{csl}$” is entirely our mistake. Here, we modified the wrong labels, and the modified Table 2 is shown below: \n   |Method|A Avg.|V Avg.|AV Avg.|ALL|\n   |:--:|:--:|:--:|:--:|:--:|\n   |w/o TA|75.17|76.05|69.86|72.44|\n   |w/o $L_{csl}$|75.73|76.14|69.35|72.28|\n   |w/o TA+$L_{csl}$|75.17|74.94|69.21|71.78|\n   |TSJTG (Ours)|**76.47**|**76.88**|**70.13**|**73.04**|\n   \n   Table R1. The modified Table 2 where the wrong label $L_{tri}$ is changed to the correct label $L_{csl}$.\n   \n   We will ensure that our final version accurately represents the ablation models and their corresponding labels in both the text and Table 2. Your feedback highlights the importance of precision in presenting our work, and we are dedicated to addressing this issue.\n4. *In line 286, what is hw? h \\times w or h_w?*\n   \n   **Response:** Thank you for bringing up the confusion regarding the notation in line 286. We apologize for any misunderstanding caused by the notation, and we appreciate your thorough analysis. \n   \n   To clarify, in line 286, the notation \"hw\" indeed represents the product of h and w, that is, $h \\times w$. In line 286, The vector $s_a \\in R^{1 \\times hw}$ signifies a row vector with a size of $h \\times w$, and the explanation you provided aligns accurately with our intent. However, we cannot use $1\\times h \\times w$ to express the size, because this represents a completely different three-dimensional tensor. Thus, We will take your feedback into consideration and ensure that our final version explicitly clarifies the notation, in order to prevent any ambiguity for our readers.\n\n*Subsection 3.3 \"Joint Audio-visual Temporal Grounding\" in line 308 is same as subsubsection \"Joint audio-visual temporal grounding (JTG) module\" in 353. It is very confusing: which part is Joint audio-visual temporal grounding?*\n\n**Response**: Subsection 3.3 describes **a process of “Joint Audio-visual Temporal Grounding” that is realized through the “Joint audio-visual temporal grounding (JTG) module”** (in line 353) **equipped with the Cross-modal synchrony loss** (in line 316).\n\n*Some typos, e.g., quotation mark mismatch throughout the paper.*\n\n**Response**: We sincerely appreciate your keen attention to detail and for identifying the typos, including the mismatched quotation marks, in our paper. \n\nWe understand the importance of maintaining a high level of accuracy and professionalism in our presentation. **Rest assured, we will thoroughly review our paper and correct all instances of quotation mark mismatches and any other typographical errors that might have arisen.** \n\nYour vigilance and feedback contribute to the overall quality of our work, and we are committed to addressing these issues to ensure a polished final version. Thank you for your valuable input.\n\n*Questions For The Authors:*\n\n*Q: please revise the weaknesses carefully before paper submission.*\n\n**A**:  We would like to express our heartfelt gratitude for your thorough review of our manuscript. Your insights have been immensely valuable, and we are truly sorry for any errors that have inadvertently been made through our initial submission. Your comments can greatly improve the quality of our manuscript. We will diligently address the identified weaknesses, e.g., typos and incorrect labels, and ensure they are thoroughly revised before submitting the final version of the paper. \n\n**We kindly request you to consider re-evaluating our article in light of the revisions we have made based on your comments.** We present a fresh perspective on the Audio-visual question answering (AVQA) challenge by proposing a target-aware joint spatio-temporal grounding network with its single-stream architecture, departing from conventional methods. Your continued engagement and thoughtful assessment are instrumental in refining our work.\n\nOnce again, thank you for your time and expertise. We look forward to your continued guidance and assessment.\n\n*Typos Grammar Style And Presentation Improvements:*\n\n*quotation mark mismatch throughout the paper.* \n\n**Response**: We will thoroughly review our paper and correct all instances of quotation mark mismatches and any other typographical errors that might have arisen. \n\nThank you for your valuable input again.\n\n*Soundness: 2: Borderline: Some of the main claims/arguments are not sufficiently supported, there are major technical/methodological problems.* \n\n*Excitement: 2: Mediocre: This paper makes marginal contributions (vs non-contemporaneous work), so I would rather not see it in the conference.* \n\n*Reproducibility: 2: Would be hard pressed to reproduce the results. The contribution depends on data that are simply not available outside the author's institution or consortium; not enough details are provided.*\n\n**Response**: We are sorry for any confusion caused by our lack of detailed descriptions. The omission of some symbol definitions and some mistakes in our manuscript have affected the readability. We hope that our responses and explanations to the aforementioned questions could change your opinion. We would like to highlight that we present a fresh perspective on the Audio-visual question answering (AVQA) challenge by proposing a target-aware joint spatio-temporal grounding (TJSTG) network with its single-stream architecture, departing from conventional methods. Our TJSTG significantly outperforms the current SOTA by 1.45% (from 71.59% to 73.04%).\n\nWe have carefully considered your feedback and observed that you mentioned some concerns about the soundness and excitement of our paper. We noticed that the claims/arguments and technical/methodological aspects weren't elaborated upon in your review. We are genuinely looking forward to you providing us with more specific details regarding the technical/methodological issues you have identified. This would greatly assist us in understanding and addressing the shortcomings you've highlighted. \n\nAs for the reproductivity, we appreciate your concern, but we want to clarify that we rely on the widely adopted publicly available dataset and will open-source our code to ensure reproducibility beyond our institution or consortium.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Results are promising.\nFigures are clear.", "reasons_to_reject": "The paper is difficult to understand since many symbols are not defined, e.g., \n1) in line 224, f_a^t \\in \\mathbb{R}^{d_a}. What is d_a? f_v^t, I think, is visual feature?\n2) in line 229, what are h, w, d_v?\n3) in ablation study, in line 488, the authors state that they remove the proposed CSL (denoted as “w/o L_{csl}\") in Table 2. However, I cannot find the ablation model (“w/o L_{csl}\"). Besides, in line 492, the authors introduce an ablation model (“w/o TA+L_{csl}). I cannot find it in Table  2.\n4) In line 286, what is hw? h \\times w or h_w?\n\nSubsection 3.3 \"Joint Audio-visual Temporal Grounding\" in line 308 is same as subsubsection \"Joint audio-visual temporal grounding (JTG) module\" in 353. It is very confusing: which part is Joint audio-visual temporal grounding?\n\nSome typos, e.g., quotation mark mismatch throughout the paper.", "questions_to_authors": "please revise the weaknesses carefully before paper submission.", "paper_topic_and_main_contributions": "This paper presents a new target-aware joint spatio-temporal grounding network for audio-visual question answering. It consists of two key components: the target-aware spatial grounding module and the single-stream joint audio-visual temporal grounding module. The first module can target on audio-visual cues relevant to the query subject by utilizing explicit semantics from the question. The second module incorporates audio-visual fusion and question-aware temporal grounding into one module with a simpler single-stream architecture.", "excitement": 1, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "quotation mark mismatch throughout the paper."}, "extra_text": {}}, {"review_id": "sKJwLGKeHe", "reviewer": "Reviewer_ctyk", "summary": "", "questions": "The word with the most significant contribution weight is selected as the target. What is the rationale behind this choice? How can we ensure that it genuinely represents the target subject information?\n\nWhy is the target information solely used to obtain target-aware visual information? Why isn't it implemented for both visual and audio modalities?\n\nIn Eqn. (13), IL refers to what?\n\nWhat is the intention of Equation 16? What would happen without it?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ctyk:\n\n*Paper Topic And Main Contributions:*\n\n*This paper delves into the domain of Audio-visual question answering (AVQA), which requires intricate spatio-temporal reasoning across both audio and visual modes. The primary concern highlighted is that existing models tend to separate audio and video channels when addressing temporal grounding, potentially leading to less accurate or intuitive AVQA solutions. To this end, this paper introduces a new grounding network that effectively synergizes spatial and temporal dimensions, ensuring a more accurate and nuanced understanding of multimodal contexts.*\n\n**Response**: Thank you for your conclusive comments! We appreciate the time and effort that you have dedicated to evaluating and improving our work. As detailed below, we have addressed each of your comments carefully.\n\n*Reasons To Accept:*\n\n1. *This paper presents a fresh perspective on the Audio-visual question answering (AVQA) challenge by proposing a target-aware joint spatio-temporal grounding network.*\n2. *The JTG, with its single-stream architecture, is a departure from conventional methods.*\n3. *The introduction of cross-modal synchrony loss (CSL) ensures better synchronization between modalities.*\n\n**Response**: Thanks for your positive comments and your recognition. We hope that the following responses can address your concerns and further improve the article as well as facilitate successful publication.\n\n*Reasons To Reject:*\n\n1. *This paper claims that \"Most of the studies (such as Xuan et al., 2020; Wu et al., 2019; Mercea 053 et al., 2022) obtain non-targeted sound-related visual regions by designing various attention schemes\". However, this paper also uses the attention mechanism to mine target information and visual information related to the target. So, how can it ensure that it does not have the aforementioned problems?*\n2. *Compared to the two-stream approach, the superiority of the one-stream approach is not clearly explained.*\n3. *Merely conducting experiments on one dataset is insufficient.*\n4. *This paper employs the CSL loss to ensure that the temporal distributions of the two modalities are closely aligned. However, visuals and audio might not always be in sync, raising questions about the appropriateness of this constraint.*\n\n**Response**: Thank you for raising these valid concerns, and please be assured that we are dedicated to addressing them in the best possible manner within the current constraints. We will respond to your comments point by point.\n\n1. **Non-targeted issue.**\n   \n   Our lack of detailed description may have caused your confusion. We hereby correct it to “*Most of the studies (such as Xuan et al., 2020; Wu et al., 2019; Mercea 053 et al., 2022) obtain non-targeted sound-related visual regions **by designing various attention schemes performing on audio-to-visual while ignoring the question-oriented information from the text modality.***” The previous method ignores the question-oriented information contained in the text modality, which leads to untargeted sound-related visual feature enhancement, and our mining and utilizing of the target information contained in the text modality addresses this issue.\n2. **Single stream network.**\n   \n   Compared to dual-stream networks, our proposed single-stream network with cross-modal synchrony loss offers more effective and efficient joint temporal grounding of audio-visual video. **Our designed single-stream network maintains both the integrity of the audio-visual content at the segment level and the relative independence between the audio and visual content at the video level.** Specifically, we design the a specialized audio-visual interleaved pattern at the segment level, i.e., $ \\textbf{IL}(\\mathbf{f}_v;\\mathbf{f}_a)=[f_v^1;f_a^1;   f_v^2;f_a^2;...;f_v^T;f_a^T]$, which exceeds concatenating them at the video level, i.e.,  $ \\textbf{Cat}(\\mathbf{f}_v;\\mathbf{f}_a)=[f_v^1;f_v^2;...;f_v^T;f_a^1;f_a^2;...;f_a^T]$, by 0.3% (from 72.74% to 73.04%). We refactor the question-related correlation of single-stream data into video-text correlation and audio-text correlation at the video level, and feed them as two distributions to our proposed cross-modal synchrony loss, which encourages the model to capture meaningful and interconnected video-level temporal patterns within audio and visual modalities.\n   \n   **Using a single-stream architecture that omits the additional fusion module yields higher accuracy with fewer parameters and simpler architecture** **than a dual-stream architecture, which demonstrates the superiority of joint audio-visual temporal grounding for the single-stream network.** As shown in the table below:\n\n|Method|Accuracy|Trainable Param.|\n|:--:|:--:|:--:|\n|dual-stream  (STG w/ TSG+$\\mathcal{L}_{csl}$)|72.89|15.3 M|\n|single-stream  (Ours)|**73.04**|**11.6 M**|\n\nTable R2. Comparison of dual-steam structure and singe-stream structure. Our TJSTG exceeds \"STG w/TSG+Lcsl\" by 0.15% with 3.7M less training parameters.\n\nSpecifically, the single-stream structure refers to our method in Table 1 in the article, denoted as “TJSTG”; the dual-stream structure refers to the results of the ablation study or our proposed TSG and Lcsl in Table 3 in the article, denoted as “STG w, TSG+Lcsl”. There is no other difference between these two methods except for the difference in structure.\n\nWe regret that due to page limitations, we were not able to fully demonstrate the superiority of the one-stream structure in the manuscript. A More detailed explanation can be found in **3. Single-Stream Network** **of the response to reviewer dLRt**. We will do our best to fully demonstrate the superiority of the single-stream approach in the final version!\n\n3. **Insufficiency of One Dataset.**\n   \n   We understand your concern regarding the limited dataset available for this relatively new task introduced in 2022. **In our case, the only publicly available dataset that aligns with the AVQA task requiring spatio-temporal reasoning ability is the one we have employed for experimentation, i.e., MUSIC-AVQA[3].** MUSIC-AVQA aligns with common practices within the field, where it serves as a standard benchmark for evaluating methods. While we acknowledge the limited scope, it's worth noting that the field's consensus often revolves around this dataset [3]\\[6]. **We focus on conducting thorough experiments to verify our motivation and draw meaningful insights from the existing dataset.** \n   \n   Nonetheless, we try to validate our method on a related dataset proposed very recently in [5]. Unfortunately, we encountered challenges in accessing the complete dataset as it was only provided via YouTube URLs, encompassing a substantial collection of 57.0K+ video data. The accessibility limitations have posed difficulties in obtaining the complete dataset for comprehensive experimentation. Despite our best efforts, the nature of this arrangement has constrained our ability to fully incorporate this dataset into our analysis in time.\n   \n   However, we want to assure you that we are actively monitoring developments in dataset creation within the community. Thank you for your understanding of the challenges we are facing in this regard. We are dedicated to addressing this limitation as effectively as possible and appreciate your consideration.\n4. **Concerns about CSL Loss for Temporal Alignment.**\n   \n   It's a valid point that visuals and audio might not always be perfectly synchronized, which raises questions about our proposed cross-modal synchrony loss (CSL).\n   \n   The CSL loss is designed to utilize the Jensen-Shannon (JS) divergence to measure the alignment between the distributions of text-audio temporal correlations and the distribution of text-video temporal correlations. **Importantly, the CSL loss is not intended to enforce strict temporal alignment in a manner that could result in the unrealistic synchronization of visuals and audio.** **Rather, it serves as a regularization technique to encourage the inherent synchrony that exists between the two modalities while allowing for some level of natural variance.** \n   \n   Take our CSL loss and strictly aligned MSE loss as an example. When the distribution of the text-visual correlation and the text-audio correlation are exactly the same, both being $w_v=w_a=[1,2,3,4,5]$, both the CSL and MSS losses are 0. When some of the segments are out of sync, e.g., when $w_a$ becomes $[1,2,1,4,5],$ the MSE loss becomes 0.8 while the CSL is only 0.003. Note that we normalize the correlation $w_v, w_a$ when calculating the loss. \n   \n   **By utilizing the CSL loss with JS divergence, we aim to capture meaningful alignment patterns while respecting the organic temporal variations that naturally occur.** In essence, the CSL loss encourages our model to identify relevant temporal associations without imposing strict constraints on the data.\n   \n   Your feedback underscores the importance of clarifying the role of the CSL loss in addressing temporal alignment while accommodating potential asynchrony. We are committed to revising our paper to provide a clearer explanation of this aspect and its implications.\n\n*Questions For The Authors:*\n\n*Q1: The word with the most significant contribution weight is selected as the target. What is the rationale behind this choice? How can we ensure that it genuinely represents the target subject information?*\n\n**A1**: As we know, the question is centered around the subject of inquiry, i.e., the target, and is broken down into its various properties, e.g., spatial and temporal. Different targets are a naturally important attribute in distinguishing between different questions. More importantly, 1) As referred to line 294-297, “*By computing text-visual attention maps, it encourages the previous TA module to capture the visual-related entity among the question.*” 2) As referred to line 430, “*Training Strategy.*”, we constructed negative samples by employing mismatched images, thereby increasing the similarity between \"positive visual samples\" and “audio and target” through $L_s$.\n\n*Q2: Why is the target information solely used to obtain target-aware visual information? Why isn't it implemented for both visual and audio modalities?*\n\n**A2**: Because of the special characteristics of audio features\\*, most of the existing audio-visual learning methods [1]\\[2]\\[3] use audio information to enhance the representation of visual features rather than vice versa. Although there are some recent methods [4] that exploit visual information to separate audio sources, they either require dense annotations or powerful specialized pre-trained models.\nIn our current approach, the utilization of target information for enhancing visual features is indeed a deliberate choice. We focus on leveraging the target information to improve the visual features due to the nature of audio-visual features.\nHowever, we acknowledge the potential benefits of implementing a similar mechanism for the audio modality. Your observation is valuable, and we will certainly explore the feasibility and implications of extending the target-aware approach to the audio modality in our future research.\n\n\\______________________________________________________\n\n\\*The commonly existing audio spectrogram overlapping phenomenon makes isolating audio elements significantly more difficult than dividing visual elements in images.\n\n[1]    Audio-Visual Segmentation. Zhou et al., In ECCV 2022. \n\n[2]    Cross-modal Background Suppression for Audio-Visual Event Localization. Xia et al., In CVPR 2022. \n\n[3]    Learning to Answer Questions in Dynamic Audio-Visual Scenarios. Li et al., In CVPR 2022. \n\n[4]    Language-Guided Audio-Visual Source Separation via Trimodal Consistency. Tan et al., In CVPR 2023.\n\n[5]    AVQA: A Dataset for Audio-Visual Question Answering on Videos. Yang et al., In ACM MM 2022.\n\n[6]    COCA: COllaborative CAusal Regularization for Audio-Visual Question Answering. Lao et al., 2023. In AAAI. \n\n*Q3: In Eqn. (13), IL refers to what?*\n\n**A3**: As referred to line 361, **IL** denotes that audio and visual features are **I**nter**L**eaved in segments, i.e.,   $ \\textbf{IL}(\\mathbf{f}_v;\\mathbf{f}_a)=[f_v^1;f_a^1;...;f_v^T;f_a^T]$. We recognize that the definition provided in the current version is not clear enough to clarify IL. We apologize for any confusion this may have caused. We will address this concern by providing a clear definition of \"IL\" in Equation (13) in the final version of our paper. We greatly appreciate your attention to detail. Thank you for your effort to improve our work.\n\n*Q4: What is the intention of Equation 16? What would happen without it?*\n\n**A4**: We deeply regret any confusion caused by our oversight, and we sincerely appreciate your attention to detail. We would like to correct Equation 16 to $f_{av}^q=f_{att}+MLP(Avg(f_{av}))$. This is a residual connection of the audiovisual features before and after they have been temporal grounded by question. This residual connection serves to capture and retain the original unaltered features while incorporating the context and alignment introduced by the temporal grounding process. Without this residual connection, the risk lies in potentially diluting the original information through the temporal grounding.  \n\n*Reproducibility: 2: Would be hard pressed to reproduce the results. The contribution depends on data that are simply not available outside the author's institution or consortium; not enough details are provided.*\n\n**Response to reproductivity:** As for the reproductivity, we appreciate your concern, but we want to clarify that we rely on the widely adopted publicly available dataset MUSIC-AVQA and will open-source our code to ensure reproducibility beyond our institution or consortium.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents a fresh perspective on the Audio-visual question answering (AVQA) challenge by proposing a target-aware joint spatio-temporal grounding network.\n\nThe JTG, with its single-stream architecture, is a departure from conventional methods.\n\nThe introduction of cross-modal synchrony loss (CSL) ensures better synchronization between modalities.", "reasons_to_reject": "This paper claims that \"Most of the studies (such as Xuan et al., 2020; Wu et al., 2019; Mercea 053 et al., 2022) obtain non-targeted sound-related visual regions by designing various attention schemes\". However, this paper also uses the attention mechanism to mine target information and visual information related to the target. So, how can it ensure that it does not have the aforementioned problems?\n\nCompared to the two-stream approach, the superiority of the one-stream approach is not clearly explained. \n\nMerely conducting experiments on one dataset is insufficient.\n\nThis paper employs the CSL loss to ensure that the temporal distributions of the two modalities are closely aligned. However, visuals and audio might not always be in sync, raising questions about the appropriateness of this constraint.\n\n", "questions_to_authors": "The word with the most significant contribution weight is selected as the target. What is the rationale behind this choice? How can we ensure that it genuinely represents the target subject information?\n\nWhy is the target information solely used to obtain target-aware visual information? Why isn't it implemented for both visual and audio modalities?\n\nIn Eqn. (13), IL refers to what?\n\nWhat is the intention of Equation 16? What would happen without it?", "paper_topic_and_main_contributions": "This paper delves into the domain of Audio-visual question answering (AVQA), which requires intricate spatio-temporal reasoning across both audio and visual modes. The primary concern highlighted is that existing models tend to separate audio and video channels when addressing temporal grounding, potentially leading to less accurate or intuitive AVQA solutions. To this end, this paper introduces a new grounding network that effectively synergizes spatial and temporal dimensions, ensuring a more accurate and nuanced understanding of multimodal contexts. ", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IqkAPrEvz7", "reviewer": "Reviewer_dLRt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer dLRt:\n\n*Paper Topic And Main Contributions:*\n\n*This paper proposes an approach which is composed of the target-aware spatial grounding module (TSG) and singe-stream joint audio-visual temporal grounding module (JTG). Specifically, TSG finds relevant audio and visual features using the query for better alignment. JTG allows a simpler module by performing audio-visual feature alignment in a single-steam architecture. Extensive experiments demonstrate the effectiveness of the proposed approach.*\n\n**Response**: Thank you for your conclusive comments! We appreciate the time and effort that you have dedicated to evaluating and improving our work. As detailed below, we have addressed each of your comments carefully.\n\n*Reasons To Accept*:\n\n1. *This paper proposes an effective single-steam alignment module which is simpler than previous works.*\n2. *The proposed approach demonstrates promising performance.*\n\n**Response**: Thanks for your positive comments and your recognition. We hope that the following responses can address your concerns and further improve the article as well as facilitate successful publication.\n\n*Reasons To Reject*:\n\n1. *Missing comparison: Table 1 should have more previous works (for example, [1]) for more thorough comparison.*\n2. *The novelty of the propose approach seems weak. Target-aware grounding module doesn’t seem to be much different from a normal attention process.*\n3. *Use of single-stream networks might not be well justified. Actually, visual and audio feature are dealt with separately in the steam and summed up.*\n   \n   *[1] COCA: COllaborative CAusal Regularization for Audio-Visual Question Answering. Lao et al, 2023*\n\n**Response**: Thank you for your valuable feedback. We will respond to your comments point by point.\n\n1. **Missing Comparison.**\n   \n    We acknowledge your suggestion to include a more comprehensive comparison with previous works, particularly with the work by Lao et al. [1]. However, it's important to note that **[1] was released on June 26, 2023, which falls after the submission deadline of June 23, 2023**. To the best of our knowledge, we have included all previous works released before the submission deadline. Given the timing constraint, we regrettably couldn't include [1] in the initial submission. However, we recognize the significance of [1] and its relevance to our paper, and we are fully committed to rectifying this in our final version. We will ensure that a detailed comparison with [1] will be included in the revised manuscript to address your concern effectively.\n2. **Novelty of the Target-aware grounding module.**\n   \n   What we adopt in the TSG module is the normal attention mechanism, but the process is significantly different from the previous approach. While previous attention processes [2]\\[3]\\[4] usually applied to only two modalities: visual-audio or visual-text, our attention process encompasses all three modalities at the same time: visual-audio-text. **In the AVQA task, we are the first to introduce textual modalities with explicit semantics during audio-visual spatial grounding, which facilitates the question-oriented comprehension of audio-visual scenes.**\n   \n   Specifically, previous AVQA approaches [2]\\[3] have usually been limited to performing cross-modal attention processes between visual and auditory modalities only or visual-audio and visual-text separately [4]\\[5]. This results in an undifferentiated focus on all sounding regions in the audio-visual scene, and the inevitable redundant audio-visual cues could affect the accuracy of the Q&A due to the untargeted modeling. Besides, most visual language learning methods, such as in [6]\\[7], directly fuse coarse-grained sentence-level textual modal features with visual features for visual feature enhancement, which is different from our question-orientated AVQA task containing a fine-grained word-level inquiring subject,  i.e., the target.\n3. **Single-Stream Network.**\n   \n   We integrate the audio and video sequences into a singular sequence after visual feature enhancement (i.e., spatial grounding), which provides the basis for a single-stream network structure. Besides, our proposed cross-modal synchrony loss establishes a link between audio and visual features instead of processing them separately as in previous dual-stream networks.\n   \n   Specifically, **our proposed cross-modal synchrony loss** **utilizes “*question*” as a medium to establish a link between audio and visual features** **by constraining the consistency of the distribution of text-audio temporal correlations with the distribution of text-video temporal correlations.** As we know, by combining vision and hearing, we can gain a more accurate understanding of a scene. For example, basses and guitars look similar but sound distinctly different. Therefore, being able to take both visual and auditory information into account simultaneously facilitates video temporal grounding. Experiments in Table 2 show that the removal of cross-modal synchrony loss caused a 0.76% decrease in accuracy (from 73.04% to 72.28%). \n   \n   Furthermore, We employed a single-stream network and designed a specialized audio-visual interleaved pattern, i.e., $ \\textbf{IL}(\\mathbf{f}_v;\\mathbf{f}_a)=[f_v^1;f_a^1;...;f_v^T;f_a^T]$, which **maintains both the integrity** **of the audio-visual content at the segment level and the relative independence** **between the audio and visual content at the video level.** This not only eliminates the need for an additional audio-visual fusion module but also improves the effectiveness of question-oriented audio-visual fusion. It is worth noting that the proposed cross-modal synchrony loss employs JS divergence to constrain the consistency of the distribution rather than crudely making the correlations exactly the same. For example, when the provided visual information is more critical to answering the question, the correlation between single-stream multimedia sequences and the “question” could be $[v_1=0.56;a_1=0.24;v_2=0.14;a_2=0.06]$, where visual features comprise a greater proportion than auditory features, and key segments have a greater proportion than the rest of the segments.\n   \n   To validate the effectiveness of the single-stream architecture we designed, we added an experimental comparison of different ways of arranging visual and audio features. The experimental results are shown in the following table:\n\n|Method|A Avg.|V Avg.|AV Avg.|ALL|\n|:--:|:--:|:--:|:--:|:--:|\n|**Cat**(A;V) w/ $\\mathcal{L}_{csl}$|75.92|76.67|69.49|72.53|\n|**Cat**(V;A) w/ $\\mathcal{L}_{csl}$|75.85|75.93|**70.23**|72.74|\n|**IL**(A;V) w/ $\\mathcal{L}_{csl}$|**76.78**|<u>76.88</u>|70.04|<u>73.04</u>|\n|**IL**(V;A) w/ $\\mathcal{L}_{csl}$(Ours)|<u>76.47</u>|**76.88**|<u>70.13</u>|**73.04**|\n\nTable R1. Effect of the feature arrangement on the accuracy(\\%). The top 2 results are highlighted. **IL** denotes that audio and visual features are **I**nter**L**eaved in segments. **Cat** denotes that audio and visual features are concatenated.\n\n  Besides, **using a single-stream architecture that omits the additional fusion module yields higher accuracy with fewer parameters than a dual-stream architecture.** As shown in the table below:\n\n|Method|Accuracy|Trainable Param.|\n|:--:|:--:|:--:|\n|dual-stream  (STG w/ TSG+$\\mathcal{L}_{csl}$)|72.89|14.6 M|\n|single-stream  (Ours)|**73.04**|**11.1 M**|\n\nTable R2. Comparison of dual-steam structure and singe-stream structure. Our TJSTG exceeds \"STG w/TSG+Lcsl\" by 0.15% with 3.7M less training parameters.\n\n  Specifically, the single-stream structure refers to our method in Table 1 in the article, denoted as “TJSTG”; the dual-stream structure refers to the results of the ablation study of our proposed TSG and Lcsl in Table 3 in the article, denoted as “STG w/ TSG+L_csl”. There is no other difference between these two methods except for the difference in structure.\n\n[1]    COCA: COllaborative CAusal Regularization for Audio-Visual Question Answering. Lao et al., 2023. In AAAI.  \n\n[2]    Learning to answer questions in dynamic audio-visual scenarios. Li et al., 2022. In CVPR.\n\n[3]    Cross-Modal Background Suppression for Audio Visual Event Localization. Xia et al.,2022. In CVPR.  \n\n[4]    Pano-AVQA: Grounded Audio-Visual Question Answering on 360$^\\circ$ Videos. Yun et al., 2021. In ICCV.  \n\n[5]    AVQA: A dataset for audio-visual question answering on videos. Yang et al., 2022. In ACM MM.  \n\n[6]    Boundary Proposal Network for Two-Stage Natural Language Video Localization. Xiao et al., 2021. In AAAI.  \n\n[7]    Structured multi-level interaction network for video moment localization via language query. 2021. Wang et al., 2021. In CVPR.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper proposes an effective single-steam alignment module which is simpler than previous works.\n- The proposed approach demonstrates promising performance.", "reasons_to_reject": "- Missing comparison: Table 1 should have more previous works (for example, [1]) for more thorough comparison.\n\n- The novelty of the propose approach seems weak. Target-aware grounding module doesn’t seem to be much different from a normal attention process. \n\n- Use of single-stream networks might not be well justified. Actually, visual and audio feature are dealt with separately in the steam and summed up.\n\n[1] COCA: COllaborative CAusal Regularization for Audio-Visual Question Answering. Lao et al, 2023", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes an approach which is composed of the target-aware spatial grounding module (TSG) and singe-stream joint audio-visual temporal grounding module (JTG). Specifically, TSG finds relevant audio and visual features using the query for better alignment. JTG allows a simpler module by performing audio-visual feature alignment in a single-steam architecture. Extensive experiments demonstrate the effectiveness of the proposed approach.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The manuscript proposes a novel idea of jointly encoding audio and visual signals. Not treating audio and video separately using joint encoding and CSL loss authors shown that the proposed approach improves on SoTA in a benchmark dataset. While mainly one benchmark available (excluding recently published dataset), this raises the question of generalizability of the approach. Some errors in technical presentation made the understanding of details harder which is very important when limited dataset available. Reviewers and authors have engaged in discussion to clarify these concerns. The final manuscript should have these discussion and corrections incorporated in the text.


A version of this manuscript appeared online before anonymity period, but the EMNLP policies indicates that authors should notify PCs about existence of such non-anonymized versions. Unfortunately, this policy is not followed and resulted in a partially non-anonymized review.
Decision: Accept-Findings