HEAR: Hearing Enhanced Audio Response for Video-grounded Dialogue
Authors: Sunjae Yoon
DaHyun Kim
Eunseop Yoon
Hee Suk Yoon
Junyeong Kim
Chang D. Yoo
Venue: EMNLP 2023 Findings (2023)
Keywords: Video-grounded Dialouge
Video Scene Understanding
Open-ended Video Question Answering
TL;DR: This paper proposes Hearing Enhanced Audio Response (HEAR) framework that improves the audibility in model agnostic manner for Video-grounded Dialouge

Abstract: Video-grounded Dialogue (VGD) aims to answer questions regarding a given multi-modal input comprising video, audio, and dialogue history. Although there have been numerous efforts in developing VGD systems to improve the quality of their responses, existing systems are competent only to incorporate the information in the video and text and tend to struggle in extracting the necessary information from the audio when generating appropriate responses to the question. The VGD system seems to be deaf, and thus, we coin this symptom of current systems' ignoring audio data as a deaf response. To overcome the deaf response problem, Hearing Enhanced Audio Response (HEAR) framework is proposed to perform sensible listening by selectively attending to audio whenever the question requires it. The HEAR framework enhances the accuracy and audibility of VGD systems in a model-agnostic manner. HEAR is validated on VGD datasets (i.e., AVSD@DSTC7 and AVSD@DSTC8) and shows effectiveness with various VGD systems.

HEAR: Hearing Enhanced Audio Response for Video-grounded Dialogue
Sunjae Yoon
Dahyun Kim
Eunseop Yoon
Hee Suk Yoon!
Junyeong Kim
Chang D. Yoo
Korea Advanced Institute of Science and Technology (KAIST)
*Chung-Ang University
(sunjae. dahyun. kim, esyoon97, hskyoon, junyeongkim@cau. ac. kr
ac.
Abstract
Video DO
Vile
srdeo-grounded Dialogue (VGD) aims to an- swer questions regarding given multi-modal input comprising video, audio, and dialogue history. Although there have been numerous efforts in developing VGD systems improve the quality of their responses, existing systems are competent only incorporate the informa- tion in the video and text and tend to struggle in extracting the necessary information from the audio when generating appropriate responses to the question. The VGD system seems to be deaf, and thus, we coin this symptom of current systems' ignoring audio data as deaf response. To overcome the deaf response problem, Hear- ing Enhanced Audio Response (HEAR) frame- work is proposed to perform sensible listening by selectively attending to audio whenever the question requires The HEAR framework en- hances the accuracy and audibility of VGD sys- tems a aggoss-g manner. HEAR val- idated on VGD datasets (i.e., AVSD DSTC7 and WSD@@DTTCS and shows effectiveness with various VGD systems.
Auda TU
('Scadds of falidag)
+
Bolaga Siaay (H glab, (g
Helags thttr itt (C io hhh... 10 (am
Un rirm
V
(0)
Dree ridmo have wand!)
people
Therr
G7 You can the shallling homs.
:r: la the inve geople spril to such otlos.
Raprddngandi
do Dlengarding andie backgroand nake
Figure I: Current VGD system's deaf responses on ques- tions about audio: (a) Audio is considered not present and (b) Audio is disregarded as background noise.
round question Q", VGD system is expected to an- swer in free-form A" to the question Q". For this VGD task, multi-moda interaction has been a pop- ular solution, including transformer (Vaswani al., 2017), where many studies concerning modality interactions are performed to boost performance and improve the efficiency of VGD systems.
Unfortunately, these multi-modas interactions focus only on finding the joint representations be- tween video and language. As a consequence, cur- rent agents tend to ignore audio in generating the response. This symptom of ignoring input audio in responding to the question will be referred to the 'deaf response'. Figure represents examples of deaf responses of existing VGD systems (Yoon et al., 2022c: et al., 2021a). To the question "Does the video have sound?" in Figure (a), the system is not able to recognize the input audio: It responds as though the audio is not present. Fur- thermore, even when the system does recognize the existence of audio, it lacks the capability to deci- pher the information within the audio accurately.
1 Introduction
One of the desiderata our vasion-language com- munity is to build conversational agents that can look, listen, think and speak as humans. These agents can potentially be deployed in various sub- sections of society, including education, security, entertainment, and visual or other impairments. To promote natural conversation between humans and the conversational agents, the Video-grounded Dialogue (VGD) task (Alamri et al., 2019) has been designed, aiming to respond to the ques- tions regarding a given multimodal input com- prising video, audio, and dialogue history. To be specific, as shown in Figure 1, given video V, audio U, dialogue history composed of cap- tion C and past rounds of Q&AA pairs H {C, (Q and current r-th
This is evident Figure I (b) where all the sounds of people talking are disregarded as background noise, resulting in incorrect responses.
Cooreespnndigg author
Our experimental evidence in Figure 2 also

---

Dastton (Trad)
2 Related works
L
2.1 Video-grounded Dialogues
$8.28
Visual Question Answering (VQA) (Antol et al., 2015; Wang et al., 2022) has been one of the proxy tasks to evaluate the multi-modal understanding of vision-language systems. In light of recent advance- ments in generative models in natural language processing (Devlin et al., 2018; Radford et al., 201 8), VQA has evolved into more general format of answering video- seteo--roonded dialogue (VGD) (Alamri al., 2019), where VGD aims gener- ale open-ended answer sentences from a question by referring to several input modalities (i.e., video, audio, and dialogue history). Many multi-modal interactions have been proposed, where various at- tention mechanisms (Sanabria et al., 2019; Le et al., 2019) have been devised to perform cross-modal interactions. To boost performances, transformer- based VGD systems (Li et al., 2021b) are utilized top of large-scale pre-trained language models (Radford al., 2019; Raffel et al., 2020). Another immense challenge is keeping track of extended dialogue context, and video, where memory net- works (Lin et al., 2019; Xie and lacobacci, 2020) and multi-step attention (Chu et al.. 2020) were introduced to efficiently store the video and long episodic dialogue. Graph representations (Kim et al., 2021; Pham et al., 2022; Le et al., 2021) were also popular solutions for holding semantic com- monalities between the dialogue and video. There have been novel structures (Hori et al., 2019b; Le et al., 2022) to enhance the multi-modas represen- tation and frameworks (Le and Chen, 2020; Lee et al., 2020; Yoon et al., 2022c) to improve the quality of responses in terms of bias or word selec- tion. As such, many advances have been made in the multi-modal understanding of VGD systems, but mainly between video and language. Thus, the VGD system's understanding of audio is still far from satisfactory. To this end, we first contribute to improving the 'listening' ability of VGD system.
Current systen perfermatcss
quedions
Current VGD systems' performances on AVSD dataset (validation); (a) Response performances according to training with and without audio, (b) Aver- age performance drops on the questions about audio.
Figure 2:
shows that the current VGD systems cannot in- corporate audio information when responding, Fig- ure 2 (a) shows the performance of response (i.e., CIDEr (Vedantam et al., 2015), ROUGE-L (Lin, 2004), BLEU (Papineni et al., 2002)) of current VGD systems according training with and with- out the audio: There are very little differences in performances between the two cases. Some met- rics even show a slightly higher performance when trained without audio. Furthermore, as shown in Figure 2 (b), when investigating the responses to audio-related questions', their performances are noticeably lower compared to the overall response performances. Therefore, existing VGD systems tend to ignore the audio and suffer from the deaf response, which leads to incorrect answers, espe- cially to questions pertinent to audio.
To overcome the deaf response problem, we pro- pose Hearing Enhanced Audio Response (HEAR) framework, which allows the VGD systems to sen- sibly listen to audio according the meaning of the question and perform enhanced listening to the au- dio. Thus, HEAR incorporates (1) Sensible Audio Listening (SAL) that selectively attends to audio according to a sensible decision of whether to focus on audio or not and (2) Reconstruetive Listening Enhancement (RLE) that improves the audibility via establishing reconstruction upper bound to connect audio with its surrounding information. For the sensible decision in SAL,, introduce two technical contributions: (1) Keyword- based Audio Sensing and (2) Semantic Neural Estimator. HEAR is applied on current runner models (Hori et al., 2019 9a; Yoon et al., 2022c: Li et al., b) in a model- aggostic manner, where the effectiveness is validated on VGD dataset (i.e., AVSD @ DSTC7, AVSD @ DSTC8) with steady performance gains on natural language generation metrics.
3 Task Definition
Video-grounded Dialogue (Alamri et al., 2019) (VGD) task aims to generate open-ended answer sentences a question regarding multimodal in- puts composed of video, audio, and dialogue his- tory. To build formal task definition of VGD, a system takes tuples (u, u, h, q²) as input and de- codes answer sentence a", where v is video, u is au-
We select the questions that contain words related to audio such as 'sound', 'speech', and 'noisc".

---

exteddig
Scoidde Andoo Lianin Xos Cs
Ean music Decoder
Eppe word
Cr Cor diba Co.
Lymrinal beal
Cur with namundag
Dialogue Langrage Model (DLM)
Audio Reconstruction
L Aud recedratien
Transformer Block N
Transformer Block N
Alternative Training
Transformer Block 1
Transformer Block
R
Neural f(x
Estinaation Score
Todao
Recestraatoo Nadling
Ressuaee srlias fermandaa Voeddng
Clasollier token
zudio
Quuction: any sounds
(a) Semsible Audio L Listening
RecaESTOREOO Gange
(b) Reconstructive Listening E Enhancement
Figure 3: Illustration of Hearing Enhanced Audio Response Framework (HEAR) for video-grounded dialogue. HEAR performs sensible listening via (a) Sensible Audio Listening that selectively attends to audio conresooning to a given question and improves audibility via (b) Reconstructive Listening Enhancement that enhances audio representations by establishing reconstruction upper bound connect audio with its surrounding information.
dio, h is dialogue history, and q" is a question asked at current round r E {1, (1. R}. The dialogue his- tory = {c, (q', a²),. (g"- } is com- posed of caption to summarize the video and set of question-answer pairs in previous rounds. For training VGD system, next-word prediction is performed, where the system predicts t-th answer word token of from the inputs of tuples (v, u, h, q²) and partial answer word tokens azzt before 1-th.
trained on the Kinetics dataset (Kay et al., 2017) to get 4096-dimensional video features v E IR 4:046 composed of rgb and optical-flow features, where L is the number of video frames. For the audio embedding, we use VGGish model (Hershey et al., 2017) pre-trained on the AudioSet dataset (Gem- meke et al., 2017) to get 128-dimensional audio features 1 where the the number of audio features The aforementioned video and audio features are concatenated along the feature dimension axis and embedded into d-dimensional space as audio-visual features u² as given below:
4 Hearing Enhanced Audio Response
Figure 3 illustrates Hearing Enhanced Audio Re- sponse (HEAR) framework designed to enhance Dialogue Language Model (DLM)² in terms of two functionalities on audio: (1) Sensibility that selec- tively attends on audio according to the meaning of the question and (2) Audibility that performs en- hanced listening to input audio. For sensibility, we propose Sensible Audio Listening (SAL) in Figure 3 (a) which trains the DLM to respond to a question by selectively weighting to audio corresponding to the audio-relatedness of the question. For audibil- ity, we devise Reconstructive Listening Enhance- ment (RLE) in Figure (b) which enhances audio representations by establishing a reconstruction up- per bound to connect audio with its surrounding information. We alternately train DLM with SAL and RLE to fully utilize video and audio modalities.
u = [ullv]] E LLd
(1)
where W E RR228 I 4096) xd d-dimensional em- bbeder and [-II-] denotes concatenation. For the text features, we tokenize all the text inputs (i.e., h, a²"a"" into a series of WordPieces (Wu et al., 2016) using the T5 Transformer (Raffel et al., 2020), such that word token representations are obtained on top of relative positional embed- dings and layer normalization (Ba al., 2016). Thus, the formal definitions of text features are as follows: history E A question E Rlyxd and answer a E glaxd, where Ln, Lq and L, are the numbers of tokens of each text".
4.2 Dialogue Language Model
Our proposed HEAR framework is performed a model-agnostic manner, such that we first de- fine Dialogue Language Model (DLM) as gen- eral VGD system. For input audio, video, and texts (i.e., history and question). DLM is trained
4.1 Input representations
We formally define input feature representations of h, d", and a" by embedding them into d- dimensional space. For the video embedding, use 13D model (Carreira and Zisserman, 2017) pre-
'Smmle rate modulated to be aligned with video frames. "We delete superscripts r in the notations for the simplicity.
Here, we refer to the general VVG system' as DLM.

---

generate next-word tokens for answer sentences a" =aff,- amn under cross-entropy loss as:
a Sarwond Way
Andiveniated quantices i wfthout wordd in the welnst
Is themonny background nels?
Is the wasking mathine on?
Can you heur any music?
W buppess bebwees these two people?
there
Are there valse?
guetiess
Does be aay thing during the video?
How old you be
(f) = log II Polafluu h, q. a<i), (2)
Aay talking sounds
what her jacket?
bu Kegwonb heod acalle umiing:
Predicted ondio related quesions
du Outilerwo bnyae vadio weoiing
where 0 is learnable weights and is the num- ber of word tokens in the answer sentence. In the following, our proposed SAL and RLE improve the DLM's sensibility and audibility for correctly responding to questions about audio.
Figure 4: Examples of questions: (a) Predicted audio- related questions by keyword- audio sensing and (b) Outliers of keyword-based audio sensing.
keyword-based audio sensing. While the keyword- based approach effectively identifies the questions that directly require information about audio, we were still able to identify outliers that were not accurately identified in Figure 4 (b). Although they do not contain keywords, the meanings of the questions are related to audio. Therefore, we further devise a semantic neural estimator So that identifies the audio-related question based on the meaning of sentences. The semantic neural esti- mator as a BERT-based classifier takes { [wets w/} to form an input instance, where Wels is token for classification and the prediction target of Wets is y E {wo, #1} denoting that y1 1 is audio- related question and yo We first train the fo with training weights e un- der L2 loss given as Ep 9)² where D is the dataset, á = fo (Gess wyy) is prediction score of the sigmoid (i.e., 0 < ÿ < 1). For labeling the y, we first include the predictions by keyword- based audio sensing as noisy label. We further include yo with (1) word-wise random shuffled ver- sions of audio-related questions and (2) random swapping versions of non-audio questions using keyword Wkey This prevents fo from simply pre- dicting based on the keywords and makes more fo- cus on the meaning of the audio-related questions in V1- After training for we calibrate the audio and video features according to the estimation of fø as:
4.3 Sensible Audio Listening
Sensible Audio Listening (SAL) is devised deter- mine whether the VGD system should listen to the audio answer given question. If the SAL deter- mines that listening is required, audio processed to be more weighted in the response. To ensure sen- sible decision-making within SAL, we introduce two technical decision rules: (1) seyword-basedd Audio Sensing and (2) Semantic Neural Estimator.
Keyword-based Audio Sensing. In many cases, unlike general questions, audio-related questions contain specific keywords (e.g.. 'sound', 'speech', 'listen') that indicate that the question concerns audio. Therefore, we conduct empirical investi- gation of these keywords. If any of these keywords are present in question, SAL identifies as an audio-related question. such cases, we mask the vidco features in the inputs, directing more atten- tion toward the audio component as given below:
is the other question.
[ullv]W V wy & Wkey [[l[mm]]W
u (
(3)
otherwise,
where iy is all word tokens in the question q and W Wkey = ('sound', 'speech', 'listen'.... } is key- word set to investigate audio-related questions. my E RLL40006 is zero padding on video. Here, we do not perform any masking when the question is not an audio-related question (i.e., Vw, Wkey). because other questions excluding audio-related questions do not show high reliance on specific modality. Thus, u' (q) L is sensible audio- visual features that make DLM selectively focus on audio for responding to the audio-related questions.
u (4) = [r x ull(1 -r) x vjW,
u
(4)
where = fo (Wels, my) R is estimation score about audio-related question and u² (2) E Lxd is our final sensible audio-visual features for SAL. Based on the sensible audio-visual feature u (q) from SAL, we train Dialogue Language Model (DLM) with cross-entropy loss as given below:
m
CSAL (0) log II
Polaf u (g) h q. act). (5)
Semantic Neural Estimator. Figure& 4 (a) il- lustrates audio-related questions identified by
AAppendix provides all the keywords that we used.
'Rogularization is also used to mitigate training imhalance.

---

Trariig
upper
Vitaa
distance
<poc
Grpa thperparanters
Htpe aurrernding reaskingg dittaee behdsiiing
Figure 6: Hyperbolic function for distance scheduling.
(b) Minimizing Reconstruction Upper Bound using surtvunding masking
(x) Reconstructive Listening Enhancement
compared to when it is not given as given below:
Figure 5: Illustrations of surrounding masking. The distance decides the extent of surrounding masking.
Car (0) < (u
um I mm v h, q):
(7)
4.4
Reconstructive Listening Enhancement
where the audio reconstruction loss based on sur- roundings Car (0) = ho uum Um V, h, q) should be lower than the reconstruction loss without sur- roundings 2 (f) ho [mm | mm ", h, ④ (i.e., upper bound). As shown in Figure 5 (a), u v audio and video features that further mask out surroundings up feature distance (e.g., n n= 3) from the target audio features. To train DLM conform this inequality condition, we calculate a reconstruction upper bound loss L in a format of ranking loss with margin 5 as given below:
For a better understanding of audio, it is crucial to improve audio representations based on common understanding of scene in a video. Therefore, our proposed Reconstructive Listening Enhancement (RLE) performs masked audio reconstruction by referring surrounding information (i.e., video and audio adjacent masked audio), which en- hances the common embedding with surround- ings. Especially to perform effective enhancement in regions closer to the masked target, we propose Reconstruction Upper Bound in the following.
Lrub(0, n)) = max((arr (0) C (0) + 8,0).
(8)
Audio Reconstruction. Audio reconstruction aims to predict masked audio based on observa- tions of their surrounding audio and other modal- ities (i.e., video and text). We randomly select input audio with a probability of p (e.g., p=10%) Ums where Um E M is the target audio features to be masked, m the set of indices for masking, and M is the number of indices. We also define u, mm E IR 121128 as surrounding audio features including masked features of zero vectors in the indices m. We introduce audio reconstruc- tion loss Car for DLM reconstruct target audio features um from the inputs of (umm- v., h q) as:
After that, we iteratively minimize the upper bound L in way of narrowing the surrounding masking with distance n² as shown in Figure 5 (b). Here, we construct masking distance scheduling g(e) according taring epoch to select progressively lower n. from higher n Therefore, the final objective of RLE is formulated as below:
min Lar (0) + Crub (O)g( 0.gm.G
(9)
where G .t+ R is set of scheduling func- tions and we select hyperbolic function as g(e) round (a - E) + as depicted Figure 6 to give stable decreasing of n with a Zim 1 Vieas and
Lar (0) ho (Um U mm v, h, q)-
(6)
(u m v, q) M regression, where um E IR M x288 is reconstructed audio on the masked indices m from the output of multi-layer perceptron on top of DLM encoder.
@
is
Tmux= =5, 115 maximum distance and epoch. Therefore, the final RLE loss the summation of two losses: CRLE = Lar + L
4.5 Optimization and Inference
Reconstruction Upper Bound. The audio recon- struction should be based on understanding of surrounding semantics rather than simply memo- rizing audio. To facilitate this, we propose Recon- struction Upper Bound (RUB) which establishes an inequality condition to ensure enhanced reconstruc- tion when the surrounding information is provided
As shown in Figure 3, we train DLM by alternately optimizing LSAL and LRLE to allow audio and
"As the surrounding masking reduced, the DLM perform the reconstruction better, thus CG decreases.
'Empirically found that low in early training (e.g.. Sie) 1) ruins training stability, as Car, L2, were almost similar. Sce more scheduling functions in Appendix B.2.

---

AVSD@DSTC
Methods
B1
B2
B3
B4
M
R
C
BE-DMN (Lin et al., 2019)
0.641
0.493
0.388
0.310
0.241
0.527
0.912
JMAN (Chu et al., 2020)
0.667
0.521
0.413
0.334
0.239
0.533
0.941
CMU (Sanabria al., 2019)
0.718
0.584
0.478
0.394
0.267
0.563
1.094
COST (Pham et al., 2022)
0.723
0.589
0.483
0.400
0.266
0.561
1.085
MSTN (Lee al., 2020)
0.377
0.275
0.566
1.115
JSTL (Hori et al., 2019b)
0.727
0.593
0.488
0.405
0.273
0.566
1.118
MIN" (Le et al., 2019)
0.731
0.597
0.490
0.406
0.271
0.564
1.127
MTN-P (Le and Chen, 2020)
0.750
0.619
0,514
0.427
0.280
0.580
1.189
VGNMN (Le et al., 2022)
0.429
0.278
0.578
1.188
SCGA (Kim et al., 2021)
0.745
0.622
0.517
0.430
0.285
0.578
1.201
RLM (Li al., 2021b)
0.765
0.643
0.543
0.459
0.294
0.606
1.308
PDC (Le et al., 2021)
0.770
0.653
0.539
0,449
0.292
0.606
1.295
THAM (Yoon et al., 2022c)
0.778
0.654
0.549
0.468
0.308
0.619
1.335
HEAR (SAL)
0.784
0.658
0.551
0.471
0.309
0.619
1.347
HEAR (SAL + RLE)
0.791
0.662
0.558
0.472
0.312
0.622
1.376
AVSD @ DSTC8
MDMN (Xie and lacobacci. 2020)
0.296
0.214
0.496
0.761
JMAN (Chu et al., 2020)
0.645
0.504
0.402
0.324
0.232
0.521
0.875
STSGR (Geng et al., 2021)
0.357
0.267
0.553
1.004
MSTN (Lee et al., 2020)
0.385
0.270
0.564
1.073
COST (Pham et al., 2022)
0.695
0.559
0.465
0.382
0.278
0.574
1.051
MTN-P (Le and Chen, 2020)
0.701
0.587
0.494
0,419
0.263
0.564
1.097
SCGA (Kim et al., 2021)
0.711
0.593
0.497
0.416
0.276
0.566
1.123
RLM (Li et al., 2021b)
0.746
0.626
0,528
0.445
0.286
0.598
1.240
PDC (Lee et al., 2021)
0.749
0.629
0.528
0.439
0.285
0.592
1,201
THAM (Yoon et al., 2022c)
0.764
0.641
0.538
0.455
0.301
0.610
1.304
HEAR (SAL)
0.767
0.646
0.541
0.459
0.302
0.612
1.324
HEAR (SAL + RLE)
0.777
0.656
0.553
0.465
0.307
0.618
1.359
Table Experimental results on AVSD@DSTC7 (test) and AVSD@DSTC8 (test) dataset. BLEU, M: METEOR, R: ROUGE-L, C: CIDEr, reported in (Kim et al., 2021)). Human Evaluation is also presented in Appendix C.
video be freely utilized for their purposes. There- fore, CSAL optimized when the training iteration number is odd, and LRLE does the even number:
tion and answer. The videos are collected from Cha- rades (Sigurdsson et al., 2016) dataset, which con- tains natural human activities. AVSD is released at Dialogue System Technology Challenge (DSTC) DSTC7 contains 7, 659, 1, 787, and 1, 710 dialogues for training, validation and test, but AVSD @ DSTCS only provides 1, 710 dialogues for the test. For the test set, 6 reference answers are available for accurate validation.
LSAL
iteration odd iteration even,
7 and 8, where AVSD@ DSTC
CHEAR
(10)
CRLE
Here, iLE only used for training. In inference, The DLM generates an answer with SAL.
5
Experiments
5.2 Metrics
5.1 Datasets
We follow official metrics for AVSD benchmark (i.e., BLEU, METEOR (Banerjee and Lavie, 2005), ROUGE-L, CIDEr) provided by AVSD challenge organizers 10 where they compute words overlaps between predicted sentence and reference answer.
AVSD@ DSTC7 and A DSTC8. (Audio Visual Scene-Aware Dialog) (Alamri et al., 2019; Hori et al., 2020) a popular benchmark for VGD task, where all videos include their summary cap-
tions and dialogue composed of 10 pairs of ques-

---

WVSD@DSTCR
Methods
B1
B2
B3
B4
METEOR
ROUGE-L
CIDEr
MTNT (Le et al., 2019)
0.357
0.241
0.173
0.128
0.162
0.355
1.249
MTN HEAR
0.382
0.264
0.193
0.144
0.179
0.374
1.314
RLM (L.i et al., 2021b)
0.765
0.643
0.543
0.459
0.294
0.606
1.308
RLM + HEAR
0.787
0.660
0.556
0.467
0.308
0.619
1.358
TSRLM (Yoon et al., 2022c)
0.767
0.644
0.542
0,461
0.296
0.608
1.311
T5RLM HEAR
0.791
0.662
0.558
0.472
0.312
0.622
1.376
AVSD @ DSTC8
MTN (Le et al., 2019)
0.689
0.571
0.470
0.404
0.251
0.551
1.049
MTN" + HEAR
0.714
0.596
0.496
0.421
0.271
0.569
1.121
RLM (Li et al., 2021b)
0.746
0.626
0.528
0.445
0,286
0.598
1.240
RLM HEAR
0.772
0.651
0.554
0.462
0.303
0.617
1.323
T5RLM (Yoon et al., 2022c)
0.749
0.631
0.529
0.445
0.290
0.600
1.263
T5RLM+ HEAR
0.777
0.656
0.553
0.465
0.307
0.618
1.359
Table 2: Experimental results on AVSD@DSTC7 (test) and AVSD@DSTCS (test) for applying HEAR on VGD runner models (B1: BLEU1, *: reconstruction-usse results, t: single reference results).
SAL
RLE
Response performances audio-related questions
BLEU1
ROUGE-L
CIDEr
1.7
k
8
Lar
Lrsb
1.6
0,302
0.348
1.367
1.5
1,402
0.310
0.355
1.427
1.4
2.00
0.316
0.366
1.434
0.329
0.375
1.467
1.3
0.311
0.357
1.415
1.2
0.341
0.384
1.492
1.1
Table 3: Ablation studies of model variants on DSTC (validation, single reference). k: Geysord-based based Audio Sensing, s: Semantic Neural Estimator, Car: Audio Reconstruction, Lrus: Recon- struction Upper Bound
(s) question
Bezmaiic Nax Edizatee) (Ko Auii Saning)
Audio-related questions (c) Audin-rraated quetions
HEAR (Durs)
Keyword
Figure 7: VGD systems' response performances on audio-related questions AVSD@DSTC7 (validation): (a) Total questions, (b) Audio-related questions pre- dicted by Semantic Neural Estimator (questions with estimation score r 0.7), (c) Audio-related questions predicted by Keyword-based Audio Sensing.
5.3 Results on AVSD benchmark
Table summarizes the experimental re- sults of HEAR on the AVSD@ DSTC7 and AVSD@DSTCB. HEAR shows state-of-the-ard performances on all the metries compared to previous works (i.e., please refer to Related Work for their detailed captions.). Our baseline DLM is T5 Transformer (Raffel et al., 2020), which is the same baseline (i.e., T5RLM) of THAM (Yoon et al., 2022c), but here, our proposed SAL shows more gains, and further improvements are also shown by applying RLE. As proposed HEAR is performed in model-agnostic manner, also validate other VGD models with HEAR in Table 2. We utilize the public codes and papers for MTN, RLM, and T5RLM, where steady gains in all metrics are shown for all the models.
5.4 Ablation Study
Table summarizes the ablative results of the pro- posed modules in HEAR framework. The first section of Table 3 is about our base DLM perfor- mances. In the second section, for the variants SAL, smaller gains are obtained when only using keyword- based audio sensing. We think that us- ing it alone was not beneficial in answering some audio-related questions that require referencing the videos, as this method unconditionally screens out video features as long as the given questions are lated to audio. the case of RLE, the audio recon- struction loss Lar generally gives a positive effect on the system performances. However, the recon-

---

Video (V)
Audio
(Punk music sound) 'r
[ (a0e
Dialogue History (H)
person watching someithing on their lepoop ploced the bitchen counter: They then open the window cleaning
wooh
Capton
Iransonatiinn Upper (eundllse
Leee any sound the video?
Figure 8: Ablative results on validation losses according to epochs: (a) SAL loss LSAL with and without RLE loss LRLE. (b) Audio reconstruction loss Car with and without reconstruction upper bound loss Lrab-
Yes, there is modio. person plays 77 music video from his laptop.
Question (Q)
Q What gatre playnng
A² HEAR Pred: can't be sure, but i think is punk music,
GT Here, think 2000 's punk rock song.
struction upper bound loss Crub becomes effective only when the Car and Cyub are used together. This denotes that the ranking loss in the Grub is founded on well-reconstruuted audio features.
Figure 9: HEAR responses audio-related question.
Questions
S
(a) Can you hear any sounds?
T
0.99
T
One may wonder if HEAR really helps answer audio-related questions. Figure 7 shows the re- sponse performances terms of VGD systems including HEAR according to questions. HEAR performs new state-of-the-art performances on top of previous runner models, but it is also meaningful that the gains are mainly obtained from improving responses to audio-related questions. From the results in Figure 7 (b), (c) and Table 2, it is con- cluded that HEAR framework can be applied to any VGD system, where exclusively contributes to improving systems' audibility and generating correct responses to given audio-related questions. Furthermore, we found that our proposed RLE also contributes to the efficient learning of sensible audio listeming (i.e., optimizing LSAL). Figure 8 (a) shows validation losses (LsA)) according to training epochs, where it can be seen that the AL. is further reduced when RLE loss (CRLE) is ap- plied. Figure 8 (b) shows the ablation studies of audio reconstruction loss Car with and without re- construction upper bound Crub- The yellow curve shows the upper bound loss Cy(e) for Crub, where the it decreases according to our masking distance scheduling g(c)" 11, The green curve denotes the Car with Lrub and it shows further optimizing com- ge()) pared without Crub as the Car decreases. This
(b) Do they speak each other?
T
0.99
T
(c) Who outside the door?
F
0.77
T
(d) Is the vacuun cleaner working?
F
0.74
T
(c) Can you tell where goes?
0.32
What color hair?
0.01
Table 4: Predictions on questions about audio related- ness. Keyword-based Audio Sensing, S: Semantic Neural Estimator, H: human rating, T: true, F: false.
5.5 Qualitative Results
Figure 9 illlustrates the HEAR's responses to audio- related question. HEAR is given the question "what genre of music is playing?" question, and it generates the answer sentence "I can't be sure, but T think it's punk music." Here, HEAR precisely predicted that the given audio was punk music despite the challenging work of dis- cerning what the sound is in the video. The more interesting fact is that HEAR represents its opinion as "I can't 't be sure" about predicting music. When listened that audio in Figure 9 (a), it was really quiet and difficult to identify what the sound is. We guess the HEAR also learned the knowledge about which sounds are difficult for humans to dis- tinguish. Table 5 shows predictions on questions by Keyword-based Audio Sensing and Semantic Neural Estimator. As the keyword based approach could not understand the meaning of the question, it shows incorrectness in some questions (i.e., (c,d,e)). 1
the audio-related
denotes that neural networks can be further opti- mized according to the training epochs by calibrat- ing their training objectives, which is also validated in other multi-modal systems (Yoon et al., 2023; Zheng et al., 2022) in other ways.
The neural estimator presents the score denoting whether the question is related to the au- dio, which provides proper distinctions between audio-related questions (i.e., (a,b,c,d)) and the oth- ers (i.e., (e.f)) based on the meaning of the question.
11 We use a hyperholic curve for g(e). Appendix also con- tains more diverse surrounding masking scheduling.

---

6 Conclusion
will also make the audio features (i.e., wav2vec 2.0 features) publicly available and perform a further study on this as our future work.
We propose Hearing Enhanced Audio Response (HEAR) framework for Video-grounded Dialogue systems. HEAR proposes Sensible Audio Listen- ing and Reconstructive Listening Enhancement, which solve the deaf response problem of VGD systems and validate model-agnostic effectiveness on top of current VGD systems.
Ethics Statement
Video-grounded Dialogue system is one of the con- versational Al agents, which designed to provide assistance to various subsections of our environ- ments including security, entertainment, education, and visual impairments. Our proposed Hearing Enhanced Audio Response framework contributes to improving responses to queries about audio by enhancing the audibility of VGD system. Recently chatbot systems (e-g., ChatGPT) has shown over- whelming performance, as such, we should also think about the potential negative societal impact of these systems. In this respect, we came up with two negative impacts: (1) unreliable vague informa- tion by conversational agents and (2) fairness issues in agents' responses. Therefore, word sense disam- biguation techniques (Yoon et al., 2022a) and multi- modal debiasing solutions (Yoon al., 2022b; Niu et al., 2021) should also be applied the dialogue systems.
Acknowledgements
This work was supported by Institute for Infor- mation & communications Technology Promo- tion(IITP) grant funded by the Korea govern- ment(MSIT) (No. 2021-0-01381, Development of Causal AI through Video Understanding and Reinforcement Learning, and Its Applications to Real Environments) and partly supported by grant of the KAIST-KT joint research project through AI2XL Laboratory, Institute of convergence Tech- nology, funded by KT [Project No. G01220646, Visual Dialogue System: Developing Visual and Language Capabilities for Al-Based Dialogue Sys- tems].
Limitations
References
Our research aims to enhance the audibility of Video-grounded Dialogue (VGD) systems, where we devise Hearing Enhanced Audio Response (HEAR) Framework. As limitation of our work, we are more focused on the methodology. We need have better understanding of the limitation of the proposed method to overcome any failure cases. As shown in the failure case in Appendix, our pro- posed method has limitations, and we would need to consider other architecture and training methods to incorporate all the necessary information that speech holds. To understand human speech, current audio features seem necessary to be trained more by large-scale audio speech recognition datasets (Panayotov et al., 2015; Garofolo, 1993). Further- more, although our proposed HEAR mitigates this problem in a methodological idea, we also think that the deaf problem can also be cured by expand- ing the audio feature dimension (i.e., 128) up to comparable scale with vidco (i.e., 4096), such that they include more detailed information. To be spe- cific, we are currently changing the current audio feature extractor (i.e., VGGish) into wav2vec 2.0 (Baevski et al., 2020), which can provide a larger di- mensional audio feature (i.e., 768 dimension). We
Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang, Anoop Cherian, Irfan Essa, Dhruv Batra, Tim K Marks, Chiori Hori, Peter Anderson, et al. 2019. Audio visual scene-aware dialog. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7558-7567.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In The JEEE International Conference on Computer Vision (ICCV).
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin- ton. 2016. Layer normalization. arXiv preprint
Alexei Baevski, Yuhao Zhou, AbdeIrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33:12449 12460.
Satanjeev Banerjec and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved cor- relation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summariza- tion, pages 65-72.

---

Joao Carreira and Andrew Zisserman. 2017. Quo vadis. action recognition? a new model and the kinetics dataset. In proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 6299-6308.
2017. The kinetics human action video dataset. arXiv preprint 1rXiv:1055006950
Junyeong Kim, Sunjae Yoon, Dahyun Kim, and Chang D Yoo. 2021. Structured eo- co-reference graph attention for video- deleodgrounded dialogue. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 35, pages 17977
Yun- Wei Chu. Kuan-Yeu Lin, Chao-Chun Hsu, and Lun- Wei Ku. 2020. Multi-step modtltoyl attention network for scene-aware dialogue system. In DSTC8 at AAA12020 workshop.
Hung Le, Nancy Chen, and Steven Hoi. 2022. Vgnmn: Video-grounded neural module networks for video- grounded dialogue systems.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lec, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint 1xXiv:1810.04800
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 3377-3393.
John S Garofolo. 1993. Timit acoustic phonetic con- tinuous speech corpus. Linguistic Data Consortium, 1993.
Hung Le and Nancy F. Chen. 2020. Multimodal trans- former with pointer network for the dstc8. In DSTC8 at AAA12020 workshop.
Jort F Gemmeke, Daniel PW Bllis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. 2017. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 786-780. IEEE.
Hung Le, Nancy F Chen, and Steven CH Hoi. 2021. Learning reasoning paths over semantic graphs for video-grounded dialogues. arXiv preprint arXiv:2103.00820
Hung Le. Doyen Sahoo, Nancy Chen, and Steven C.H. Hoi. 2019. Multimodal transformer networks for end- to-end video-grounded dialogue systems. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL).
Shijie Geng, Peng Gao, Moitreya Chatterjee, Chiori Hori, Jonathan Le Roux, Yongfeng Zhang. Hong- sheng Li, and Anoop Cherian. 2021. Dynamic graph representation learning for video dialog via multi- modal shuffled transformers. In Proceedings of the AAA/ Conference on Artificial Intelligence, vol- ume 35, pages 1415-1423.
Hwanhee Lee, Seunghyun Yoon, Franck Dernoncourt, Doo Soon Kim, Trung Bui, and Kyomin Jung. 2020. Dstc8-avsd: Multimodal semantic transformer net- work with retrieval style word generator. In DSTC8 at AAA12020 workshop.
Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort Gemmeke, Aren Jansen, R Channing Moore. Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Sey- bold, et al. 2017. Can architectures for large-scale audio classification. In 2017 ieee international con- ference on acoustics, speech and signal processing (icassp), pages 13 135. IEEF.
Zekang Li, Zongjia Li. Jinchao Zhang, Yang Feng, and Jie Zhou. 2021a. Bridging text and video: A univer- sal multimodal transformer for audio-visual scene- aware dialog. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:2476-2483.
C Hori, A Cherian, T Hori, and TK Marks. 2020. Au- dio visual scene-aware dialog (avsd) track for natural language generation in dstc8, In AAAI-DSCC8 Work- shop.
Zekang Li, Zongjia Li, Jinchao Zhang, Yang Feng, and Jie Zhou. 2021b. Bridging text and video: A univer- sal multimodal transformer for audio-visual scene- aware dialog. IEEE/ACM Transactions Audio, Speech, and Language Processing, 29:2476-246
Chiori Hori, Huda Alamri, Jue Wang. Gordon Wich- ern, Takaaki Hori, Anoop Cherian, Tim K. Marks, Vincent Cartillier, Raphael Gontijo Lopes, Abhishek Das, Irfan Essa, Dhruv Batra, and Devi Parikh. 2019a. End-to-end audio visual scene-aware dialog using multimodal attention-based video features. IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
Chin-Yer Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74 81.
Kuan-Yert Lin, Chao-Chun Hsu, Yun-Nung Chen, and Lun-Wei Ku. 2019. Entropy-enhanced multimodal attention model for scene-aware dialogue generation. In DSTC7 at AAA12019 workshop.
Chiori Hori, Anoop Cherian, Tim K. Marks, and Takaaki Hori. 2019b. Joint student-teacher learning for audio-visual scene-aware dialog. In Proceedings of the Interspeech.
Ilya Loshchilov and Frank Hutter. 2017. pled weight decay regularization. arXiv preprint wrXiv:1711.05101.
Decou-
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier. Sudheendra Vjjayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al.
Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian-Sher Sheng Hua, and Ji-Rong Wen. 2021. Counter- factual vqa: A cause-effect look at language bias. In

---

Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 12700- 12710.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe Le, Mohammad Norouzi. Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, al. 2016. Google's neural machine translation system: Bridging the gap between human and machine trans- lation. arXiv preprint arXiv: 1609001144
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an ast cor- pus based on public domain audio books. In 2015 TEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206-5210. IEEE.
Huiyuan Xie and Ignacio Iacobacci. 2020. Audio visual scene-awan dialog system using dynamic memory networks. In DSTC8 at AAA12020 workshop.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th anmual meeting of the Association for Computa- tional Linguistics, pages 31 1-318.
Hee Suk Yoon, Eunseop Yoon, John Harvill, Sun- jac Yoon, Mark Hasegawa-Johnson and Chang D Yoo. 2022a. Smsmix: mannsmanint sentence mixup for word sense disambiguation. arXiv preprint arXiv:2212.07072
Hoang-Anh Pham, Thao Minh Le, Vuong Le. Tu Minh Phuong, and Truyen Tran. 2022. Video dialog as conversation about objects living in space-time. In European Conference Computer Vision, pages 710-726. Springer.
Sunjae Yoon, Ji Woo Hong. Eunscop Yoon, Dahyun Kim, Junyeong Kim, Hee Suk Yoon, and Chang D Yoo. 2022b. Selective query-guided debiasing for video corpus moment retrieval. In European Confer- ence on Computer Vision, pages 185- 2000. Springer.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language under- standing by generative pre-training.
Sunjae Yoon, Gwanhyeong Koo, Dahyun Kim, and Chang D Yoo. 2023. Scanet: Scene complexity network for weakly-supervised video moment retrieval. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 13576- 13586.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, 1lya Sutskever, 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9.
Sunjae Yoon, Eunseop Yoon, Hee Suk Yoon, Junyeong Kim, and Chang Yoo. 2022c. Information-theoree text hallucination reduction for video-grounded di- alogue. In Proceedingy of the 2022 Conference on Empirical Methods in Natural Language Processing, pages
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 21(140):1-61
Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Ramon Sanabria, Shruti Palaskar, and Florian Metze. 2019. Cmu sinbad's submissino for the dstc7 avsd challenge. In DSTC7 at AAAI2019 workshop.
Minghang Zheng, Yanjie Huang, Qingchao Chen, Yuxin Peng. and Yang Liu. 2022. Weakly supervised tem- poral sentence grounding with gaussian- based con- trastive proposal learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pat- tern Recognition, pages 15555-15664
Gunnar A Sigurdsson. Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. 2016. Hol- lywood in homes: Crowdsourcing data collection for activity understanding. In European Conference on Computer Vision, pages 510-526. Springer.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devin Parikh. 2015. Cider: Consensus-basse image de- scription evaluation. Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages
Wenhui Wang, Hangbo Bao, Li Dong. Johan Bjorck, Zhiliang Peng. Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. 2022. Image foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint 2208. 10442.

---

A Experimental Details
1.0%
60%
Training. Proposed HEAR trained on NVIDIA
20.9%
6.9%
Quadro RTX 8000 (48GB of memory) GPU.
The optimization details are as follows. The
10.1%
Adam W optimizer (Loshchilov and Hutter, 2017) is used with parameters as 31 = 0.9, f2 0.999,
18.2%
14.0%
and 10c-8. Due to the efficient usage of re- sources. the previous question-answer pairs (i.e.. (Q--2,44 2) (Q"=),
18.3%
{C,
talk hsar sound
noise andio speak etc
as a dialogue history are introduced into HEAR for answering current questions Q" For the learning rate, we first set as Ir 6.24e - 5 and it linearly decreases following a piecewise linear curve up to Ir = 3.63e 10 and the model is trained the model during 15 epochs. The hyperparameters are 8 0.05 for the margin in CRLE For the joint d-dimensional space, all modalities (i.e., video, audio, words) are embedded space. The best model is decided by the lowest loss of the validation set on AVSD@DSTC7 (i.e., AVSD@ DSTC8 contains only test set for evaluation.). It takes 14 hours to finish the training and the model is fully optimized in about 10 hours.
Prupartion audio-related axsttooss keywurds
Figure 10: Chart representing the proportion of audio-related questions according keywords in W Wkey
Number keywords
BLEU
ROUGE-L
CIDEr
17
0.327
0.367
1,458
18
0.333
0.368
1.466
19
0.335
0.371
1.469
768 dimensional
20
0.331
0.366
1.468
21
0.324
0.363
1.465
22
0.321
0.361
1.460
Table 5: Performance variations according num- ber of keywords for Keyword-based Audio Sensing AVSD@ DSTC7 validation set
Inference. In the inference, answer word tokens are generated in a sequence with a probability, where the beam search applied avoid prompt word selection with a beam size of 5. The max length for the answer word tokens is set to 20 with a length penalty of 0.3. In the Experiment of the main paper, every performance of HEAR is aver- aged 15 times with random seed number.
tion of audio-related questions according cach keyword, where 'talk", 'hear', and 'sound' are the most related words to the audio questions. We ad- ditionally considered more keywords (e.g., 'tell', 'background'), but they rather degrade the motiva- tion of keyword- -bseed audio sensing by finding other questions (e.g., 'can you tell me..', 'can you something in the background') excluding audio- related questions. To be specific, we extend the evaluations for SAL in terms of Keyword- basee Audio Sensing (Table A). We investigate the per- formance variations according to the quantity audio-related keywords utilized, The results show positive correlation between the number of kcy- words and the enhancement. However, surpassing specific number (i.e., 19) leads to reversal in this trend, revealing detrimental correlation. This implies that the pool of keywords starts to include words that lack sensibility in effectively categoriz- ing audio-related questions. Thus, we employ 19 keywords for SAL listed in Appendix B.
B Additional Experiments
To improve the reproducibility of proposed mod- ules (i.e., SAL, RLE) in HEAR, we give detailed explanations about them with additional results and illustrations of the method.
B.1 Keyword lists for keyword-based audio sensing
For details of Scction 4.3, we list the keywords that we used for keyword-based audio sensing as given below:
W Wkey (noise, sound, voice, speech,
speak, talk, listen, hear, say, sing, music, audio, call, hum, loud, tones, utter, volume, song),
(11)
B.2 Surrounding Masking Scheduling
For the reconstruction upper bound in Section 4.4. we provide further ablation studies about distance scheduling for surrounding masking. The surround- ing mask is designed for promoting ranking loss
where we also considered all the keywords' plural forms in W Wkey- key- Figure 10 summarizes the propor-

---

NTmxx (1
))
+1
round (alog (e)) mmax
round @max e)
Fmax
Rynax
Amax 1
log (fmax
a
v Vemax
dasane 90
(pcch
spoch (c)
Faxax
(a] Distance scheduling (linear curve)
Distance scheduling (logistic
Distanca scheduling (hyparbolic curve)
lpper bound
Audio recondrracion los
Audio recorsinction loss
Upper bound with
Upper bound with
lnnarr curve schotulin
053
now
bypecho curee schedrigg
0.5
0.55
6:5
0.33
655
Audooroonsrutttnn loss
Upper bound with loghtk carve sthollling
044
spech
epech
epoch
(4) Audio reconstruction losses with Rocostrruction Upper Sound (linar curve shccuduling)
Audio reconstruction losses Reconstruction
(1) Audio reconstruction losses Reconstruction
Upper Bound (logistic curve scheduling)
Upper Bound (hyperbolic curve scheduling!
Figure 11: Illustrations of distance scheduling for surrounding masking. The distance decides how much mask video and audio features for establishing upper bound e (a) linear curve, (b) logistic curve, (c) hyperbolic curve. Audio reconstruction losses with reconstruction upper bound using (d) linear curve scheduling, (e) logistic curve scheduling, (f) hyperbolic curve scheduling,
Crub by ensuring that the audio reconstruction far based on the surrounding information has an im- proved quality than audio reconstruction Pür with- out the surrounding information. To make more effective frub, C can be designed to minimize its loss by By reducing the distance n of the surround- ing mask. To this end, we leverage the extent of the surrounding mask under our designed various mod- elings n = g() for masking scheduling Figure 11. We narrowed down the extent of surrounding masking based on three different curves: linear curve, logistic curve, and hyperbolic curve. The hy- perbolic curve and linear curve show effectiveness in optimizing validation loss. However, the logistic curve shows some deterioration in the optimization. We think this is because the logistic curve makes the surrounding masking applied to a very narrow area from the beginning of training, which acts like a hard negative that is almost similar to the posi- tive reconstruction (i.e., reconstruction from only masked target audio). Providing hard negatives in early training is considered to hinder optimization when the learnable weights of the model are not properly trained. Therefore, surrounding masking with hyperbolic scheduling is the most effective.
Tmmax
BLEU1
ROUGE-L
CIDEr
2
0.310
0.362
1.436
3
0.348
0.371
1.475
4
0.341
0.384
1.492
5
0.316
0.369
1.438
0.314
0.368
1.436
Table Ablation study on the distance of negative masking of HEAR valid split of DSSD.
AV @ DSTC7 under distance scheduling with the hyperbolic curve. When 7amax is small (e.g., 7max 2 or 3). Für makes audio reconstruction based the nearest video and audio from the target au- dio. This can be effective in terms of improving the connectivity of neighboring video and audio at a narrow distance, however since the distance is too close, there is no significant difference from the positive audio reconstruction, which hinders reconstruction. Based on results in Table 6, neg- ative masking was effective when Amaa = 4 or 5. In this case, the surrounding modalities' features for audio reconstruction were properly removed by masking, in the meanwhile the surroundings were not masked excessively. Thus the masking contributes on hard negative audio reconstruction. When Rexax is large (e.g., Pimas = 6), the perfor-
Table 6 summarizes the results of HEAR ac- cording to the distance n on the validation split of

---

Type HEAR
THAM
RLM
GT
Video (V)
semantic 2.41
2.15
1.81
2.86
grammatical
2.35
2.24
2.16
2.76
fluency 2.62
2.58
2.41
2.81
Audio
(TV sounds)
Table 7: Human evaluation responses to 50 audio- related questions with respect to semantic adequacy, grammatical correctness, fluency. The scores as- signed as ("1: not at all", "2: neutral", "3: correct"), GT: Ground- -runh response.
Dialogga History (H)
Guy ditting aod and tume of with the sumotr cuntrol, He and ques wisttver the floor to cnanzine it.
How manyy people the room?
(1 heme is jost one man theme.
Qusstoon (Q)
Q
mance again degrades. For this case guess that the negative reconstruction is quite different from the positive, so would not be an effective nega- tive. These studies were also conducted for larger maxx but no further improvement was confirmed.
A" HEAR Pred; Yes, can hear the tv.
GT Yes, you can hear the
Figure 12: Illustration of additional results of HEAR
Vidoo (V)
C Human Evaluation
To assess the improvement of HEAR framework in the responses audio-related questions, duct Human Evaluation. We selected a total of
Audio (U)
(Huuman speech)
Dialogue History (H)
50 audio-related questions and generate their re- sponses from models (i.e., HEAR, THAM, RLM). We evaluate all responses based on scale of to 3, considering three key perspectives: semantic adequacy, grammatical correctness, and fluency, Each score denotes that "1: not at all", "2: neu- tral", "3: correct". Table shows the human eval- uation scores based on 11 evaluators. Evaluators the responses based on three categories, where HEAR outperformed recent runner models all three categories, receiving higher ratings. Further- more, through empirical validation, we have con- firmed the presence of inappropriate answer re- sponses within the Ground- Truth. As result, the human evaluation scores appropriately reflect this by not consistently obtaining score of 3.0 value close to it.
starding lagrop, mesting, ditiigg plastng
caplicn
canting.
0²
backgrourd
Judk pesson talling
Question (Q)
Q' you understand speech, said?)
A' HEAR Pred: No, i can 't understand the speech.
GT The person saying 1560 and 25
Figure 13: Illustration of failure case of HEAR
itations in accurately comprehending the language within audio inputs. Our empirical studies found that this challenge is prevalent among various di- alogue language models, including our model. It is noted that the current audio feature used the HEAR framework is pre-trained with the environ- mental sounds dataset, such as AudioSet (Gem- meke et al., 2017). However, it appears that further data training is necessary to enhance the system's ability to understand speech effectively. To address this challenge, our future work will focus in- corporating audio features, such as wav2vec 2.0 (Baevski et al., 2020), which have been trained specifically for audio speech recognition tasks. By leveraging these advanced audio features, we aim to enhance the model's ability to accurately under- stand and process speech-related information.
D Additional results and Failure case
Figure 12 illustrates additional results of HEAR framework. When presented with the question, "Can you hear the TV?" our HEAR provides an affirmative response, stating, "Yes, can hear the TV." This response is generated by considering both the audio input of the TV and the visual con- of the TV image. Although our proposed HEAR improves the understanding of audio, there were instances of failure when comes to ques- tions pertaining to speech recognition. As shown in Figure 13, the HEAR framework demonstrates lim-


Reviews:
[{"review_id": "cpq8dPBfos", "reviewer": "Reviewer_c3vW", "summary": "", "questions": "The training process of SAL is not very clear, is it trained alone and freezed?\nThe Keyword-based Audio Sensing mentioned in the SAL seems to serve as a method for collecting training data of Semantic Neural Estimator, is that true? \nThe training data for  Semantic Neural Estimator is pure synthetic, without any human annotation?\nWhat is the difference between SAL and Semantic Neural Estimator?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer c3vW, We address your comments and questions below. \n\n**[Q1] Evaluation for SAL is limited.**\n\n**[A1]** We extend the evaluations for SAL in terms of Keyword-based Audio Sensing (Table A) and Semantic Neural Estimators (Table B). We investigate the performance variations according to the quantity of audio-related keywords utilized. The results show a positive correlation between the number of keywords and the enhancement. However, surpassing a specific number (i.e., 19) leads to a reversal in this trend, revealing a detrimental correlation. This implies that the pool of keywords starts to include words that lack sensibility in effectively categorizing audio-related questions. Thus, we employ 19 keywords for SAL listed in Appendix B.1. Additionally, Figure 7 in the main paper also provides a qualitative demonstration of effectiveness on audio-related questions according to both the Keyword-based Audio Sensing and Semantic Neural Estimator.\n\n**[Q2] The improvement from SAL is very marginal as shown in Table 1.**\n\n**[A2]**\n- The improvement from SAL is not marginal. For the metrics (B1,B2,B3,B4,M,R,C) on AVSD@DSTC7 and 8 in Table 1, over 25 runs with p-value <0.05, our average increments (+7.2,+6.3,+3.6,+3.4,+2.2,+2.0,+23.6)x10^(-3) of HEAR (SAL) validates statistical significance from the previous SOTA model. \n- Moreover, SAL is one of the components in the HEAR framework. Our proposed HEAR framework contains two main components: (1) Sensible Audio Listening (SAL) and (2) Reconstructive Listening Enhancement (RLE). When considering the entire HEAR framework (SAL + RLE), the overall improvement is substantially more pronounced, as evident in both Table 1 and Table 3 in the main paper. Additionally, the gains are primarily centered around audio-related questions, as depicted in Figure 7. With these factors in mind, we politely request reviewer c3vW to reconsider the assessment, taking into account the combined contributions of Reconstructive Listening Enhancement.\n\n**[Q3] It is unclear what causes the deaf problem of VGD system.**\n\n**[A3]** Many questions (about 76%) in the training dataset are based on visual information in a video, which causes a current VGD system to mainly rely on a video modality as a bias and is attributed to the key factor contributing to the deaf problem.\n\n**[Q4]  It is unclear if the HEAR framework is helping or just being a rule that helps the VGD system remember the answer for certain types of questions.**\n\n**[A4]** In order to discern whether the HEAR framework offers assistance or merely facilitates answer memorization for specific question types, such as audio-related questions, we conducted the following experiments. We first intentionally designed a memorizing language model that memorizes the answers to audio-related questions. To build this memorizing model, we provide only a question as a model input without other modalities when the input question is audio-related, after that we train the model to predict the answer. This makes a shortcut to memorize answers corresponding to the input questions due to the deficiency of clues to answer. In Table C, we present response evaluation on audio-related questions about (1) HEAR (Ours) with ground-truth, (2) the memorizing model with ground-truth, and (3) the memorizing model with HEAR predictions. The results of (2) denote that memorizing the answer sentences in the training dataset is not effective for answering in the validation set. The results of (3) show how similar the responses are between HEAR and the memorizing model, where the low scores denote that our model does not follow the way of memorizing answers in responding.\n\n**[Q5] The training process of SAL is not very clear, is it trained alone and frozen?**\n \n**[A5]** We train a dialogue language model by alternately optimizing SAL and RLE to allow audio and video to be freely utilized for their purposes. (i.e., Line 366-368). Thus SAL is applied when the number of training iterations is odd and RLE is applied when the iteration number is even as given below:\n\n$\nL_{HEAR} = \\begin{cases}\n    \\displaylines{ L_{SAL}, & \\text{iteration} = odd \\\\\\ L_{RLE},  & \\text{iteration} = even}\n  \\end{cases}\n$\n\n In the case of the Semantic Neural Estimator in SAL, it is frozen after pre-training.\n\n**[Q6] The Keyword-based Audio Sensing mentioned in the SAL seems to serve as a method for collecting training data of the Semantic Neural Estimator, is that true?**\n\n**[A6]** Yes, it is true. However, to prevent the estimator from overfitting to the predictions of Keyword-based Audio Sensing (KAS), we have taken additional precautions. We further augment predictions of KAS to include samples of non-audio-related questions by applying (1) word-wise random shuffled versions of audio-related questions of KAS and (2) random swapping versions of non-audio questions using the keywords. (i.e., Line 280-284) This ensures a more robust training process for the estimator and reduces its vulnerability to overfitting.\n\n**[Q7] The training data for Semantic Neural Estimator is pure synthetic, without any human annotation?**\n\n**[A7]** Yes, it is synthetic.\n\n**[Q8] What is the difference between SAL and Semantic Neural Estimator?**\n\n**[A8]** The Semantic Neural Estimator is one of our methods to perform Sensible Audio Listening (SAL).\n\n**[Q9] Front in figure 4 5 6 8 is too small.**\n\n**[A9]** We will make more space to present the figures in high resolution.\n\n\n***Table Specifications***\n\nBLEU1, ROUGE-L, CIDEr scores on AVSD@DSTC7 validation set\n\n\n**Table A.**\nKeyword-based Audio Sensing\n| number of keywords | BLEU 1 | ROUGE-L | CIDEr |\n|--------------------|--------|---------|-------|\n| 17                 | 0.327  | 0.367   | 1.458 |\n| 18                 | 0.333  | 0.368   | 1.466 |\n| 19                 | 0.335  | 0.371   | 1.469 |\n| 20                 | 0.331  | 0.366   | 1.468 |\n| 21                 | 0.324  | 0.363   | 1.465 |\n| 22                 | 0.321  | 0.361   | 1.46  |\n\n**Table B.**\nSemantic Neural Estimator. The keywords are used for training Semantic\nNeural Estimator.\n| number of keywords | BLEU 1 | ROUGE-L | CIDEr |\n|--------------------|--------|---------|-------|\n| 17                 | 0.339  | 0.382   | 1.489 |\n| 18                 | 0.34   | 0.383   | 1.49  |\n| 19                 | 0.341  | 0.384   | 1.492 |\n| 20                 | 0.339  | 0.385   | 1.489 |\n| 21                 | 0.339  | 0.381   | 1.483 |\n| 22                 | 0.337  | 0.379   | 1.483 |\n\n**Table C.**\nResponse evaluation on audio-related questions about HEAR and memorizing language model. BLEU1, ROUGE-L, CIDEr scores on AVSD@DSTC7 validation set\n\nHEAR: Our model\n\nMLM: Memorizing Language Model (Memorizing answer in training dataset)\n\nGT: ground-truth\n\n|| BLEU 1 | ROUGE-L | CIDEr |\n|----------------------------|--------|---------|-------|\n| Evaluation (1) (HEAR, GT)  | 0.372  | 0.398   | 1.661 |\n| Evaluation (2) (MLM, GT)   | 0.193  | 0.195   | 0.862 |\n| Evaluation (3) (MLM, HEAR) | 0.212  | 0.208   | 0.933 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed HEAR frameworks fits most VGD systems, improving their performance specifically on the audio related question while not hurting the performance of the other questions.", "reasons_to_reject": "The Sensible Audio Listening (SAL) proposed is confusing. \nEvaluation for SAL along is limited and the improvement from SAL is very marginal as shown in Table 1. \nIt is unclear what causes the deaf problem of VGD system and if the HEAR framework is helping or just being a rule that helps the VGD system remember the answer for certain type of questions.", "questions_to_authors": "The training process of SAL is not very clear, is it trained alone and freezed?\nThe Keyword-based Audio Sensing mentioned in the SAL seems to serve as a method for collecting training data of Semantic Neural Estimator, is that true? \nThe training data for  Semantic Neural Estimator is pure synthetic, without any human annotation?\nWhat is the difference between SAL and Semantic Neural Estimator?", "paper_topic_and_main_contributions": "The paper introduces HEAR framework to help making Video-grounded Dialogue systems better perceive and understand audio. HEAR framework is model-agnostic and experiments show that it improves the performance of VGD systems on audio-related questions.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Front in figure 4 5 6 8 is too small."}, "extra_text": {}}, {"review_id": "k79mKOoqM2", "reviewer": "Reviewer_9PUj", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 9PUj, We address your comments and questions below.\n\n**[Q1] The Audio Reconstruction task is very similar to the masked pre-training in NLP or masked auto-encoder in CV. In fact there are several publications on the two methods for audio data ([1][2][3]). Especially, the Audio-visual pre-training has been explored in [4]. Given the reconstructive listening enhancement (RLE) component plays an important role according to Table 3, how effective the proposed method is compared to other reconstruction pre-training methods needs to be validated.**\n\n**[A1]** Certainly, we acknowledge the significance of comparing our proposed method to other established audio reconstruction methods. In Table A, we present the results of this comparison, considering the approaches outlined in [1], [2], [3], and [4]. For the task of video-grounded dialogue, the reconstructions (i.e., RLE (Ours), [4]) based on multi-modalities (i.e., audio reconstruction based on audio and visual modality) are more effective than reconstruction (i.e., [1],[2],[3]) based on single audio modality. Furthermore, our proposed RLE stands out as even more effective than the reconstruction approach in [4]. This distinction underscores the significance of our proposed \"Reconstruction Upper Bound\" of RLE which contributes to the refined optimization of audio reconstruction throughout the training process by integrating contrastive learning objectives. Thank you for your attention to this matter. We will reflect on these results in the main paper.\n\n***Table Specifications***\n\nBLEU, METEOR, ROUGE-L, CIDEr scores on AVSD@DSTC7 test set\n\n[1] Huang, Po-Yao, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, and Christoph Feichtenhofer. \"Masked autoencoders that listen.\" Advances in Neural Information Processing Systems 35 (2022): 28708-28720. \n\n[2] Hsu, Wei-Ning, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. \"Hubert: Self-supervised speech representation learning by masked prediction of hidden units.\" IEEE/ACM Transactions on Audio, Speech, and Language Processing 29 (2021): 3451-3460. \n\n[3] Baevski, Alexei, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. \"wav2vec 2.0: A framework for self-supervised learning of speech representations.\" Advances in neural information processing systems 33 (2020): 12449-12460. \n\n[4] Shi, Bowen, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed. \"Learning audio-visual speech representation by masked multimodal cluster prediction.\" arXiv preprint arXiv:2201.02184 (2022).\n\n**Table A.** \nComparisons according to the audio reconstructions in work [1], [2], [3], [4]\n(*For the work [1], since the feature masking is not available to the frequency level, we omit the frequency masking. We consider that k-means clustering on MFCC in [2] seems not effective due to the environmental noise in the AVSD dataset.)\n\n|                                  | BLEU 1 | BLEU 2 | BLEU 3 | BLEU 4 | METEOR | ROUGE-L | CIDEr |\n|----------------------------------|--------|--------|--------|--------|--------|---------|-------|\n| HEAR + RLE                       | 0.791  | 0.662  | 0.558  | 0.472  | 0.312  | 0.622   | 1.376 |\n| HEAR + Audio Reconstruction [1]  | 0.787  | 0.657  | 0.552  | 0.47   | 0.309  | 0.619   | 1.358 |\n| HEAR + Audio Reconstruction [2]  | 0.764  | 0.646  | 0.545  | 0.451  | 0.291  | 0.603   | 1.311 |\n| HEAR + Audio Reconstruction [3]  | 0.785  | 0.657  | 0.553  | 0.469  | 0.309  | 0.618   | 1.349 |\n| HEAR + Audio Reconstruction [4]  | 0.789  | 0.659  | 0.556  | 0.47   | 0.31   | 0.619   | 1.363 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is the first to improved the video-grounded dialogue system by improving the audio modality.\n- The sensible audio listening (SAL) component explicitly indicate the model to focus on audio feature which achieves better performance compared to feeding both audio and video features to the transformer blocks.\n", "reasons_to_reject": "- The Audio Reconstruction task is very similar to the masked pre-training in NLP or masked auto-encoder in CV. In fact there are several publications on the two methods for audio data ([1][2][3]). Especially, the Audio-visual pre-training has been explored in [4]. Given the reconstructive listening enhancement  (RLE) component plays an important role according to Table 3, how effective the proposed method is compared to other reconstruction pre-training methods needs to be validated.\n\n[1] Huang, Po-Yao, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, and Christoph Feichtenhofer. \"Masked autoencoders that listen.\" Advances in Neural Information Processing Systems 35 (2022): 28708-28720.\n[2] Hsu, Wei-Ning, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. \"Hubert: Self-supervised speech representation learning by masked prediction of hidden units.\" IEEE/ACM Transactions on Audio, Speech, and Language Processing 29 (2021): 3451-3460.\n[3] Baevski, Alexei, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. \"wav2vec 2.0: A framework for self-supervised learning of speech representations.\" Advances in neural information processing systems 33 (2020): 12449-12460.\n[4] Shi, Bowen, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed. \"Learning audio-visual speech representation by masked multimodal cluster prediction.\" arXiv preprint arXiv:2201.02184 (2022).", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces a new framework to solve the \"deaf response\" problem in video-grounded dialogue system, where the audio information is ignored even though the question is related to the sound/speech. The proposed Enhanced Audio Response (HEAR) framework consists of two components: a sensible audio listening component that predicts whether the question is related to the audio or not; a reconstructive listening enhancement component that forces the model to reconstruct the masked frames of the audio feature given the context frames. The experiments show the proposed HEAR framework outperforms other baselines on AVSD@DSTC7 and AVSD@DSTC8 datasets.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SWbkCPexJW", "reviewer": "Reviewer_Zsad", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Zsad, We address your comments and questions below.\n\n**[Q1] Figure 5 is too small to get a clear understanding. The main components in Figure 5 show a lot of similarities with Figure 3. Thus, some repetitive parts can be omitted in Figure 5 to make it more concise.**\n\n**[A1]** Yes, we will make more space for Figure 5 to have a high resolution and simplify the redundant parts with Figure 3. Thank you!\n\n**[Q2] As the authors mentioned in Line 250-254, they only perform masking when the question contains audio-related words, and other questions excluding audio-related questions do not show high reliance on a specific modality. Is there any experimental evidence to support this?**\n\n**[A2]** Table A shows the comprehensive overview of which modality gives evidence to answer the audio-related questions and non-audio-related questions. Since there is no supervision about the evidential modality, we investigate each 180 questions in the dataset (AVSD@DSTC7). In the case of audio-related questions, it shows a high dependence (71%) on an audio modality. However, the other questions (i.e., non-audio-related questions) show more diverse preferences on many other multiple modalities.\n\n**[Q3] Furthermore, the authors claim that only a few words can be identified as audio-related questions, which is limited.**\n\n**[A3]** Yes, the keywords of Keyword-based Audio Sensing (KAS) have a limitation in that they cannot recognize audio-related questions at a sentence level. To this end, we further design the Semantic Neural Estimator (SNE) that takes an input sentence about a question and predicts the score about audio-relatedness. Henceforth, as shown in Figure 7, the SNE shows more improved responses in audio-relate questions than KAS.\n\n**[Q4] The overall loss function is not clear. How authors combine these loss functions and balance them is missing.**\n\n**[A4]** We train a dialogue language model by alternately optimizing SAL and RLE to allow audio and video to be freely utilized for their purposes. (i.e., Line 366-368) Therefore the total loss is defined as given below:\n\n$\nL_{HEAR} = \\begin{cases}\n    \\displaylines{ L_{SAL}, & \\text{iteration} = odd \\\\\\ L_{RLE},  & \\text{iteration} = even}\n  \\end{cases}\n$\n\nWe will reflect on the formulation of the total loss in the main paper. Thank you!\n\n**[Q5] The novelty of this paper is incremental. The methods proposed by the authors are similar to previous works. Furthermore, as for audio-visual question answering, the authors should focus more on the interaction between audio and visual modality to get a full understanding of the inputs.**\n\n**[A5]** \n- Our proposed Reconstructive Listening Enhancement (RLE) is designed to promote the interaction between audio and video modality. Table 1 and Figure 7 demonstrate the effectiveness of the RLE and especially high-performance improvement in audio-related questions. \n- The approach is completely new and effective. We first propose Reconstruction Upper Bound to further enhance the optimization of the audio reconstruction objective based on visual modality. The learning curve in Figure 8 also demonstrates the effectiveness of introducing Reconstruction Upper Bound compared to merely optimizing audio reconstruction.\n\n\n**Table A.** Ratio of evidential modality to answer questions\n\n|| audio | video | diaolgue | audio + video | audio + dialogue | video + dialogue | audio + video + dialogue |\n|----------------------------|-------|-------|----------|---------------|------------------|------------------|--------------------------|\n| audio-related question     | 71%   | 1%    | 1%       | 18%           | 5%               | 3%               | 1%               |\n| non-audio-related question | 1%    | 32%   | 8%       | 18%           | 4%               | 26%              | 11%         |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors propose Hearing Enhanced Audio Response (HEAR) framework which incorporates (1) Sensible Audio Listening (SAL) that selectively attends to audio according to a sensible decision of whether to focus on audio or not and (2) Reconstructive Listening Enhancement (RLE) that improves the audibility via establishing a reconstruction upper bound to connect audio with its surrounding information. For the sensible decision in SAL, the authors introduce two technical contributions: (1) Keyword-based Audio Sensing and (2) Semantic Neural Estimator", "reasons_to_reject": "1.\tFigure 5 is too small to get a clear understanding. The main components in Figure 5 show a lot of similarities with Figure 3. Thus, some repetitive parts can be omitted in Figure 5 to make it more concise. \n2.\tAs the authors mentioned in Line 250-254, they only perform masking when the question contains audio-related words, and other questions excluding audio-related questions do not show high reliance on a specific modality. Is there any experimental evidence to support this? Furthermore, the authors claim that only a few words can be identified as audio-related questions, which is limited.\n3.\tThe overall loss function is not clear. How authors combine these loss functions and balance them is missing. \n4.\tThe novelty of this paper is incremental. The methods proposed by the authors are similar to previous works. Furthermore, as for audio-visual question answering, the authors should focus more on the interaction between audio and visual modality to get a full understanding of the inputs.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The topic is video-grounded dialogue.\nThe main contribution is that the authors introduce audio into video-grounded dialogue", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes the Hearing Enhanced Audio Response (HEAR) framework that helps Video-grounded Dialogue (VGD) systems to better perceive and understand audio. The framework consists of two main components: Sensible Audio Listening (SAL), and Reconstructive Listening Enhancement (RLE). SAL predicts whether the question about the video clip is related to the audio track or not. For questions related to the audio track, RLE forces the model to reconstruct masked frames of the audio given the context frames.

Contributions:
- The HEAR framework is model-agnostic and can be applied to most VGD systems. 
- The SAL component explicitly directs the model to focus on audio features, this achieves better performance compared to feeding both audio and video features to the transformer blocks. Reducing VGD "deafness".
- Experiments show that it improves the performance of VGD systems on audio-related questions.

The reviewers generally saw the contributions as being incremental with respect to existing work. That said, the authors provided additional comparisons as part of the rebuttal process which demonstrate the effectiveness of multi-modal reconstruction vs single-modal reconstruction, and importantly, the effectiveness of their approach vs a prior multi-modal reconstruction approach.
Decision: Accept-Findings