Boosting Audio Visual Question Answering via Key Semantic-Aware Cues

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-03 06:19:14

Abstract: The Audio Visual Question Answering (AVQA) task aims to answer questions related to various visual objects, sounds, and their interactions in videos. Such naturally multimodal videos contain rich and complex dynamic audio-visual components, with only a portion of them closely related to the given questions. Hence, effectively perceiving audio-visual cues relevant to the given questions is crucial for correctly answering them. In this paper, we propose a Temporal-Spatial Perception Model (TSPM), which aims to empower the model to perceive key visual and auditory cues related to the questions. Specifically, considering the challenge of aligning non-declarative questions and visual representations into the same semantic space using visual-language pretrained models, we construct declarative sentence prompts derived from the question template, to assist the temporal perception module in better identifying critical segments relevant to the questions. Subsequently, a spatial perception module is designed to merge visual tokens from selected segments to highlight key latent targets, followed by cross-modal interaction with audio to perceive potential sound-aware areas.Finally, the significant temporal-spatial cues from these modules are integrated for answering the question. Extensive experiments on multiple AVQA benchmarks demonstrate that our framework excels not only in understanding audio-visual scenes but also answering complex questions effectively.

Main paper: 
 related works 2.1 audio visual scene understanding: 
 Inspired by the multisensory perception of humans, the community has paid more and more attention to audio-visual scene understanding in recent years. [34] . Compared to other modalities, visual and auditory modalities possess unique characteristics such as cognitive foundation, semantic consistency, spatial consistency, temporal consistency, and rich support from real-world data. It includes various interesting tasks such as sound source localization [12, 13, 31] , action recognition [10] , event localization [3, 33] , video parsing [26, 28, 32] , segmentation [20, 24, 41] , dialog [1, 30] , etc. These studies integrate rich audiovisual information within multimodal scenes to overcome limitations in perception inherent to single modalities, thereby utilizing both auditory and visual modalities to explore finer-grained scene comprehension. Apart from the above methods that facilitate scene understanding by excavating and analyzing different modalities, a unified multimodal model should also be able to reason their spatio-temporal correlation. Therefore, we focus on the audio-visual question answering task [6, 19, 35, 37] and explore spatiotemporal perception and reasoning in the audio-visual context.Audio-visual question answering, which exploits the natural multimodal medium of video, is attracting increasing attention from researchers [4, 23] . It requires comprehensive understanding and integration of diverse modalities, leading to precise responses to distinct questions. To explore the above AVQA task, Yun et al. [37] proposed the Pano-AVQA, which includes 360-degree videos and their corresponding question-answer pairs, aimed at exploring understanding of panoramic scenes. Li et al. [19] presented that the MUSIC-AVQA has become a strong benchmark for promoting spatio-temporal reasoning research in dynamic and long-term audio-visual scenes. Considering that real-life scenarios contain greater variety of audio-visual daily activities, AVQA benchmark is proposed in [35] , which further expands the audio visual scene coverage of AVQA task. Recently, LAVISH [23] have dedicated to exploring improvements in audio-visual association and enhancing training efficiency, resulting in satisfactory outcomes. Above research extract audio and visual features globally, without considering the importance of local feature representation. Chen et al. [5] consider the importance of the given question, which guides the feature extraction of both audio and visual signals. And then the PSTP-Net [18] is proposed explore critical temporal segments and sound-aware regions among the complex audiovisual scenarios progressively. However, aligning questions with video semantics is challenging due to the non-declarative nature, making it hard to identify key relevant segments. Our work focuses on empowering the model to gradually perceive essential visual and auditory cues for audio-visual scene understanding. 
method: 
 To solve the AVQA challenges, we propose an effective Temporal-Spatial Perception Model (TSPM) to achieve fine-gained audiovisual scene understanding, thus answering questions accurately. An overview of the proposed framework is illustrated in Fig. 2 .Given an input audio-visaul video sequence, we first divide it into 𝑇 non-overlapping audio and visual segment pairs {𝑎 𝑡 , 𝑣 𝑡 } 𝑇 𝑡 =1 , where each segment is 1𝑠 long. Subsequently, we partition each visual frame into 𝑀 patches and append a special [CLS] token to the beginning of the first patch. The question sentence 𝑄 is tokenized into 𝑁 individual words {𝑞 𝑛 } 𝑁 𝑛=1 .Audio Representation. For each audio segment 𝑎 𝑡 , we use the pre-trained VGGish [11] model to extract the audio feature as 𝑓 𝑡 𝑎 ∈ R 𝐷 , where 𝐷 is the feature dimension. The pretrained VGGish model is a VGG-like 2-D CNN network that trained on the large-scale AudioSet [11] dataset, employing over transformed audio spectorgrams. Then the features at the audio spectrgram second-level can be interpreted as 𝐹 𝑎 = {𝑓 1 𝑎 , 𝑓 2 𝑎 , ..., 𝑓 𝑇 𝑎 }. Visual Representation. A fixed number of frames are sampled from each visual segment 𝑣 𝑡 . Then we apply pre-trained CLIP [29] , with frozen parameters, extract both frame-level and token-level features as 𝑓 𝑡 𝑣 and 𝑓 𝑡 𝑝 on video frames, respectively, where 𝑓 𝑡 𝑣 ∈ R 𝐷 , 𝑓 𝑡 𝑝 ∈ R 𝑀 ×𝐷 and 𝑀 are token numbers of one frame. Finally, the visual frame-level and token-level features can be denoted as𝐹 𝑣 = {𝑓 1 𝑣 , 𝑓 2 𝑣 , ..., 𝑓 𝑇 𝑣 }, 𝐹 𝑝 = {𝑓 1 𝑝 , 𝑓 2 𝑝 , ..., 𝑓 𝑇 𝑝 }, respectively. Text Representation. Given an asked question 𝑄, we represent each word 𝑞 𝑛 in a fixed length vector with word embeddings, and then feed it into the pre-trained CLIP [29] model to get the question feature 𝐹 𝑄 , where 𝐹 𝑄 ∈ R 𝐷 . Note that the first token pooling is used for extracting question features.To hightlight the 𝑇𝑜𝑝 𝑘 crucial temporal segments that relevant to the question, we propose a Temporal Perception Module (TPM) with a carefully designed text prompt. While previous works [5, 18] have considered identifying key segments through the semantic similarity between question and temporal visual segments, aligning questions with visual frame semantics poses a significant challenge due to the non-declarative sentence of the questions. Therefore, the key of TPM lies in constructing a declarative sentence, aligning it effectively with the semantic content of the video, and facilitating the identification of critical segments.To achieve this, we devised a Text Prompt Constructor (TPC) with the goal of generating declarative statements based on input questions. This help semantic alignment between the generated statements and the visual frame, facilitating to identify key temporal segments to enhance the model's temporal perception ability. Since the input question do not contain answers, directly transforming them into declarative statements pose difficulties. Hence, we approached the problem from a new perspective, considering the design of statements that exclude irrelevant segments, guiding the model's attention toward temporal content relevant to the questions. Illustrated by the example in Fig. 2 , for input question "Where is the first sounding instrument?", the objective is to identify the moment when the first instrument starts playing. Considering that instruments in the video do not play simultaneously but follow a sequential order, we direct the model's attention to segment in the video where instruments do not play simultaneously. This way directs the model focus on segments where there are changes in the order of instrument sounds, identifying crucial segments. Leveraging the TPC, we manually transform the question into a declarative sentence "The instruments in the video do not sound at the same time.", denote as TPrompt, aligning its feature representation well with the semantic content of the video. This allows us to locate segments related to TPrompt and subsequently locate temporal segments relevant to the question.For a given declarative sentence TPrompt, its feature embedding 𝐹 𝑇 𝑃𝑟𝑜𝑚𝑝𝑡 using the same encoder as the given question. Concretely, we firstly use one linear projection layer Key(•) to transform indexing visual features 𝐹 𝑣 to indexing keys k. Then we get an attention score for each indexing key in the video temporal sequence. A Softmax layer normalizes the attention scores and generates an attention weight vector 𝑊 by:EQUATIONwhere k 𝑗 = Key(𝐹 𝑗 𝑣 ), 𝑗 ∈ {1, 2, ...,𝑇 }, and 𝑑 is the dimensionality of the key vector. Considering that the higher weight indicates a • • • Set B Set A • • • • • • Set B • • •Step 1Step 2Step 3Input imageTokens in Set BFigure 3 : Spatial Perception Module. Similar tokens are merged. For example, for a given complex sense, the man is playing flute if merged into a single token, and the woman is playing violin is merged into a single token. Following this, the proposed model identify the sounding instrument, thus inferring the correct answer to the input question.stronger correlation between the video content and TPrompt, we conduct 𝑇𝑜𝑝 𝑘 feature selection over 𝑇 segments. To be specific, we employ a temporal selection operation algorithm, denoted as Ψ, which is implemented by the sorted algorithm for ranking and sorting to pick out the crucial relevant segments with the highest attention weights and their corresponding indices:EQUATIONwhere Ψ is shown in Algorithm 1 , Ω 𝑇 𝑃𝑀 is the index position corresponding to the 𝑇𝑜𝑝 𝑘 highest weights, Ω 𝑇 𝑃𝑀 ∈ {0, 1, ..., 𝑘 -1} 𝑇 𝑜𝑝 𝑘 , 𝐹 ′ 𝑎 , 𝐹 ′ 𝑣 is selected temporal feature, 𝐹 ′ 𝑎 ∈ R 𝑇 𝑜𝑝 𝑘 ×𝐷 , 𝐹 ′ 𝑣 ∈ R 𝑇 𝑜𝑝 𝑘 ×𝐷 . Note that the 𝑇𝑜𝑝 𝑘 temporal audio segments are corresponding to positions on the 𝑇𝑜𝑝 𝑘 visual segments relevant to the question.To identify visual regions that are pertinent to the key instrument, the Spatial Perception Module (SPM) is designed to merge visual tokens in selected temporal segments based on similarity, preserving their semantics, and subsequently engages in cross-modal interaction with audio to enhance audio-visual association. Given previous works [18] attempt to identify crucial regions by leveraging the semantic similarity between questions and visual tokens, the lack of semantic about objects within these tokens presents a challenge in establishing effective correlations with sound.To address this, we enhanced the preservation of semantic information in visual tokens along selected key temporal sequences. We achieve this by merging similar tokens within each visual frame, resulting in merged tokens that carry richer semantic information about objects. Specially, given the visual token-level embedding 𝐹 𝑝 and 𝑇𝑜𝑝 𝑘 curious temporal segment index, we obtain the temporal visual token-level features as follow:EQUATIONwhere 𝐹 ′ 𝑝 ∈ R 𝑇 𝑜𝑝 𝑘 ×𝑀 ×𝐷 . and Φ the algorithm that determines the temporal position in 𝐹 𝑝 based on the index in Ω 𝑇 𝑃𝑀 . For the given selected visual token-level feature 𝐹 ′ 𝑝 , we employ a token-merging strategy to enhance the semantic features between the attention and MLP branches of each transformer block. Then, similar tokens are merged in each transformer block with per layer, and the merged visual token-level feature F𝑝 as:EQUATIONwhere F𝑝 = { f 1 𝑝 , f 2 𝑝 , ..., f 𝜆 𝑝 }, F𝑝 ∈ R 𝜆×𝑆 ×𝐷 and 𝜆 is selected temporal segments' moment, 𝑆 is merged tokens number. Specifically, as shown in Fig. 3 , evenly divide the 𝑀 tokens in 𝐹 ′ 𝑝 into two subsets 𝐴 and 𝐵 of roughly equal size in Step 1. Then, for one subset 𝐴, calculating the similarity between each token and every token in the other subset 𝐵, drawing an edge for each calculated similarity. Subsequently, apply mean fusion to the tokens connected by the similar edges. And in Step 2, concatenating the two subsets to generate a merged visual token-level feature F𝑝 . It's worth noting that between each transformer block's attention branch and MLP branch, Step 1 through Step 3 of the visual token merging process are executed, resulting in the creation of multiple merged tokens with semantic representations.Then, considering that the sound and the location of its visual source usually reflects the patial association between audio and visual modality, we leverage the powerful cross-modal perception ability to interact between selected visual merged token-level features and audio embeddings F𝑝 , 𝐹 ′ 𝑎 . This enables concrete audio-visual correlation which performs attention-based patch-level merged tokens sound source perception. Denote Attn(•) to be the scaled dot-product conducted on the query, keys, and values, the aggregated feature can be obtained by:EQUATIONwhere 𝐹 𝑝 ∈ R 𝑇 𝑜𝑝 𝑘 ×𝑆 ×𝐷 . Thus far, we have progressively identified the key temporal segments that are most relevant to the input question, and its potential sound-aware areas.To achieve AVQA task, we concatenate the updated visual features 𝐹 ′ 𝑣 , 𝐹 𝑣 , and the audio features 𝐹 ′ 𝑎 obtained from TPM and SPM, respectively. Then the visual fusion feature 𝐹 𝑎𝑣 obtained by a linear layer. To verify the audio-visual fusion of our proposed effective Temporal-Spatial Perception Model, we employ a simple elementwise multiplication operation to integrate the question feature 𝐹 𝑞 and the previously obtained audio-visual fusion embedding 𝐹 𝑎𝑣 . And it can be formulated as:EQUATIONEQUATIONwhere ⊙ is element-wise multiplication operation, 𝛿 and 𝐹𝐶 represent 𝑇 𝑎𝑛ℎ activation function and linear layer, respectively. Then a Softmax function is used to output probabilities 𝑝 ∈ R 𝐶 for candidate answers, where 𝐶 is the size of the pre-defined candidate answer vocabulary pool. With the predicted probability vector and the corresponding groundtruth label y, we optimize it using a crossentropy loss: L 𝑞𝑎 = -𝐶 𝑐=1 𝑦 𝑐 𝑙𝑜𝑔(𝑝 𝑐 ). During testing, we can select the predicted answer by ĉ = arg max 𝑐 (𝑝). 
experiments 4.1 datasets: 
 MUSIC-AVQA [19] , it contains 9,288 videos covering 22 different musical instruments, with a total duration of over 150 hours and 45,867 question-answering pairs. The questions are designed under multi-modal scenes containing 33 question templates covering nine types, i.e., the audio-visual, separate visual, and separate audio, depending on which modalities are used to discover question-related clues for answer prediction. The diversity question-answering pairs which occupy a large portion of the entire dataset, and there are five audio-visual question types referring to existential, counting, location, comparative, and temporal. The large-scale MUSIC-AVQA dataset is well suited for studying temporal-spatial perception for dynamic and long-term audio-visual scenes. AVQA [35] , is designed for audio-visual question answering in general real-life scenario videos. It contains 57,015 videos from daily audio-visual activities, along with 57,335 question answering pairs designed relying on clues from both modalities, where information from a single modality is insufficient or ambiguous.For both datasets, we adopt the official split of the two benchmarks into training, evaluation, and test sets.For the visual stream, we divide the video into 1-second segments and frames sample rate at 1𝑓 𝑝𝑠. We utilize the CLIP-ViT-L/14 [29] model pre-trained on ImageNet to extract 512-𝐷 feature representations for each visual segment, where [𝐶𝐿𝑆] token denote visual frame-level features. For the audio signal, it is sampled at 16kHz, which is a standard sampling rate for audio. we use the VGGish network pre-trained on AudioSet to extract 128-𝐷 features. For each input question sentence, we extract its feature same as visual frame-level encoder to obtain 512-𝐷 feature vector. In all experiments, we use Adam optimizer with an initial learning rate of 1𝑒-4 and will drop by multiplying 0.1 every 10 epochs. The batch size and number of epochs are set to 64 and 30, respectively. The constructed statement templates are provided in the supplementary code files. We use the 𝑡ℎ𝑜𝑝 library in PyTorch to calculate the model's parameters and FLOPs. Our proposed model is trained on NVIDIA GeForce RTX 3090 and implemented in PyTorch.To verify the effectiveness of the proposed TSPM, we compare it with multiple existing mehtods: AVSD [30] , Pano-AVQA [37] , AVST [19] , LAVISH [23] , COCA [16] , PSTP-Net [18] , etc. Tab. 1 indicate that the TSPM outperforms all comparison methods. Specially, the TSPM method shows significant improvements in the subtask types of Audio-visual, including Localization, and Temporal. Specifically, compared to the recent PSTP-Net [18] , the model achieves remarkable improvements of 0.92% (73.51% and 71.26%) in the above-mentioned complex audio-visual question types. It is worth noting that the model shows a performance boost of 5.14% (82.29% and 77.15%) and 7.54% (84.90% and 77.36%) in the Counting and Localization subtasks of the visual modality, respectively, when compared to PSTP-Net [18] . The significant performance improvements indicate that our TSPM is effectively identifies crucial temporal segments and spatio tokens in videos. Moreover, in comparison to LAVISH [23] , which fine-tunes large pretrained models, our model demonstrates superior efficiency without the need for fine-tuning. We conducted tests with an equal number of epochs under the same hardware configuration, and it was observed that LAVISH incurred a cost 14× higher than our model. Additionally, we observed limitations in the performance of Comparative type questions, and we consider this may be attributed to the challenges of separating multiple sound in complex audio-visual scenes. This motivates us to explore strategies (such as dynamic fusion) in future work that can achieve better performance on both single-modality and multi-modality aspects.To further validate the capabilities of the proposed TSPM, in contrast to splitting by Question ID, we also partitioned the MUSIC-AVQA dataset based on Video ID, apportioning it into training, validation, and test sets in a ratio of 7 : 1 : 2. As shown in Tab. 2, it can be seen that our proposed TSPM achieves the best overall performance compared to the latest AVQA methods. Particularly, in the three subtask types of audio, visual, and audio-visual, our TSPM outperforms others significantly, showcasing the excellent generalization capability and performance of the proposed TSPM.In summary, the TSPM offers significant improvements over existing approaches and provides a novel insight into questionoriented audio-visual scene understanding.In this subsection, we delve into examining the impact of various modules within the TSPM on the performance of the MUSIC-AVQA.Audio To verify the effectiveness of the proposed components, i.e., TPM, SPM, TPC, Tokens merge, etc., we remove them from the primary model and re-evaluate the new model's performance. Tab. 3 shows that after removing a single component, the overall model's performance decreases, and different modules have different performance effects. The specific analysis is as follows: In general, each module contributes to the better performance. When all modules are present, the TSPM achieves best result on the MUSIC-AVQA dataset. Similarly, we explored the impact of key parameter configurations on model performance. As shownEnsemble Total Accuracy (%) HME [7] HAVF [35] 85.0 PSAC [22] HAVF [35] 87.4 LADNet [21] HAVF [35] 84.1 ACRTransformer [38] HAVF [35] 87.8 HGA [14] HAVF [35] 87.7 HCRN [17] HAVF [35] 89.0 PSTP-Net [18] -90. in Tab. 4, when the 𝑡𝑜𝑝 𝑘 value is large, it may introduce temporal redundancy. When there are too many tokens, semantic merging on the token is not thorough enough; conversely, an excessive merging may result in semantic loss. The model achieves optimal performance when 𝑡𝑜𝑝 𝑘 = 10 and 𝑡𝑜𝑘𝑒𝑛𝑠 = 14, respectively.Tab. 5 illustrates the computational costs of TSPM compared with ST-AVQA [19] , PSTP-Net [18] and LAVISH [23] . It can be observed that TSPM has fewer training parameters, lower FLOPs, and higher accuracy compared to ST-AVQA. Although PSTP-Net boasts lower computational costs, our TSPM achieves superior results at extremely low computational costs. LAVISH achieves a well accuracy, but its parameter is more than three times that TSPM. This is because LAVISH fine-tunes large pretrained models, whereas TSPM achieves comparable results without fine-tuning. In summary, our proposed TSPM achieves high performance at relatively low cost, fully demonstrating the effectiveness and efficiency of the model.To verify the generalization capability of the proposed TSPM, we compared it with multiple existing AVQA-based methods, including ACRTransformer [38] , HCRN [17] , PSTP-Net [18] , etc., on the AVQA dataset. As illustrated in Tab. 6, the TSPM exhibits remarkable performance compared to recent methods. Specifically, our approach outperforms PSTP-Net [18] by 0.6% (90.8% and 90.6%), demonstrating notable superiority over earlier methods such as ACRTransformer [38] . Furthermore, while the performance improvement of TSPM on the AVQA dataset seems limited compared to its performance on the MUSIC-AVQA dataset, we attribute this primarily to the AVQA dataset's shorter duration (10s . 60s) and simpler audio-visual components. Despite these differences, our TSPM maintains its effectiveness even in this scenario. Notably, in Tab. 6, the PSTP-Net [18] achieved a 1.2% improvement over the HCRN [17] , while our TSPM exhibited a more substantial 1.8% enhancement, indicating the significant effectiveness of TSPM's performance boost. Additionally, ablation studies further confirm the effectiveness of both TPM and SPM components. Moreover, for the experimental settings on the AVQA dataset, we selected the Top-𝑘 = 8 temporal segments relevant to the given question, with a merged token count of 𝑡𝑜𝑘𝑒𝑛𝑠 = 14. It's worth noting that HAVF [35] in Tab. 6, serving as the baseline [18] . It can be observed that TSPM can progressively select relevant temporal segments and locate potential sound-aware areas, thus accurately providing correct answers to the given questions. This process vividly demonstrates TSPM's effective spatiotemporal perception capabilities in complex audiovisual scenarios. method for the AVQA dataset, includes three fusion modalities and integrates their outputs using an averaging strategy to generate answers. In summary, the proposed TSPM effectively demonstrates both its effectiveness and generalization.To showcase the temporal and spatial perception capabilities of the proposed TSPM, we provide two examples contrasting with the recent AVQA-related method PSTP-Net in Fig. 4 . In Example 01, when presented with the question "Where is the first sounding instrument?", the TPM first identifies the temporal indices relevant to the question. Subsequently, the SPM sequentially locates potential sound-aware areas, with the heatmap indicating these regions. In this example, it becomes apparent that initially, only the "flute" on the right side is playing, but as time progresses, the violin on the left side also begins playing. The heatmap effectively illustrates the variation in multiple instruments playing within this dynamic and complex audio-visual scene. Consequently, it can be inferred that the correct answer to the question is the instrument on the "right" side. Similarly, in Example 02, the progressive temporal-spatial perception process is aptly demonstrated, resulting in the correct answer. These visualizations indicate that the proposed TSPM can effectively perceive the temporal segments relevant to the question and the spatial areas associated with sound, showcasing its efficacy in audio-visual question answering task. 
conclusion: 
 In this work, we propose an effective Temporal-Spatial Perception Model framework for addressing complex question-answering task in dynamic audio-visual scenarios. It includes a temporal perception module with a declarative sentence text prompt and a spatial perception module incorporating token merging. These modules are employed to locate temporal segments relevant to the question and enhance spatial audio-visual associations, thereby facilitating fine-gained audio-visual scenes understanding. Extensive experiments demonstrate that the proposed framework achieves precise temporal-spatial perception on multiple benchmark, effectively showcasing the reasoning process involved in answering questions. We believe that our work will serve as inspiration researchers in the field of audio-visual scene understanding. 
: 
 The instruments in the video do not sound at the same time.What is the first sounding instrument?Where is the clarinet?clarinet t=1 t=3Input Question: Where is the first sounding instrument?The instruments in the video do not sound at the same time. and separation [9, 39, 40] , event localization [3, 33] , video parsing [26, 28, 32] , segmentation [20, 24, 41] , dialog [1, 30] , question answering [6, 19, 35, 37] , etc., towards audio-visual scene understanding. Particularly, the Audio-Visual Question Answering (AVQA) task, involving the fine-gained spatio-temporal perception and reasoning of complex audio-visual scenes, has emerged as valuable and challenging focus of research interest.TextTo solve above AVQA task, Li et al. [19] build a large-scale MUSIC-AVQA dataset as a strong benchmark and propose a spatiotemporal grounding model to achieve scene understanding and reasoning over audio and visual modalities. Yun et al. [37] and Yang et al. [35] also introduce the Pano-AVQA and AVQA dataset to explore panoramic and real-life scene, respectively. Recently, LAVISH [23] introduced a novel parameter-efficient framework for encoding audio-visual scenes using off-the-shelf pretrained vision transformers, achieving notable progress. Moreover, researchers [5, 18] have considered the significance of the given question, attempting to achieve precise perception of relevant temporal segments and spatial sound sources using the question as a guiding factor, leading to promising results. Clearly, these endeavors have markedly propelled the progress of research in audio-visual question answering.Despite the significant progress made in AVQA, there are still several challenges that need to be addressed. For temporal perception, this task involves understanding long audio-visual videos, which suffer from heavy information redundancy. Existing works [19, 23, 27] typically employ a uniform sampling strategy to reduce redundancy and computational costs, but may lead to the loss of crucial information. Others [5, 15, 18] attempt to use the CLIP pretrained model as a feature extractor by measuring the similarity between given questions and video frames to select temporally relevant segments. However, the given question's expression format is not declarative, inconsistent with the textual format used in the CLIP model, making it difficult to effectively align with the semantic content of video frames, and thus challenging the search for temporally relevant segments related to the input question. Regarding spatial perception, the lack of supervised information for spatial visual objects and sound makes it challenging for models to associate visual targets with sounds in the video, thereby making it difficult to identify potential sound-aware areas. While existing pre-trained object detection models used in visual question answering task can excel in key object localization, the absence of certain specific categories (eg., suona, guzheng, etc.) in the AVQArelated datasets, adding difficulty to locate relevant areas. Some explorations [18] have employed ViT to convert video frames into token sequences, utilizing semantic similarity calculations between questions and tokens to perceive the most relevant tokens as key targets. Nevertheless, the absence of objects' semantics in visual tokens makes establishing an effective correlation with questions challenging, thus rendering accurate localization difficult. This imprecise spatial perception makes it difficult to establish effective associations with sounds in the video, thereby complicating the localization of potential sound-aware areas. Hence, as shown in Fig. 1 , it is crucial to enable machines to perceive complex audiovisual scenes in a manner akin to human cognition and accurately infer answers to questions.To address these challenges, we propose an effective Temporal-Spatial Perception Model (TSPM) to perceiving crucial visual and auditory cues related to the questions in complex audio-visual scenes. Firstly, the content related to the question is usually scattered in partial segments of the video instead of the whole sequence. Hence, we design a Text Prompt Constructor (TPC), which construct a declarative sentence text prompt derived from the question template, effectively aligning it with the semantic content of the visual frames. Following this, we introduce a Temporal Perception Module (TPM) that utilizes cross-modal attention mechanisms to identify the key temporal segments relevant to the given question. Secondly, identifying key visual areas and their corresponding sound source positions within critical segments, which can help to learn audio-visual associations in complex scenarios. To achieve this, the Spatial Perception Module (SPM) is designed to merge visual tokens on selected temporal segments to multiple joint tokens, thus preserving the semantic information of potential targets. Then, these joint tokens interact cross-modal interaction with audio to perceive potential sound-aware areas. Finally, the above critical temporal segments and sound-aware regions' features are fused to obtain a joint representation for question answering. Extensive experiments on multiple benchmark demonstrate that our proposed approach achieves precise temporal-spatial perception, highlighting its immense potential in tackling audiovisual question-answering task. Our contributions can be summarized as follows:• The temporal perception module designed in TSPM transforms questions into declarative prompts using a constructed declarative sentence generator, facilitating better alignment with the semantic of visual frames and effectively identifying key temporal segments relevant to the given question. • The spatial perception module introduced in TSPM merges visual tokens on selected temporal segments to preserve key potential targets, then engages in cross-modal interaction with audio, effectively perceiving potential sound-aware areas.• Extensive experiments on multiple benchmarks demonstrate that the proposed TSPM achieves precise spatio-temporal perception, showcasing its significant potential in addressing audio-visual question answering task. 
 
 Appendix:  


Decision: The reviewers are satisfied with the authors' rebuttal and have no doubt that the paper quality is acceptable for the conference. The authors should incorporate the reviewers' insightful feedback including writing issues into their revision.

Reviews: review: paper_summary: paper_summary: This paper focuses on audio-visual question answering problems, proposes a Temporal-Spatial Perception Model (TSPM), which is composed of Temporal Perception module and Spatial Perception module respectively. By enhancing features in specific time steps and spatial positions, it achieves better performance on AVQA tasks.
strength_weakness: strength_weakness: 1. The model explicitly can intuitively locate and enhance features of specific time steps and spatial positions that have gain for the final performance. 
2. The model achieves the best performance and surpasses the second most by a wide margin on MUSIC-AVQA. 
3. The structure of the paper is clear and the writing is smooth.
limitations: limitations: 1. The author did not provide technical details about the Text Prompt Constructor (TPC).
2. The Token Merge process disrupts the spatial order of the image tokens. Considering that this method uses a Transformer-based encoder, it is necessary for the authors to elaborate further on whether this process occurs before or after Position Encoding and its impact on sequence processing.
3. Algorithm 1 appears to be similar to the conventional top-k algorithm. Why is it listed separately?
4. Compared to PSTP-Net, this paper upgrades the visual and textual feature extractors from CLIP-ViT-B/32 to CLIP-ViT-L/14, which results in stronger representation capabilities and increased computation cost. Comparing their performance and parameters on this basis is unfair. The authors should either provide results using the same encoder or justify the necessity of this upgrade.
paper_summary: paper_summary: This paper proposes a new AVQA method that transforms non-declarative questions into declarative prompts to better identify relevant segments. While the idea is promising, the article lacks sufficient explanations for many key points, which need to be supplemented.
strength_weakness: strength_weakness: By constructing declarative prompts corresponding to the questions, it aligns well with the data format used by the CLIP model, which intuitively aids in segment matching. The motivation behind this approach is reasonable.
The experimental section is comprehensive, providing ample comparisons and evaluations.
limitations: limitations: The key methods lack specific explanations, and some of the explanations for certain questions are somewhat vague.

Questions:
1.	In the context of Figure 1, the mention of "the absence of objects' semantics in visual tokens" is not clearly expressed in the figure, and I don't quite understand it.
2.	Regarding the crucial TPC technique, only one example is provided, and there is no description of the method or process for constructing declarative sentence prompts based on a given question. 
3.	Algorithm 3 mentions Φ without further explanation or reference.
4.	Step 3 in the process of merging visual tokens was not explained. If we follow the approach shown in Figure 3, why do we ultimately obtain an unequal number of a set tokens, b set tokens, and merge tokens
5.	Is there any other method of using prompt in the AVQA field? If there are, there is a lack of comparison between other methods and this article.
paper_summary: paper_summary: This paper proposes an end-to-end method integrating audio and visual features for the Task of AVQA. Specifically, the authors first utilize a Text Prompt Constructor to transform the prompt to a declarative one, then design a Temporal Perception Module to identify the significant frames, lastly adopt a Spatial Perception Module with multimodal interaction to derive the final representation. The proposed model achieves great performances on the AVQA dataset, across Audio, Visual and Audio-Visual subtasks.
strength_weakness: strength_weakness: 1.	This paper proposes an end-to-end approach to integrate multimodal features, i.e., audio and visual.
2.	This paper conducts comprehensive experiments on AVQA datasets, and performs extensive ablation studies to verify the effectiveness of each component.
limitations: limitations: 1.	This paper overall seems like an add-on study of a previous work PSTP-Net[1]. Except for the Text Prompt Constructor, the author needs to specify the novelty and the difference between this work and [1]
2.	The authors did not give the concrete structure of TPC. The authors should specify how it is implemented. Sec 3.2 only gives a vague statement of “Leveraging the TPC, we manually transform the question into a declarative sentence”
3.	Some of the equations may not be properly written. For example, in Equation (5), why does $F_v$ only depends on $f_p^{\lambda}$ ?
4.	According to Table 3, the avg result achieves 73.35 even when all components are removed. This result beats or is comparable to almost all of the baselines in Table 1, including PSTP-Net. The authors may need to analyze this phenomenon.

[1] Li, G., Hou, W., & Hu, D. (2023, October). Progressive Spatio-temporal Perception for Audio-Visual Question Answering. In Proceedings of the 31st ACM International Conference on Multimedia (pp. 7808-7816).
paper_summary: paper_summary: This paper proposes a method for audio-visual question answering, utilizing text prompts to guide the model in capturing temporal segments based on audio. The motivation sounds reasonable. The results surpass those of LAVISH, presented at CVPR 2023, and the ablation experiments validate the effectiveness of the results. However, there are some minor errors in the formulas and figures, and some parts of the description are unclear. See limitations.
strength_weakness: strength_weakness: 1. The motivation of the paper is sound, addressing the limitation of audio-visual question answering tasks where non-declarative questions are hard to align with visual semantics.
2. The design of modules for query rewriting and the merging of similar tokens at the spatiotemporal level is quite innovative (though I am not fully aware of the latest advances in AVQA, so there might be some omissions).
3. The method achieved good results on two public AVQA datasets.
4. The code is provided, which should ensure reproducibility.
limitations: limitations: Overall quality is acceptable, but there are some areas I don’t quite understand. I'm not sure if it is due to my oversight:

1. How exactly is TPC performed? Is it manually designed? What are the design principles? How is the alignment between the rewritten TPrompt and visual semantics ensured? I believe this alignment is crucial for selecting key temporal segments, but the authors seem to have not provided a detailed explanation.
2. What are the dimensions of the aggregated feature $F_v$ in formula (5)? Why is it $F_p$ below it? If $f_p^{\lambda}$ and $F_v$ sizes are not consistent, it seems an equal sign cannot be used here.
3. In Section 3.4, line 514 mentions the activation function $\sigma$ and Tanh. Where are these used?
4. In line 548 of section 4.2: "we utilize the CLIP-ViT-L/14 model pre-trained on ImageNet." The authors should carefully check this statement.
5. VGGish is 128-d. Transforming 128-d to 512-d looks a bit odd; doing the reverse seems more usual. Is this approach consistent with other papers?
6. The top-k values for MUSIC-AVQA and AVQA are different. Is this related to the average length of the dataset videos?