Weakly Supervised Gaussian Contrastive Grounding with Large Multimodal Models for Video Question Answering

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-17 09:32:03

Abstract: Video Question Answering (VideoQA) aims to answer natural language questions based on the information observed in videos. Despite the recent success of Large Multimodal Models (LMMs) in image-language understanding and reasoning, they deal with VideoQA insufficiently, by simply taking uniformly sampled frames as visual inputs, which ignores question-relevant visual clues. Moreover, there are no human annotations for question-critical timestamps in existing VideoQA datasets. In light of this, we propose a novel weakly supervised framework to enforce the LMMs to reason out the answers with question-critical moments as visual inputs. Specifically, we first fuse the question and answer pairs as event descriptions to find multiple keyframes as target moments and pseudo-labels, with the visual-language alignment capability of the CLIP models. With these pseudo-labeled keyframes as additionally weak supervision, we devise a lightweight Gaussian-based Contrastive Grounding (GCG) module. GCG learns multiple Gaussian functions to characterize the temporal structure of the video, and sample question-critical frames as positive moments to be the visual inputs of LMMs. Extensive experiments on several benchmarks verify the effectiveness of our framework, and we achieve substantial improvements compared to previous state-of-the-art methods.

Main paper: 
 introduction: 
 Video Question Answering (VideoQA) stands at the forefront of developing intelligent systems that can reason about causal and temporal relations and answer natural language questions in videos, which is an essential manifestation of human intelligence. Despite significant advancements have been made by self-supervised pretraining and transformer-style architectures [34, 37, 38, 47, 48, 50] in recent years, VideoQA remains a challenging problem that requires models to comprehensively understand and dynamically align the semantics of both the visual world and natural language.Figure 1 : We argue that the information in uniformly sampled frames is insufficient for LMMs to answer the question correctly. Therefore, we utilize the fused event description to provide additional weak supervision and generate weight distributions for each video moment. We align the positive description-moment pairs while pushing away negative ones. In this way, we can effectively select question-critical moments for LMMs to reason out the answer.With the progress in vision-language pre-training techniques [22, 34] and Large Language Models (LLMs) [3, 35] , Large Multimodal Models (LMMs) [8, 21, 30] , as the further development of Large Language Models (LLMs) [3, 35] , have showcased impressive capabilities across various image-language tasks. These LMMs share very similar architecture and paradigms. They first extract visual features with an image encoder, and the encoded features will be sent into a connection module to obtain a set of visual tokens that are in the same feature space as the LLM. Then, the visual tokens are concatenated with the input text embeddings together, to be fed into the LLM to decode the target text sequence. However, limited by the long sequence frames in videos and computation costs, current LMMs fall short when applied to VideoQA. They simply concatenate the visual tokens of uniformly sampled, sparse frames (e.g., 4 frames) as the visual inputs for answer prediction. Such a sampling strategy does not consider the specific question at hand, treating all frames equally and introducing redundancy, potentially distracting the model from discovering true answers.Therefore, it's necessary to localize the moments crucial for answering the question for LMMs (as the positive moments shown in Figure 1 ). Notably, different from the task of Temporal Sentence Grounding (TSG) [13, 54] which aims to localize a video moment described by a declarative sentence, the grounding mechanism in VideoQA features some unique challenges. First, questions in VideoQA are interrogative sentences, and they lack explicit information about the answer content needed to be grounded. For instance, in Figure 1 , there is a semantic gap between the interrogative question [How does the boy react after opening the present?] and the declarative description [The boy carries up the dog after opening the present.]. Thus, models are required to both localize the moment [after opening the present] explicitly shown in the question and identify the implicit answer moment [carry up the dog], demanding the causal-temporal reasoning. Second, VideoQA aims to correctly answer the questions of videos, rather than solely grounding specific video moments, and there are no human annotations for the timestamps of question-critical moments in existing VideoQA datasets.To address these challenges, we introduce a weakly supervised framework by discovering question-critical moments with Gaussianbased Contrastive Grounding (GCG). As labeling the timestamps of question-critical moments is labor-intensive and subjective, we leverage the powerful visual-language alignment capability of the CLIP models [10, 34] to provide timestamps of keyframes. In detail, we fuse the textual question and answer to generate a declarative sentence as the event description, and then compute the similarities between the description and each frame. Frames with the highest scores will be the keyframes of target moments. We observe that LMMs with these pseudo-labeled keyframes as visual inputs showcased significant improvements on a wide range of VideoQA tasks (as shown in Figure 4 ), which also indicates a great potential to localize question-critical moments for LMMs. To equip LMMs with such ability to automatically find these question-critical moments, motivated by more recent research [41, 56, 57] which has highlighted the superiority of end-to-end Gaussian mask learning in video grounding tasks, we use multiple Gaussian masks to characterize the inherent temporal structure of the video. Differently, we explicitly introduce additional objectives as weak supervision to help generate more suitable Gaussians for LMMs. With this new design, our GCG will distinguish the positive video moments (green in Figure 1 ) from negative video moments (orange and yellow in Figure 1 ). The positive moments are crucial for answering the question and will be the visual inputs of LMMs for answer prediction. Moreover, to ensure the selected positive moments are closest to the event description, our GCG also includes a contrastive objective [14] that can align the positive description-moment pairs while pushing away negative ones. Notably, different from previous works like SeViLA [52] which pre-train an additional LMM as the keyframe localizer with other datasets like QV-Highlights [19] , our method is lightweight and flexible for end-to-end training with LMMs.To summarize, we make the following contributions: (1) We propose a weakly supervised grounding framework for VideoQA, by utilizing the alignment capability of the CLIP models to provide pseudo-labeled timestamps of keyframes without human-labor annotated costs. (2) We devise the Gaussian-based Contrastive Grounding (GCG) for weakly-grounded selection of question-critical moments, enhancing the effectiveness and interpretability of LMMs when applied to VideoQA, by revealing which visual scenes result in the predicted answers. (3) We conduct extensive experiments to verify the effectiveness of our proposed method, and achieve substantial improvements on six challenging VideoQA benchmarks including NExT-QA, Causal-VidQA, Intent-QA, ActivityNet-QA, MSVD-QA, and MSRVTT-QA. 
related works 2.1 large multimodal models (lmms): 
 LMMs [1, 8, 21, 30, 51] in their current form primarily function as image-to-text generative models, taking images as input and generating text sequences. These models have demonstrated strong capabilities in image-language understanding and reasoning by adapting frozen language models to frozen image encoders with trainable connection modules, following large-scale image-text pretraining. The connection module can either be a transformer-based architecture like Q-former in InstructBLIP and BLIP-2 [8, 21] , Perceiver Resampler in Flamingo [1] , or a simple linear layer in LLaVA [30] . Most current LMMs are essentially image-based models, and they simply concatenate the visual tokens extracted from uniformly sampled, sparse frames as visual inputs for video-language tasks. This results in a lack of temporal modeling ability and emphasizes the necessity of selecting specific video moments, particularly for addressing the demands of reasoning-based VideoQA tasks. In this paper, our goal is to enhance the causal-temporal reasoning abilities of LMMs without additional pretraining on video-text corpora, by discovering the question-critical moments with our weaklysupervised Gaussian-based Contrastive Grounding.Early VideoQA benchmarks [16, 44, 53] focus on descriptive questions (e.g., [what's the man doing]) within short video clips, rarely going beyond a recognition of the objects and actions. Instead, more recent VideoQA benchmarks [23, 24, 40] like NExT-QA [40] emphasize counterfactual, temporal, and causal reasoning involving multiple entities and relations, demanding the ability to uncover the causes behind specific events within longer videos, necessitating localizing a text query to specific moments.In light of this, ATP [4] utilizes the tool of atemporal probe to select a single frame without temporal information for downstream tasks. MIST [11] and TranSTR [28] fuse frames with the mechanism of adaptive temporal rationalization and iterative spatial-temporal attention, respectively. NExT-GQA [41] constructs grounding labels in the test set of NExT-QA and uses a single Gaussian mask to fuse frames along the temporal dimension. SeViLA [52] , similar to us, utilizes the LMM (BLIP-2) for VideoQA. However, SeViLA uses two LMMs to generate pseudo-labels and answer questions respectively, with extra pre-training on TSG datasets [19] and a multi-stage training scheme. Different from previous works, we utilize the CLIP [10, 34] models to automatically provide weak supervision for grounding, and our lightweight GCG module learns multiple Gaussian masks to generate both positive and negative moments in an end-to-end manner, with an additionally contrastive objective to distinguish positive ones from negative ones for frame selection. 
preliminary: lmms for videoqa: 
 We take InstructBLIP [8] as an example to illustrate how LMMs deal with VideoQA. Similar to other LMMs, InstructBLIP approaches VideoQA as a text generation task conditioned on the question Q and video V with 𝑇 frames, and predicts the answer A by the following three steps:(1) The Vision Transformer [9] in EVA-CLIP [10] serves as the frozen image encoder to extract embeddings of each frame individually, and obtains E = {e 1 , e 2 , • • • , e 𝑇 } , E ∈ R 𝑇 ×𝑁 𝐼 ×𝐷 𝐼 , e 𝑡 ∈ R 𝑁 𝐼 ×𝐷 𝐼 , where 𝑡 denotes the 𝑡-th frame, 𝑁 𝐼 is the patch number of each frame (including the class token), and 𝐷 𝐼 is the embedding dimension. To mitigate computational costs, existing LMMs uniformly sample 𝐾 frames (𝐾 ≪ 𝑇 ) to represent the video, resulting in the sampled Ê ∈ R 𝐾 ×𝑁 𝐼 ×𝐷 𝐼 .(2) A trainable Q-former serves as the connection module to bridge the modality gap between vision and language. It takes frame embeddings Ê as inputs and outputs a set of fixed-length frame tokensF = {f 1 , f 2 , • • • , f 𝐾 } , F ∈ R 𝐾 ×𝑁 𝐶 ×𝐷 𝐶 , f 𝑡 ∈ R 𝑁 𝐶 ×𝐷 𝐶 ,where 𝑁 𝐶 is the token number of each frame (𝑁 𝐶 ≪ 𝑁 𝐼 , e.g., 𝑁 𝐶 = 32 and 𝑁 𝐼 = 257 in InstructBLIP), and 𝐷 𝐶 is the dimension of the connection module.(3) Each f 𝑡 in F are concatenated together to obtain the flattened F ∈ R (𝐾 •𝑁 𝐶 ) ×𝐷 𝐶 , followed with a fully-connected layer to project F into the LLM's dimension 𝐷 𝐿 . At last, the final projected F ∈ R (𝐾 •𝑁 𝐶 ) ×𝐷 𝐿 is fed into the frozen LLM (e.g., FLAN-T5 [7] or Vicuna [55] ) serving as soft prompts, together with the word embeddings of question Q, to generate the answer text A.The model is trained by optimizing the trainable parameters 𝜃 of the model 𝑃 with the autoregressive language modeling objective:EQUATIONwhere A 𝑡 is predicted autoregressively at position 𝑡, and 𝐿 𝑎 is the sequence length of the ground truth answer text A. Our motivation is to replace the uniformly sampled frames Ê in step (1) with question-critical frames as visual inputs. 
method: 
 Figure 2 (a) gives an overview of our framework. After extracting the frame embeddings E as in Section 3, step (1), our GCG will select the most question-critical 𝐾 frames Ê from E, as the visual inputs for the LMM. To ensure the selected frames are most relevant to answering the question, the GCG module will be additionally optimized by the pseudo-labels of question-critical moments, resulting in the regression objective L 𝑟𝑒𝑔 from weakly grounded timestamps, and the contrastive objective L 𝑐𝑜𝑛 aligning the paired description-moment pairs while pushing away unpaired ones.For video representations, along with the frame embeddings E, we also extract the corresponding class tokens E [𝐶𝐿𝑆 ] ∈ R 𝑇 ×𝐷 𝐼 from E for further pseudo-labels generation and contrastive grounding.For language representations, we tokenize the question Q into a sequence of words and then feed them into the text encoder of EVA-CLIP, to get word-level embeddings Q = q 𝑡 𝐿 𝑞 𝑡 =1 ∈ R 𝐿 𝑞 ×𝐷 𝐼 , where 𝐿 𝑞 denotes the sequence length of the question. We get the embeddings of the fused event description (detailed in 4.2) the same way as Q, but only retain the class token d [𝐶𝐿𝑆 ] ∈ R 𝐷 𝐼 to represent it for further pseudo-labels generation. Considering the powerful visual-language alignment ability of EVA-CLIP, as in Figure 2 (b), we utilize its joint-trained image and text encoder to provide pseudo labels for timestamps of question-critical moments as weak supervision.Event Description Generation. To adapt the textual representation for better event description and reduce the semantic gaps, we directly prompt the LLM inside the LMM (e.g., Vciuna [55] ) to fuse the question and answer pairs with hand-written demonstrations. For example, the QA-pair [Q: How does the boy react after opening the present? A: carry up the dog] will be transformed into the declarative event description [The boy carries up the dog after opening the present.]. Since the event description is composed of simple changes in the grammatical structure of the question Q and answer A, most opensourced or API-based LLMs can easily achieve this. Notably, the event descriptions provide more accurate textual descriptions for question-critical moments because the answer content is included.Pseudo Labels Generation. As in Figure 2 (b), we represent the video and event description with the class tokens E [𝐶𝐿𝑆 ] ∈ R 𝑇 ×𝐷 𝐼 and d [𝐶𝐿𝑆 ] ∈ R 𝑇 ×𝐷 𝐼 respectively, to obtain the weakly labeled question-critical timestamps. In detail, we compute the cosine similarities between E [𝐶𝐿𝑆 ] and d [𝐶𝐿𝑆 ] and get the similaruity scores 𝑠 ∈ R 𝑇 , which recording the relevance between each frame and the event description. Then, we choose the indexes of the highest Top-𝐾 scores in 𝑠 as 𝑤 ∈ R 𝐾 , where each element 𝑤 𝑘 ∈ {1, 2, • • • ,𝑇 }, to be the timestamps of question-critical frames. We verify the effectiveness of the weakly grounded 𝑤 in the analysis 5.3.Recent research [41] highlights the superiority of end-to-end Gaussian mask learning for grounding in VideoQA. Motivated by this, we design the effective weakly supervised Gaussian-based Contrastive Grounding (GCG). However, unlike [41] to generate a single Gaussian mask, we generate multiple Gaussian masks to characterize the multi-event temporal structure of the video. Moreover, our GCG is optimized from both the QA supervision L 𝑣𝑞𝑎 and weakly labeled supervision L 𝑟𝑒𝑔 and L 𝑐𝑜𝑛 , which is more effective for discovering the question-critical moments.Specifically, we utilize the Gaussian generator to obtain 𝐾 Gaussian masks 𝑔 = {𝑔 1 , • • • , 𝑔 𝐾 } , 𝑔 𝑘 ∈ R 𝑇 , depending on the video and question. These Gaussian masks will be combined into an overall weight distribution 𝑝 ∈ R 𝑇 , to indicate the importance of each video moment. Notably, 𝑝 tends to have 𝐾 peaks, representing the most question-critical 𝐾 frames, with corresponding indexes to be the centers of each Gaussian function 𝑔 𝑘 .As in the right part of Figure 3 , the Gaussian generator consists of a cross-modal embedding layer and a transformer encoder [36] . The cross-modal embedding layer is a down-sampling linear layer with learnable modal-type embeddings and positional embeddings. The concatenated multimodal embeddings M = [E [𝐶𝐿𝑆 ] ; Q] ∈ R (𝑇 +𝐿 𝑞 ) ×𝐷 𝐼 serve as inputs for the Gaussian generator:M = Linear(M), M ∈ R (𝑇 +𝐿 𝑞 ) ×𝐷 𝐺 M[: 𝑇 ] = M[: 𝑇 ] + Type 𝑉 + Pos; M[𝑇 :] = M[𝑇 :] + Type 𝑇 (2)Next, a standard transformer encoder is adopted to establish the cross-frame dynamics and cross-modal interactions, which takes the embedded M and yields M ∈ R 𝑇 ×𝐷 𝐺 (only reserve the first 𝑇 embeddings). Then, we use attention pooling to summarize the outputs M along the temporal dimension and derive the global video representations G ∈ R 𝐷 𝐺 . As G integrates all the video and question information, we predict the centers 𝜇 ∈ R 𝐾 of 𝐾 learnable Gaussian functions weighting over the entire video sequence, through G with a fully connected head activated by Sigmoid function:EQUATIONGiven predicted 𝜇, we get 𝐾 Gaussian functions 𝑔 = {𝑔 1 , • • • , 𝑔 𝐾 } as masks, parameterized with (𝜇, 𝜎):EQUATIONwhere 𝜎 is a hyperparameter controlling the width of the Gaussian curve. Then, the weight distribution 𝑝 ∈ R 𝑇 of each video moment is generated by summing each 𝑔 𝑘 :EQUATIONNorm(•) scales values into the range [0, 1]. As the 𝐾 peaks in 𝑝, whose corresponding indexes tend to be {𝜇 1 , • • • , 𝜇 𝐾 }, represent the most question-critical 𝐾 frames, we optimize the Gaussian generator with the regression objective to measure the discrepancy between the predicted centers 𝜇 ∈ R 𝐾 and the weakly grounded timestamps 𝑤 ∈ R 𝐾 by smooth 𝐿1 loss:EQUATIONContrastive grounding aims to ensure the selected moments are most relevant to the event description. To achieve this, we learn a cross-modal embedding space, where the embeddings of the event description d [𝐶𝐿𝑆 ] should be well aligned with the selected positive frames E 𝑝𝑜𝑠 ∈ R 𝐾 ×𝐷 𝐼 , which are derived from the weight distribution 𝑝, and far away from those irrelevant ones. E 𝑝𝑜𝑠 are also the class tokens of the selected frame embeddings Ê ∈ R 𝐾 ×𝑁 𝐼 ×𝐷 𝐼 , which will be the visual inputs of LMMs for final answer prediction. Positive Moments Selection. Since the distribution 𝑝 weights each video moment based on its contribution to the question, we select the Top-𝐾 elements from E [𝐶𝐿𝑆 ] according to 𝑝, and obtain E 𝑝𝑜𝑠 ∈ R 𝐾 ×𝐷 𝐼 as the positive frames. However, the selection via vanilla hard Top-𝐾 produces a discrete selection, making it inapplicable for end-to-end training. We address this issue by adopting a differentiable Top-𝐾 using the perturbed maximum method [2] .Negative Moments Mining. To distinguish highly confusing scenes, we mine negative moments within the same video as intranegative frames E 𝑖𝑛𝑡𝑟𝑎 ∈ R 𝑁 𝑖𝑛𝑡𝑟𝑎 ×𝐷 𝐼 , by sampling frames with the lowest 𝑁 𝑖𝑛𝑡𝑟𝑎 weights in 𝑝 from E [𝐶𝐿𝑆 ] . We also use 𝑁 𝑖𝑛𝑡𝑒𝑟 frames randomly sampled from other videos within the same batch to serve as inter-negative frames E 𝑖𝑛𝑡𝑒𝑟 ∈ R 𝑁 𝑖𝑛𝑡𝑟𝑎 ×𝐷 𝐼 . These negative samples from both the same video and other videos can provide richer information. The objective is described as an infoNCE loss:L 𝑐𝑜𝑛 = - 1 𝐾 𝐾 ∑︁ 𝑘=1 log exp(d [𝐶𝐿𝑆 ] ⊗ E 𝑝𝑜𝑠 𝑘 /𝜏 ) exp(d [𝐶𝐿𝑆 ] ⊗ E 𝑝𝑜𝑠 𝑘 /𝜏 ) + SUM SUM = 𝑁 𝑖𝑛𝑡𝑟𝑎 ∑︁ 𝑖=1 exp(d [𝐶𝐿𝑆 ] ⊗ E 𝑖𝑛𝑡𝑟𝑎 𝑖 /𝜏 ) + 𝑁 𝑖𝑛𝑡𝑒𝑟 ∑︁ 𝑗 =1 exp(d [𝐶𝐿𝑆 ] ⊗ E 𝑖𝑛𝑡𝑒𝑟 𝑗 /𝜏 ) (7)𝜏 is the temperature factor and ⊗ is the dot product. Contrastive grounding can maximize the similarity between the query d [𝐶𝐿𝑆 ] and a group of corresponding positive video moments E 𝑝𝑜𝑠 under the joint embedding space while pushing away negative ones.With the distribution 𝑝 optimized by both L 𝑟𝑒𝑔 and L 𝑐𝑜𝑛 , we select the most weighted 𝐾 frame embeddings from E ∈ R 𝑇 ×𝑁 𝐼 ×𝐷 𝐼 based on the 𝑝, and obtain the selected frame embeddings Ê ∈ R 𝐾 ×𝑁 𝐼 ×𝐷 𝐼 . This process replaces the uniform sampling, and the same perturbed maximum method is adopted for differentiability. At last, we feed Ê into the Q-Former and LLM as the steps (2) and (3) in Section 3 to autoregressively predict the answer A. During training, the whole pipeline is optimized by the joint objective:EQUATION𝛼 1 and 𝛼 2 are the hyper-parameters to control the strengths of GCG. During the inference process, GCG only generates the distribution 𝑝 as in Section 4.3, and obtains the most weighted 𝐾 frame embeddings Ê via a fully discrete Top-𝐾 selection. 
experiments 5.1 datasets: 
 NExT-QA [40] We choose InstructBLIP [8] and BLIP-2 [21] as our LMM for their representative structure and widespread use, with EVA-CLIP [10] as the image encoder, Q-former [21] as the connection module, and Vicuna [55] or FLAN-T5 [7] as the large language model. Following We first explore the performance of InstructBLIP with different frames as visual inputs to verify the effectiveness of the pseudolabeled 𝑤: Pseudo-labeled (E) means we choose the 𝐾 frames whose indexes correspond to 𝑤 ∈ R 𝐾 as visual inputs (detailed in Section 4.2). Pseudo-labeled (Q) is obtained the same way as Pseudo-labeled (E) but with the pure question for similarity computation. Baseline means we uniformly sample 𝐾 frames as visual inputs.Figure 4 shows that Pseudo-labeled (E) exhibits significantly improved performance, particularly in benchmarks featuring longer videos and more complicated questions (+6.6% for NExT-QA and +6.8% for Intent-QA). This performance gap emphasizes the need and potential for more future work to effectively localize questioncritical frames as visual inputs when using LMMs in video-language tasks. This also verifies the effectiveness of using the event description to provide pseudo-labels as weak supervision. Moreover, Pseudo-labeled (E) performs much better than Pseudo-labeled (Q). This can be explained from two perspectives: (1) The event descriptions include the contents of the answers needed to be grounded, filling the semantic gap between the pure question and the answer. (2) The CLIP models are mostly pre-trained on images and declarative texts, therefore the declarative event descriptions are more suitable for similarity computation to decide keyframes.Table 1 and 2 show the superiority of our method. For the multichoice setting, we achieve an accuracy of 74.6%, 72.1%, and 73.1% in NExT-QA, Causal-VidQA, and Intent-QA respectively. For the openended setting, we achieve an accuracy of 61.7%, 49.5%, and 49.9% in MSVD-QA, MSRVTT-QA, and ActivityNet-QA respectively. To ensure a fair comparison, we also apply the same settings to get the results for the vanilla InstructBLIP on these datasets as baselines, with 𝐾 = 4 frames uniformly sampled from the 𝑇 = 32 frames as visual inputs. Although the baseline InstructBLIP performed fair on these datasets, our proposed GCG showed a significant improvement, particularly in questions that require complex causal-temporal reasoning (+2.1% and +2.7% for Tem and Cau in NExT-QA, +2.3% for Cou in Causal-VidQA). When both using BLIP-2 as the baseline model, GCG still performs better than SeViLA [52] (73.6% vs 71.5% in NExT-QA) without extra pre-training or a multi-stage training scheme. Moreover, despite not having undergone video-text pretraining, our method still surpasses those large-scale pre-trained models (e.g., HiTea, COSA, VALOR) on MSVD-QA, MSRVTT-QA, which primarily feature straightforward questions and short videos (10-15s) . We also observe that the improvements on ActivityNet-QA (+3.6%) are much larger than the MSVD-QA (+2%) and MSRVTT-QA (+2.5%). This can be attributed to the average video length of ActivityNet-QA being 180 seconds, which is much longer than MSVD-QA (10s) and MSRVTT-QA (15s), emphasizing the necessity of discovering question-critical moments more with our method.We investigate the role of our framework with different variants and hyperparameters of GCG, by using InstructBLIP as the baseline model, and NExT-QA and MSVD-QA as the default benchmarks. Table 3 : Ablation studies on loss components of GCG. Loss components in GCG. We exhaust the combination of different loss components in GCG. Table 3 shows the results:• L 𝑣𝑞𝑎 solely hardly outperforms the baseline, because the Gaussian generator can not identify the causal scene without supervision of question-critical moments. This reflects our motivation in using the event descriptions to generate pseudo labels as weak supervision. • L 𝑣𝑞𝑎 + L 𝑟𝑒𝑔 and L 𝑣𝑞𝑎 + L 𝑐𝑜𝑛 match equally in accuracy that consistently surpasses baseline and L 𝑣𝑞𝑎 . L 𝑟𝑒𝑔 is responsible for regularizing the indexes of peaks in 𝑝 to approximate the timestamps of weakly grounded 𝑤, and L 𝑐𝑜𝑛 ensures the maximization between the selected moments and event descriptions. • L 𝑣𝑞𝑎 + L 𝑟𝑒𝑔 + L 𝑐𝑜𝑛 is the complete GCG, which further boosts the performance significantly in all cases, showing that L 𝑟𝑒𝑔 and L 𝑐𝑜𝑛 contribute in different aspects and their benefits are mutually reinforcing.Number of frames 𝑇 and Gaussian widths 𝜎. We evaluate how the number of overall sampled frames 𝑇 and the widths 𝜎 of Gaussian functions will influence the performance. Figure 5 (a) indicates that performance improves as more frames are included, however, beyond a certain threshold (𝑇 = 48), there is a performance drop. This suggests that too many frames may introduce redundancy and noise, while too few frames miss important information. As for 𝜎 in Equation 4, it essentially decides the width and the degree of dispersion of the Gaussian distribution 𝑔. A larger 𝜎 generates more dispersed 𝑔 with a more exploratory 𝑝 and vice versa. In Figure 5 (b), we vary 𝜎 from 0.1 to 0.7 and observe that the performance fluctuates in a range of [73.9, 74.6] for NExT-QA and [61. 4, 61.7] for MSVD-QA. The performance of NExT-QA is more sensitive to 𝜎, and we argue that this is because the average video length of NExT-QA (44s) is much longer than MSVD-QA (10s). 65 0 4 8 16 32 64 60.7 61.4 61.7 61.5 60 65 4 8 16 24(a) effect of N_intra (b) effect of N_inter Ablations on negative sample numbers 𝑁 𝑖𝑛𝑡𝑟𝑎 and 𝑁 𝑖𝑛𝑡𝑒𝑟 . We also investigate how the number of intra-negative samples 𝑁 𝑖𝑛𝑡𝑟𝑎 and inter-negative samples 𝑁 𝑖𝑛𝑡𝑒𝑟 influence the performance. As verified in Figure 6 (a), sampling enough intra-negative moments is beneficial to mining the positive moments for reasoning. Figure 6 (b) shows that the performance gains also increase as the number of inter-video negative moments increases. Moreover, the impact of intra-negative moments within the same video is larger than inter-negative moments, because the scenes in the same video are more confusing to distinguish the true question-critical moments. We also observe that the performance decreases after the 𝑁 𝑖𝑛𝑡𝑟𝑎 and 𝑁 𝑖𝑛𝑡𝑒𝑟 reach a specific value of 16 and 32 respectively. We argue that selecting excessive negative moments tends to distract positive moments and therefore degrades the performance. We finally set the 𝑁 𝑖𝑛𝑡𝑟𝑎 = 16 and 𝑁 𝑖𝑛𝑡𝑒𝑟 = 32 which balance well the performance and computational cost.Ablations on hidden size 𝐷 𝑅 and layer number 𝑁 . To determine the optimal configuration for our Gaussian generator, we set different layer number 𝑁 and hidden size 𝐷 𝑅 for the transformer Figure 7 : Qualitative results on NExT-QA test set. The frames selected by our method are highlighted in blue dashed lines. The ground truth answers are in green. We also display the weakly grounded similarity scores of each frame.encoder in the Gaussian generator. The results of these variants on NExT-QA are shown in Table 4 . We can see that when the layer number of the encoder is fixed to 𝑁 = 2, the model achieves the best performance of 74.8% with 𝐷 𝐺 = 1024. However, such improvement in performance comes at the cost of a significant increase in the number of parameters (27.6M). To ensure the flexibility and adaptability of our method, we choose 𝐷 𝐺 = 256 as our default setting, which achieves a balance between fair performance (74.6%) and much fewer parameter quantities (2.0M). We can also observe that when the hidden size is fixed to 𝐷 𝐺 = 256 and the layer number 𝑁 ≥ 2, the change in 𝑁 has a relatively minor impact on performance. For better computation efficiency, we adopt 𝑁 = 2 for our Gaussian generator. Further LoRA tuning to push better results. We note that LMMs with GCG can be considered a strong model for VideoQA. To achieve better performance, we further add per-task LoRA tuning [15] for the frozen language model in LMMs (using a rank of 16), yielding new SOTA results on these VideoQA benchmarks. These new SOTA results in Table 5 indicate the extensibility and flexibility of the design of our GCG, which can be easily combined with other components for LMMs.We also present qualitative results in Figure 7 , along with the frames identified by GCG (bounded with blue dashed lines) and the weakly grounded similarities scores. Both cases show semantic correspondence between the question and the selected moments, enhancing the interpretability of LMMs for VideoQA by revealing which visual scenes result in the answers. In the upper case, the frames identified by GCG precisely correspond to the event [The baby rolled the roller on the toy when she reached it] with relatively higher similarity scores, demonstrating the ability of GCG to localize video moments crucial for answering the question. By leveraging the information from this localized segment, the LMM can successfully arrive at the correct answer [roll the roller on the toy]. In contrast, without our GCG, the presence of massive redundancy in the video overwhelms the reasoning process and leads to a false prediction of [laugh loudly]. 
conclusion: 
 In this paper, we have studied the problem of discovering questioncritical moments in videos when adapting LMMs for VideoQA tasks. To address the shortcomings of the uniform sampling strategy and the absence of human annotations for question-critical timestamps in VideoQA datasets, we introduce a weakly-supervised framework to force the LMMs to reason out the answers by grounding questioncritical moments as visual inputs. To achieve this, we utilize the CLIP models to automatically provide the pseudo-labeled timestamps of keyframes. With these keyframes as additional weak supervision, we propose the Gaussian-based Contrastive Grounding, a flexible and lightweight method to dynamically select questioncritical moments with end-to-end training. Through a series of experiments and analyses, we have demonstrated the effectiveness of our approach in various challenging VideoQA tasks, particularly excelling in causal-temporal reasoning. Limitations. Despite the significantly improved performance on several VideoQA datasets with our method for LMMs, it's essential to acknowledge the potential presence of language bias in the frozen language models. In our future work, we plan to mitigate these biases and enhance the reasoning ability of current LMMs further. 
 
 Appendix:  


Decision: The paper discusses a weakly supervised video grounding method for the task of Video Question Answering (VideoQA). The approach uses  CLIP models to generate pseudo-labeled timestamps and introduces a Gaussian-based Contrastive Grounding method for selecting  question-critical keyframes. Experiments and analyses demonstrates consistent performance improvements across multiple datasets.

(+) On the positive side, the reviewers found the method to be interesting and effective and appreciate the impressive results on several popular  datasets. 

(-) On the negative side, there are still some concerns on the  limited novelty, insufficient evaluation, hyperparameter selection, and clarity of the writing.

Several technical questions and suggestions were raised by the reviewers. The authors have taken these into consideration. Overall, all reviewers agreed to accept the paper after the rebuttal. Therefore, the AC recommends accepting the paper.

Reviews: review: paper_summary: paper_summary: This paper proposes a weakly supervised grounding framework that compels LMMs to deduce answers by grounding question-critical moments as visual inputs. This approach utilizes CLIP models to provide pseudo-labeled timestamps and devises a Gaussian-based Contrastive Grounding (GCG) method for weakly-grounded selection of question-critical moments. Experiments and analyses demonstrate the effectiveness of this approach in various challenging VideoQA tasks, particularly excelling in causal-temporal reasoning.
strength_weakness: strength_weakness: The paper presents an effective model, Weakly Supervised Gaussian Contrastive Grounding with Large Multimodal Models for Video Question Answering. The model has been rigorously validated on five different VideoQA datasets. Extensive experiments verify the effectiveness of the framework, and the proposed model achieves substantial improvements compared to previous state-of-the-art methods.
limitations: limitations: •	The novelty of the paper is limited, as the main idea of weakly supervised grounding is similar to [41]. Furthermore, the proposed model appears to function as a pre-processing module for VQA, with almost no specifically designed modules for the VQA model itself.
•	The review of the related work is not sufficient. The paper should add a section in related works about weakly supervised grounding.
•	Some experimental results are not very convincing. For example, the paper did not provide the results on Causal-VidQA in Figure 4.
paper_summary: paper_summary: This paper aims to enhance VideoQA through two approaches. The first involves selecting question-critical keyframes, rather than uniformly sampled frames, and using CLIP scores to generate pseudo-labels. The second strategy introduces Gaussian-based Contrastive Grounding (GCG) to learn multiple Gaussian masks, capturing the video's temporal structure. These methods are incorporated into two VideoQA approaches, demonstrating consistent performance improvements across multiple datasets.
strength_weakness: strength_weakness: 1.	Selecting question-relevant keyframes improves grounding by narrowing the receptive field for LMMs, enhancing the localization of correct timestamps.
2.	Learning multiple Gaussian masks to generate positive and negative moments introduces a new contrastive objective, improving grounding performance.
limitations: limitations: 1.	The method uses both question and answer to generate event descriptions during keyframe selection, but during inference, only the question is available. If the inference relies solely on the question for event description (as depicted in Figure 3), how does the approach address the absence of an answer, and how are performance changes evaluated?

2.	In test cases (Figure 7), answers are used to generate event descriptions. Additionally, what do the displayed scores (ranging from 10 to 30+) represent? Are these CLIP scores?
paper_summary: paper_summary: VideoQA aims to answer questions using video information. Large Multimodal Models (LMMs) have achieved success in image-language understanding but struggle with VideoQA, often overlooking important visual cues. Existing VideoQA datasets lack human annotations for crucial timestamps. To address this, a weakly supervised framework is proposed, utilizing CLIP models to identify key frames as target moments and pseudo-labels. A Gaussian-based Contrastive Grounding (GCG) module is introduced to sample question-critical frames as visual inputs for LMMs. Experimental results demonstrate the improvements over previous methods.
strength_weakness: strength_weakness: - The motivation is stated clearly. 
- More attention to question-relevant visual clues will promote the power of LLM for VideoQA.
- The experiments demonstrate the effectiveness of the proposed method.
limitations: limitations: 1. The effectiveness of the proposed method heavily relies on selecting $\alpha$ in Equation (8). Conducting a sensitivity analysis of $\alpha$ is necessary, and discussing insights into selecting this hyper-parameter is essential.



2. Despite the proposed method demonstrating enhancements compared to other state-of-the-art methods, it incurs additional costs in localizing question-critical moments. Further discussion is warranted regarding this trade-off.



3. Some grammar errors and typos (e.g., ...when both using L677) persist in the current manuscript and should be corrected.