Conditional Diffusion Model for Open-ended Video Question Answering

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 07:08:45

Abstract: Open-ended VideoQA presents a significant challenge due to the absence of fixed options, requiring the identification of the correct answer from a vast pool of candidate answers. Previous approaches typically utilize classifier or similarity comparison on fusion feature to yield prediction directly, lacking coarse-to-fine filtering on numerous candidates. Gradual refining the probability distribution of candidates can achieve more precise prediction. Thus, we propose the DiffAns model, which integrates the diffusion model to handle open-ended VideoQA task, simulating the gradual process by which humans answer open-ended question. Specifically, we first diffuse the true answer label into a random distribution (forward process). And under the guidance of answer-aware condition generated from video and question, the model iteratively denoises to obtain the correct probability distribution(backward process). This equips the model with the capability to progressively refine the random probability distribution of candidates, ultimately predicting the correct answer. We conduct experiments on three challenging open-ended VideoQA datasets, surpassing existing SoTA methods. Extensive experiments further explore and analyse the impact of each modules, as well as the design of diffusion model, demonstrating the effectiveness of DiffAns. Our code will be available.

Main paper: 
 introduction: 
 Video Question Answering (VideoQA) seek to explore technique that combines video when answering question [50] , representing one of the challenging multimodal tasks. The VideoQA task can be categorized into two types: multi-choice (several candidate answers are provided for each question) [23, 44] and open-ended (a very large candidate answers set are constructed for all questions) [18, 48, 49] . In this paper, we focus on the open-ended VideoQA which is more challenging than multi-choice VideoQA. Open-ended VideoQA is popularly set as a multi-class classification problem, where the means of answer generation are mainly divided into two categories: classifier and similarity method. Classifier method directly use a classifier to predict the answer [12, 27, 45, 51] , as illustrated in Figure 1(a) . Initially, the features of video and question are modeled for interaction and fusion, followed by the usage of a classifier to classify the fused feature on a large candidate answer set. Cross-entropy is employed as the loss function during training, while the candidate corresponding to the highest probability score is selected as the answer during prediction. This method is straightforward but may lose the textual semantic information of answer while treating candidates solely as classification labels. Therefore, some recent methods [5, 46, 47] attempt to encode the textual content of candidates as features, aiming to optimize the answer prediction semantically, as depicted in Figure 1(b) . During training, they employ contrastive learning to minimize the distance between the fused features of video-question pairs and the features of ground truth answers. Then they select answer with the highest similarity to the fused features as the prediction.These two methods, while both achieving decent results, select only one answer from a vast candidate set directly without coars-to-fine filtering, inevitably leading to significant inaccuracy in answer prediction. According to the general intuition when facing open-ended question, human typically continuously search for the direction, narrow down the scope of candidates and finally respond more precisely. For example, given the video and question in Figure 1 (c), we can first come up with some more relevant answers (with a higher probability), and then lock in the correct answers ("phone"). This characteristic is what the model precisely needs when handling open-ended VideoQA. Excitingly, the recent diffusion models [33, 38] , which are popular and applied in various fields [6, 20, 25, 43] , precisely align with this advantage. Their notable effectiveness lies in progressively generating vivid images from noisy images, approaching real scenes step by step. Thus, we incorporate the gradual refining from random to precise features of diffusion models to simulate human intuition for open-ended VideoQA, as shown in Figure 1(c) . Firstly, we interact with the video and question to generate condition aware of answer. Then, we introduce a random probability distribution of candidate answers and use a denoising network based on diffusion models to incrementally refine the probability distribution of answers, guided by the condition possessing fusion information of video and question. In this way, the model continuously iterates to search for the direction and narrow down the scope of answers until finally locking onto the answer accurately. This achieves a more precise process of selecting the unique correct answer from a vast candidate set.Therefore, to model the open-ended VideoQA process mentioned above, we propose DiffAns to tackle open-ended VideoQA with Diffusion Model. Specifically, we first embed the video and question using pre-trained image and text backbones, and achieve semantic projection of the visual modal and contextual perception of question. Next, we construct the Answer-aware Condition Generator for generating condition aware of answre for the subsequent diffusion model. It handles the video and question features through interaction and fusion, extracting the key information required for answering which ensures the correct distribution updating during denoising. After that, the proposed Answer Denoiser encode the answer probability distribution and denoises it with noise intensity guided by answer-aware condition. Then, it decodes the denoised feature yielding the distribution at next step. Iterate that process and finally obtain the correct answer probability distribution.Our contributions can be summarized as follows:•We tackle the open-ended VideoQA task with diffusion model and propose DiffAns. It emulates human intuition by iterative seeking answer direction and narrowing down scope of the answer to select the unique correct answer from a large candidate set. The coarse-to-fine refining process achieving a more precise answer prediction. • We design the Answer Denoiser to enhance the denoising process of diffusion model. It entails encoding and decoding answer probability distribution and incorporates noise intensity to refine the noisy distribution under the guidance of answer-aware condition.used and challenging open-ended VideoQA datasets, demonstrating the superiority of our method over existing SoTA approaches. Additionally, our extensive experiments further explore and analyse the impact of each modules, as well as the design of diffusion model. 
related work 2.1 open-ended videoqa: 
 Open-ended VideoQA is aimed at comprehending video content to answer to related questions without provided answer options, unlike multi-choice VideoQA [50] . Thus, a vast answer set is often constructed as global answer options specific to the dataset. Consequently, each answer is treated as class label and the task is transformed into a multi-label classification task [50] . In recent years, two predominant solutions have emerged: classifier method and similarity method. Classifier method is more prevalent, involving various forms of interaction modeling between video and question, which can be spatio-temporal [7, 11, 19, 21] , hierarchical [7, 22, 29, 34, 45] , multi-scale [12, 34] , multi-granularity [45] , or Transformerbased [27, 28, 51] . And the final fusion of feature is fed into a classifier. This method typically employ cross-entropy loss for model training, and select the label with the highest probability score as the predicted answer. However, such straightforward approaches to a certain extent sacrifice the intrinsic semantic information of each answer text. Recently, some endeavors [5, 46, 47] have sought to harness the semantic of answers to guide the training and inference process of model. During the training phase, they employ contrastive learning as a loss function to steer the distribution of fusion features and answer features after interacting and fusing the video and the question. This method treats the label answer as positive sample while considering other answers as negative samples, aiming to align the distributions of fused feature with the label answer closer and diverge from other candidate answers.In the inference phase, they compute the dot product similarity between the fused feature and all candidate answers, selecting the answer with the highest similarity as the prediction. Despite the respectable performance achieved by both methods, the process of directly selecting the sole answer from an extensive answer set in one step raises doubts. We aspire for models to progressively refine and determine answers step by step to possess the coarse-tofine filtering of singular answer selection. Thus, we leverage the corresponding properties of diffusion models to tackle open-ended VideoQA. We realize an intuitive approach, iteratively narrowing down the range of answers and ultimately predicting the correct answer. We use denoiser of diffusion model to mimick the process of iteratively refining answers from the vague recesses of the mind in accordance with human intuition.The diffusion model [17, 33, 41] belongs to generative models, which simulate diffusion processes in physical thermodynamics. During the training process, it initially adds noise to the labels through a forward process until they approximately conform to a Gaussian distribution. Subsequently, in the reverse process, it learns to reverse this noisy process through a denoiser to obtain the original labels. In recent years, the remarkable success of this model in the generative domain has been exhilarating, especially in the field of visual generation [2, 38, 40] , which have generated highquality images that are difficult to distinguish between real and fake, completely overturning previous generative models. Recently, in the domain of video generation, models [3, 16] have similarly produced stunning generated videos using diffusion models. Due to its progressive refinement of labels and the continuous improvement in training and inference efficiency, it has also been applied in various fields such as Image Detection [6] , recommend system [43] ,Video-Text Retrieval [20] , and Video-Text Localization [25] . In the early stages, the denoising networks of diffusion models were typically dominated by U-Net [39] , but recently, Transformer [42] have been gradually explored and applied, achieving remarkable results. However, to our knowledge, researchers have not yet utilized diffusion models to tackle VideoQA. Therefore, to address this gap, we fully leverage the characteristics of diffusion models to handle open-ended VideoQA in a more precise manner. Meanwhile, we implement the denoiser of the diffusion model using transformerbased modules. To the best of our knowledge, we are the first to adapt the diffusion model for VideoQA. 
prelimilaries: 
 To better comprehend our utilization of the diffusion model in achieving a coars-to-fine answer process for open-ended VideoQA, we first describe the definition of open-ended VideoQA, and then clarify the theory of the forward and reverse processes within the diffusion model.For open-ended VideoQA, the input typically comprises an untrimmed video (i.e., V = {𝑣 𝑖 } 𝑁 𝑣 𝑖=1 ) and a corresponding question (i.e., Q = {𝑞 𝑖 } 𝑁 𝑞 𝑖=1 ), where 𝑁 𝑣 and 𝑁 𝑞 denote the number of frames and words respectively. Concurrently, there exists an extensive answer set (i.e., A = {𝑎 𝑖 } 𝑁 𝑎 𝑖=1 ) aiming to encompass all possible answers specific to dataset, where 𝑁 𝑎 represent the number of candidate answers. The model necessitates modeling the function Ω process to achieve the selection of the correct answer 𝑎 * = Ω 𝜃 (V, Q), where 𝑎 * ∈ A and 𝜃 denotes the parameters of model.Forward process is essentially a noisy process that requires continuously adding noise to the original label information 𝑥 0 ∼ 𝑞(𝑥 0 ) to obtain noisy data 𝑥 𝑇 after 𝑇 steps, where 𝑇 represents the intensity of the noise. As depicted in Figure 2 , in each step, the forward process of adding noise [17] can be represented asEQUATIONwhere 𝛽 denotes the variance schedule. The process to get 𝑥 𝑇 in 𝑇 steps can be defined asEQUATIONThrough reparameterization technique, the final noisy distribution 𝑥 𝑇 can be directly represented asEQUATIONwhere the 𝜖 𝑇 ∼ N (0, 𝐼 ) denotes the noise and 𝛼 𝑇 = 𝑇 𝑖=1 (1 -𝛽 𝑖 ). Reverse process is a denoising process, which continuously denoises the noised distribution 𝑥 𝑇 to obtain the original distribution 𝑥 0 , asymptotically. As depicted in Figure 2 , the reverse process of each step can be represented asEQUATIONwhere the 𝜇 𝜃 (𝑥 𝑡 , 𝑡) is predicted mean, 𝜃 is model parameter and 𝜎 2 𝑡 is associated with 𝛽 𝑡 . Similarly, in our work, we employ the Answer Denoiser to achieve this reverse process. However, we predict the distribution of candidate answers instead of 𝜇 𝜃 (𝑥 𝑡 , 𝑡), like [25] . 
method: 
 The main structure of DiffAns is illustrated in Figure 3 , comprising three modules. Firstly, Encoders achieve the encoding of both video and question, effectively mapping the visual modality into the semantic space of text modality, and facilitating contextual understanding of question. Subsequently, the Answer-aware Condition Generator interact and fuse the video and question to obtain answer-aware condition, essential for guiding the subsequent denoising process. Lastly, Answer Denoiser iterate to denoise the noised probability distribution integrating noise intensity, which simulate the coars-to-fine filtering on vast candidate answers.For the textual input (e.i., question Q), we introduce the robust RoBERTa [31] in NLP as our language backbone. Specifically, Q is initially treated as a text sequence of length 𝑙 𝑡 . If the sequence length is insufficient, padding is applied; otherwise, the sequence is truncated. Subsequently, they are tokenized and passed through the pre-trained RoBERTa, where the output of the last hidden layer serves as the representation of question (e.i., 𝑄 𝑒𝑚𝑏 ∈ 𝑅 𝑙 𝑡 ×𝑑 𝑡 ), where 𝑑 𝑡 denotes the hidden size of RoBERTa. To mitigate potential biases introduced by the pre-trained RoBERTa, we employ Encoder in Transformer [42] (e.i., a multi-head attention layer (ATTN) followed by a feedforward network (FFN)) to perceive the contextual information of question. This can be defined asEQUATIONwhere 𝐿𝑁 denotes a regularization method (i.e. LayerNorm). The definition of ATTN is:EQUATIONEQUATIONwhere 𝑊 𝑖 ∈ 𝑅 𝑑 ×𝑑 are trainable parameters, and 𝑑 is the size of our model dimension, which is equal to 𝑑 𝑡 in our implementation. 𝑆𝑜 𝑓 𝑡𝑚𝑎𝑥 is an activation function, 𝜏 denotes the temperature of attention. The 𝑎𝑡𝑡𝑛(𝑄, 𝐾, 𝑉 ) represents scaled dot product attention with 𝑄, 𝐾 and 𝑉 as the query, key and value respectively. And the definition of FNN is: where 𝑊 𝑖 ∈ 𝑅 𝑑 ×𝑑 are trainable parameters, 𝑅𝑒𝐿𝑈 represents an activation function.EQUATIONFor the visual input (e.i., video V), we employ the ViT [9] as the visual backbone, which is a widely used and effective model in CV. Specifically, we first uniformly sample 𝑙 𝑣 frames from the video. For video with fewer than 𝑙 𝑣 frames, we duplicate the last frame to fill the gap. Subsequently, each frame is embedded by the ViT initialized by CLIP [37] , resulting in video representation (e.i., 𝑉 𝑒𝑚𝑏 ∈ 𝑅 𝑙 𝑣 ×𝑑 𝑣 ). In order to align the visual and textual features in a unified semantic space, we froze the ViT and introduced projecter, a linear layer followed by LN. It project the visual feature into textual semantic space, which can be represented as follow:EQUATIONwhere 𝑊 ∈ 𝑅 𝑑 𝑣 ×𝑑 and 𝑏 ∈ 𝑅 𝑑 are trainable parameters.We aspire to achieve a comprehensive interaction and fusion of video 𝑉 𝑒𝑛𝑐 and question 𝑄 𝑒𝑛𝑐 , incorporating fusion features as condition for subsequent denoising process. This condition should encompass information for answering question, guiding the denoiser to refine candidates probability distribution more rapidly and accurately from random distributions. A intuition solution is the Decoder in Transformer [42] , which comprises a self-attention layer, a cross-attention layer and a FFN layer. Specifically, it first learns potential query semantic of 𝑄 𝑒𝑛𝑐 through self-attention to facilitate subsequent capture of video related information, which can be denoted as:EQUATIONwhere 𝐴𝑇𝑇 𝑁 (𝑄, 𝐾, 𝑉 ) denotes the attention layer. Then, the crossattention is employed with 𝑉 𝑒𝑛𝑐 as key and value and 𝑄 𝑐 as query respectively. It utilizes attention mechanism to focus on video information relevant to the question. By removing redundant information, the model obtains crucial feature for answering question. This process can be defined asEQUATIONFinally, we further explore deep semantic information through a FFN layer followed by LN to abstract the fusion of video and question, obtaining a more refined and higher-order condition. This can be denoted asEQUATION)Due to the truth that different videos and questions require different answers to be generated. The denoising process heavily relies on the quality of the generated condition (e.i., 𝐶). To further ensure the quality of 𝐶, we retain the cross-entropy loss to constrain the condition generation process. We perform mean pooling on the fused features 𝐶 to obtain the global information. Then, we map it to the dimension of the answer distribution through MLP. These can be defined asEQUATIONwhere 𝑊 1 ∈ 𝑅 𝑑 ×𝑑 and 𝑊 2 ∈ 𝑅 𝑑 ×𝑁 𝑎 are trainable parameters, 𝑝𝑜𝑜𝑙𝑖𝑛𝑔 denotes the mean pooling on token length dimension. 𝐸𝐿𝑈 denotes an activation function. Then, we employ cross-entropy loss to constrain the relationship between this distribution and the gold label, with the formula asEQUATIONwhere 𝑧 𝑖 = 1 if 𝑖 is the index of ground-truth answer 𝑎 * and 0 otherwise. 𝑁 𝑎 denotes the size of candidate answer set. ⨁ Feed-Forward ⨁ Q K, V N K, V Q Figure 4:The Answer Denoiser module, refining the noised answer distribution aggregating the intensity, under the guidence of answer-aware condition.Obviously, through these operations, the 𝐶 is expected to yield real answers 𝑎 * . Therefore, the condition 𝐶 inevitably possesses complete information for answer generation, bringing superior guidance to the subsequent denoiser.To mitigate the difficulty caused by selecting a single answer 𝑎 * from a vast answer set A, we introduce the conditional diffusion models, which progressively refine a random probability distribution 𝑥 𝑇 to the correct distribution 𝑥 0 over multiple steps 𝑇 using the Answer Denoiser (AD), as shown in Figure 4 . At each iteration, AD gradually refine the noised probability distribution, mimicking the direction selection and narrowing of the answer scope. Specifically, this process models 𝑝 𝜃 (𝑥 𝑡 -1 |𝑥 𝑡 ) by mapping 𝑥 𝑡 to denoised 𝑥 𝑡 -1 . AD mainly comprises three components: Answer Encoder module maps the candidates probability distribution 𝑥 𝑡 into the semantic space. Denoising Block module refines the feature of the noised distribution under guidance of condition 𝐶, and finally Answer Decoder module decodes the embedded distribution feature to obtain the denoised probability distribution 𝑥 𝑡 -1 after the current step.For Answer Encoder, initially, during the forward process we need to continuously add noise into original answer distribution. Unlike generative tasks [41] , the ground truth distribution is a onehot representation in scope of [0, 1]. Therefore, in order to align it with noise in the distribution, we first scale the distribution to the [-𝜆, 𝜆] to better fit the Gaussian distribution, following previous method [6, 25] . This is represented asEQUATION)After noise addition from 𝑥 0 to 𝑥 𝑇 , it is rescaled back to the original [0, 1] which can be denoted asEQUATION)We encode the distribution of noisy 𝑥 𝑡 using a linear layer, mapping it to the semantic space of the model, denoted asEQUATIONwhere 𝑊 ∈ 𝑅 𝑁 𝑎 ×𝑑 and 𝑏 ∈ 𝑅 𝑑 are trainable parameters.For Denoising Block, we introduce information of noise intensity 𝑇 to aid in the denoising process. We encode 𝑇 using a sinusoidal mapping and add it into 𝑥 𝑒𝑛𝑐 , obtaining the intensity-aware noisy answer encoding 𝑥 𝑖𝑛𝑡 . There we stack 𝑁 blocks based on crossattention and FFN to achieve denoising of 𝑥 𝑖𝑛𝑡 under guidance of condition 𝐶. In each block, we treat 𝑥 𝑖𝑛𝑡 as query and 𝐶 as key and value, respectively. These are interacted via cross-attention, yielding an answer distribution 𝑥 𝑐 fused with conditional information. This is formulated asEQUATIONThen, we refine the answer distribution through a FNN layer, represented asEQUATIONIn the above two formulations, we omitted LayerNorm for simplicity, which specific position in module can be seen in Figure 4 .Undergoing 𝑁 blocks, we achieve the denoising process for the step 𝑡, obtaining the refined answer encoding for 𝑥 𝑟 . For Answer Decoder, after a LayerNorm, we decode the encoded information 𝑥 𝑟 with MLP, transforming 𝑥 𝑟 into 𝑥 𝑡 -1 . This process can be represented asEQUATIONwhere 𝑊 1 ∈ 𝑅 𝑑 ×𝑑 and 𝑊 1 ∈ 𝑅 𝑑 ×𝑁 𝑎 are trainable parameters. After multiple iterations of denoising over several time steps 𝑇 , we derive the predicted x0 . Then, we employ the Kullback-Leibler divergence as loss of x0 and 𝑥 0 , aiming to minimize the distributional gap between them. This can be denoted asEQUATIONDuring the training process, the true label distribution 𝑥 0 is initially mapped to a distribution 𝑥 𝑇 close to Gaussian noise through a forward process. Subsequently, Answer Denoiser is applied in the reverse process, effectively denoising 𝑥 𝑇 under guidance condition 𝐶. After multiple iterations to obtaine the predicted probability distribution x0 , the diffusion loss 𝐿 𝑑𝑖 𝑓 is calculated. Additionally, to ensure the quality of the generated condition, 𝐿 𝑐 is incorporated into Answer-aware Condition Generator. Therefore, the overall loss during training can be expressed asEQUATION)where 𝛼 is hyparameter to control the weight of 𝐿 𝑐 .During the inference process, without ground truth answer, only the reverse process is feasible. The corresponding noisy distribution 𝑥 𝑇 is replaced with randomly generated Gaussian noise. The model graduallty refines the random distribution through the reverse process and obtains the final predicted probability distribution x0 . The ultimate prediction class label â is the candidate answer with the highest probability score, represented as follows: â = arg max( x0 ).(23) 
experiments 5.1 experimental settings: 
 5.1.1 Datasets. We evaluated our method with three widely used and challenging datasets in the open-ended VideoQA: MSVD-QA [48] , MSRVTT-QA [49] and TGIF-FrameQA [18] . The statistics on these three datasets can be seen in Table 1 . MSVD-QA [48] encompasses a plethora of topics, including humans, animals, various actions, and unique scenarios, providing rich content for evaluation. It comprises approximately 2k videos, with around 50k samples (i.e., QA pairs) in total. On average, each video has 25 related questions. The average video length is 10 seconds, and the average number of words in question is 6.6. The size of the candidate answer set is around 2k.MSRVTT-QA [49] represents a broader compilation, showcasing a more diverse visual content, including the most comprehensive categories. It contains around 10k videos, with approximately 244k samples in total. Similar to MSVD-QA, each video has an average of 24.4 questions. The average video length is 15 seconds, the longest among the three datasets. The average question length is 7.4, longer compared to MSVD-QA. The size of the candidate answer set is 4k, which is the largest of these dataset.TGIF-FrameQA [18] features videos sourced from real internet GIFs, accompanied by human-generated questions and answers, enhancing the authenticity. It comprises an astounding number of approximately 39.5k videos, with around 53.1k samples in total, implying that each video has fewer than 2 related questions on average. The average video length is 3 seconds due to the GIF format, and the average question length is 9.8, the longest among the three datasets. The size of the candidate answer set is 1,541.For all three datasets, we followed previous works and evaluated the models using Accuracy(%) as the metric.In the feature representation module, we sample 𝑙 𝑣 = 16 frames for video and fix the maximum length of input sequences to 𝑙 𝑡 = 20 tokens. Subsequently, we employ ViT-L14 initialized by CLIP [37] and RoBERTa-base [31] as the visual and language backbones respectively, both with the hidden dimension size 𝑑 𝑣 = 𝑑 𝑡 = 768. We froze the ViT and fine-tune the RoBERTa. The model is configured with a same dimension size 𝑑 = 768 of language backbone. For the MSVD-QA, MSRVTT-QA, and TGIF-FrameQA datasets, the numbers of denoising blocks 𝑁 are set to {4, 2, 3} respectively, with a loss weight 𝛼 = 1. Within the diffusion model, we utilize DDIM [41] as the diffusion strategy, cosine schedule for noise, with the scale 𝜆 = 0.5. And the sampling steps (e.i. noise intensity) 𝑇 = 50. During training, the model's learning rate is 1𝑒 -4, with a batch size of 128, and a maximum iteration of 10.We compared our DiffAns model with recent SoTA methods on three datasets in Table 2 . To highlight the competitiveness of our method, we analyze the result from three perspectives.SoTA methods addressing openended VideoQA are classifier-based, exhibiting commendable effectiveness. However, some recent approaches such as CoVGT [47] and ATM [5] , have also proposed the utilization of similarity-based method. Unlike other methods, they both have all embraced pretraining strategy. It is evident that similarity method is more suitable for models with pre-training process; otherwise, classifier method are sufficient to achieve commendable results, such as TranSTR [27] and KPI [24] , which achieve the best performance. Given the large size of answer set in open-ended VideoQA, both of the aforementioned models inevitably introduce difficulty by selecting a unique answer from such a multitude of candidate answers in a single step. Therefore, we opt to utilize a diffusion-based method to refine the predicted answer gradually, marking the first employment of diffusion methods in VideoQA. Ultimately, our results surpass all recent SoTA methods, whether classifier-based or similarity-based. This demonstrat the effectiveness of our proposed DiffAns model.As time goes on, an increasing number of advanced backbones have been proposed and applied in the field of videoQA. In the early time, SoTA methods mostly employ CNNbased model [1, 4, 10, 13, 14] as visual backbone and utilize word embedding [36] to encode question. With the success and application of Transformer [42] , it has also been introduced into VideoQA, achieving remarkable performance. Many methods have begun to use them as the backbone, whether in the field of visual [9] or language [8, 15, 31] domain. The two modal backbones utilized in our model are respectively the commonly used and efficient ViT [9] and RoBERTa [31] . Compared to our model, TranSTR [27] incorporates Faster R-CNN [1] for extracting object features in addition to ViT for the visual aspect. Moreover, a more advanced language model(e.i., Deberta [15] ) is utilized as language backbone. However, their ultimate performance still falls short of our model's. This further substantiates the effectiveness of the proposed diffusion-based paradigm.Size. Meanwhile, our analysis reveals that the difficulty of these three datasets precisely corresponds to the size of the answer set (lower accuracy indicating higher difficulty). TGIF-FrameQA [18] has the smallest answer set, with the highest accuracy, while MSRVTT-QA [49] has the largest answer set, with the lowest accuracy. Thus, it can be concluded that the broader answer scope in open-ended VideoQA leads to increased interference in model selection of the correct answer from numerous candidates, rendering the dataset more challenging. The proposed DiffAns focuses on coarsto-fine filtering candidate answers to address open-ended VideoQA. Experimental results demonstrate that our approach surpass SoTA methods, validating the effectiveness of multi-step denoising for selecting correct answer from large answer sets. Finally, an appropriate 𝛼 can make the two losses more compatible, leading to optimal result. 5.3.3 Denoising Block. We explore the impact of 𝑁 , which denotes the number of denoising blocks, as shown in Table 5 (b). It proves that too large or too small 𝑁 leads to decrease of model performance. This phenomenon may arise from insufficient layers causing inadequate denoising and excessive layers leading to overfitting. indicated, DDIM still outperforms DDPM in performance, thus we adopted DDIM. (4) The schedule of 𝛽 in diffusion model determines how the step size increases. When replacing cosine with linear, the decrease of model performance is observed, particularly on MSRVTT-QA. Therefore, cosine schedule is more suitable and is utilized in our model.To better align the probability distribution with Gaussian distribution for enhanced noise integration, we applied scaling process. The results under different 𝜆 are presented in Table 4b . It is observed that the model achieved optimal performance when 𝜆 = 0.5, indicating its suitability. Our analysis suggests that excessively low 𝜆 lead to information loss, while overly high 𝜆 hinder the model to develop robust denoising capabilities.5.4.3 Impact of steps𝑇 . We investigated the impact of varying𝑇 on the model's performance, as depicted in Table 4c , where t denotes the average time of inferring one sample on TGIF-FrameQA. A higher 𝑇 incurred greater computational costs. It was observed that our model achieve optimal performance when 𝑇 = 50. This contrasts with tasks such as Image Generation, where 1000 or more steps are commonly utilized. This difference can be attributed to the requirement of precise pixel-level generation in image tasks [41] , whereas our model focuses on deriving a probability distribution.DiffAns differs from other conventional methods in that model iteratively denoising the answers through multi-step refinement, gradually shaping the correct probability distribution. Consequently, we visualize the distributions at different step of inference on MSVD-QA dataset, as depicted in Figure 6 . Due to space constraints, we illustrate the probability of top 8 candidate answers, which is refined from random to biased during inference. The phenomenon meet expectation that firstly the answers relevant to video or question get top probability, but finally ground truth is higher while others decrease. In that practice, we also find that less step is enough, which reason may be same to 5.4.3. 
conclusion: 
 Open-ended VideoQA involve a vast candidate answer set. Selecting a single answer in a one-step manner without coarse-to-fine filtering leads to inaccuracy. Hence, we introduce a diffusion model that gradually refines answers through multi-step denoising, achieving a more precise answer generation process. This marks the first application of diffusion model to VideoQA, and our experiments corroborate the efficacy of our approach. We aspire that our innovative work will bring insights to open-ended VideoQA field. 
 
 Appendix:  


Decision: This paper presents a conditional diffusion model for open-ended video question answering. It receives two borderline accept and one borderline reject. The response well addresses the reviewers' concerns. All reviewers give the borderline accept ratings after the rebuttal. The merits, including motivation, interesting ideas, good writing, and good results, are well recognized by the reviewers. Please also incorporate the analysis in response into the camera-ready version. Overall, I think the current manuscript meets the requirements of this top conference.

Reviews: review: paper_summary: paper_summary: This paper proposes DiffAns as the first diffusion-based method in open-ended video question answering. In the forward pass, the true answer label is diffused into a random distribution. Then the model iteratively denoises to obtain the correct probability distribution under the guidance of answer-aware condition generated from video-question information. Experiments are conducted to validate the proposed method.
strength_weakness: strength_weakness: 1.The motivation of the paper is clear. Employing a diffusion model to construct the multi-step reasoning process of open-ended VideoQA is intuitively correct and reasonable.

2.This work introduces diffusion models into the domain of open-ended video question answering, which makes a relatively large technical contribution. Combining diffusion models with generative multimodal models such as VideoQA models can be an interesting and meaningful future research area that deserves further exploring.

3.The paper is well written and easy to follow.
limitations: limitations: 1.Although integrating diffusion models with downstream VideoQA models is of value, exploring the combination of LLMs/LVLMs with the proposed diffusion-based framework can be more interesting and more promising.

2.This work verifies the idea and effectiveness of their designs on three datasets. I suggest more datasets can be included to further enhance the quality of the paper.
paper_summary: paper_summary: The paper introduces a novel model named DiffAns, which employs a conditional diffusion model to address the challenge of open-ended Video Question Answering. Unlike multiple-choice VideoQA, open-ended VideoQA requires identifying the correct answer from a vast pool of potential answers without any fixed options. The DiffAns model simulates the human process of gradually refining the probability distribution of candidate answers to achieve more precise predictions. It integrates a diffusion model that first diffuses the true answer into a random distribution and then iteratively denoises it to retrieve the correct probability distribution under the guidance of an answer-aware condition.
strength_weakness: strength_weakness: (1) The DiffAns model seems to pioneer the use of diffusion models in the field of VideoQA, providing a new perspective on how to handle the open-ended nature of the task.

(2) The paper's experimental results appear promising, indicating potential for further exploration and application in the field.
limitations: limitations: (1) The motivation of this paper is unclear. The authors claim that gradual refining of the predicted distribution can achieve a more precise prediction for VQA since this process simulates human intuition. However, this explanation lacks sufficient (theoretical and/or empirical) evidence. The authors only show the denoising process at very limited steps (t=50, 25, and 0) in Figure 6, which cannot directly illustrate why and how the proposed method works well (by simulating human intuition). The lack of explanation of motivation and method makes it difficult to judge the scope and applicability of the proposed method.

(2) The English writing of this article leaves much to be desired. I think the organization and logic of the article are clear. However, there are too many grammatical mistakes. For example, “seek” should be “seeks” in line 40. “use” should be “uses” in line 95. “search” should be “searches” in line 115. “encode” should be “encodes” in line 149. “simulate” should be “simulates” in line 220. “e.i.” should be “i.e.” in lines 320, 326 and 329. “demonstrate” should be “demonstrates” in line 655. I encourage the authors to polish their writing carefully.

(3) Since DeBERTa is a more advanced language model than Roberta (as stated in line 669), why not use it as a language backbone in this paper?

(4) Why are the results of RaFormer and TranSTR, two current SOTA methods, on TGIF-Frame missing?
paper_summary: paper_summary: The paper proposes a diffusion-based method to apply the Open-ended VideoQA task by simulating the gradual process by which humans answer open-ended questions. The difference between the method compared with traditional methods such as the classifier method and similarity method is the model iteratively denoises to obtain the correct probability distribution. In the train procession, the true answer labels are fusion in the video and questions encoder to help better get the final correct answer label. The method was implemented in the three open-ended VideoQA datasets. The extensive experiment results verify the superiority of this method.

Questions
1.How does the answer denoiser reflect the coarse-to-fine filtering on vast candidate answers? I don't seem to see this process of "filtering".
2.In the main experimental results presented in Table 2, it is observed that for the largest answer set scale dataset, MSRVTT, the proposed method did not show significant improvement compared to the classifier method TranSTR. This makes it difficult to convincingly demonstrate the capability of the proposed method in handling datasets with large answer sets. It is recommended to provide comparisons with more similar datasets to further substantiate these findings.
strength_weakness: strength_weakness: 1.The approach innovatively proposes solving the open-ended VideoQA task using the diffusion-based model, which has better interpretability than traditional approaches.
2.This paper demonstrates a well-structured framework, with clear organization and logical flow throughout the content.
limitations: limitations: 1.In the experimental section, the dataset used is too limited. It would be more persuasive if more datasets were utilized.
2.In the comparative experiments, the classifier method was extensively utilized, but there was insufficient comparison for the similarity method.
3.It fails to delve deeper into the reasons that would better illustrate the effectiveness of this method.