Learning from Inside: Self-driven Siamese Sampling and Reasoning for Video Question Answering

Abstract: Recent advances in the video question answering (i.e., VideoQA) task have achieved strong success by following the paradigm of fine-tuning each clip-text pair independently on the pretrained transformer-based model via supervised learning. Intuitively, multiple samples (i.e., clips) should be interdependent to capture similar visual and key semantic information in the same video. To consider the interdependent knowledge between contextual clips into the network inference, we propose a Siamese Sampling and Reasoning (SiaSamRea) approach, which consists of a siamese sampling mechanism to generate sparse and similar clips (i.e., siamese clips) from the same video, and a novel reasoning strategy for integrating the interdependent knowledge between contextual clips into the network. The reasoning strategy contains two modules: (1) siamese knowledge generation to learn the inter-relationship among clips; (2) siamese knowledge reasoning to produce the refined soft label by propagating the weights of inter-relationship to the predicted candidates of all clips. Finally, our SiaSamRea can endow the current multimodal reasoning paradigm with the ability of learning from inside via the guidance of soft labels. Extensive experiments demonstrate our SiaSamRea achieves state-of-the-art performance on five VideoQA benchmarks, e.g., a significant +2.1% gain on MSRVTT-QA, +2.9% on MSVD-QA, +1.0% on ActivityNet-QA, +1.8% on How2QA and +4.3% (action) on TGIF-QA.


Reviews and author responses:

 My question has been addressed. I will keep my original rating.


 I would like to keep the Rating as 5: Marginally below the acceptance threshold. The proposed method is more like a generic video understanding method. It can be applied to most video understanding tasks to learn from the inside by using soft labels, such as action recognition. In my opinion, if a paper is targeting the problem of Video Question Answering, it should not simply give a solution on Video Understanding. This paper solves the task by giving pseudo-labels based on the visual similarity of sampled frames. The solution seems to be not specially designed for the task.

 Dear Authors of NeurIPS 2021 Submission #7610,

Thanks for bringing up this question.

(+) I acknowledge the novelty and the competitive performance of this work. The method is new on the Video Question Answering task, and it is validated on five commonly-used VideoQA benchmarks. 

However, I still have a concern about this work.

(-) **The proposed method is more like a generic video understanding method.** It can be applied to most video understanding tasks to _learn from the inside by using soft labels_, such as action recognition. In my opinion, if a paper is targeting the problem of Video Question Answering, it should not simply give a solution on Video Understanding. This paper solves the task by giving pseudo-labels based on the visual similarity of sampled frames. The solution seems to be not specially designed for the task. 



 Dear AC and all reviewers:

Thanks again for all of your constructive suggestions, which have helped us improved the quality and clarity of the paper!

Since the discussion phase has started for over one week, we have not heard any post-rebuttal response yet.

Please don’t hesitate to let us know if there are any additional clarifications or experiments that we can offer, as we would love to convince you of the merits of the paper. We appreciate your suggestions.

Best regards,

Authors of NeurIPS 2021 Submission #7610


 Because combining siamese clips with the anchor clip brings extra computation and memory during the inference. We don’t want to bring any extra burden (e.g., computation, memory, and parameters) with the inference.  As we know, the ensemble approach usually requires large computation and memory, which is not good for inference and real-world applications.

 We replace our Siamese clips with multiple copies of anchor clips to analyze the effect of SKG. Since we used 8 siamese clips in our paper, we replaced the siamese clips with 8 copies of anchor clips during the training. We execute this ablation study on the MSVD-QA dataset. This ablation study achieves 41.3%. It promotes the baseline (39.4%) with 1.9% performance. It can validate the power of the ensemble way. But only using the anchor clips for ensemble can’t bring visually contextualized knowledge. We skillfully exploit the visually contextualized knowledge via the siamese clips, which obtains 45.5% accuracy. Hence, the improvement of our method mainly comes from the SKR and SKG by using Siamese clips rather than the simple ensemble.

 There are some constraints for selecting the starting sampling index of the anchor clip. It should be in the first third of the video to guarantee the equilibrium of sampling.  The “nearby” means the index of the Siamese clip should be next to the index of the anchor clip. The pace of sampling is consecutive before or after the anchor index. We will add more details in this section in our final version. Thanks for your reminder.

 We will add the full notations of each vector dimension in Section 3.1, and inject pseudo-code algorithm into the method section to further improve the paper readability in our final version. Thanks for your suggestions.

 One of the limitations of our proposed method is only validated on the video question answering. Currently, we mainly focus on the VideoQA task. We will carefully consider more video-related tasks in our further work. 

One of the potential risks of the proposed system is the data biases. Our system tends to distinguish the wedding scene with a white wedding dress. As we know, the white color is a classical topic for western weddings. However, at the eastern wedding, the popular color should be red. And eastern people in the eastern funeral usually wear white clothes. Hence, our system sometimes mistakenly judges the eastern funeral as a wedding.  


 We really appreciate your constructive and insightful comments. Indeed, we are working on the extension of our siamese sampling and knowledge by using self-supervised learning. We have to say you are very professional in this research area. We will release the reproducible code for future research purposes.

 As we know, our method showed superior performance on five different datasets due to our Siamese Sampling and Reasoning. The soft labeling is a strong method to improve model training. But it is the key for how to design a suitable algorithm of combing contextualized knowledge with soft labeling to improve the VideoQA. We introduce such a novel algorithm in our paper. We have compared our proposed method with other popular VideoQA methods as shown in Table3, Table4, and Table5. Since the previous methods on VideoQA are non-soft-labeling, it can further demonstrate the novelty and effectiveness of our method to use the soft labels.

 We don’t think that our method is a bit mismatch to the video question answering task. Our proposed SiaSamRea is a tentative attempt for video question answering, which also can look through from the paper title. As far as we know, how to explore the visually contextualized knowledge from inside in the video-based multimodal research has not been explored yet. And video question answering is a classical task that can be well explored. Hence, we validate our method on five VideoQA datasets. Different from current methods mainly look for insights from outside data and powerful pre-trained models. Our work proposes a totally different insight called learning from the inside by using soft labels. 

The action classification seems to be a good task that can be validated in our future work. We have validated our method on five commonly-used VideoQA benchmarks. The sufficient experimental results and analysis can be seen in Section 4. By the way, we don’t find any similar works to perform on the video-based multimodal tasks.


 We have conducted experiments on the MSVD-QA dataset to analyze the effect of these learnable weights (W1 and W2). Our SiaSamRea without W1 performs 44.9% on MSVD-QA, which heavily drops 0.6% performance compared with our whole SiaSamRea (45.5%). One possible reason is that the W1 can make the propagated prediction matrix learnable and adaptively propagate the predictions between all the samples.  When performing our model without W2, it achieves 45.2%. The performance without W2 is slightly decreasing (~0.3%). When it comes to our method without using W1 and W2, it gets 44.5%. As we can see, the W1 and W2 both can bright improvement. Hence, it can further demonstrate the importance and necessity for the generation of expected soft labels by correctly merging several predicted candidates. 

This paper tackles the video question answering task. It proposes a model with context reasoning within the same video, which is realized by generating pseudo-labels based on the visual similarity of sampled frames. Results are reported on five video question answering benchmarks, MSRVTT-QA, MSVD-QA, ActivityNet-QA, How2QA and TGIF-QA.

  The proposed model is technically sound. The paper is clearly written. The key components in the proposed model (e.g. the number of sampling siamese clips, siamese knowledge generation and siamese  knowledge reasoning) are ablated. Extensive evaluations are conducted on five benchmarks.
 The Equation (5) for generating the soft learning target Q contains trainable weights W1 and W2, what is the effect for these learnable weights? What if we directly use the rest part in Equation (5) (without learnable weights W1 and W2) as soft label?
 

This paper proposes a self-driven siamese sampling and reason (SiaSamRea) method for the video question answer task. The task is essentially solved by a classification or regression model based on the problem settings. The key idea of SiaSamRea is to sample more clips in the training phase so that these extra clips can provide soft labels, which are similar to pseudo labels, for another supervision. The Siamese Knowledge Generation (SKG) module builds an interdependent matrix to measure the correlation between each clip-pair of the video, and the Siamese Knowledge Reasoning (SKR) module aggregates the prediction from these extra clips based on the correlation matrix from SKG. Finally, the aggregation in SKR is used as a soft label and included in the loss term. By doing so, the method achieves state-of-the-art performance on five VideoQA benchmarks.  (+) The idea of this paper is intuitive and easy to follow. This is also because of the clear writing and illustrative features. 

(+) The SKR module seems to be very effective according to the ablation study. This might be due to the soft label is usually useful to optimize the model. The supervision from Siamese Clips's prediction provides more in-the-domain information to the model.

(+) This work reaches state-of-the-art performance on five different datasets. If the code gets released as the authors promised in the paper, it will be a good resource for the video question answering community.

(-) The SiaSamRea method is a bit mismatch to the video question answering task. This method should be aligned to a more generic video understanding task such as action classification, or even self-supervised learning such as SimCLR, SwAV, and SimSam, because the proposed SKG and SKR modules only require the input should have video. 

(-) Although it shows superior performance on five different datasets, the comparisons are not very fair. As we know, soft labeling is a strong method to improve model training. According to the ablation experiments, we can also see that only use SKR can give a big boost (~4%). However, the author is comparing their methods with other non-soft-labeling ones. 

 I would like to suggest comparing the effect of soft labeling with the proposed SKG and SKR modules. Soft labeling, or generating pseudo labels from the training set, usually gives an extra boost to the model performance. 

This submission addresses the problem of Video Question Answering (VideoQA). Compared to previous method, the authors propose to sample siamese samples from same video and apply siamese knowledge as a soft label to self-supervise train the network. Experiments show that the proposed method achieves state-of-the-art performance on five VideoQA datasets.   ### Strengths

* The novelty is clear and motivation is reasonable. The technical details are explained clearly, with self-explained figures as figure 1 and figure 2

* The SimSamRea method achieves state-of-the-art performances on five popular VideoQA datasets, which verify the generalizability and effectiveness of the method. 

* The siamese sampling and knowledge based self-supervised learning can be applied to future researches.  * It would be better to  release the reproducible code for future research purpose. 

The authors address the task of video question answering and extend the state-of-the-art framework using the concept of siamese sampling.
They build on the work of ClipBERT which proposed that using sparsely sampled frames from videos improve performance and efficiency compared to densely sampled frames.
This work performs multiple such sparse samples keeping temporal ordering (called as siamese samples) and combines prediction from the samples using graph convolutional layers leading to a pseudo label.
The final optimization function uses a combination of two loss functions: (i) loss between ground truth and predictions from anchor clip (a single sparse clip), and (ii) loss between predictions from anchor clip and pseudo labels generated from siamese samples.
They have shown that the model outperforms state-of-the-art significantly on several commonly used data sets.  1. It's unclear why the authors used the predictions from siamese clips as pseudo labels instead of combining it with anchor clip predictions perhaps using the same graph convolutional architecture or other ensemble approach.
2. The ablation studies section contains ablation with just Siamese Knowledge Reasoning (SKR) and both SKR and SKG. However, it's unclear if the gain in either of these cases is due to additional information from clips or just additional parameters and power of ensemble. The authors should perform ablation using multiple copies of anchor clip as input to SKG and understand the effect.
3. The siamese sampling section (Section 3.2) is slightly unclear. Firstly, they mention they randomly select the starting sampling index of anchor clip. Do they have any constraints that it should be in first half of the video or based on the sampling stride the end index should be in range? Secondly, they mention the siamese samples occur "nearby" the index of anchor clip. Please define "nearby" concretely. Can the index be before the anchor clip? Is it consecutively after the anchor index?
4. Notations in preliminary section (Section 3.1) do not specify the vector dimensions. Please specify them to increase clarity. Adding Algorithm can further improve readability. Limitations not discussed. The authors can do qualitative error analysis and downsides of using their approach.

Meta-review: The paper received mixed recommendations from the reviewers, one reviewer increasing their score after the author response leading to 2x5, 1x6, and 1x7 rating.

One concern remaining after the author response was the question if the novel approach is specific enough to VideoQA or should be evaluated on other video recognition tasks.
However, I think
(a) answering this question is outside the scope of this paper, and remains to be seen, as the question/task conditioning might be an important component which makes the proposed approach so successful and if future work shows this is generalizable to other tasks will only increase the impact of this work further.
(b) the strong results (in combination with the novel approach in the videoQA field) are of interest to the community

Given the interesting model and strong results (outperforming recent work, including CoMVT [34] from CVPR 21) on 5 datasets, and solid ablation study, I recommend acceptance under the expectation that the authors revise the paper according to the author response and address the reviewers concerns, including, but not limited to:
1) ablating W1, W2
2) ablation with multiple copies
3) release code
4) additional details about where the sampling of the anchor and Siamese clips is 

Additional I would like the authors to add the missing comparison to [22] on TGIF and recommend proofreading the paper for English fluency.

Decision: accept