Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies VideoQA through the lenses of causality where it attempts to break the spurious correlations caused by biases (e.g., linguistic biases, visual biases etc.) when predicting answers. It proposes to identify these biases by forcing VideoQA models to respond to unanswerable questions obtained by pairing videos and question-answer pairs of different random samples. The proposed method demonstrates its effectiveness on several VideoQA datasets.
strength_and_weaknesses: 1. Strengths:

+ The paper is well written and easy to follow. Although similar works have been done in VQA, not much work on debiasing and robustness has been done in VideoQA settings.
+ The proposed method achieves state-of-the-art performance on major VideoQA datasets.

2. Weaknesses:

+ Similar studies have been done in VQA and image captioning [1, 2]. The idea of sampling “unanswerable” QA pairs to detect the shortcuts that VQA models exploit when answering questions has been studied. The proposed method would be more interesting if it focuses on identifying biases that are unique to video data.

[1] Wen, Zhiquan, et al. "Debiased Visual Question Answering from Feature and Sample Perspectives." Advances in Neural Information Processing Systems 34 (2021): 3784-3796.
[2] Yang, Xu, Hanwang Zhang, and Jianfei Cai. "Deconfounded image captioning: A causal retrospect." IEEE Transactions on Pattern Analysis and Machine Intelligence (2021).

+ It is unclear what biases are detected and reduced by the proposed method. Examples of only “how many” question type do not seem convincing enough. 

+ The explanation of how the authors detect visual biases and linguistic biases could be clearer. They also should provide more detailed analysis of how reducing the negative impacts of these biases impacts the overall performance.

+ How do the authors make sure the newly created pairs of video-question are unanswerable? What if questions are general-purpose such as “what is in the video?” or “what action is performed in the video?”?

+ I have a doubt about the results of TGIF-QA dataset as if you train VideoQA models on each question type one by one, what biases do you detect? What makes the improvements so large? What about the Count questions? I do not see it in Table 1.
clarity,_quality,_novelty_and_reproducibility: This paper is technically sound but similar ideas have been used in related tasks.
summary_of_the_review: The paper has contributions to VideoQA task specifically but similar ideas have been used in related tasks.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper tries to fix bias in VideoQA models through causally approaches. The authors noticed that VideoQA models tend to just learn the dataset statistics ("How many ..." --> "2") and look to rectify that by forcing the model to answer questions that don't make sense for the input in order to disrupt the spurious correlations. The goal here is to enforce that the model only takes advantage of causal relations between the question and the video and makes use of both modalities.

They go on to run experiments on three benchmark VideoQA datasets and show favorable results, as well as apply GradCAM to assess what parts of the modalities are being taken into account.
strength_and_weaknesses: Strengths:
1. A strength of this approach is that it appeals to a well founded base vis a vis the causal methods.
2. I'm not sure how novel the larger approach is, but applying this to VideoQA in order to debias the question and the video is a good idea that I have not seen before.

Weaknesses:
1. Boy oh boy is this hard to read. The intro section has three different summaries. It's not until page 6 that the authors finish explaining how their method works, and it starts on page 3. Figure 3 does not help clarify much. In fact, I think it made my understanding even worse. Why do you even need Figure 2 showing the graphical interpretation of the do calculus? In fact, why do you need most of the causal inference preliminaries? A lot of what the authors write is fairly basic and can be summarized / pointed to in other works. Figure 4 is super confusing. Why are there arrows going in both directions? I have rarely, if ever, seen a NN diagram that operates like that. The notation used to explain this method is tremendously difficult to read. I *strongly* suggest improving this because this paper is just about unreadable given how many times it takes to go through and properly glean what's going on. 
2. That's not even to mention that it would fail for anyone who isn't looking on an interface w color or to anyone who is color blind to blue/red. You should use color only to highlight, not for something with real purpose and meaning.
clarity,_quality,_novelty_and_reproducibility: 1. Clarity: This paper gets a 0 for clarity. Please make this easier to read. It's very challenging at the moment

2. Quality: The paper seems to have a good idea with good empirical and qualitative evaluation. I don't have any qualms there.

3. Novelty: This application of causal methods is novel in so far as I understand. 

4. Reproducibility: I cannot reproduce this given that I am uncertain about my understanding of the actual approach re the lack of clarity.
summary_of_the_review: My review of this paper is overwhelmed by how hard it is to grok the idea. This doesn't pass the bar and would take great revision to fix it. I do actually think it's possible in this round, but it's going to take a wholesale rewrite of the first 2/3. I strongly suspect the paper will be 100x better after doing that.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a debasing approach to better video QA performance that utilizes learnt confounders to discover and fix spurious correlations. The confounders include both data-agnostic item, i.e. a learnable base (Z) and a data-specific item that depends on the video and textual input (Z'). The model jointly learns to output the right answer with the original input (X) and output the same prediction with created unanswerable input (X') using the confounders. 
strength_and_weaknesses: Strengths: 
(1) A novel model architecture for debasing video QA 
(2) Separate out the spurious relations with confounders 
(3) Detailed explanations for the algorithms designs, and implementation details
(4) Good results on various benchmarks

Weakness:
(1) Nearly Four-times additional computational cost during training due to unanswerable Xq, Xv. in the original network f and the confounder encoder g; Good to include training times 
(2) Good to have ablation study on the two component of Z (visual and textual)separately, currently only have them as a whole.
(3) Good to have ablation study on the number of the confounders


clarity,_quality,_novelty_and_reproducibility: The paper conduct a good original research on debasing video QA. It is relatively easy to reproduce the model architecture based on the descriptions in the paper. 
summary_of_the_review: The paper introduces an approach to using confounders to separate out spurious relations in order to debasing video QA. The idea is novel and obtains good results, however with large additional computational cost.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a framework for debiasing video question answering (VideoQA) with casual intervention. The novelty relies in that the confounders do not need to be know in advance. Instead, the framework uses pairs of "unanswerable" questions at training time to find them. Experiments are conducted on three standard VideoQA datasets, achieving better accuracy than previous methods on all of them. Ablation studies and quantitative results are also provided.
strength_and_weaknesses: Strength
- Casual intervention is gaining popularity as a way to debias vision-and-language models that leverage statistical correlations to make predictions. An important limitation with casual intervention is that the so-called confounders need to be known in advance. Previous work approximates those condounders with, for example, the list of objects in COCO datasets. In contrast, this paper proposes to learn confounders by using unmatching question-video pairs and forcing the model to make predictions based on their statistical correlations.
- The experiments show that the method achieves high accuracy. Ablation studies shows that all the parts of the proposed framework are relevant.

Weaknesses
- A question that arises and it is not very clear in the paper is how to ensure that the unmatching question-video pairs are really unrelated. Just choosing random pairs does not ensure that the video and the question are irrelevant. For example, a "how many people" question can still be relevant in a different video.
clarity,_quality,_novelty_and_reproducibility: The paper is clear. For reproducibility may need the authors to open-source the code.
summary_of_the_review: The paper proposes an interesting solution to the problem of causal intervention in VideoQA without knowing confounders in advance, which is a more realistic setting than using a list of objects as confounders. Experiments are technically sound and show better performance than previous work.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper was reviewed by four experts in the field and received a mixed score. The main concerns are the limited novelty, unconvincing experiments, and lack of clarity. The authors did a good job of rebuttal and addressed many of the concerns. However, the reviewers (including all positive ones) still feel that more work is needed to get it to the best version. AC also agrees that this work can be much stronger with additional experiments. While this paper clearly has merit, the decision is not to recommend acceptance. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.


justification_for_why_not_higher_score: 1. Unconvincing experiments.
2. The quality of the presentation does not meet the standard of ICLR.
justification_for_why_not_lower_score: NA
Decision: Reject