Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper targets the bias problem in visual question answering task where the models may only capture the biases instead of showing real reasoning abilities. The paper introduces a method called D-VQA which tries to alleviate this problem from feature and sample perspective. With two unimodal bias detection modules, D-VQA recognizes and removes the negative biases while authors also construct two types of negative samples to increase sensitiveness of VQA models to both modalities by minimizing the possibility of predicting correct ground-truth answer of the positive sample. The results on VQA-CP and VQA v2 dataset show the effectiveness of the approach.
main_review: The paper introduces an approach called D-VQA which targets the bias problem in VQA. The paper is well written and is easy to follow and targets an important task which can help alleviate problems in the data collection methods such as bias that can occur naturally. My comments follow below:

1) For feature based debiasing, the model has two unimodal bias detection modules which pass image alone and text alone through the model to obtain the bias feature. This bias feature is removed from the multimodal feature and optimized for binary cross entropy loss for answers. The original multimodal feature is brought close to bias free multimodal feature through contrastive loss making sure that no extra parameters are used during inference.
2) For sample based debiasing, the authors sample negative sample for positive pairs by replacing either the text or the image. The binary cross entropy loss for the negative pairs is maximized so that the model is sensitive to the multimodal input and not focus on bias in the unimodal input.
3) For the ablations, they should have also been conducted on VQA v2 so that reader can understand the impact and differences on the well-calibrated benchmark. At the moment, it is hard to say how exactly the ablations work in a setting with actual bias.
4) For the LXMERT+D-VQA option, it would have been useful to have numbers on VQA v2 to understand if D-VQA introduces a drop or gain in performance for the models which are already performing very high on VQA v2 task.
5) The crucial details for the training have been added into the supplementary but instead should be including in the main paper as the paper feels incomplete without them. Also, is the LXMERT backbone pretrained or not? If yes, how does D-VQA affect pretraining?
6) From the ablations, it looks like that the sample based debiasing on vision side results in the most gain on the final results. There is no independent evaluation of just using feature based "question-to-answer" and "vision-to-answer" debiasing and understand how impactful those are. This is in contrast with how paper is structured mostly around feature-based debiasing even though sample-based debiasing which provides the most gain is only discussed for one paragraph. This questions the novelty of the paper as the sample based debiasing on vision side is obvious given numerous past study on the question side bias and is simple to add.
7) Table 2, Row 1 is technically simple UpDn. Why is that number different compared to Table 1, Row 3?
8) Authors stress on the point that they don't use any extra data for their augmentations. But contrary, why not improve results possibly by using techniques from MUTANT etc. It would have been good to see if extra data augmentations techniques can work well with DVQA.
9) Figure 1 necessarily doesn't show the problem of bias in VQA clearly and probably should something more concrete along with questions.
10) The name D-VQA is too general and I would suggest authors to rename title of their paper to be something more specific as it doesn't completely solved the problem of bias in VQA as it currently seems like reading the paper title.
11) The paper only tests the debiasing on images from COCO domain specifically. It would be good to see generalization to other domains such as TextVQA as well.


Post Rebuttal:
Please check my response to the authors.

limitations_and_societal_impact: The broader impact statement has been included and is reasonable.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work addresses a known bias in visual question answering tasks, in which networks learn to pick an answer based on a single modality without perceiving the entire input.  The work novelty comes from proposing a set of regularization strategies s.t: (1) reducing "bad" bias that is not a useful commonsense knowledge;  (2) reducing computational inferences through contrastive loss; (3) considering an image-only bias; (4) employ negative sampling to increase attention sensitiveness on both modalities. In both VQA and VQA-CP, the approach shows state-of-the-art performance.
main_review: Originality: A known bias with VQA models is that they do not perceive all inputs and take shortcuts. There have been various attempts to fix this issue through training an additional unimodal classifier and removing those biases (e.g., RUBi, LMH). However, this work offers some novel ideas, like negative sampling and contrastive loss. Further, the proposed approach allows the choice of unimodal features that are not contributing to the joint task, while previous works argue that unimodal features should be completely removed. 

Quality: Most of the techniques used are conventional (e.g., negative sampling, contrastive loss). Therefore, I consider the approach to be technically sound.  I recommend providing more qualitative examples, particularly negative ones. Also, in my view, since the bias-detection module is a novelty, every supporting evidence for bias-detection in the supplementary material (e.g., Tab 1, Fig 2) is appropriate for the main paper. 

Clarity: The submission is clear, and the ideas are intuitive. The appendix provides implementation details but no code. A point I find quite confusing is the bias detection model's ability to distinguish between a 'good' and a 'bad' bias. A 'good' bias, according to the authors, helps the joint task. However, the joint task does pick a 'bad' bias as well. I would appreciate a clarification on this. 

Significance: Visual question answering is an important and popular task. The bias discussed in this paper is a major concern in any VQA model, which is why the popular VQA-CP dataset was developed.  The proposed techniques should aid in advancing the goals of answering questions using reasoning instead of unaware biased correlations.  Importantly,  the experiments show a significant improvement in the performance of the VQA-CP task (61.91 vs. 57.59)

To summarize, here's what concerns me:
[W1] Are there any negative effects for the proposed methods?
[W2] Can the authors comment on the impact of different f_q/f_v? For instance, f_g can be an LSTM encoder for textual input. 
[W3] Why the b_q layer picks only 'bad' bias? 


limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative social impact of their work
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces “D-VQA”, a training approach dedicated to remove harmful biases (both language and vision ones) in VQA models. D-VQA performs in two ways, namely: feature-wise and sampling-wise.

On the features side, during its training, D-VQA attempts to de-bias the features learned by a VQA model by subtracting the (deliberately) biased features learned by unimodal language-only and vision-only models. Moreover, in order to get rid of the unimodal branches during the inference, D-VQA training includes an additional contrastive loss which aims to bring the original VQA features closer to the debiased ones (i.e. the ones obtained after feature subtraction with the unimodal branches).

On the sampling side, D-VQA employs an existing approach proposed in [42] (which is properly mentioned and recognized by the authors) introducing negative sampling triplets for each positive triplet (by randomly changing either an image or a question in a positive triplet).

Combining the 2 aforementioned techniques allows D-VQA to obtain the state-of-the-art results on both VQA-CP and VQAv2 comparing to the existing bias-reduction methods (while using the UpDn model as a backbone). In addition to that, the authors demonstrate that their approach is model-agnostic (for example, it is compatible with a more recent and powerful LXMERT backbone).
main_review: *Originality*

The paper rather exploits and successfully combines already-existing ideas for bias-reduction in VQA (by clearly citing the original works) than proposes radically new ones.

Thus, the sampling of negative triplets was proposed in [42]. In this work, the authors straightforwardly extend this idea by randomly switching both images and questions.

Similarly, de-biasing the VQA features via learning deliberately biased unimodal ones was also proposed in many recent papers (for example, in [9]).

In my opinion, the main originality of the present work is the idea of the contrastive learning (the loss L_c) which makes the VQA model independent of its unimodal branches at the inference stage (and, therefore, accelerates and simplifies the final model).

*Quality*

To the best of my knowledge, the significant related work has been properly cited and compared with the proposed approach. The claimed contributions are supported with conclusive experimental results and ablation studies on the VQA-CP and VQAv2 datasets.

*Clarity*

The paper is clearly written and easy to follow. Some important implementation details can be found in Supplementary Materials.

*Significance*

All in all, D-VQA can be considered as a significant contribution as it demonstrates the complementarity of two (almost orthogonal) branches of the research on bias reduction in VQA, namely: feature de-biasing and training balancing.

The results could have been even more convincing had the authors tested their approach on other unbiased datasets than VQA-CP (for example, GQA-OOD: https://github.com/gqa-ood/GQA-OOD - the dataset has been published after the NeurIPS deadline so the lack of evaluation on it cannot be considered as a problem).

*Minor remarks / questions*

1.	In the ablation study reported in Table 3, you demonstrate that amplifying the unimodal biased feature vectors f_q and f_v with the estimated real-values b_q and b_v (obtained via “bias detection modules”) improves the performances. Would not it work even better if b_q and b_v were vectors (as in [9], for example) rather than simple real values (in this case, f^{bias}_q could be a pointwise product between f_q and b_q)?

2.	I am not sure to understand the Equation 5. Does “a” denote a vector or a single real value? Indeed, on the one hand, you state that “m_a” outputs a single real value. But on the other hand, “a” is an output of the softmax operation and is also transposed in Eq. 5 indicating that it is rather a vector.

*Typos*

1.	Line 103 (and also in other parts of the article): you employ the word “auxiliary” as a verb. I am not a native speaker, but I do not understand what that means, and I am not sure that it is grammatically correct.

2.	Line 195: “contains” -> “containing”.


limitations_and_societal_impact: The "broader impact" statement has been included in the article and seems reasonable.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper tackle explores how to partially annealed dataset biases when training VQA models.
 To do so, they applied to debiasing strategy:
  - they generate free negative samples to tentatively debased the answer distribution
 - they use a neural pipeline to tentatively debiased the multimodal features
Interestingly, both approaches are model-agnostic (up to $f = F(h,g)$), making the approach generic.

The paper has many merits and solid results. Multiple complementary experiments are insightful.
Yet, there are a few remaining ambiguities that prevent me from fully advocating for acceptance.
 I would be more than happy to increase my score if they are correctly addressed:
main_review: 

Key remarks:

A) The authors emphasize the debiasing aspect of the neural pipeline. It includes equations 3-7. My main concern is about equation 6 and the loss $L_d$.
First, can you elaborate on the $L_d$? What are the target labels of $L_d$ (or is it equivalent of $L_{vqa}$)? More generally, why this specific loss would only keep the “good” biases? 
Second, as $f$ and $f_{bias}$ are trained independently (and thus, they do not live in the same space), why computing the difference would debias $f$?

Third, some design choices are not well-justified, e.g. why b_q.f_q instead of directly using $f_q$? Why using attention to combine $fˆ^ q_{bias}$ and $f^v_{bias}$? Even if I understand the intuition of those patterns, they remain arbitrary design choices that increase the complexity of the model. I would thus recommend the authors ablate them to justify them.
 
 B) In light of Table 2, the sampling procedure looks crucial. However, no ablation only contains feature-debiasing to determine its impact correctly. Therefore, it is hard to estimate whether feature debiasing has a marginal effect or not. Second, the negative samples are not correctly put in the perspective of the literature. It is mentioned that the process is inspired from 32 and 42 without more details. This is unfortunate as the current paper does show that sampling jointly questions and images is beneficial (which was not observed in 32 and 42). It took me quite some time to be sure that 3.3 was an actual (simple but valid) contribution to the paper.

 Potential improvements:

C) I understand the need to have $L_{sam}$ to remove the feature debasing pipelines. However, this aspect can be further study: 1) What happens if the loss is removed and the pipeline is removed at test time? Is there an implicit alignment of features or the debiasing signal is just lost? 2) What happens if the full pipeline is preserved at test time (more parameters), is the final higher as there is no proxy? 

D) Positive biases are quite a controversial	topic; I would recommend detailing a bit more this section with the relevant literature. On the other side, there is no need to have such an extended related work section on self-supervised learning… as there is no self-supervised learning pipeline in the paper. Self-supervised learning is not CPC! Contrastive losses existed far before SimCLR!

Other remarks:
 - Overall, the paper is clearly written. I may recommend making figure 2, a bit bigger while showing the equation. Another idea would be to summarise all the equation 3-7 into a single one to help the global understanding of the model (in Appendix?).
Related work is correct. As mentioned, there is no need for detailed self-supervised learning literature. 
L123: please add citations after “most previous methods.”
L218: are \hat{q} and \hat{v} from the same question-image pair?
Table1 is complete. Well appreciated
As mentioned in B, please complete the ablation in Table 2.
Tables 4 and 6 are persuasive.
Do you. Have some piece of intuition why D-VQA is so robust with few labels? Is it because of negative sampling (potential additional experiment)


Conclusion:
 This paper is consistent and has a pure-ML approach that I really appreciate. It also has solid results and convincing complementary experiments.
However, there are still some obscure points that I encourage the authors to clarify before I can recommend acceptance. 
Besides, if some of the experiments I recommended were performed (e.g, Complete Tab2, ablate $L_sim$, ablate neural choice), I am eager to increase my score even more.

limitations_and_societal_impact: LGTM
review_title: null

Meta-review: The paper proposed a new method for debiasing VQA models. A consensus for acceptance emerged very quickly in the discussion phase, helped by a swift rebuttal by the authors and additional experiments on new datasets.

The AC concurs.
Decision: Accept (Poster)