Counterfactual Vision-Language Data Synthesis with Intra-Sample Contrast Learning
Authors: Zhecan Wang
Yicheng He
Wenhao Li
Haoxuan You
Long Chen
Noel C Codella
Yulei Niu
Kai-Wei Chang
Shih-Fu Chang
Venue: Submitted to ICLR 2023 (2023)
Keywords: counterfactual
data augmentation
vision language
kowledge distillation
vcr
vqa
visual question answering
commonsense reasoning
multimodal
robust
domain-shift
debiased

Abstract: Existing Visual Learning (VL) benchmarks often contain exploitative biases. Most former works only attempted to mitigate biases in semantically low-level and conventional visual-question-answering typed datasets like VQA and GQA. However, these methods cannot generalize to recently emerging highly semantic VL datasets like VCR and are also difficult to scale due to many severe problems like high-cost labors, drastically disrupting the data distribution\textit{, etc.}To resolve those problems and also address other unique biases on VCR-like datasets, we first conduct in-depth analysis and identify important biases in VCR dataset. We further propose a generalized solution that synthesizes counterfactual image and text data based on the original query's semantic focus while producing less distortion to the data distribution. To utilize our synthesized data, we also design an innovative intra-sample contrastive training strategy to assist QA learning in Visual Commonsense Reasoning (VCR). Moreover, our synthesized VL data also serve as a highly-semantic debiased benchmark for evaluating future VL models' robustness. Extensive experiments show that our proposed synthesized data and training strategy improve existing VL models' performances on both the original VCR dataset and our proposed debiased benchmark.

Under review as a conference paper at ICLR 2023
COUNTERFACTUAL VISION-LANGUAGE DATA SYN- THESIS WITH INTRA-SAMPLE CONTRAST LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Existing Visual Learning (VL) benchmarks often contain exploitative biases. Most former works only attempted to mitigate biases in semantically low-level and conventional visual-question-ansvee answering typed datasets like VQA and GQA. However, these methods cannot generalize to recently emerging highly semantic VL datasets like VCR and also difficult to scale due many severe problems like high-cost labors, drastically disrupting the data distribution, etc. To resolve those problems and also address other unique biases on VCR-like datasets, we first conduct in-depth analysis and identify important biases in VCR dataset. We further propose generalized solution that synthesizes counterfactual image and text data based on the original query's 's semantic focus while producing less dis- tortion to the data distribution. To utilize our synthesized data, we also design innovative intra-sample contrastive training strategy to assist QA learning in Visual Commonsense Reasoning (VCR). Moreover, our synthesized VI. data also serve highly-emmantic debiased benchmark for evaluating future VL models' robustness. Extensive experiments show that our proposed synthesized data and training strategy improve existing VL models' performances both the original VCR dataset and our proposed debiased benchmark.
INTRODUCTION
Why is everyone on the porch staring at [person6]
persons
0.03
a) There are many people outside.
b) The street cleaning,
0.00
Everyone on the porch are surprised to see [person6].
0.01
d) [person8] and [person10] are outside.
Baseline
Baseline CDS
Baseline CDS-ICL
example from VCR and the paired visual Grad-CAM result from a finetuned VL BERT, Su et al. (201)). With training, we expect the VL model to integrate mulitmodal information and commonsense when select the correct answer (labelled by a green check). For in- stance, to answer this question, we expect the model to focus [person6] on the left and people in the center. However, the model fails to pick up the correct visual clue and focuses on the irrelevant entity, the window in the background. The orange words are the overlapping words between the correct choice and the question.
Figure 1:
Recently many works have explored Vision Language (VL) models' learning high semantics from image and text data. As result, many VQAT-like benchmarks such GQAHudson & Manning 2011), VQALi et al. (2018). VCRZellers et al. (2019) and ENLI-VEXie et al. (2019) were pro- posed to evaluate models' abilities in learning visual commonsense and reasoning. Despite recent

---

Under review as a conference paper at ICLR 2023
successes on aforementioned VL tasks, severe drawbacks due biases in these evaluation still exist and remain unsolved. These biases prevent existing VL models from understanding the image and text but encourage them to learn shallow mappings based on repeated patters spurious distribu-
tions Cao et al. (2020); Manjunatha et al. (2019); Ye & Kovashka (2021]; Kovaleva et al. (0019;;
Ramakrishnan et al. (011)); (Sllaarau al. (2020) Agrawal al. (2018).
Among those datasets, Visual Commonsense Reasoning (VCR) Zellers et al. (2019) has become one of the most recognized highly-semantic VL benchmarks with heavy manual annotations, over the past years. Despite its intention, VCR also suffers from various visual and language biases as illustrated [i Firstly. the correct choice shares the most number of overlapping words with the question Ye & Kovashka (202I); Secondly. the distractors (incorrect choices) are low-quality and often not related to the image and question at all. Lastly, as pointed out by previous works, VL models often fail to understand the visual scene and thus maintain very low utilization in visual data Cao et al.] (2020): Ye & Kovashka (202T); Agrawal et al. (2088); Ramakrishnan et al.) (2018). As shown by the Grad-CAM results Selvaraju al. (2016) in Fig. I models like VL-BERT often focus on regions irrelevant to the question while deducing the correct answer. As a result these biases in VL datasets like VCR, existing VL models can easily learn shortcuts without utilizing commonsense and reasoning. Therefore, a slight Homain-shiffe in the question, answer or image may drastically affect VL models' predictions Dancette et al. (2021), , leading problems such as prediction inconsistency Selvaraju et al. (2020) Ray et al. (201 Ribeiro et al. (2019).
Different from conventional datasets like VQA, GQA but similar to other highly semantic Tapaswi et al. (2016); Park et al. (2020), VCR has more complex images, diverse question types and follows the standard Multiple-Choice-Qune Choice-Question (MCQ) format where each question has mul- tiple sample-specific choices in sentences. Therefore VCR-like dataset may have specific hidden biases unseen before. However, former works only focus on countering biases in the conven- tional VL datasets which makes them not applicable to VCR. For instance, (Agrawal et al. (2018); Dancette et al. (202T)'s methods for balaneing answer distribution does not apply to VCR since VCR has guestion-specific choices in sentences; directly placing occlusions or maskings on im- age or text Chen et al. (2020a): Liang et al. (2020) may drastically disturb the data distribution in VCR;additional high-cost manual annotations
not practical Ray al. (2019); Selvaraju et al. (016): Ribeiro et al. (2019); VCR does not have questions asking only about colors or numbers Gokhale al. (2020); Inter-sample contrastive learning between image and text pairs may distract models from intra-sammi sample differentiation among VCR's 's sample specific choices, etc.. [Chen et al. 2020a) Liang al. (2020); Gokhale et al. (2020);
Many problems like the above mentioned prevail in former methods and prevent them from gen- eralizing to highly semantic VI datasets like VCR. To raise the community's attention in biases of highly semantic VL datasets like VCR and countering them, in this work, we first conduct in- depth analysis and identify unique biases in VCR. Second, we propose a generalized Counterfactual Data Synthesis (CDS) method to help counter the identified biases. CDS utilizes adversarial models modify images and answer choices to create synthesized positive and negative image and text data without drastically disturbing data distribution like direct occlusions. Further, we prove that CDS's synthesized data can compliments VCR data to effectively mitigate our identified biases and even integrate to a debiased evaluation benchmark to evaluate future models's 's robustness.
To better leverage our synthesized data in training, we also propose Intra- natra-sample Contrastive Learn- ing (ICL) framework to assist existing VL models focus on intra-sample differentiation among an- swer choices and images. Unlike [hen et al. (2020a); Gookaae et al. (2020), ICL frees us from creating paired answers for negative synthesized images. With extensive experiments, we demon- strate that ICL with synthesized data can help existing VL models to be more robust in terms of domain-shifts in data.
In conclusion, our contributions are four-folds.
Firstly, we identify significant biases in VCR and analyze VL models' over-reliance on text data.
Secondly, we propose an innovative counterfactual VL data synthesis method, CDS, to mitigate the dataset biases. This is the first work propose an adversarial VL data synthesis method in VCR.
Thirdly, to better leverage our synthesized data in training, we further propose an intra-sample contrastive learning mechanism to assist the conventional QA learning with cross entropy loss. To
2

---

Under review as a conference paper at ICLR 2023
the best of our knowledge, are the first to adopt contrastive learning strategy with counterfactual VL data in highly-semantes VL datasets like VCR and prove its effectiveness.
Lastly, synthesized VL data complements the VCR data and mitigate biases. Therefore, the gen- crated data also serve as a highly semantic VL debiased benchmark on top of VCR for evaluating VL models' performance in more challenging setting. We also conduct extensive experiments prove that both our synthesized data and training strategy massively improve VL models' performance on the original VCR and our debiased evaluation benchmarks.
RELATED WORK
2.1 BIASES IN VISION- DATASETS
Many works have explored biases in VL dataset, especially in the conventional ones like VQA Li eet al. (2018) and GQA Hudson & Manning (2019). Since they do not have sample-specific dis- tractors, all questions share the same set of answer choices. AAgrwaal et al. (2018); Dancette et al.] (202I]: Zhang et al. (2016) point out that shallow biases exist in the question-ans answer distribution with such limited variance. Ramakrishnan al. 2018) ManjunMaaaaa et al. 2019) focus on effects of language priors VQA like shortcuts from the prior words of questions. Due to lazy human anno- tations and less complexity in text than visual data, VL models can easily learn shortcuts from text biases. Therefore, simple changes in text may lead to significant performance drops and prediction inconsistency (Shah al. (2019); Li al. (2018); Selvaraju et al. (2020); Riibeir et al. (2019); Ray et al. (2019). On the other hand. Cao et al. (2020); Wang et al. (002) also confirms that existing VL models tend to under-utilize visual information compared
text. However, nearly all former works focus on conventional VQA, GQA and other similar VL datasets. Recent highly semantic VI dataset like VCR Zellers et al. (2019); Tapaswi et al. (2066) with MCQ format have answer distribution with larger variance, more diverse question types and complex images. Therefore many of the explored bias analysis may not apply directly, Only one former work, Ye Kovashka (2021), mentions the bias problem in VCR. However, it suffers from very limited scope when only looking at pronoun words. In this work, we provide an in-depth analysis of biases in an existing highly semantic VL dataset, VCR.
2.2 MITIGATING BIASES
Various methods have been proposed to counter biases. Chen et al. (220a): Liang et al. 2020) Gokhale al.) (2020) place occlusions or maskings on images and questions create synthesized data. However, occlusions and maskings drastically disturb data distribution while leading to non- sensical synthesized answers. Additionally, these methods do not apply to VCR due to their need for inter-sample contrastive learning. Agrawal et al. (2018); Dancette et al. 2 categorize answer choices of the whole VQA and rearrange the overall distribution. However, these methods do apply to VL dataset with sample-specific distractors. Ray al. (2019;; /Selvarauu et al. 22020) Ribeiro jet al. (2019) utilize annotated sub-questions to enhance models' prediction consistency. Neverthe- less, these methods have high manual cost and hard generalize Gokhale et al. (2020) applies ad- versarial methods to modify the images but only limit to questions asking about colors and numbers.
Other methods Niu et al. T2 Wang al. (2022); Zhang al. (20216); Niu & Zhang (2021): Gupta et al. (2022) either require largely pretrained VL models, additional data resources or focus only on one modality and hence are hard generalize. Differently. CDS produces synthesizd pos- itive and negative data for both images and texts. ICL can further provide an effective intra-sample
contrastive learning strategy with simple setups while maintaining base models' structures.
DATASET BIASES AND SHORTCUTS
Former works Cao et al. (020): Ye & Kovashka (202T); Agrawal et al. (2018); Ramakrishnan et al. 2018 have identified that existing VL models are vulnerable to over-reliance on data from text modality and under- utilization from visual modality. Different from the conventional VL datasets like VQA.Antol et al. (2015) and GQA Hudson & Manning (2019), questions and answers in highly semantic VL datasets Zellers et al. (2019). Xie et (2019): Park et al. (2020); Tapaswi et al. (2016) contain much more implicit semantic information and images also complex understand.
3

---

Under review as a conference paper at ICLR 2023
Hence, it may be more difficult create those highly semantic dataset without biases and supervise VL models' training on them without picking up shortcuts. To further quantify this issue, we finetune two T5-Base Su et al. (2019) models on the training set of both MQA-Multiple-Chonl Choice and VCR with only text data. Comparing the validation results (21% on VQA-Multipe--Cooice and 58% on VCR), we notice that language-only model can achieve higher performance via only relying on text information (including question and answer choices) on the highly semantic VL dataset, VCR. To our surprise, language-only model can achieve more than 50% much higher than the accuracy of random guessing, 25%. This indicates potential learnable strong correlations between only texts and groundtruth labels. The correlation concerns us and may also counter the original objective for evaluating VL models' abilities in visual commonsense reasoning via utilizing both image and text.
In this work, we define "shortcuts" in a more generalized form comparing with Ye & Kovashka (202]:: way of selecting the correct answer by reliance on simple (lexical) patterns, matching repeated references without requiring to fully understand the given image and text. In the following, we share two identified shortcuts based on our analysis.
Shorteuts of Overlapping Words: As in Tab. D for each ceesttoo-annser-choiiee pair in VCR, we first preprocess them via simple tokenization and lemmatization into tokens. Through exact matching, we cumulatively count the number of overlapping words between the question against the correct and the incorrect answer choices respectively. Comparing the top two rows, we discover that the correct answer choices have much higher frequency for containing more overlapping words against the question compared with the incorrect ones. We further experiment measure the val- idation accuracy of always selecting the choice with the most overlapping words. To our surprise, in the highly curated VCR dataset costing millions, this simple strategy can already bring us an accuracy of 53.14%. For verifying that if it is plausible for existing models to take this shortcut, we further extract a subset from the full VCR data with which the former finetuned language-ons T5 model can achieve more than 90% confidence in prediction. After conducting the same analysis on this subset, as shown the third and fourth rows of Tab. 1 we realize that, within 77.81 % of this subset, the correct answer choices have the most overlapping words across all choices. This obviously verifies that existing models are capable of utilizing this shortcut.
Shortcuts of Low-quality Distractors: Due to the nature of MCQ, the quality of incorrect answer choices (distractors) also profoundly affect the difficulty of selecting the correct choice. Based on our experiences of dataset annotation, cognitively it is often much more difficult to come up with quality distractor than the obvious correct choice even for human annotators. Referring Zellers et al. 2019), for every VCR question, the distractors are actually derivation of correct choices of other questions after modification with heuristic rules. With the well-konwn lazy annotation biases, it is not surprising for distractors. We started with transforming questions and answer choices into a common feature space via a pretrained BERT model Reimers & Gurevych (201)). Then, for every image-question pair: We calculate the cosine similarity of its paired correct answer choice against all the answer choices of other inage-question pairs in the dataset and sort them based on the similarity. Similarly, we also calculate the cosine similarity between every correct choice against the three incorrect ones then average them all across samples. We find out that for every sample, the average similarity score between the correct and the incorrect is 0.31. To our surprise, it is even lower than the score between the 1000th ranked similar choice sorted across the dataset and the correct choice. This implics that existing VCR incorrect choices even outside the top 1000th window of the most similar choices against the correct answer. With applying a simple K-Nearest-Neighbor algorithm Kozma (2008), further cluster all the answer choices into 20 clusters. For every correct choice, we calculate the similarity score of it against the furthest choice in the same cluster. The average score is 0.34 and it is also higher than the average score between the correct and the incorrect ones for every -mage-question pair. Regarding this, we believe existing distractors in highly semantic VL dataset like VCR suffer biases of low-quality and this may lead to shortcuts models can easily pick up.
realize that most questions in VCR may be paired with much low-quality
4 METHOD
There are two approaches in terms of countering biases and learning shallow shortcuts: the first one is to directly modify the data to mitigate biases in the dataset and the second one is to modify the training strategy of models to avoid learning the shorteuts. In this work, we propose Counterfactual
4

---

Under review as a conference paper at ICLR 2023
Vision-Language Data Synthesis and tackle those two aspects in a unified way. Given an input image I, question Q, and answer A, apply CDS to generate positive and (osuttive(cuumtteracttuuaa) synthesized image and answer data, (I+, - and (A+, A-). Unlike Chen et al. (2020a); |iang et al. (2020); Gokhale et al. (2020), Intra- sample Contrastive Learning (ICI.) helps bypass the need of creating paired synthesized answers for counterfactual images.
Contrastive Learning (CDS-ICL) framework that
4.1
DOS-TEXT-POSITIIE
External Knowledge: As discussed in the previous section, correct answers often share more over- lapped words and syntactical structures with the question. To mitigate this dataset bias well as derive more synthesized choices, our strategy is to replace words, phrases of the correct choice with semantic-related alternatives to create variations with fewer overlapped words. Unlike former meth- ods Chen et al.] (2020a); Liang et al.] (2020); Gokhale et al.] (2020) that simply place masking tokens and drastically disturb the original data distribution, with semantically consistent noises. We first utilize some external database to retrieve alternative words replace words in the correct choice, especially the overlapped ones. The alternative words consist of synonyms and hypernyms from a lexical database, WordNet [Felbbuum (2010), and also connected concept-words, from ConceptNet[Spect et al. (2017) through a set of selected relationships (Referring to Appendix).
synthesized text data is numan-readalee
Pretrained Language Model: To increase the variation in our synthesized choices and avoid naive and biased templates, we further utilize pretrained language models: We inference a La-Le model Raffel et al. (2020) finetuned on TaPaCo Scherrer (aPherr/ccerrer (2020) to generate additional paraphrased variations of the correct choices; We also remove stop words from existing correct answers and keep the rest as input of keywords to a pretrained T5-large model finetuned on CommonGen Lin et al. sentences.
2019) to generate
For ensuring the quality of our created choices and mitigating the biases, lastly, we apply adversarial filterings. First, we calculate the average number of total overlapped words between the distractors and the image-question pair. To mitigate the bias having more overlapped words in correct answer choices, have the same number of overlapped words as the average number of incorrect choices. Lastly, combine question-coorect-answee pairs into statements/captions, culate the sentence similarity Reiimers& & Gurevych (2019) between each of the generated choices filtered from the last step against the corresponding statement and only keep the top three our final synthesized positive answer choices.
Adversarial Filtering:
choose to reserve generated positive choices that
via heuristic rules"] and cal-
4.2 CDS-TEXT-COMIE TTEATTCOONTERFFCTUUAN
For creating counterfactual answer choices and mitigate the biases of low- quality distractors, generate alternative distractors that are more similar to the correct answers while preserving the semantic difference. Therefore, we utilize both the original correct and in- correct choices generate counterfactual distractors. If we modify based the correct choice, we would only replace the target word with antonyms from WordNet and connected concept words from ConceptNet via selected relationships If we modify based on the incorrect ones, also utilize synonyms and hypernyms from WordNet and connected concepts via another larger selected set of relationships (referring Appendix) in ConceptNet. Following Ye Kovashka (202I), we always align the pronouns in the generated incorrect choices against the pronouns in the question.
External Knowledge:
Pretrained Language Model: Additionally, we also paraphrase each original incorrect choice with pretrained language models to generate more variations as before.
Adversarial Filtering: Lastly, after grammar filtering, we group all the synthesized counterfac- tual choices derived from both original correct and incorrect choices together to apply filtering. We first calculate the average number of overlapped words between the correct choice and the image-question pair. To mitigate the low-quality distractor bias, we reserve generated counterfac- tual choices that have the same number of overlapped words as the original correct choice. Then,
Details of hcuristic rules is listed in Appendix (DistinctFrom, Antonym)
5

---

Under review as a conference paper at ICLR 2023
based sentence semantic similarity, we rank the generated choices by their maximum similarity measured against any of the original incorrect choices. After selecting the top twenty choices, we re-rank them based their similarity against the corresponding statement (combination between question ans answer) and finally only reserve the top three. Hence, we expect the final selected generated distractors to be more relevant to the original question-correct-anssee pair.
SPL
1st Plsss
Triple
2nd Pasts
3nd
Figure 2: Diagram of Coarse-to-Fine Region Removal.
4.3
CDS- IDAGE
Preventing existing models from neglecting visual information and assisting them focus relevant regions, we propose a region removal method that generates both It and I- to emphasize the correct visual dependency. To generate It, remove image regions that are irrelevant to the question and answer so that the semantic integrity of the image is preserved. On the contrary, we remove relevant image regions create I-. To minimize disturbance in the image distribution, we design two-stage coarse-to-fine visual in-painting steps.
Many methods have been proposed select the relevant regions in the image given the question and answer. Das et al. (2017); [Selvarauu et al. (2019) use collected human attention maps select relevant regions on VQA (Antol al. 2015). However, since hu- man attention maps are often hard to collect, some alternative methods such as Grad-CAM Chen al. (2020a); Zhang al.) (202Tb) and visual-textual co-attention map Lu et al. (2016) are pro- posed. Nevertheless, as these methods obtain relevant information from the embedding space of finetuned model, we are concerned that they may already suffer from learning dataset biases based on our analysis before. To ensure correctly determining the relevant entities and regions, we adopt a conditional selection mechanism. At first, we prioritize directly match overlapped tokens between VCR's images and texts via keywords- similar to Gupta et al. (2022). For (1, Q, A) pair, we construct the image token set S₁ by combining the annotated ground-truth object labels and the detected object labels from Anderson al. (2018). We also further build the textual object set Sy via extracting all the tokens present in the question, the correct answer, and rationale except stop words. We compare every element of S, against every element of ST. When comparing two objects, we exhaustively apply keywords-matchin stems, and substrings. If there exists any directly matched entity, we prioritize its region as the rel- evant region. If none is matched, we then apply soft matching via inferencing an off-shelf concept similarity model Zhu Iglesias (2017) to measure the semantic similarity of every possible pair between the two sets and obtain the scores, Csem- Further, we also obtain the statements/caattions as a combination between Q and data mentioned before. Afterward, we utilize pretrained CLIP model Radford et al. 22221) to measure the visual relevance between the region of every en- tity from S and the statements and obtain the scores, C... Finally, we define relevant regions as: REL {R (s:)
Relevant Region Selection:
between them and their corresponding lemmas,
S,), if 3sc, (st, st) 1, Csem (ss, C, (R (sx). T. R(s) denotes the region for the object and ff represents the wwwords-matchhnng procedure. When equals 1, the two entities are matched. T is hyperparameter threshold.

---

Under review as a conference paper at ICLR 2023
Coarse-to-Fine Region Removal: After determining the relevant regions in images, following the structure of SPL Zhang et al. (202aa), we design a Coarse-to- torree-t--Fine In- in-painting Generative Adver- sarial Network (IGAN framework remove relevant and irrelevant regions to generate realistic natural images, I+/- In this framework, we reserve two TGANs, an TGANp pretrained on Places2 Zhou al. (2017) in Zhang et al.] (2021a) and another IGAN; finetuned on VCR. We first trieve the segmented polygons (if' the entity is provided by VCR annotation) or bounding boxes (if generated) of the selected relevant entities. After determining the dimensions of the maximum inscribed rectangles within the polygons or boxes, we create corresponding rectangle maskings in ratios, (0.7, 0.5, 0.3) of the maximum dimensions. Similarly, we also calculate the maximum di- mensions of inscribed rectangles in regions that have no entity overlapped on top at all and create maskings of different ratios within those regions. When fine-tuning IGAN, y on the VCR training set, we input images with regions masked by one of those maskings and supervise TG (GAN, to re- construct the masked region. Therefore, essentially 1GAN, is trained to reconstruct the interior of either an entity an open background region based on its neighboring non-masked pixels and patterns. In order to create It, in inferencing, we filter to irrelevant regions, R (s:) & R E L and create maskings of the minimum circumscribed rectangles around the boxes polygons. We then feed images masked by those maskings into IGANs to remove the irrelevant entities. Similarly, For creating I-, we create similar maskings over the relevant regions from REL inference remove the relevant entities. Some examples of the reconstructed images by IGAN are shown in
In order to produce fine- greeigrrind images and avoid drastically disturbing existing image distribution and bringing obvious artifacts/biases as occlusion boxes do in Chen et al.) (2020a); Liang et al. 2020); Gokhale et al. (2020). in practice, during infereneing, we apply coarse-to-reline strategy by first passing the masked images into IGAN,, that was pretrained on a larger dataset and then feed the reconstructed output to ,GAN, that was finetuned more specifically to refine the reconstruction. As inferencing in IGAN,, we additionally refine the image via a triple-grid mechanism. As in Fig. 2 for a given masked region, we evenly split into M blocks and N blocks respectively where 2 M < N. In the first pass, we allow IGAN, to reconstruct the whole masked region in one pass and then revisit the same region with smaller maskings in the following two passes. In the second pass, we take turns turn each of the M blocks in order into smaller masking, from the top left to the bottom right, and accordingly reconstruct each masked region to refine. Note that when reconstruct the first block from the top left of M blocks, the visual regions of the rest M - 1 blocks are not masked but in-painted with results from the former pass as placeholders. Therefore, we cumulatively inference M times to refine the whole region in the 2nd overall pass. A similar procedure is carried out for the third pass with N blocks. This method allows the framework to maintain the global consistency in reconstructed visual patterns while obtaining the flexibility in refining smaller regions.
4.4
CONTRASTIVE LEARNING IN VCR
4.4.1
ANSWER-FOCUSBD CONTRASTIVE LEARNING
In former works resolve Visual- -uuustion-Answeeinn--yyyceo model as maximizing a probability of answer selection conditioning on the given image and ques- tion. P(a I.Q). As shown in Fig. 3 it can be essentially regarded as problem of mapping from (T,Q) to A. Hence, former works Chen et al. (2020a); Liang et al. (2020); Gokhale al. (2020) have to create heuristic methods to create paired answer for every created counterfactual image. This unfortunately may result in incorrect or counter- intuitive answers with strong biases like "Not "Not Green". " In this work, due to ICL (discussed later), we do not have to create paired answers for coutnerfactual images. Therefore, we only need to focus on differentiating positive and negative answer choices within each VCR sample. Considering this, we follow the conventional methods to include the Cross Entropy loss of mapping (I, Q) to A via:
tasks, it is conventional to
K
LOE = Σ yd log (o (P(((alL,0)) I.Q)))
(1)
where y¹ is the groundtruth label, K is the total number of samples and o is the softmax function. During QA training, within each batch, we conduct softmax bewteen the logit of each correct answer
7

---

Under review as a conference paper at ICLR 2023
Answer Socsee Connrsstive Loming Loaming
origia
(Ac, 3A3
[A+, 3AI]
Ui+ 3A-1
[Ac 340
Irage-focioed lntre sample
Inve (peronnd roing
Helsgpnn jeeronce
[penand))
Woott persond coin? All hhisatecigeetaa seanc (panord)
Our
Ac Hevill joicat milpnnen)
Figure 3: Diagram of the combinations of (1, Q. A) pairs utilized in training. The pairs form the top block are utilized in QA classification training. The pairs in the bottom block are used in intra- sample contrastive learning. The orange lines indicate the overlapping/repeated pairs. In practice, only pass those pairs once in the feedforward operations. Ac represents the correct choices. Ai represents the incorrect choices.
and three other incorrect ones. As in Fig. within this training strategy, we can directly augment T+ and A /- into QA training.
4.4.2 IMAGE-FOCUSED INTRA SAMPLE CONTRASTIVE LEARNING
However, is not straightforward how to apply the counterfactual image, I- into cross entropy classification loss above. Hence, we decide to transform the problem of mapping (IQ) to A, (/Q A) into another related problem mapping (QA) to I. (QA I).
Conventional VQA, GQA datasets do not have sample-specific distractors, and therefore, resolving their tasks, IQ + mapping is essentially learning to differentiate the current sample's answer from other samples' based the given (I, Q) data. In other words, their tasks focus inter- sample differentiation. Hence, learning other inter-sample mappings like QA - with contrastive loss (learning differentiate images of different samples based the given (Q, A) data) may help resolve the original VQA tasks Li et al. (2018). Differently, VCR has sample-specific distrac- tors. Therefore it focuses more intra-sample differentiation and learning inter-sample mapping of QA + I may not directly help optimize solving the VCR task. On that account, we limit the QA + / mapping problem into the intra-sample pairing problem between each (Q', A') pair against (T', I'+, ('+, with a loose contrastive loss. As mentioned before, existing VL models under-utilize visual information such that they fail understand and pick up relevant visual clues from images. Since the contrast among (I', 1'+, P-) -) is the presence of relevant visual regions to the question and answer, constraining the learning to would regulate models to better utilize the visual data in VCR. As shown in Fig. 3 the model would be reinforced to realize the intra- visual difference, the relevant entity to the question and answer. the cup. Even though it is not directly optimizing for selecting the answer but it is aiming to select the critical entity the answer mentioned.
We further follow the conventional methods with VL transformer-bas based models and extract the [CLS] token's 's hidden feature, in the end represent the whole input (I, Q, A) pair, e.g. positive pairs like (I, Q, A) and (I+, Q, A) and negative pair (I_,Q, Q. A). Lastly, the contrastive learning is via calculating the InfoNCE loss Van den Oord et 2018):
exp ( (z, z,) /T)
LNCE = loog
(2)
exp ( (z, pp) /T) + exp ( (z, zn) /T)
8

---

Under review as a conference paper at ICLR 2023
where & measures the cosine distance, is hyperparameter temperature, 2p is the [CLS] token feature for positive pair, (I,S) or (I+, and Zn is for a negative pair. (I-
Finally, the overall objective function is given by:
X LeE A2LNGE
where A and X2 are hyperparameter weights.
5 EXPERIMENT
In this section, we first describe the datasets including debiased evaluation benchmark. We then give quantitative analysis on the statistics of biases, ablation, benchmark evaluation and debiased evaluation.
5.1 BASE MODEL
CDS-ICL is generalizable method that can be applied on different VL models. clearly demon- strate its effectiveness, in this work, we evaluate it with three high-performing baseline methods: one two-stream transformer- based VL model, LXMERT Tan Bansal (2019) and two one-stream transformer-basee VL models, VL-BERT Su et al. (2019) and UNITER Chen et al. (2020)).
5.2 DATASET
5.2.1 VCR AND VCR-PRONOUN-SHIER
We conduct training and evaluation on the train and validation subset of VCR data consisted of around 290K pairs of image, question, and four-way answers. Additionally, for comprehensive comparison in domain-shife scenarios, we adopt the evaluation with rule-based pronoun-shift from Ye & Kovashka 2021) as one of our evaluation settings, "CR-Pronoun-
5.2.2 VCR-AS
To fairly compare VL models' capabilities in highly semantic VL data with domain-shift and min- imized artifact/noises, we only include the synthesized data samples with synthesized answer data in our own debiased evaluation benchmark. Hence, we include sample data with combinations: (Q(ee, 3A,). IQ(A+, (AA) (Q(A,, 3A"). IQ(A+, 3A-) in our debiased evaluation benchmark with more generalized domain-shin scenarios of answer choices, VCR-Answer-Shhe (VCR-AS).
5.3 STATISTICS AND BIASES
As in Tab. π with CDS-Text the difference of the average number of overlapping words (against either the VCR question or object labels from the VCR image) between the correct and incorrect choices becomes much less comparing with before. The percentage incorrect choices that have more overlapping words against the correct ones also increases. Furthermore, after CDS-Texi, the averaged semantic similarity between every correct answer choice against the incorrect ones also increases from 0.31 to 0.4 after using CDC-Text.
5.4 ABLATION
According the first three rows in Tab. after CDS-Text brings synthesized positive and negative answer choice data and CDS- CDS-Image brings synthesized positive image data into the QA classifica- tion training with cross entropy loss, the baseline model produces obvious and consistent improved performance. With intra-sample CL, synthesized negative image data is also augmented into the training with contrastive loss and the baseline model can achieve an overall improvement of 1.36%. However, after Inter-sample CL, adding images of other samples into the contrastive learning, this distracts the model's 's attention in intra-sample differentiation especially among answer choices. The severe performance drop confirms with our hypothesis before.

---

Under review as a conference paper at ICLR 2023
Choice
VCR Data
Text
Overlapping Words
w/ Question
w/ Image
Avg. Num.
More( (%)
Avg. Num.
More(S))
Incorrect
Train + Val
N
1.80
38.16
0.95
32.23
Correct
Train Val
N
2.02
44.44
0.97
45.75
Einguage-biased
Incorrect
N
1.65
12.11
0.83
28.84
Subset
Language-biasee biased
Correct
N
3.14
77.81
0.98
48.16
Subset
Incorrect
Train + Val
Y 1.90
40.38
0.94
35.10
Correct
Train Val
1.95
42.81
0.95
42.17
Table I: Average number of overlapped words correct and incorrect answer choices against the questions and object labels from images.
CDS- Text
CDS-Image
Intra-sample
Inter-Sample
75.53
75.84
76.30
76.89 (+1.36)
74.16
Table 2: Ablation Study, The baseline model is VLBERT Large-
5.5
EVALUATION WITH VL BENCHMARK METHODS
As Tab. for clearly demonstrating the effectiveness of CDS-ICL, apply it top three top- performing VL models on VCR and witness consistent performance boosts.
Model
Q2A
QA2R
Q2AR
VL-BERT, Suet al.
2019
75.5
77.9
58.7
VL-BERT, + CDS
ICL
76.9
78.8
60.6
UNITER Chenet al.
20206
76.7
80.0
61.4
UNITER, CDS
ICL
77.3
80.8
62.3
VILLA Ganet 2020
78.2
82.2
64.3
VII.IA, CDS
78.7
82.6
65.0
(+0.7)
Table 3: Comparing against benchmark methods. The results are based on our re-implementation.
5.6 DEBIASED VL EVALUATION
CDS-ICL not only can assist existing VL models improve performances in the standard VCR evalu- ation but also can enhance their robustness against domain-shien shifts in data. In Tab. 2 with CDS-ICL, base models consistently suffer less performance drops.
Model
CDS-ICL
VCR
VCR-PS
VCR-AS
VL-BERT
75.5
71.1
70.6
76.9
74.2
73.7
UNITER,
N
76.7
72.8
72.0
77.3
75.4
74.7
Table Comparing against benchmark methods. The results are based our re-implementation.
6 CONCLUSION
In this paper, propose universal multimodal contrastive learning framework with novel data augmentation that jointly learn debiased VL representatiio for iments demonstrate that both the additional generated data and our contrastive learning framework improve models' performance on both the original VCR and our debiased VCR evaluation set while also helping models focus relevant regions in the image.
tasks. Our exper-
10

---

Under review as a conference paper at ICLR 2023
REFERENCES
Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4971-4980, 2018.
Peter Anderson, Xiaodong He, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answer- ing. In CVPR, 2018.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433, 2015.
Jize Cao, Zhe Gan, Yu Cheng, Licheng Yu, Yen-Chun Chen, and Jingjing Liu. Behind the scene: Revealing the secrets of pre-trained vision-and- and-lanuuage models. In European Conference Computer Vision, pp. 565-580. Springer, 2020.
Long Chen, Xin Yan, Jun Xiao, Hanwang Zhang, Shiliang Pu, and Yueting Zhuang. Counterfactual samples synthesizing for robust visual question answering. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 10800-10809, 2020a.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference computer vision, pp. 104-120. Springer, 2020b.
Corentin Dancette, Remi Cadene, Damien Teney, and Matthieu Cord. Beyond question-based bi- ases: Assessing multimodal shorteut learning in visual question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1574-1583, 2021.
Abhishek Das, Harsh Agrawal, Larry Zitnick, Devi Parikh, and Dhruv Batra. Human attention in visual question answering: Do humans and deep networks look at the same regions? Computer Vision and Image Understanding, 163:90-100, 2017. 2017
Christiane Fellbaum. Wordnet. In Theory and applications of ontology: computer applications, pp- 231-243. Springer, 2010.
Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adver- sarial training for vision-and-language representation learning. Advances in Neural Information Processing Systems, 33:6616-66288 6-6628, 2020.
Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang. Mutant: A training paradigm for diitriiitdttiiitiinnn generalization in visual question answering. arXiv preprint arXiv:2009 2020.
Vipul Gupta, Zhuowan Li, Adam Kortylewski, Chenyu Zhang, Yingwei Li, and Alan Yuille. Swap- mix: Diagnosing and regularizing the over-reliance on visual context in visual question answer- ing. arXiv preprint w:Xiv:2204.02285, 2022.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets of bert. arXiv preprint arXiv:1908.088: 085933 2019.
Laszlo Kozma. k nearest neighbors algorithm (knn). Helsinki University of Technology, 32, 2008
Qing Li. Qingyi Tao, Shafiq Joty, Jianfei Cai, and Jiebo Luo. Vqa-e: Explaining, elaborating, and enhancing your answers for visual questions. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 567-567, 2018.
Zujie Liang, Weitao Jiang, Haifeng Hu, and Jiaying Zhu. Learning contrast the counterfactual samples for robust visual question answering. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pp. 3285-3292, 2020.

---

Under review as a conference paper at ICLR 2023
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. Commongen: A constrained text generation challenge for generative commonsense reasoning. arXiv preprint arXiv:1911.03705 2019.
Jiasen Lu, Jianwei Yang. Dhruv Batra, and Devi Parikh. Hierarchical question-image co-attention for visual question answering. Advances in neural information processing systems, 29, 2016.
Varun Manjunatha, Nirat Saini, and Larry S Davis. Explicit bias discovery in visual question an- swering models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 9562-9571, 2019.
Yulei Niu and Hanwang Zhang. Introspective distillation for robust question answering. Advances in Neural Information Processing Systems, 15:16292-16300 2021.
Yulci Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian-Sheng Hua, and Ji-Rong Wen. Counter- factual vqa: A cause-effect look at language bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12700-1277 2021.
Jae Sung Park, Chandra Bhagavatula, Roozbeh Mottaghi, Ali Farhadi, and Yejin Choi. Visualcomet: Reasoning about the dynamic context of a still image. In European Conference Computer Vision, pp. 508-524. Springer, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 3748-8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with unified text-to-text transformer. Mach. Learn Res., 21(140):1-67, 2020.
Sainandan Ramakrishnan, Aishwarya Agrawal, and Stefan Lee. Overcoming language priors in visual question answering with adversarial regularization. Advances in Neural Information Pro- cessing Systems, 31, 2018.
Arijit Ray, Karan Sikka, Ajay Divakaran, Stefan Lee, and Giedrius Burachas. Sunny and dark out- side?! improving answer consistency in vqa through entailed question generation. arXiv preprint arXiv: 04696, 2019.
Nils Reimers and Iryna Gurevych. Sentence-bert. Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL httpps//aarvvvorrg/b///////
10084
Marco Tulio Ribeiro, Carlos Guestrin, and Sameer Singh. Are red roses red? evaluating consistency of question-answering models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6174-61 6184, Florence, Italy, July 2019, Association for Computa- tional Linguistics. doi: 1/119655/1/111--122.. 1621. URL Jutps://cclathooloy.orrPPP 1621
Yves Scherrer. TaPaCo: A corpus of sentential paraphrases for 73 languages. In Proceedings of the 12th Language Resources and Evaluation Conference, pp. 6868-6873, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-3444, URL https: /accanttology..rgg/2oo.cccr 1.848
Ramprasaath R Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Why did you say that? arXiv preprint 1rXiv:1611.07450 2016.
Ramprasaath R Selvaraju, Stefan Lee, Yilin Shen, Hongxia Jin, Shalini Ghosh, Larry Heck, Dhruv Batra, and Devi Parikh. Taking hint: Leveraging explanations make vision and language models more grounded. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 2591-2600, 2019.
12

---

Under review as a conference paper at ICLR 2023
Ramprasaath R Selvaraju, Purva Tendulkar, Devi Parikh, Eric Horvitz, Marco Tulio Ribeiro, Be- smira Nushi, and Ece Kamar. Squinting at vqa models: Introspecting vqa models with sub- questions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recog- nition, pp. 10003-10011, 2020.
Meet Shah, Xinlei Chen, Marcus Rohrbach, and Devi Parikh. Cycle-consistency for robust visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8649-6658, 2019.
Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In Thirty-first AAAI conference on artificial intelligence, 2017.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv: (9xXiv:/00..000 190800850000... 2019.
Hao Tan and Mohit Bansal. Lxmert: Learning cross poss-modality encoder representations from trans- formers. arXiv preprint arXiv: 177400900,, 2019.
Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. MovieQA: Understanding Stories Movies through Question-Answering. In IEEE Con- ference Computer Vision and Pattern Recognition (CVPR), 2016.
Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv e-prints, pp. arXiv-1807, 2018.
Zhecan Wang, Noel Codella, Yen-Chun Chen, Luowei Zhou, Xiyang Dai, Bin Xiao, Jianwei Yang, Haoxuan You, Kai-Wei Chang, Shih-fu Chang, et al. Multimodal adaptive distillation for lever- aging unimodal encoders for vision-language tasks. arXiv preprint arXiv:2204. 2022.
Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine- grained image understanding. arXiv preprint arXiv:1901 19XX::1901.06066, 2019.
Keren Ye and Adriana Kovashka. A case study of the shorteut effects in visual commonsense reason- ing. In Proceedings of the AAAI conference artificial intelligence, volume 35, pp. 3181-3189, 2021.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6720-6731, 2019.
Peng Zhang, Yash Goyal, Douglas Summers-Star Dhruv Batra, and Devi Parikh. Yin and yang: Balancing and answering binary visual questions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5014-5022, 2016.
Wendong Zhang, Junwei Zhu, Ying Tai, Yunbo Wang, Wenqing Chu, Binghing Ni, Chengjie Wang. and Xiaokang Yang. Context-aware image inpainting with learned semantic priors. arXiv preprint arXiv:2100..7722, 202
Xi Zhang, Feifei Zhang, and Changsheng Xu. Multi-level counterfactual contrast for visual com- monsense reasoning. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 1793-1802, 2021b.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on patlern analysis and machine intelligence, 40(6):
2017.
Ganggao Zhu and Carlos A Iglesias. Sematch: Semantic similarity framework for knowledge graphs. Knowledge-Bacd Based Systems, 130:30-32. 2017.
A
APPENDIX
You may include other additional sections here.
13


Reviews:
[{"review_id": "zrGvjZE74Q", "reviewer": "Reviewer_rKpc", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the biases in the VCR dataset. It first identifies two important bias problems, by exploiting which significantly better accuracy can be obtained than the random guess. The authors then propose to synthesize counterfactual text and image data to debias the VCR dataset. An intra-sample contrastive learning strategy is used to augment the VQA model training.", "summary_of_the_review": "The paper should be significantly revised.", "strength_and_weaknesses": "Strengths:\n1. The findings of the biases in the VCR dataset is inspiring for the entire community. While significant progress has been made and the best accuracy is getting closer to human level, little work has focus on the bias issues (I have to admit I am not an expert in the bias/debias area though. I'll rely on other reviewers' comments to update this claim if it is not correct.).\n\n2. The proposed debias approach shows improved accuracy on the VCR dataset.\n\nWeaknesses:\nMy major concern is the presentation of the paper. It is not clearly written and hard to follow. \n    a) Most importantly, there are no examples of what the altered candidate answers look like. Without them, it is really hard to understand the techniques presented in the paper.\n    b) The images in Fig. 2 and 3 are too small. It is hard to clearly see the content.\n    c) The usage of the math symbols is a bit arbitrary. For example, $I_+$ and $\\textbf{I}_+$ are used in mixture. In the Coarse-to-Fine Regional Removal part, the symbol $REL$ is not defined. \n    d) On the bottom of Page 5, it says details of heuristic rules is listed in Appendix, which is totally missing.\n    e) It might be a better idea to move Table 1 closer to Section 3 so that the readers can better understand the statistics of the bias in the VCR dataset.\n\nWith so many important details missing, I am unable to correctly gauge the technical content of the proposed approach.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "j-mjwnEYoNh", "reviewer": "Reviewer_nbLD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on addressing the biases in the VCR dataset. It found that the correct answer has the most overlapping words with the question and the incorrect choices have low bert--similarity with the correct ones. The authors propose to rewrite all answer choices, generate counterfactual images, and train the model with contrastive loss. Firstly, they replace the words with synonyms/antonyms or related concepts, generate new ones with pretrained LM, and filter the results by considering the number of overlapped words and their similarity to the questions. Secondly, the authors generate the counterfactual images by masking and impainting the question relevant/irrelevant regions. Lastly, they minimize the infoNCE loss given the generated positive (I+, Q, A) and negative (I-, Q, A) samples.", "summary_of_the_review": "Overall, the proposed method neither convince me nor was validated by the experiments. Thus I do not recommend acceptance.", "strength_and_weaknesses": "Strength:\n\nThe paper proposes a new region removal method through impainting instead of masking.\n\nWeakness:\n\nThe paper proposes to increase the number of overlapped words in incorrect choices but decrease the number in the correct ones. It may introduce a new bias that the correct answer choices have fewer overlapped words.\n\nReplacing words with antonyms is not guaranteed to generate an incorrect answer. For example, in Figure 1, the changed correct choice \"everyone on the porch is unsurprised to see [person6]\" still seems a correct answer compared with the other three. The authors do not provide any generated examples to show results are reasonable.\n\nEquation 2 lacks detail. One can generate multiple I+ and thus have more than one positive z_p in equation 2. However, the infoNCE loss can only have one positive z_p and multiple negative z_n. Do the authors randomly select one I or I+ as a positive sample?\n\nThe experimental results are weak. For example, the performance of UNITER Large is 77.3%, 80.8%, and 62.8% on the VCR leaderboard. The proposed method does not improve performance.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HIvJcRbvRvH", "reviewer": "Reviewer_BL7G", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper exceeds page limits (10 pages versus 9 pages). It should be desk-rejected if we follow the rules. \nThus, my rankings below should not be taken into account.\n\n", "summary_of_the_review": "The paper exceeds page limits (10 pages versus 9 pages). It should be desk-rejected if we follow the rules. \nThus, my rankings below should not be taken into account.", "strength_and_weaknesses": "The paper exceeds page limits (10 pages versus 9 pages). It should be desk-rejected if we follow the rules. \nThus, my rankings below should not be taken into account.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "2jE9ayoniG", "reviewer": "Reviewer_EKkA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "They proposed two methods with different aspects for debiased learning of multimodal learning tasks, e.g., VCR. One is counterfactual vision-language data synthesis (CDS) with linguistic prior knowledge, pretrained lanugage model, and adversarial filtering for textual CDS. And coarse-to-fine In-painting Generative Adversarial Network is used to generate CDS images. While the other is inter-sample contrastive learning exploiting the synthesized dataset. Using the top-performing VL models, they evaluate the proposed method on the VCR benchmarks.\n", "summary_of_the_review": "A major part of the paper is allocated to explain the counterfactual vision-language data synthesis; however, since experimental validation is weak, the novelty and significance of this work are incoherent.", "strength_and_weaknesses": "#### **Strength**\n\nFor the counterfactual vision-language data synthesis, they carefully generated CDS-Text and CDS-Image for ICL and semantical debiased benchmarks.\n\n#### **Weakness**\n\n- **W1. Evaluation and justification.** The current form of the manuscript incurs many questions. W1-1) How many samples are there in the original VCR and how many counterfactual samples are generated? W1-2) From this, how much it takes to train the model? W1-3) How do we decide this approach is preferable despite of increased training time and computational costs? W1-4) Ablation study for the isolated contribution of CDS would be significant (no ICL, only data augmenting)? W1-5) The improvements in Tables 3 and 4 are statistically significant? W1-6) What was the confidence interval? Therefore, the evaluation and justification of the proposed method are short of expectations for publication.\n\n- **W2. Writing.** There are many minor mistakes in writing and use the notations which are not correctly defined. This makes it hard to follow the content. The paper could be significantly improved by reorganizing and including extensive ablation studies to see the contributions of CDS and ICL and their competitiveness.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject