Training and Inference Instructions
All models are evaluated using beam search with n=3 , temperature 0 , max length 256 . In the case of generating scene graphs for the compositional chain-of-thought method, we set the max length to 256 for the graph generation step as recommended by Mitra et al. ( 2023 ) . API models are evaluated with default hyperparameters.
We format all fine-tuning data in the instruction format following LLaVA Liu et al. ( 2023a ) . To avoid overfitting on a particular instruction for this task, we generate 20 similar instructions using an LLM (ChatGPT-4) and randomly assign one of them to every instance in the training, validation, and testing set. Same instructions were sampled for the e-ViL dataset. Table 8 shows the 20 instructions used.

No. Instruction 1 Does the image’s narrative confirm or disprove the claim REPLACE_CLAIM? Discuss your reasoning and identify it as either entailment or contradiction. 2 Does this image confirm or deny the claim REPLACE_CLAIM? Discuss your reasoning and determine a label: entailment or contradiction. 3 Is the image’s message supporting or opposing the claim REPLACE_CLAIM? Discuss your rationale and determine the appropriate label: entailment or contradiction. 4 Is there agreement or disagreement between the image and the claim REPLACE_CLAIM? Provide your analysis and choose between entailment or contradiction. 5 Does the visual evidence support or counter the claim REPLACE_CLAIM? Provide your explanation and assign it a label of entailment or contradiction. 6 Does the image agree with or dispute the claim REPLACE_CLAIM? Explain your analysis and mark it as entailment or contradiction. 7 Does the illustration affirm or contest the claim REPLACE_CLAIM? Provide your argument and choose a label: entailment or contradiction. 8 Is the visual content in agreement or disagreement with the claim REPLACE_CLAIM? Offer your explanation and categorize it under entailment or contradiction. 9 Is the image in harmony with or in conflict with the statement REPLACE_CLAIM? Explain your justification and label it as entailment or contradiction. 10 Is the portrayal in the image consistent with or contradictory to the claim REPLACE_CLAIM? Offer your insights and select between entailment or contradiction. 11 Does the image’s depiction validate or refute the claim REPLACE_CLAIM? Explain your point of view and select a label: entailment or contradiction. 12 Is the content of the image endorsing or challenging the claim REPLACE_CLAIM? Justify your position and label it as entailment or contradiction. 13 Is the image consistent with the statement REPLACE_CLAIM? Justify your answer and classify it as either entailment or contradiction. 14 Does the illustration affirm or negate the claim REPLACE_CLAIM? Articulate your reasoning and apply a label: entailment or contradiction. 15 Does the picture support or refute the assertion REPLACE_CLAIM? Offer your rationale and select a label: entailment or contradiction. 16 Is the visual portrayal compatible with or adverse to the claim REPLACE_CLAIM? Justify your viewpoint and label it as entailment or contradiction. 17 Does the image corroborate or dispute the claim REPLACE_CLAIM? Outline your reasoning and categorize it under entailment or contradiction. 18 Is the depiction aligned with or against the claim REPLACE_CLAIM? Share your evaluation and identify it as either entailment or contradiction. 19 Does the image entail or contradict the claim REPLACE_CLAIM? Explain your reasoning and provide a label between entailment or contradiction. 20 Can the image be seen as validating or opposing the claim REPLACE_CLAIM? Explain your thought process and assign a label of entailment or contradiction 21 Is the image’s representation supportive of or contradictory to the claim REPLACE_CLAIM? Articulate your analysis and assign the label: entailment or contradiction.