RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
What Errors Do Models Make?
We also analyze to understand what type of errors do each model make when they are considered not adequate in the above evaluation. In Figure 7 , we illustrate the normalized frequency of error types when both annotators agree that the explanation is not adequate (i.e., out of all errors for this model, what percentage is each type of error?). In general, annotators did not consider verbosity to be a major issue of the systems. For GPT-4, the leading error type is hallucination, indicating the need to improve faithful image recognition even in the most advanced models.
For the fine-tuned model and LLaVA-34B-SG, the main error type is unsound reasoning, indicating that it is challenging for the models to reason about multimodal figurative inputs consistently.
How Well Does the Explanation Score Predict Human Judgment on Adequacy?
We explore whether the proposed explanation score can capture human judgement of explanation adequacy. We collect all instances where both annotators agreed on the adequacy judgement for the explanation.
We evaluate if the explanation score described in Section 4.2 can act as a good predictor for the human adequacy judgment. We find that the area under the Precision-Recall curve is 0.79, and the maximum F1 score is 0.77, obtainable at the explanation score threshold of 0.53. Hence, we use this threshold to report the results in Table 3 . We also use the threshold of 0.6 since it maximizes F1 such that both precision and recall are above 0.75.
Conclusions
We introduce a high-quality dataset for understanding figurative phenomena in multimodal input, V-FLUTE, framed as an explainable visual entailment.
Our dataset consists of 6,027 < image, claim, label, explanation > instances spanning a variety of figurative phenomena such as metaphor, idiom, simile, sarcasm, and humor. We use this dataset to benchmark the performance of state-of-the-art vision-language models using both automatic and human evaluation and to identify critical areas of improvement for VLMs for this task.
B.1 Claude
Model Name: claude-3-opus-20240229

Max Tokens: 256

Images greater than 5MB were resized maintaining aspect ratio
B.2 GPT-4
Model Name: gpt-4-1106-vision-preview

Max Tokens: 256

Seed: 42

Image URL detail: ’high’
B.3 Gemini
Model Name: gemini-pro-vision

Max Tokens: 256

Safety Settings: ’BLOCK NONE’

Images greater than 5MB were resized maintaining aspect ratio
Fine-tuning
Seed: 42

Vision Tower: openai-clip-vit-large-patch14-336

Number of Training Epochs: 3

Train Batch Size (per device): 16

Eval Batch Size (per device): 4

Learning Rate: 2e-5

Weight Decay: 0

Warmup Ratio: 0.03

Scheduler Type: cosine

Number of epochs: 4 for eViL and eViL + vFLUTE, 10 for VFLUTE

mm-projector-type: mlp2x gelu

mm-vision-select-layer: -2

mm-use-im-start-end: False

mm-use-im-patch-token: False

image-aspect-ratio: pad

group-by-modality-length: False
LoRA
lora r: 128

lora alpha: 256

mm-projector-lr: 2e-5
Deepspeed Configuration
FP16 enabled: auto

BF16 enabled: auto

Micro Batch Size Per GPU: auto

Train Batch Size: auto

Gradient Accumulation Steps: auto

Zero Optimization Stage: 3
D.1.1 One-shot Prompt for generating explanations
We describe our one-shot prompts given to an LLM ( gpt-3.5-turbo-instruct-0914 ) for generating explanations of entailment-contradiction relationship. Refer to Table 9 for the detailed prompt.
D.2.1 Zero-shot Prompt for generating explanations
We provide our zero-shot prompt given to an LLM ( gpt-4-vision-preview ) for generating the entailment explanations given the claim and the image. Refer Table 10 for the detailed prompt.
D.3.1 Few-shot Prompt for generating opposite claims
We provide our few-shot prompt given to an LLM ( (gpt-4-0613) ) for generating the opposite claims. Refer Table 11 for the detailed prompt.
D.3.2 Zero-shot Prompt for Rephrasing
We provide our zero-shot prompt given to an LLM ( gpt-4-vision-preview ) for rephrasing the explanations given the claim and the crowd worker explanation. Refer Table 12 for the detailed prompt.
D.4.1 Few-shot Prompt for generating entailing claims
We describe our few-shot prompts given to an LLM ( gpt-4-0613 ) for generating entailing claims as part of the pipeline. Refer to Table 13 for the detailed prompt.
D.4.2 Zero-shot Prompt for validating the entailing claims
We describe our zero-shot prompt given to an LLM ( gpt-4-vision-preview ) for validating the claims generated in the previous step. Refer Table 14 for the detailed prompt.
D.4.3 Few-shot Prompt for generating opposite claims
We provide our few-shot prompt given to an LLM ( (gpt-4-0613) ) for generating the opposite claims. Refer Table 15 for the detailed prompt.
D.4.4 Zero-shot Prompt for generating explanations
We provide our zero-shot prompt given to an LLM ( gpt-4-vision-preview ) for generating the entailment explanations given the claim and the image. Refer Table 16 for the detailed prompt.
Figure 1 : Explainable figurative visual entailment task: given an image (premise) and a claim (hypothesis), output whether the image entails or contradicts the claim along with a textual explanation.; Explainable figurative visual entailment task: given an image (premise) and a claim (hypothesis), output whether the image entails or contradicts the claim along with a textual explanation.
Figure 2 : Creation of V-FLUTE instances for methaphors and similies from HAIVMet.; Creation of V-FLUTE instances for methaphors and similies from HAIVMet.
Figure 3 : Creation of V-FLUTE instances for metaphors, similes, idioms from IRFL.; Creation of V-FLUTE instances for metaphors, similes, idioms from IRFL.
Figure 4 : Creation of V-FLUTE instances for sarcasm from MuSE.; Creation of V-FLUTE instances for sarcasm from MuSE.
Figure 5 : Creation of V-FLUTE instances for humor from MemeCap.; Creation of V-FLUTE instances for humor from MemeCap.
Figure 6 : % Drop in F1 score for various models by source dataset between 0 to 0.6. Higher drop indicates higher proportion of wrongly generated explanations.; % Drop in F1 score for various models by source dataset between 0 to 0.6. Higher drop indicates higher proportion of wrongly generated explanations.
Figure 7 : Normalized frequency of main error types in the explanation by model.; Normalized frequency of main error types in the explanation by model.
Figure 8 : Distribution of lengths of claims by source dataset.; Distribution of lengths of claims by source dataset.
Figure 9 : Distribution of lengths of explanations by source dataset.; Distribution of lengths of explanations by source dataset.
Figure 10: Taxonomy of models used for the study.; Taxonomy of models used for the study.
(a) Metaphors and Similes; (b) Humor; (c) Sarcasm and Idioms; Performance of the models by phenomenon.
V-FLUTE dataset composition: 5 figurative phenomena, source datasets, and our contributions. E denotes number of entailment instances, C - contradiction.
Sample dataset instances form V-FLUTE corresponding to the source datasets displaying images (premise), claims (hypothesis), labels, and explanations [Row 1-5].
F1 Score results for different models across thresholds 0.0, 0.53, and 0.6 for explanation score. Best result overall is in bold, best result in each category is underlined.
Human baseline results (F1@0) by phenomenon and source dataset.
Adequacy and Preference rates for generated explanations.
Data counts per phenomenon and dataset.
One shot prompt given to an LLM ( gpt-3.5-turbo-instruct-0914 ) for generating explanations of entailment-contradiction relationship of the HAIVMET dataset.
Zero shot prompt given to an LLM ( gpt-4-vision-preview ) for generating explanations of entailment-contradiction relationship of the IRFL Dataset. The dataset contains similes, metaphors and idioms. For metaphors and idioms, the word simile in the prompt is replaced with the corresponding type.
Few shot prompt given to an LLM ( gpt-4-0613 ) for generating opposite claims utilizing the sarcastic claim and crowd worker explanation.
Zero shot prompt given to an LLM ( gpt-4-vision-preview ) for rephrasing the explanations given the claim and the.
Two shot prompt given to an LLM ( gpt-4-0613 ) for generating entailing claims utilizing the meme captions part of the MemeCap dataset.
Zero shot prompt given to an LLM ( gpt-4-vision-preview ) for validating the claims generated in D.4.1 . The corresponding meme image is also attached with the prompt.
Few shot prompt given to an LLM ( gpt-4-0613 ) for generating opposite claims utilizing the generated claim and explanation.
Zero shot prompt given to an LLM ( gpt-4-vision-preview ) for generating the entailment explanations. The corresponding meme image is also attached with the prompt.