Title: SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterations
Benchmark: SugarCrepe++
Summary: SugarCrepe++ is a multimodal benchmark for evaluating semantic and lexical understanding. The benchmark improves upon prior compositional reasoning tasks by having the model choose between two semantically equivalent but lexically dissimilar correct captions, and one lexically similar but semantically dissimilar hard negative caption for an image. The benchmark is publicly available, human-validated, and can be used to evaluate multi-modal and unimodal LLMs. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: semantic understanding, compositional reasoning, compositionality, 
Phenomenon defined by authors: Yes
Phenomenon definition: Semantic equivalence is when two sentences convey the same meaning, and semantic similarity is when two sentences describe the same topic. Lexical refers to words/vocabulary and hence, lexical-similarity compares a pair of sentences at the word-level. In particular, a higher overlap of vocabulary and order of occurrence should lead to higher lexical similarity. 
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: No
Phenomenon contested: Contested
Purpose notes: The two sub-elements are semantic equivalence detection and lexical sensitivity. The appendix also defines syntactic similarity metrics, but they relate to the dataset's construction instead of the dataset's purpose. 
Task definition: Models are given an image, and must choose between three captions: a pair of semantically equivalent but lexically different correct captions, and one hard negative caption. The triplet ensures there are pairs of semantically-equivalent, semantically-opposite, lexically-similar, and lexically-dissimilar sentences. Both multimodal and unimodal language models are evaluated. 
Task item definition: Each sample in SugarCrepe++ dataset consists of an image and a corresponding triplet of captions: a pair of semantically equivalent but lexically different positive captions and one hard negative caption.
Task definition detail: The appendix reports several custom metrics to measure the syntactic and lexical similarity (SLS) between two sentences, in addition the VERA and grammar model scores of the benchmark. 
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: SugarCrepe++ uses image-caption pairs from MS-COCO, and uses fine-tuned Mistral 7B to generate a lexically different but semantically equivalent image caption, and then a coherent and fluent hard negative (incorrect) caption, that are automatically and manually validated. 
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: The paper does not justify the caption multiple-choice task as a complete real task nor ground the evaluation in a real-world scenario, but given its prevalence in the compositional reasoning literature, it could be more representative then presented. 
Dataset size: 4757
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric definition detail: The paper defines two custom metrics: ITT_{hit} for multi-modal image-to-text (ITT) models, and TOT_{hit} for uni-modal text-only (TOT) models. I is the image, (P1 and P2) are the positive captions, and N is the hard negative caption. Both metrics are binary. ITT_{hit} is 1 when (p(P1|I) > p(N|I)) ∧ (p(P2|I) > p(N|I)) for the likelihood p. TOT_{hit} is 1 when  (p(P1|P2) > p(N|P2)) ∧ (p(P2|P1) > p(N|P1)), also with likelihood p. The log likelihood is proportional to the cosine similarity between the respective embeddings. 
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: The type of hard negative: Swap Object, Swap Attribute, Replace Object, Replace Attribute, Replace Relation
Metric access: Model access required (e.g. logits)
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: The authors highlight that given the sensitivity of LLMs to prompt-formatting and and adversarial prompting, lexical structure likely influences semantic understanding. However, most benchmarks evaluate semantic similarity without considering lexical influence, and fail to investigate how models understand semantic equivalence given controlled lexical constraints. The task itself -- choosing the correct caption -- appears to be standard for the compositional reasoning literature, but the paper did not ground the task in existing literature beyond its use in current benchmarks. 
Taxonomy: Language Modelling / Robustness
New taxonomy: General Capability / Reliability