Title: SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterations	
Benchmark: SUGARCREPE++
Summary: In this paper, we introduce the SUGARCREPE++ dataset to analyze the sensitivity of VLMs and ULMs to lexical and semantic alterations.

We comprehensively evaluate VLMs and ULMs that differ in architecture, pre-training objectives and datasets to benchmark the performance of SUGARCREPE++ dataset. 

Experimental results highlight the difficulties of VLMs in distinguishing between lexical and semantic variations, particularly to object attributes and spatial relations.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: The sensitivity of VLMs and ULMs to lexical and semantic alterations.
Phenomenon defined by authors: Yes
Phenomenon definition: Semantic text similarity is one of the oldest metrics to evaluate language understanding and despite recent evidence of lexical sensitivity, large benchmarks evaluate semantic similarity without explicitly considering the lexical influence. In this work, we aim to address this gap by proposing a dataset to perform joint evaluation of semantic understanding — through the semantic equivalence detection task (elaborated below) — and lexical sensitivity in language models.' (page 2)
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: The task is to evaluate whether language models can accurately detect semantic equivalence or non-equivalence between pairs of captions that differ lexically and syntactically. 

Each input consists of an image and a triplet of captions: two semantically equivalent but lexically different captions (positives), and one semantically different caption (negative), forming a 3-way semantic (in)equivalence classification task.
Task item definition: A multimodal input pair of an image and a caption (semantically or lexically modified), and a binary label indicating whether the model ranks the original caption higher than the altered one.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 550,000 image-caption pairs
Dataset metadata available: Yes
Dataset metadata detail: Semantic and lexical transformations applied to the original image-text pairs: Swap Object, Swap Attribute, Replace Object, Replace Attribute, Replace Relation
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: By transformation type: Swap Object, Swap Attribute, Replace Object, Replace Attribute, Replace Relation
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Language Modelling / Robustness
New taxonomy: General Capability / Reliability