Title: Do Vision-Language Models Understand Compound Nouns?
Benchmark: Compun
Summary: We curate Compun, a novel benchmark with 400 unique and commonly used Compound Nouns (CN), to evaluate the effectiveness of VLMs in interpreting CNs. 

We perform an in-depth analysis to highlight CLIPs’ limited understanding of certain types of CNs. 

We present an alternative framework that moves beyond hand-written templates for text prompts widely used by CLIP-like models. We employ a Large Language Model to generate multiple diverse captions that include the CN as an object in the scene described by the caption. Our proposed method improves CN understanding of CLIP by 8.25% on Compun.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: VLMs' ability to understand compound nouns
Phenomenon defined by authors: Yes
Phenomenon definition: A compound noun (CN) is a noun formed from two or more words combined to create a single noun with a new meaning. 

Interpreting the meaning of CNs by decoding the implicit semantic relation between their constituent nouns has attracted interest in NLP for decades.

Though extensively studied in NLP, whether modern vision-language models (VLMs) understand CNs is under-explored. Their paper fills in this gap. 
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: No
Phenomenon contested: Widely-agreed
Task definition: Each instance in Compun corresponds to a unique compound noun and includes one image representing the compound noun (CN), along with two additional distractor images. These distractor images depict the individual constituent nouns that form the CN.

Given the class name (or the CN), the task of a VLM is to retrieve (or select) the correct image among the distractors.
Task item definition: A unique compound noun, one image representing the compound noun (CN), two additional distractor images that depict the individual constituent nouns that form the CN.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 400
Dataset metadata available: No
Sampling method: Random sample (creators defined a task space and sampled from it)
Split reviewed: Test
Response format: Multiple choice
Response format detail: Multiple choice as in the VLM picks up one image out of three options
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: NLP / Understanding
New taxonomy: General Capability / Natural Language Understanding