Title: Do Vision-Language Models Understand Compound Nouns?
Benchmark: Compun
Summary: Compun is a multimodal benchmark to assess how models understand compound nouns using text-to-image retrieval tasks. The dataset is publicly available, manually curated, and focuses on noun+noun compound nouns. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: compound nouns
Phenomenon defined by authors: Yes
Phenomenon definition: "A compound noun (CN) is a noun formed from two or more words combined to create a single noun with a new meaning." (519). 
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: No
Phenomenon contested: Widely-agreed
Purpose notes: Compun focuses "primarily on the noun + noun type" of compound nouns (519). 
Task definition: The task is "text-to-image retrieval where, given a text prompt with a CN [compound noun]" the model must "select the correct image that shows the CN among a pair of distractor images that show the constituent nouns that make up the CN" (519). 
Task item definition: "Each instance in Compun corresponds to a unique compound noun and includes one image representing the compound noun, along with two additional distractor images" (520). 
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: The paper defines a new pipeline to generate prompts for text-to-image retrieval, where given a compound noun, a LLM generates "multiple diverse captions" where "each caption describes a scene with the compound noun as a key object in it. Finally, the captions are used to construct a custom prompt for text-to-image retrieval" (520). Expert annotators manually collect the images per instance, and MTurk was used to decide the most common meaning if a compound noun had several interpretations. 
Task ecology: Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 400
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric definition detail: The paper defines a binary accuracy metric, where the score is 1 if the cosine similarity of the prompt and positive caption is greater than the similarities between the prompt and the negative captions, and 0 otherwise. 
Metric aggregation: Simple Mean
Metric statistics: Average
Metric subscores: No
Metric access: Model access required (e.g. logits)
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: The authors ground the text-to-image retrieval task in previous cognitive science research, and discuss how the benchmark can be improved by expanding to different types fo compound nouns and by using novel metrics for retrieval. 
Taxonomy: NLP / Understanding
New taxonomy: General Capability / Natural Language Understanding