Title: We're Afraid Language Models Aren't Modeling Ambiguity
Benchmark: AmbiEnt (Ambiguity in  Entailment)
Summary: This paper examines the ability of LMs to handle ambiguity. The authors model ambiguity via its effects on entailment relations and collect a dataset of NLI examples, each annotated with a set of labels (potentially reflecting underlying ambiguity) and disambiguating rewrites. LMs perform substantially worse on AmbiEnt than humans.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: They measure the ability of LMs to recognize and disentangle possible meanings of ambiguous sentences.
Phenomenon defined by authors: Yes
Phenomenon definition: "Formally characterizing ambiguity requires a choice of meaning representation to distinguish between possible interpretations, and enumerating the full set of interpretations can be tricky or impractical. Thus, we adopt a functional approach: using the natural language inference (NLI) task format, we characterize ambiguity in the premise and/or hypothesis by its effect on entailment relations." (p. 791)
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: No
Phenomenon contested: Widely-agreed
Task definition: The authors use AmbiEnt for three tasks (called "tests" in the paper): (i) generating disambiguations, (ii) recognizing the validity of plausible interepretations, and (iii) modeling open-ended continuations reflecting different interpretations.
Task item definition: Each item consists of:
- a premise/hypothesis pair;
- a set of labels (entail, contradict, neutral);
- disambiguating rewrites (for items with more than one label in the label set).
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 1,545
Dataset size notes: Validation: 100
Dataset metadata available: No
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Validation
Response format: Multiple choice, Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), Distribution (perplexity, calibration, correlation), Correlation (Matthew's correlation, Pearson's r), edit-f1
Metric definition detail: Model access required for one of the three tasks.
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: Yes
Human baseline: Yes
Authors discuss validity: No
Taxonomy: NLP / Understanding
New taxonomy: General Capability / Natural Language Understanding