Title: CLAMBER: A benchmark for identifying and clarifying ambiguous information needs in LLMs
Benchmark: CLAMBER
Summary: CLAMBER is a 12 K‑example benchmark that probes two complementary abilities of large language models (LLMs): (1) detecting whether a user query is ambiguous and (2) asking a helpful clarifying question when ambiguity is present. The dataset is organised around an eight‑category taxonomy that spans epistemic misalignment, linguistic ambiguity, and aleatoric output‑related ambiguities, enabling fine‑grained evaluation of current open and closed LLMs under several prompting regimes.
Contribution: - Introduces the first LLM‑oriented ambiguity taxonomy that explicitly separates unfamiliar‑knowledge and contradiction cases (epistemic) from lexical/semantic (linguistic) and missing‑facet (aleatoric‑output) cases.
- Releases ∼12 K validated examples drawn from and re‑worked out of eight existing resources plus GPT‑4 generation, each annotated with gold binary ambiguity labels and reference clarifying questions.
- Provides the first systematic comparison of ambiguity identification versus clarification across five popular LLMs (Vicuna/Llama2 variants and ChatGPT), revealing over‑confidence effects of CoT / few‑shot prompting and large performance gaps on semantic‑referent and contradiction ambiguities.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Ambiguity Detection, Query Clarification
Phenomenon defined by authors: Yes
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Task 1 – binary prediction of whether a query is ambiguous; Task 2 – generate one clarifying question aimed at resolving that ambiguity.
Task item definition: Each item in the dataset consists of a user query (question), optional context (context), a binary label indicating whether the query is ambiguous (require_clarification), the category and sub-type of ambiguity (category, subclass), and a gold standard clarifying question (clarifying_question).
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: The benchmark is built by repurposing and synthesizing examples from multiple prior datasets targeting specific ambiguity types (e.g., AmbigQA, AmbiCoref, AmbiTask, AmbER, ALCUNA), which were then transformed into a unified format using a consistent taxonomy. Ambiguous and unambiguous queries were manually or programmatically crafted or modified, and additional examples (e.g. clarifying questions) were generated via GPT-4 and validated by linguistic experts to ensure consistency and accuracy.
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 12,079
Dataset metadata available: Yes
Dataset metadata detail: ambiguity category
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), Human ratings (text quality, preference, NOT manual scoring of other metrics)
Metric aggregation: Simple Mean
Metric statistics: Mean and standard error; calibration (ECE) and AUROC for confidence analyses.
Metric subscores: Yes
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: The authors explicitly discuss the construct validity of CLAMBER by acknowledging that their results are sensitive to prompt selection and that only five LLMs were tested due to resource constraints. They also validate their data via expert annotation and revision, ensuring high quality of labels and clarifying questions. Furthermore, they provide a comprehensive taxonomy and perform both automated and human evaluation of model outputs, indicating a deliberate effort to ensure that the benchmark actually measures what it intends to i.e., LLM capability to detect and resolve ambiguity.
Taxonomy: Knowledge / Conflicts
New taxonomy: General Application / Knowledge Conflicts