Title: Task Ambiguity in Humans and Language Models
Benchmark: AmbiBench
Summary: The paper investigates how both humans and models behave in the face of such task ambiguity by proposing AmbiBench, a new benchmark of six ambiguously-specified classification tasks. The paper also evaluates humans and models on AmbiBench by seeing how well they identify the intended task using 1) instructions with varying degrees of ambiguity, and 2) different num-
bers of labeled examples.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: how both humans and models behave in the face of such under-specified tasks. 
Phenomenon defined by authors: Yes
Phenomenon definition: Language models have recently achieved strong performance across a wide range
of NLP benchmarks. However, unlike benchmarks, real world tasks are often poorly specified, and agents must deduce the user’s intended behavior from a combination of context, instructions, and examples.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: detect a simple linguistic feature in an input sentence—for example, whether an outdoor location or an animal was mentioned—and then output the appropriate classification letter (X or Y).
Task item definition: Input: Output ’X’ if the sentence has a certain linguistic feature and ’Y’ otherwise. + A sentence

Output: "X" or "Y". 
Task source: Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 240
Dataset size notes: Train: 272
Dataset metadata available: Yes
Dataset metadata detail: Linguistic feature type. 
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric subscores: Yes
Metric subscores detail: By linguistic feature: human subject, indoor location, religious leader, pronoun gender, proper noun, negation
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: No
Taxonomy: Language Modelling / Robustness
New taxonomy: General Capability / Reliability