Title: Larger Probes Tell a Different Story: Extending Psycholinguistic Datasets Via In-Context Learning
Benchmark: NEG-1500-SIMP, ROLE-1500
Summary: Psycholinguistic datasets for negation and role reversal, which extend existing smaller benchmarks using GPT-3. Evaluation of multiple LMs on these extended benchmarks reveals performance drops.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: negation, role reversal
Phenomenon defined by authors: Yes
Phenomenon definition: Negation sensitivity (LMs ability to understand negation);
Role reversal (ability to understand reversing semantic roles)
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Accurately predicting masked/target words in constructed sentence pairs - given negation and role reversal.
Task definition detail: Pair of sentences, e.g., an affirmative sentence and its negated counterpart.
Task source: Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 3000
Dataset metadata available: No
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall), Distribution (perplexity, calibration, correlation)
Metric aggregation: Simple Mean
Metric statistics: Simple means, McNemar test, Minimum detectable effect (MDE)
Metric subscores: Yes
Metric subscores detail: negation, role reversal, models, dataset versions
Metric metascoring: pass@k (any correct answer in k trials)
Metric access: Model access required (e.g. logits)
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: "We validated our new datasets through human evaluation. For this, we randomly selected 100 samples from each of the extended datasets"
Taxonomy: NLP / Understanding
New taxonomy: General Capability / Natural Language Understanding