Title: ScoNe: Benchmarking Negation Reasoning in Language Models With Fine-Tuning and In-Context Learning
Benchmark: ScoNE-NLI, ScoNE-NLG
Summary: This paper examines how well LMs handle natural language negation. To this end, the authors introduce two benchmarks: ScoNE-NLI, derived from the existing benchmark MoNLI, contains contrast sets of six examples with up to two negations where either zero, one, or  both negative morphemes affect the NLI label; ScoNE-NLG is specifically designed for generative models like InstructGPT and contains contrasting triplets of half-completed narratives that have different completions depending on the presence and scope of negation.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: They want to measure whether "models are truly able to handle nested negation and scope" (p. 1803).
Phenomenon defined by authors: Yes
Phenomenon definition: The authors provide the following explanation of what aspect of negation they focus on:

"[N]egation is not simply present or absent in a sentence. Rather, negation morphemes are semantic operators that take scope in complex ways, as we see in clear contrasts like \textit{the person who was at the talk wasn't happy} and \textit{the person who wasn't at the talk was happy.}" (p. 1803)

The focus on the interaction of negation and scope sets the introduced benchmarks apart from prior work.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: ScoNE-NLI: this is an NLI task where the correct label depends on parsing the interaction of negation and scope correctly.
ScoNE-NLG: this is a sentence completion task where plausible continuations depend on parsing the interaction of negation and scope correctly.
Task item definition: ScoNE-NLI: premise-hypothesis pairs where both can contain negations with different scopes.
ScoNE-NLG: half-completed narratives that can contain negations with different scopes.

Task definition detail: Items form sets (ScoNE-NLI: six items; ScoNE-NLG: three items), where the presence/scope of negation is systematically varied.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language)
Task source detail: Source of ScoNE-NLG is not directly specified, so I assume the 74 triplets are hand-crafted.
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: ScoNE-NLI: 1,200; ScoNE-NLG: 222.
Dataset size detail: The authors say they use the MoNLI train and test splits (which is the dataset ScoNE-NLI is based upon), but they do not provide any further details in the paper. I determined the numbers by manually examining the actual benchmark.
Dataset size notes: ScoNE-NLI train: 6,012.
Dataset metadata available: No
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train
Response format: Multiple choice, Free response (e.g. summary paragraph, executable code)
Response format detail: ScoNE-NLI: multiple choice; ScoNE-NLG: free response.
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric definition detail: Metric is nowhere mentioned in the paper, so I assume they use accuracy (standard for NLI evaluations).
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: They provide metrics for the different negation-scope combinations (six subsets for ScoNE-NLI, three subsets for ScoNE-NLG).
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: They find that InstructGPT fails on ScoNE-NLI, which they interpret as evidence that the benchmark is not suitable for that model. They then introduce ScoNE-NLG, which they claim is better for measuring the capabilities of generative LMs on scoped negation.
Taxonomy: NLP / Understanding
New taxonomy: General Capability / Natural Language Understanding