Title: This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models
Summary: The paper introduces a large semi-automatically generated dataset of circa 400,000 descriptive sentences about commonsense knowledge that can be true or false in which negation is present in about 2/3 of the corpus in different forms. 
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: negation probing
Phenomenon defined by authors: Yes
Phenomenon definition: Although large language models (LLMs) have apparently acquired a certain level of grammatical knowledge and the ability to make generalizations, they fail to interpret negation, a crucial step in Natural Language Processing. 
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Take a natural language sentence, and classify its truth value. 
Task item definition: A brother is never a female person who has the same parents as another person.  True. 
Task source: Expert-crafted task examples (e.g. hand-written examples)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 90,281
Dataset size notes: Train: 268,505; validation: 2,514
Dataset metadata available: Yes
Dataset metadata detail: types and amount of negative knowledge
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train, Validation
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric subscores: Yes
Metric subscores detail: By types and amount of negative knowledge
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: 3.2 Dataset quality assessment: Human Evaluation addresses the validation of the
generation process and the different templates used, that is to say, whether the sentences in the dataset are grammatical and that overall represent true and false knowledge as expected.
Taxonomy: NLP / Understanding
New taxonomy: General Capability / Natural Language Understanding