Title: Diagnosing the First‑Order Logical Reasoning Ability Through LogicNLI
Benchmark: LogicNLI
Summary: The paper introduces LogicNLI, a NLI‑style benchmark crafted to diagnose large language models’ first‑order logic (FOL) reasoning abilities. It disentangles logic from commonsense by procedurally generating facts, rules and statements covering seven FOL operators, and evaluates models along four axes: accuracy, robustness, generalization and proof‑based traceability. Experiment results show substantial gaps to human performance and highlight weaknesses in negation and universal quantification.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: (logical) reasoning
Phenomenon defined by authors: Yes
Phenomenon definition: Authors define FOL reasoning as multi‑step inference using the seven foundational logical operators applied to simple propositions expressed in natural language.
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: Given a set of natural‑language facts and rules, model is expected to predict whether a hypothesis is entailment, contradiction, neutral, or paradox relative to the premise.
Task item definition: premise texts (facts + rules), hypothesis statement, groud-truth label, proofs
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 2000
Dataset size notes: 16000 (train), 2000 (validation)
Dataset metadata available: Yes
Dataset metadata detail: label balance, hop counts, subject/predicate/vocab sizes, sequence lengths
Sampling method: Random sample (creators defined a task space and sampled from it), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: robustness, generalization, traceability
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: No, link is broken
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: complementary test suits and analyses per operator
Taxonomy: Reasoning / Logical
New taxonomy: General Capability / Logical Reasoning