Title: Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models
Benchmark: Multi-LogiEval
Summary: Multi-LogiEval evaluates language models' multi-step logical reasoning across propositional, first-order, and non-monotonic logic with varied inference rules and depths. Tests on leading models reveal substantial performance drops as reasoning complexity increases, exposing critical gaps in logical reasoning capabilities.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: reasoning, multi-step logic
Phenomenon defined by authors: Yes
Phenomenon definition: The ability to perform multi-step reasoning – drawing conclusions from provided multiple premises -- is a hallmark of human intelligence. Our work aims to bridge these gaps by creating a more comprehensive and logically complex evaluation dataset by incorporating varying numbers of reasoning depths (i.e., multi-steps) to reach conclusions. Our work systematically evaluates multi-hop logical reasoning over various inference rules and their combinations.  (section 1)
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Given a context (story with logical statements) and a question (candidate conclusion), the model is expected to predict Yes/No. (Binary entailment classification)
Task item definition: Context, question, answer, metadata (e.g., logic type, logic depth)
Task source: Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 1552
Dataset metadata available: No
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean 
Metric subscores: Yes
Metric subscores detail: 15 types (3 logics × 5 depths)
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Error analysis, validity discussion
Taxonomy: Reasoning / Logical
New taxonomy: General Capability / Logical Reasoning