Title: MetaLogic: Logical Reasoning Explanations with Fine-Grained Structure
Benchmark: MetaLogic
Summary: This paper proposes MetaLogic, a benchmark designed to evaluate models' logical reasoning by generating detailed explanations called "logic metagraphs". These metagraphs extend typical reasoning chains by including rebuttal conditions, internal logical formulae based on modal logic, and degrees of certainty for each statement. Based on 1,000 logical passages from the ReClor dataset, MetaLogic challenges models to produce these fine-grained structures. Experimental results show current models struggle significantly with this task.
Contribution: Key contributions include: (1) Proposing the "logic metagraph", a novel, fine-grained explanation structure for logical reasoning, incorporating rebuttal, internal formulae, and certainty, inspired by cognitive science and logic theories. (2) Creating the MetaLogic dataset annotated with these structures using passages from ReClor. (3) Defining the task of generating logic metagraphs. (4) Benchmarking sequence-to-sequence models and demonstrating the significant challenge posed by the task.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Logical reasoning; Generation of fine-grained logical explanations.
Phenomenon defined by authors: Yes
Phenomenon definition: The ability to parse a natural language passage containing a logical argument and represent its underlying structure as a "logic metagraph". This involves identifying statements (nodes), their inferential relationships (support/rebut edges), the internal logical composition of each statement (formulae using variables and modal/logical operators), and the certainty level associated with each statement.
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Purpose notes: To create a benchmark that captures finer details of logical reasoning present in real-world arguments (like rebuttal and certainty) compared to previous simpler chain-of-reasoning datasets, leveraging established logical and argumentation theories.
Task definition: Logic Metagraph Generation: Given a logical passage (pre-segmented into statements and atomic sentences), generate the full logic metagraph including the support/rebut relationships between statements, the internal logical formula (using modal logic) for each statement, and the degree of certainty for each statement.
Task item definition: An instance consists of a logical passage from the ReClor dataset (which includes context, question, and options, though the task focuses on the passage logic) with pre-identified statements (nodes) and atomic clauses (variables). The target output is the fully annotated logic metagraph for the passage detailing node relationships (support/rebut), internal node formulae, and node certainty levels.
Task definition detail: The task involves generating a complex structured output. Internal node formulae use propositional logic extended with modal operators (necessity □, possibility ◇) based on the S5 system. Certainty is mapped to 5 discrete levels derived from modal logic.
Task source: Human exam questions (e.g. GRE questions), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language)
Task source detail: Passages from the ReClor dataset (GMAT/LSAT problems). Initial segmentation automatic. Crowdworkers annotated the meta-graph structure (support/rebut links) and internal binary logical relations. Unary operators and certainty labels were derived semi-automatically using keyword heuristics and dependency parsing, followed by checks.
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: Focuses on extracting and formalizing the logical structure inherent in complex argumentative texts, a core component of analytical reasoning.
Dataset size: Total: 1,000 passages/metagraphs. Test set: 200 passages/metagraphs.
Dataset size notes: Train: 600 passages. Dev: 200 passages. Total meta nodes: 3,609. Total formulae: 1,500.
Dataset metadata available: Yes
Dataset metadata detail: Source Passage Text, Segmented Statements (Nodes), Segmented Atomic Sentences (Variables), Meta Edges (Support/Rebut links), Node Formulae (Modal logic representation), Node Certainty Degree (5 levels).
Sampling method: Random sample (creators defined a task space and sampled from it), Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Free response (e.g. summary paragarph), Structured response (e.g. valid JSON, API call alone)
Response format detail: Models output a linearised text sequence encoding the metagraph's components: meta-structure edges, node formulae (using specific tokens for operators), and node certainties, separated by delimiters.
Metric definition: Exact Match (accuracy, F1, precision, recall), Macro-F1 for the multi-class certainty prediction
Metric definition detail: Component-wise F1 and AllCorrect scores for Meta Structure (Nodes, Steps), Formulae. Accuracy, AllCorrect, and Macro-F1 for Certainty. Overall AllCorrect score for the entire metagraph.
Metric aggregation: Simple Mean
Metric statistics: F1 score, AllCorrect (Exact Match), Accuracy, Macro F1
Metric subscores: Yes
Metric subscores detail: Performance reported per component (Meta Structure, Formula, Certainty). Analysis by inference type (Support vs Rebut) and per logical operator. Analysis across varying training data sizes. Error analysis categorizes mistakes within each component.
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Logic metagraph format based on Toulmin model and S5 modal logic. Annotations performed by trained crowdworkers with quality checks (IAA reported as high/very high). Detailed statistics on graph complexity, formulae, and certainty distribution provided. Error analysis identified specific challenges for models.
Taxonomy: Reasoning / Logical
New taxonomy: General Capability / Logical Reasoning
Procedural notes: Source data from ReClor dataset. Details the semi-automatic annotation of unary operators using dependency parsing and keyword indicators. Provides details on model implementations (T5, MetGen), training setup (batch size, epochs, optimizer, GPUs), and evaluation metrics. Error analysis categories are defined. Annotation interface shown.
Other notes: The standout feature is the highly detailed "logic metagraph" structure, aiming to capture logical nuances like rebuttal and certainty often missed in simpler explanation formats. The results strongly indicate that generating such complex structured logical representations from text remains a major hurdle for current generative models.