JustLogic: A benchmark for natural language deductive reasoning

Authors: Michael K. Chen, Xikun ZHANG, Dacheng Tao
Primary area: datasets and benchmarks
Keywords: benchmark, logical reasoning, LLM, natural language processing (NLP), propositional logic

Abstract:
Logical reasoning is a critical component of Large Language Models (LLMs), and substantial research efforts in recent years have aimed to enhance their deductive capabilities. However, existing deductive reasoning benchmarks, which are crucial for evaluating and advancing LLMs, are inadequate due to their lack of task complexity, presence of prior knowledge as a confounder, and superficial error analysis. To address these deficiencies, we introduce JustLogic, a synthetically generated deductive reasoning benchmark designed for rigorous evaluation of LLMs. JustLogic is (i) highly complex, capable of generating a diverse range of linguistic patterns, vocabulary, and argument structures; (ii) context-independent, eliminating the advantage of models possessing prior knowledge and ensuring that only deductive reasoning is used to answer questions; and (iii) capable of in-depth error analysis on the heterogeneous effects of reasoning depth and argument form on model accuracy. Our experimental results on JustLogic reveal that the performance of most state-of-the-art (SOTA) LLMs, specifically Llama3-8B (57.8\%), Llama3-70B (64.6\%), and GPT-4o (65.6\%), is significantly worse than the average human performance (73.0\%). A recently released reasoning model, OpenAI o1-preview, performed substantially better, with an accuracy of 81.0\%. However, it still lags behind the human ceiling of 100.0\%. These results demonstrate that the JustLogic benchmark is realistic and achievable for both humans and models and that there is still substantial room for improvement in the deductive reasoning capabilities of LLMs. We posit that the use of context-dependent and relatively simplistic benchmarks has misrepresented the reasoning abilities of many SOTA models. We release our open-source dataset to provide accurate evaluations of model performance in deductive reasoning and to facilitate LLM advancement through in-depth error analysis.

Decision: reject
Reviewer scores: 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730611511927

thread_type:
official_review

<thread_object>
creator: reviewer_lxqp

<content>
summary:
The paper introduces a synthetic benchmark JustLogic to assess the deductive reasoning capabilities of LLMs. Key contributions include
* Benchmark Design: JustLogic is a benchmark specifically for evaluating LLMs' logical reasoning skills. It minimizes the influence of prior knowledge, ensuring that models rely on logical deduction alone rather than external knowledge.
* Evaluation of Existing Models: JustLogic's experiments reveal that current LLMs, such as GPT-4 and Llama3 variants, perform significantly below human accuracy on deductive tasks, indicating substantial room for improvement in LLM reasoning capabilities.
* Error Analysis: JustLogic allows detailed analysis of model errors, exploring how factors like argument form and reasoning depth affect model accuracy, thus offering insights into specific areas of reasoning weakness in LLMs.

strengths:
* Originality: This paper contributes a well-defined deductive reasoning benchmark addressing limitations in prior benchmarks by focusing on memorization-aware design and error analysis. The originality lies in synthesizing complex deductive tasks without relying on real-world facts, thus eliminating the influence of prior knowledge, which is usually ignored in existing benchmarks.

* Quality : JustLogic’s design is grounded in logical principles, using well-established argument forms (e.g., modus ponens) to construct queries. It provides a comprehensive evaluation across multiple LLMs and include human baselines, showing the benchmark’s realism and effectiveness. The extensive error analysis further strengthens the paper.

* Clarity: The paper is clearly written and organized into motivation, dataset construction, and evaluation. 

* Significance: This work contributes to LLM reasoning evaluation, as it introduces a benchmark that better reflects true deductive reasoning without overly reliance on memorized facts.

weaknesses:
1. “it is important to note that, despite being drawn from real-world data, the statements are generally factually inaccurate. This is intentional.” “By using real-world yet factually inaccurate statements, we combine realism and context-independence.”

It does not make much sense to “intentionally” choose factually inaccurate statements. The authors claim to study reasoning from a perspective of “eliminating the influence of prior knowledge and ensuring context independence” (L93). However, the effect of prior knowledge is exerted from both the positive and negative sides, i.e. certain knowledge could align with or disagree with a given query. For example, if a model is already pretty sure about the fact that “Japan is in Asia”, then this degree of certainty would likely to affect the reasoning about the question that leads to “Japan is not in Asia”, so the influence of prior knowledge is not “eliminated”. In this sense, the topic would be counterfactual reasoning, which has been studied by a few previous work [1]. A more reasonable principle of choosing the statements would be complete irrelevance of real-world knowledge, e.g. “My cat Fluffy likes paper bags”. 

2. “The variable(s) within each expression is ultimately replaced by randomly selected generic, realworld sentences from GenericsKB-Best”

It would be better to include more details about the GenericsKB-Best dataset. One of the concerns would be: Is it assured that any two statements in the dataset are semantically independent with each other? For example, if statement A is “Japan is in Asia”, while statement B is “Japan is not in Asia”, then “A and B” will actually reduce to False. This problem would be more pronounced when the “argument complexity” is scaled up, as there would be more statements involved.

3. Error analysis is somewhat superficial. It is not surprising to see a decrease in performance as the reasoning depth increases. A more interesting investigation would be regarding at which step the model starts to fail and why, as it would be a direct guidance of steering the model towards the correct reasoning path and avoid straying from the subject. 

4. It would be better to add another section to prove the robustness of the benchmark by finetuning a model on the synthetic deductive data, showing whether the benchmark can be hacked by training on test data.


[1] Wu, Zhaofeng, et al. "Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks." arXiv preprint arXiv:2307.02477 (2023).

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730645705395

thread_type:
official_review

<thread_object>
creator: reviewer_3ipx

<content>
summary:
This paper describes a procedure for synthesizing data to test the logical reasoning abilities of language models.
The proposed method consists of:
1. Constructing random propositional logic arguments
2. Substituting natural language statements drawn at random from GenericsKB for the propositional variables
3. Sampling a truth value for the example (from True/False/Uncertain) and generating a query statement accordingly (either the conclusion of the argument, its negation, or a random related statement from GenericsKB, respectively)

The authors use this procedure to construct a challenge dataset, JustLogic, and evaluate several LLMs on it, in addition to conducting a human baseline study on MTurk. The human average is 73%, while OpenAI o1-preview gets 81%. The authors state that the 'human ceiling' is 100%, but do not clarify whether this is the accuracy of the best-performing participant, or just implying that every question was answered correctly by at least one participant.

strengths:
The authors rightly state that conflating prior knowledge and inferred knowledge is an issue with certain ubiquitous benchmarks, and causes problems when they are used to make claims about reasoning. It is helpful to have more challenging datasets that are controlled to ensure performance reflects models' ability to reason from provided context, rather than other extraneous features.

weaknesses:
Two of the properties of JustLogic claimed as virtues by the authors are somewhat dubious:
- W1: **The truth value of a statement is unrelated to its truth value in the real world** (termed "context-independence" by the authors). This seems to me to be mostly a source of confusion for both models and human labelers, as the dataset tacitly overrides the truth values of very basic facts. The value of "context-independence" is ensuring that answers can't be guessed based on questions in the absence of premises (as the authors test in 5.1). However, this can be accomplished without contradicting world knowledge - see for example MuSR (Sprague et al., 2024), in which premises involve world knowledge but answer options are controlled so that correctness solely reflects argument structure.
- W2: **High "natural language complexity", by which the authors mean vocabulary size.** I think this is a straightforward overclaim: the linguistic complexity of the task does not increase just because propositional variables have been substituted for longer or more diverse spans. A nontrivial notion of natural language complexity would imply a higher degree of complexity in the _relationship between_ the natural language and the logical argument structure, i.e. comprehending a wider range of linguistic structures would be required in order to recover the logical structure. In my opinion this would also imply moving beyond propositional logic, as natural language largely revolves around predicate-argument structure.

Additionally:
- W3: The 'future-proofness' of the dataset is called into question by the fact that o1-preview outperforms the human average. Additional results showing that the dataset's complexity knobs can be turned up to counteract this would mitigate this weakness. As it stands, the authors claim that the dataset can be scaled to match stronger models but do not actually demonstrate this for the strongest model.

questions:
**Suggestion:** It would be helpful for clarity if you could use a term other than 'context-independence' to describe the property of answer correctness not being influenced by prior knowledge, as 'context-independent' is ambiguous and somewhat counterintuitive: in fact, answers in JustLogic are _only_ sensitive to the question and context (JustLogic accuracy goes to random chance when C is omitted from the CQO tuple in 5.1). The clarity of the paper's arguments would be improved by avoiding the use of 'context' to refer to both world knowledge and question premises.

**Q1:** How is the human ceiling computed?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703781257

thread_type:
official_review

<thread_object>
creator: reviewer_nseq

<content>
summary:
The paper introduces JustLogic, a benchmark designed to test LLMs' deductive reasoning without the bias of prior knowledge. It highlights the shortcomings of the exitsting logical benchmarks based on complexity and error analysis -- showing some of the sythetic datasets, while may not rely on prior knowledge, are not complex enough. Other datasets, are complex but may be solve using prior human knowledge. Authors curate a large number of logical derivations, using a mix of LLM and human generated templates and propositions from GenericsKB, authors create a logical benchmark, which is synthetic, (possibly) not true in the real world and complex.  Despite improved performance by models like OpenAI o1-preview, they still lag behind human reasoning. JustLogic aims to drive deeper evaluation and enhancement of LLM capabilities.

strengths:
1. The paper highlights flaws in current reasoning benchmarks: low complexity, prior knowledge dependence, and limited error analysis.
2. It addresses the flaws highlighted in point 1 while constructing the new dataset JustLogic that have been unaddressed in datasets like FOLIO, ProofWriter, etc.
3. The solution to the flaws like focusing on argument complexity and Prior knowledge independence looks interesting.
4. Very minimal manual effort is required to construct the dataset.

Authors also aspire to make it more future-proof by leveraging synthetic creation abilities.

weaknesses:
1. There are several points which are claimed but do not have enough justification, for example, it may not be naturally true that a "conclusion" is untrue in the real world just because it is synthetic. No effort have been made to enforce factual inaccuracies -- they should also compare with PrOntoQA, which created a "Fictonal" ontology and a " false" ontology for this purpose. I believe, such a process is required, if you really want to ensure that prior knowledge will not affect the results.
2. Overall the conclusion that every LLM except o1 is lagging -- seems to not add much to the ongoing body of knowledge. Why are some of the derivations difficult vs easy. Only 1/2 line of explanations are given, that too in a hand-wavy manner (L513)
3. There are some issues with authors arguments about "flaws" in the benchmarks. See questions.

questions:
1. Context-ndependent is mentioned in abstract/intro many times. It is unclear how the authors ensured. Seemed like nothing special has been done.
2. Why is the average human performance so low? If you recruit more trained puzzle solvers, such as say linguistic olympiad participants or winners, I believe they may do much better, In that case, this false supremacy of o1 should not be there.
3. L100: Why is it novel? I did not understand what is claimed to be novel here at all.
4. Tab 1: It will be better to clarify what does complexity mean in statistical terms, number of clauses, words etc.
5. L290: This Step 3 is written in a very convouted way or is it the process? Are we saying we have a conclusion and we do not know using logic, whether the conclusion is entailed or not. This can not be true.
Then why a paragraph is randomly assigned an answer? Please explain clearly.
6. L317: How is the number of domains measured? Is it from GenericsKB? 
The vocab seems to have been created by a mix of chosen templates and words from GenericsKB. Hard to say that it is significantly complex -- give GKB sentences are quite simple and there are limited templates authors curated.
7. L333: This may not be that simple. One may be able to prove the negation in a smaller set of steps. Is that considered as well? Or, the dataset does not have such examples?
8. L350-353: Very interesting paragraph. Again how do authors ensure the factual incorrectness of the conclusions? 
9. L442:Why is the performance on CLUTTRR so low compared to JustLogic?
10. L514: For the apparently long tail argument forms, have you qualitatively evaluated the CoT or some other ways to actually see what reasoning is being done by the LLMs?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731817136361

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to Reviewers

comment:
## Summary

Thank you to all reviewers for your thoughtful reviews and feedback! We appreciate your recognition of the significant shortcomings in existing deductive reasoning benchmarks and your positive assessment of how JustLogic solves them through its novel dataset construction method.

We respond to individual reviews below. This general response summarizes the main changes to the paper. All changes are colored blue.

## Highlighting JustLogic’s contributions

We further clarified JustLogic’s contributions to the literature by refining the introduction (L66-71) and conclusion (L530-538).

Logical reasoning is one of the key challenges for LLMs. Despite great advances made in recent years, it remains controversial whether deductive reasoning abilities have genuinely progressed. This is due to the lack of a reliable benchmark that accurately and comprehensively evaluates deductive reasoning. JustLogic timely addresses this critical research gap.

**Importance of a reliable deductive reasoning benchmark**: Existing benchmarks used to evaluate LLMs’ reasoning abilities obfuscate their true deductive reasoning capabilities. This is because they simultaneously evaluate many types of reasoning, e.g. inductive, deductive, and analogical, and involve many types of knowledge, e.g. commonsense, math, and scientific knowledge (Section 2.1). 

In response, benchmarks that solely test for deductive reasoning have attracted considerable interest. However, their reliability is questionable due to several fundamental issues, i.e. prior knowledge dependence, lack of complexity, and lack of error analysis (Table 1).

**JustLogic** addresses the limitations of existing benchmarks. Specifically, we ensure…
- Prior knowledge independence by generating arguments that do not align with real-world knowledge. We prove this via our prior knowledge independence test.
- Argument complexity by synthetically generating argument structures
- Natural language complexity by leveraging GenericsKB sentences and curated templates
- Error analysis capabilities due to JustLogic’s high degree of flexibility, e.g. ability to adjust reasoning depth.

## Change of Terminology

The term “context-independent” has been changed to “prior knowledge independence” for better clarity. The term "context" now exclusively refers to the paragraph of premises provided by each JustLogic question.

## New Experiment: Impact of Factual Accuracy on Model Performance (Appendix C)

We conducted a new empirical study to investigate whether the factual accuracy of conclusions affects model performance. This is highly relevant to comments by reviewers NSeq, 3iPx, and LXqP.

The results show that there is no consistent relationship between factual accuracy of conclusions and model performance. The evidence does not support the idea that inaccurate conclusions confuse models and factually accurate conclusions give models an advantage.

This is because (i) our prompts explicitly ask models to avoid prior knowledge, and (ii) how LLMs treat factual accuracy when reasoning deductively depends on the LLM’s training. Further, we posit that the ability to perform deductive reasoning while ignoring prior knowledge is a fundamental human skill, and models should be evaluated in this regard.

Therefore, we conclude that the inclusion of both factually accurate and inaccurate conclusions is a justified way of ensuring prior knowledge independence. Further details on the experiment and subsequent analysis can be found in Appendix C.

## New Analysis: Sample Texts from Deductive Reasoning Benchmarks (Appendix D)

We recognize that the metrics for natural language complexity, i.e. vocabulary size and number of domains, do not fully capture the range of linguistic patterns in datasets. Thus, we show sample texts from CLUTRR, ProofWriter, ProntoQA-OOD, LogiQA 2.0, FOLIO, and JustLogic. This is relevant to reviewers NSeq, 3iPx, and LXqP.

By examining the texts, we highlight JustLogic’s superior or comparable natural language complexity to other reasoning benchmarks.

## New Section: Future Works (Appendix E)

The reviewers provide insightful comments on how JustLogic can be further developed. While these are beyond the scope of this work, we intend to explore these ideas further in the future. We therefore outline the following in the Future Works section:
1. Further enhance JustLogic’s natural language using LLM-generated text, as suggested by reviewer 3iPx.
2. More comprehensive error analysis on how and where exactly LLMs fail, as suggested by reviewers NSeq and LXqP.
3. Scaling JustLogic to incorporate more logical reasoning-related question types.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732532937481

thread_type:
official_comment

<thread_object>
creator: area_chair_cbf3

<content>
title:
Action Required: Respond to Author Rebuttals - Nov 27

comment:
Dear ICLR Reviewers,

The author discussion phase is ending soon. Please promptly review and respond to author rebuttals for your assigned papers. Your engagement is critical for the decision-making process.

Deadlines:

November 26: Last day for reviewers to ask questions to authors.
November 27: Last day for authors to respond to reviewers.
November 28 - December 10: Reviewer and area chair discussion phase.
Thank you for your timely attention to this matter.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces JustLogic, a synthetically generated benchmark designed to test LLMs' deductive reasoning capabilities while eliminating the influence of prior knowledge. The authors evaluate various LLMs including OpenAI o1-preview on JustLogic, achieving 81% accuracy compared to a human average of 73%.

The reviewers value the paper's motivation to address the flaws in current reasoning benchmarks and its comprehensive evaluation with multiple LLM models. Key concerns discussed include: the decision to use factually inaccurate statements (though authors showed empirically this does not affect model performance), limited investigation of failure modes in the error analysis, and questions about whether vocabulary diversity truly constitutes meaningful linguistic complexity compared to existing synthetic benchmarks such as SimpleLogic.

The authors have engaged constructively with reviewer feedback, providing additional empirical analysis on factual accuracy effects and clarifying their benchmark design choices. While some concerns about error analysis depth and linguistic complexity remain to be addressed in future work, I give a rejection as the paper's contribution to deductive reasoning evaluation, while valuable, needs further development in terms of thoughtful error analysis and linguistic complexity before meeting ICLR's standards.