LogicPro: Logical Reasoning Enhanced with Program Examples
Authors: Jin Jiang, Yuchen Yan, Liu Yang, Yonggang Jin, Shuai Peng, Zhi Tang, Xunliang Cai, Yixin Cao, Mengdi Zhang, Liangcai Gao


Reviewer paper summaries: This paper proposes to enhance LLM's logical reasoning ability in a data-centric sense, by constructing a large-scale dataset LogicPro. LogicPro is constructed from code problems such as LeetCode, which is composed of five steps - build test sample inputs, logical reasoning problems, test sample code, rewrite code, and construct the final answer. The authors test LogicPro by training with different SFT sources for LLMs and testing them on various benchmarks. Experiment results verified the effectiveness of LogicPro.
Summary of strengths: - The idea of leveraging program problems to construct logical reasoning data samples is interesting.
- The experiments and analyses are solid and comprehensive.
- The constructed dataset, LogicPro is large-scale and effective.
Summary of weaknesses: - The paper is not well-motivated. From the introduction, I could not understand why improving LLM's complex reasoning ability has to do with constructing logical reasoning synthetic data. Also, I cannot understand the three points mentioned in lines 124-125. It might be better if the authors could discuss how the three points are proposed and how the previous benchmarks are insufficient w.r.t. these aspects.
- Some of the design choices in Section 2 are not well explained. For example, why 30 in line 179? If an LLM is prompted with the same query to generate 30 times, I don't think the diversity would be good. Some qualitative analysis here would be helpful.
- I don't think the majority of program problems have explicit relations with logical reasoning. For example in Figure 3, I think this is a complex reasoning problem, but not necessarily a *strict* logical reasoning problem that could be categorized into inductive, deductive, or abductive.
- Finally, since the paper is submitted to the Resources and Evaluation track, I think a human evaluation of the data quality is important.
Comments, suggestions and typos: - The writing of this paper needs significant revision. The authors should avoid sentences that are too long and cause confusion, e.g., lines 126-132. Also, the citation format should be corrected, e.g., line 128.
Overall assessment scores: 2.5