Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpus

Authors: terufumi  morishita
gaku  morio
atsuki  yamaguchi
yasuhiro  sogawa
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-06 07:00:01

Abstract: Large language models (LLMs) are capable of solving a wide range of tasks, yet they have struggled with reasoning.
To address this, we propose $\textbf{Additional Logic Training (ALT)}$, which aims to enhance LLMs' reasoning capabilities by program-generated logical reasoning samples.
We first establish principles for designing high-quality samples by integrating symbolic logic theory and previous empirical insights.
Then, based on these principles, we construct a synthetic corpus named $\textbf{Formal} \ \textbf{Logic} \ \textbf{\textit{D}eduction} \ \textbf{\textit{D}iverse}$ (FLD$^{\times2}$), comprising numerous samples of multi-step deduction with unknown facts, diverse reasoning rules, diverse linguistic expressions, and challenging distractors.
Finally, we empirically show that ALT on FLD$^{\times2}$ substantially enhances the reasoning capabilities of state-of-the-art LLMs, including LLaMA-3.1-70B.
Improvements include gains of up to 30 points on logical reasoning benchmarks, up to 10 points on math and coding benchmarks, and 5 points on the benchmark suite BBH.

Main paper: 
 introduction: 
 Knowledge and reasoning have long been considered essential elements for achieving artificial intelligence (McCarthy, 1959; Weizenbaum, 1966; Winograd, 1971; Colmerauer and Roussel, 1973; Shortliffe, 1976; Elkan and Greiner, 1993) . Knowledge refers to facts about the world, e.g., "objects with mass generate a gravitational field" and "the Earth has mass." Reasoning involves combining multiple facts according to specific rules to obtain new knowledge. For example, the new knowledge that "the Earth generates a gravitational field" can be derived from the aforementioned two facts.Recent observations suggest that LLMs can solve problems using memorized knowledge of similar samples seen during pre-training, but they cannot solve novel, unknown problems that require reasoning (Hodel and West, 2023; Dasgupta et al., 2023; Zhang et al., 2024) . For instance, LLMs can solve famous arithmetic problems as is but not when the numbers are changed (Razeghi et al., 2022) , and they can solve coding tests from past years before the "knowledge cutoff" but not from the present year (Mitchell, 2023) . This bias towards knowledge has been observed even in state-of-the-art LLMs such as GPT-4 (Liu et al., 2023b; Wu et al., 2023; Dziri et al., 2023) .LLMs' poor reasoning capabilities can stem from the lack of high-quality reasoning samples in the pre-training corpus, which primarily consists of human-written texts (Betz et al., 2021; Morishita et al., 2023) . Indeed, reasoning samples in human-written texts often exhibit low quality, as evidenced by fallacies and biases commonly found in online debates (Hansson, 2004; Guiaşu and Tindale, 2018; Cheng et al., 2017) . This is unsurprising given that humans usually think reflexively rather than through rigid reasoning (Kahneman, 2011; Sunstein and Hastie, 2015; Paglieri, 2017) . Thus, a Each benchmark set, such as "Logic" and "Math", comprises various benchmarks in that domain. Tables 2, 4 shows the details.straightforward strategy to improve LLMs' reasoning capabilities is to prepare many high-quality reasoning samples and train LLMs on them.We propose one such approach, Additional Logic Training (ALT), which utilizes high-quality samples of logical reasoning, the most fundamental form of reasoning. To prepare such samples, we utilize synthetic generation (Clark et al., 2021; Betz et al., 2021; Tafjord et al., 2021; Morishita et al., 2023) , where computer programs generate deductive reasoning samples in which a given hypothesis is proven or disproven by combining given facts following rigid reasoning rules. We overview ALT in Figure 2 .In synthetic generation, computer programs generate samples according to pre-designed patterns, so this design largely determines the quality of these samples by nature. Thus, we start by discussing what is the ideal design for synthetic logic samples, incorporating symbolic logic theory and empirical findings (Section 2). The essence of logical reasoning lies in its ability to handle unknown facts, unlike knowledge, which deals solely with established facts, such as commonsense facts; therefore, samples must cover reasoning with unknown facts. Samples must include both illogical and logical reasoning to enable LLMs to distinguish between them. The samples must cover various patterns regarding a comprehensive set of reasoning aspects, such as reasoning rules and linguistic expressions of logical statements. We summarize these discussions into design principles, which guide the design of synthetic logic samples. Finally, based on these principles, we construct a synthetic corpus named Formal Logic Deduction Diverse (FLD ×2 ), comprising numerous samples of multi-step deduction with unknown facts, diverse reasoning rules, diverse linguistic expressions, and challenging distractors (Section 3).We then empirically verify that ALT can enhance LLMs' reasoning capabilities (Sections 4, 5) . Using 31 benchmarks covering diverse tasks, we observed that ALT on FLD ×2 substantially boosts state-of-the-art LLMs' reasoning capabilities. Even LLaMA-3.1-70B, the largest LLM pre-trained on over 15 trillion tokens, shows substantial improvements with ALT (Figure 1 ). Among synthetic logic corpora with different sample designs, FLD ×2 yielded the largest performance gains, validating our proposed design principles. Moreover, we discovered that employing a knowledge-forgetting prevention method during ALT is critically important, as it likely prevents the LLM's knowledge of established facts from being displaced by the unknown facts included in synthetic logic corpora.Finally, we analyze which task-solving capabilities ALT can enhance and why (Section 6). We observed a substantial improvement of up to 30 points on logical reasoning tasks (Table 4a ). Surprisingly, we also observed improvements in abductive reasoning tasks, which go beyond the synthetic logic corpora's original deductive reasoning tasks. Case analyses indicate that these improvements result from LLMs having acquired the fundamentals of the logic reflected in the design principles. We also observed improvements of up to 10 points on math and coding tasks, indicating the generalizability of the obtained reasoning capabilities (Tables 4b, 4c ). We also observed improvements of up to 6 points on natural language inference (NLI) tasks (Table 4d ). Case analyses suggest that LLMs successfully integrated the commonsense knowledge they had originally acquired during pre-training with the logical reasoning capabilities newly acquired from ALT. Improvements across various other tasks (Table 4e ) demonstrate the broad benefits of the obtained reasoning capabilities beyond standard reasoning tasks, though the modest improvements of up to 2 points indicate the need for future research on more effective application of these capabilities. Our contributions are summarized as follows:• We propose Additional Logic Training (ALT) and empirically verify that it can enhance the reasoning capability of state-of-the-art LLMs across various sizes, from 7B to 70B.• We establish systematic design principles for synthetic logic samples; then, we construct a synthetic corpus named Formal Logic Deduction Diverse (FLD ×2 ), comprising numerous samples of multi-step deduction with unknown facts, diverse reasoning rules, diverse linguistic expressions, and challenging distractors. We empirically verify that Formal Logic Deduction Diverseindeed leads to the largest improvements among corpora with different sample designs.• We demonstrate that LLMs enhanced by ALT can solve not only the original logical reasoning tasks present in synthetic logic corpora but also other tasks, such as math and coding tasks, and notably NLI tasks, which require integrating knowledge and reasoning. This finding underscores the potential for advancing truly versatile AI possessing both knowledge and reasoning capabilities. We release the corpus, code, and the trained model under a permissive license1 .2 How Should Synthetic Logic Samples Be Designed?In synthetic generation, computer programs generate samples according to pre-designed patterns, so this design largely determines the quality of the samples. While Previous studies have examined several designs (Clark et al., 2021; Betz et al., 2021; Tafjord et al., 2021; Morishita et al., 2023) , these designs were not systematically discussed, so they may not be the most effective ones. Thus, we start by discussing how to optimally design synthetic logic samples. To this end, we consider symbolic logic theory as suggested by Morishita et al. (2023) and integrate empirical findings from previous studies. First, we observe that the essence of logical reasoning, based solely on the logical relationships between facts, lies in its ability to handle unknown facts, unlike knowledge, which by definition deals solely with established facts (Section 2.1). Therefore, we argue that samples should cover reasoning with unknown facts to represent this essential aspect of logical reasoning. We also observe that logical reasoning involves various other aspects, such as illogical reasoning, reasoning rules, and linguistic expressions that represent logical statements (sections 2.2 to 2.4). The samples should cover various patterns regarding these aspects to enable LLMs to solve various reasoning problems. We summarize these discussions into the following design principles, which guide the design of synthetic logic samples. 
teaching reasoning with unknown facts: 
 We first explore the essence of logical reasoning that differentiates itself from knowledge. Consider the following logical step:The Earth orbits the Sun.If the Earth orbits the sun, the Earth has four seasons. The Earth has four seasons.(1)This step is valid because the conclusion is logically derived from the two premises. Next, consider another logical step:The Earth orbits the Sun.If the Earth orbits the sun, the Earth does not have four seasons. The Earth does not have four seasons.(EQUATIONThe second premise and consequently, the conclusion, is factually wrong. Nevertheless, if the premise was hypothetically correct, the conclusion could be logically derived. Therefore, step ( 2) is also logically valid. Finally:1. A Foo star exists.2. If a Foo star exists, a Bar star also exists.A Bar star exists.(3) "Foo star" and "Bar star" are unknowns; nonetheless, we can still determine that step (3) is logically valid. Steps ( 1) to (3) above can be abstracted into a deduction rule, i.e., modus ponens, using symbols:EQUATIONAs we have seen, the logical validity of a deduction rule depends solely on whether the conclusion is logically derived from the premises, not on the factual correctness of the contents of F and G.Therefore, the contents of F and G can be arbitrary.we consider what kind of samples would be needed to teach the deduction rule (4) to LLMs. We assume a task to generate the conclusion given the premises as prompt inputs. If the learner were human, they would be able to infer the underlying deduction rule ( 4) by observing samples such as ( 1) to (2) . As a result, they would become able to solve the unknown problem (3).However, from a purely inductive perspective, samples (1) to (2) cannot simply be generalized to the deduction rule (4) . This is because the samples (1) to (2) themselves do not contain the information that the contents of F and G are arbitrary. In fact, one could generalize samples (1) to (2) to other rules; for example, the conclusion G can be derived if F and F → G are given as premises and F and G include 'Earth' as their contents. Innumerable such deduction rules can be inductively inferred from the given samples. In other words, induction has arbitrariness (Hume, 1748; Goodman, 1954; Quine, 1969) .Humans prefer simpler rules (Bertrand; Wittgenstein, 1922) , so they boldly induce up to the deduction rule (4) . However, it is unclear how purely inductive learners such as LLMs, which extract only what can be inferred from samples without prior preferences, induce up to (4). For example, if only specific contents such as "Alice is kind" and "Bob is smart" are assigned to F and G in training samples, an LLM could develop into a machine that generates the conclusion G only when the input contains the specific contents. In order for LLMs to accurately induce that F and G are indeed arbitrary:Design Principle 1 (Reasoning with Unknown Facts). Prepare many samples assigning arbitrary contents to F and G. They will make LLMs accurately induce F and G are indeed arbitrary, ultimately enabling them to reason with unknown facts.Suppose we have LLMs trained on a large number of samples as follows:F ∧ G (F ∧ G) → H H (5)where ∧ denotes logical conjunction, and arbitrary contents are assigned to F, G, H. Suppose that we give this LLM a problem such as:EQUATIONSince the premises are insufficient for logically deducting the conclusion, outputting nothing is the correct answer.Unfortunately, an LLM could output H, which was indeed often observed in our preliminary experiments. This is because while the LLMs can induce from sample (5) that it can generate the conclusion H when the two premises of (5) are given, the LLMs cannot induce from the sample that it is not allowed to generate the conclusion H when the premises of (6) are given, as such information is not included in the sample (5) itself. Therefore, Design Principle 2 (Illogical Reasoning). Include negative samples such as (6). These samples will make LLMs induce that conclusions cannot be derived from insufficient premises.Deduction rules other than (4) exist:(F∧G) F (F∧G) ∧elimination G (F → G)∧(G → H) syllogism F→ H F→ G contraposition ¬G→ ¬F ¬(F∨G) ¬F∧¬G ¬(F∧G) De Morgan's laws ¬F∨¬G (7)where ∨ denotes logical disjunction and ¬ negation. Since there are infinitely many possible logical formulas that can appear as premises and conclusions, there are infinitely many deduction rules. Providing LLMs with these infinite deduction rules is obviously intractable.Instead of directly providing these infinite deduction rules, we can take another approach. Consider multi-step deductive reasoning (Figure 2 left), where multiple deduction rules derive a conclusion. Notice that the syllogism in ( 7) can be expressed by multi-step deductive reasoning using more "atomic" deduction rules. Indeed, there exists a set of atomic deduction rules called the axioms that satisfies the following: Theorem 2.1 (Completeness of First-Order Predicate Logic Gödel (1930) ). Any valid deduction rule can be expressed by multistep deductive reasoning constructed from the axioms.In contrast to the axioms, the 'compound' deduction rules, such as syllogism, contraposition, and De Morgan's laws, are called theorems. According to the completeness Theorem 2.1, if we can handle the axioms, we can effectively handle other deduction rules as well. Indeed, Morishita et al. (2023) empirically verified that a language model trained on the axioms generalizes to handle other deduction rules more effectively than those trained on non-axiom deduction rules. Therefore, Design Principle 3 (Diverse Reasoning Rules). Samples should express multi-step deduction constructed from the axioms. They will effectively teach LLMs diverse deduction rules (Morishita et al., 2023) In multi-step deductive reasoning, the number of logical steps s from premises to a conclusion can vary largely depending on the problem. Therefore:Design Principle 3' (Diverse Reasoning Rules). Samples should include diverse numbers of logical steps s.Ideally, this would be sufficient, but empirical evidence has shown that LLMs struggle with constructing multi-step deductive reasoning with large steps s (Gontier et al., 2020; Morishita et al., 2023) . Consequently, LLMs would not excel at handling theorems that require a large number of steps s when expressed by the axioms. Therefore, as an additional countermeasure:Design Principle 3" (Diverse Reasoning Rules). Samples should also include representative theorems, such as syllogism, contraposition, and De Morgan's laws.There are various linguistic structures for expressing the logical relationship F → G, such as "If F then G", "F leads to G", and "F results in G". If we only include specific expressions in the corpora, LLMs may only learn to react to these specific expressions, which has been observed in previous experiments (Zhang et al., 2022; Yuan et al., 2023) . To prevent this, Design Principle 4 (Diverse Linguistic Expressions). Samples should include diverse linguistic expressions that represent logical statements.In this chapter, we have established the principles to guide the design of synthetic logic samples. Next, we construct a synthetic logic corpus based on these principles. FLD ×2 ablation corpora → 100 → w/o DP1 not used → w/o DP2 2 (implication) → w/o DP3.rules 1 → w/o DP3.steps 1 → w/o DP4 
creating a synthetic corpus based on design principles: 
 To prepare diverse samples reflecting the design principles 1 to 4 (DP1-4), we built a novel sample generator by extending the previous one by Morishita et al. (2023) and then generated the synthetic logic corpus named FLD ×2 (Formal Logic Deduction Diverse). Figure 2 shows a schematic of our generator and a deduction sample. Table 1 compares FLD ×2 with existing corpora. More specifically, our generator generates deduction samples through the following steps. First, the generator randomly generates a sample of multi-step deductive reasoning written in logical formulas, as shown on the left side of Figure 2 , where a conclusion is derived from premises using multiple deduction rules (See Appendix D.3 for more details of this generation procedure). At this time, the generator also generates 'distractor' logical formulas, which express negative premises of DP2. Next, the generator converts each logical formula into English expressions. To achieve this, the generator first randomly selects a template from pre-defined options, such as "If F, then G," "F leads to G," or "F results in G," for the logical formula "F → G." It then assigns English content randomly constructed from a vocabulary, such as "(that) a Foo star exists" and "(that) a Bar star exists," to each symbol, such as F and G. Finally, it converts the multi-step deduction into a deduction sample (right side of Figure 2 ) by using the premises as 'facts', the conclusion as 'hypothesis', and the intermediate logical steps as 'logical steps'. The deduction sample requires LLMs to generate logical steps that derive a given hypothesis based on the given facts.Table 1 outlines the comparison of FLD ×2 with other existing corpora (Clark et al., 2021; Bao et al., 2022; Morishita et al., 2023) in terms of DP1-4, which is detailed as follows:• DP1: We assign F and G content randomly constructed from a vocabulary. While the existing corpora used small-sized vocabulary of up to 15k, we use a large vocabulary of around 100k words built from WordNet (Miller, 1995) . This will teach LLMs that F and G are truly arbitrary, ultimately enabling them to reason with unknown facts. • DP2: The existing corpora used randomly generated logical formulas as distractors. In contrast, we implement adversarial distractors. For example, for a premise F ∧ G, we use F with missing information (see Equations ( 5), ( 6)), and for a premise F → H, we use F ∧ G → H with missing information as distractors. These distractors teach LLMs precisely when a conclusion can and cannot be derived. As with previous corpora, we include a variable number of distractors in each sample, randomly chosen from a range of 0 to 20. • DP3-3": While the existing corpora used a small number of deduction rules of up to 13 (refer to 2023)), we include diverse deduction rules, encompassing the axioms and representative theorems, such as modus ponens, syllogisms, and contraposition, totaling about 50 rules. We include samples with up to s = 8 logical steps, following (Morishita et al., 2023) . • DP4: We manually craft several more English templates per logical formulas than those used in FLD. Since the templates have a nested structure, they yield combinatorially more diverse English expressions. While counting the exact number of the resulting expressions is intractable, we observed at least dozens of expressions per logical formula, including minor variations. See Appendix D.4 for details. 
experimental setup: 
 We briefly explain the experimental settings. Refer to Appendix E for the details. Synthetic Logic Corpora: We examine the proposed FLD ×2 and previous corpora (Table 1 ).LLMs: We used the state-of-the-art LLM, LLaMA-3.1 (8B and 70B) (AI@Meta, 2024).Training Settings: We trained the LLMs by a method similar to supervised fine-tuning; as illustrated in Figure 2 , we used the facts and hypothesis as inputs and logical steps and additional answer label (see Appendix D.1) as outputs. We excluded loss computation for the inputs to prevent LLMs from learning to generate unknown facts. We trained the LLMs for 1 epoch on 100k samples (∼ 0.1B tokens) from the training split of each corpus, with a batch size of 256, resulting in 390 steps, with a linear warmup for 200 steps. We used the learning rate of 2e-05 for the 8B model and 3e-06 for the 70B model. We used Huggingface (Wolf et al., 2020) for implementation.Prevention of Knowledge Forgetting by Recall Adam Optimizer: Synthetic logic corpora include many samples with unknown facts, so training on them should cause LLMs to forget their knowledge of existing facts. To prevent this, we employed the Recall Adam optimizer (Chen et al., 2020) , which regularizes parameter updates to avoid deviating too far from the pre-training parameters.Recall Adam stands out for LLM training for several reasons (see Appendix E.0.1 for details). We used our re-implemented version2 . The hyperparameters were: β 1 = 0.9, β 2 = 0.999, ϵ = 10 -6 , fisher coefficient = 4000 for the 8B model and 2000 for the 70B model.We evaluated the trained LLMs on 31 benchmarks shown in Table E .7 using 5-shot in-context learning, except for BBH and AbuductionRules, which used 3-shot in-context learning. These benchmarks cover a wide range of tasks and are prominent in LLM evaluation. Note that we excluded the synthetic logic corpora used for training, as training on them often leads to overfitting to their superficial and statistical cues (Zhang et al., 2022; Yuan et al., 2023) , failing to measure truly generalizable reasoning capabilities. We used lm-evaluation-harness (Gao et al., 2023) and bigcode-evaluation-harness (Ben Allal et al., 2022) for the implementation. 
can additional logic training enhance llms' capabilities?: 
 Table 2 show the performance of LLMs before and after ALT. Most LLMs trained with ALT outperformed their counterparts without ALT. Notably, ALT yielded substantial gains of up to 10 points even for LLaMA-3.1-70B, the largest LLM pre-trained on over 15 trillion tokens. These results verify that ALT can enhance the capabilities of state-of-the-art LLMs.Among the LLMs trained with ALT, the one trained on FLD ×2 (i.e., ⊕ALT-FLD ×2 ) achieved the highest generalization performance across the benchmarks. Table 3 shows the performance of the LLMs trained on ablated FLD ×2 corpora, each of which lacks one of the design principles. As seen, ablating any design principle almost always led to performance degradation. These results demonstrate that the proposed design principles are critical to obtaining the maximum possible gain from ALT, and each principle is indispensable.Table F .8 shows that the LLMs trained with ALT without preventing knowledge forgetting by Recall Adam optimizer underperformed compared to their counterparts trained with knowledge forgetting prevention and even the LLM without ALT. This behavior presumably occurred because the unknown facts included in synthetic logic corpora displaced the LLM's knowledge of existing facts. Therefore, knowledge-forgetting prevention is critically important for the success of ALT.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: we release the code, data, and model.principles:• The answer NA means that paper does not include experiments requiring code. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
what capabilities can additional logic training enhance and why?: 
 We analyze the results on each benchmark or each case and discuss whether and why the LLM's capabilities to solve the tasks can or cannot be enhanced by ALT.Table 4a shows that ALT substantially boosted LLaMA-3.1-70B's performance by up to 30 points on various benchmarks dealing with logical reasoning tasks. Surprisingly, we also observed improvements on abductive reasoning tasks, which go beyond the original deductive reasoning tasks in synthetic logic corpora. Abductive reasoning involves guessing the missing premises that caused the observed conclusion rather than deriving a conclusion from the premises. For example, from the observed conclusion, "the window glass at home was broken and the room was ransacked," we guess the premise "a burglar broke in." The improvements would be due to the fact that, while the surface form of abductive reasoning problems differs from that of deductive reasoning, they share the fundamentals of logic reflected in the design principles.Next, we conduct case analyses to see whether the LLM enhanced by ALT acquired the abilities intended by the proposed design principles (DP1-4). Table 5 shows problems where LLaMA-3.1-70B's errors have been corrected by ALT. The first problem is very simple, so it is surprising that LLaMA-3.1-70B failed to solve it, indicating the inherent difficulty of learning logical reasoning solely from pre-training. In contrast, ⊕ALT-FLD ×2 , which was additionally trained on FLD ×2 , solved the problem correctly. The premises of the problem are randomly constructed to express unknown facts. Therefore, the result suggests that ⊕ALT-FLD ×2 acquired genuine logical reasoning ability, which can handle unknown facts (DP1).In the second problem, ⊕ALT-FLD ×2 correctly answered "neutral", indicating that it successfully learned that conclusions cannot be derived from insufficient facts (DP2).The third problem comes from the FOLIO benchmark. To solve this problem, LLMs must use syllogism at the first step as follows: "All eels are fish, and no fish are plants. Therefore, all ells are not plants." ⊕ALT-FLD ×2 answered this problem correctly, suggesting that it successfully learned diverse deduction rules (DP3).FOLIO problems are created based on Wikipedia topics, describing them in more natural and realistic linguistic expressions than in other benchmarks. As seen in the fourth problem, ⊕ALT-FLD ×2 understands such expressions, suggesting the effect of diverse expressions from DP4 and/or that LLMs can integrate their original linguistic ability with the newly acquired logical reasoning ability. Tables 4b, 4c shows that ALT substantially boosted the LLaMA-3.1-70B's performance by up to 7 and 10 points on math and coding tasks, respectively. The math improvements are reasonable, as understanding predicate logic is a prerequisite for solving mathematical problems. For coding, some recent studies have verified the opposite direction, namely, that training on coding data improves logical reasoning abilities (Jiang et al., 2024b; MA et al., 2024; Uchiyama et al., 2024) .Table 4d shows that ALT substantially boosted the LLaMA-3.1-70B's performance by up to 6 points on various natural language inference (NLI) benchmarks. NLI is similar to deductive reasoning in assessing whether a premise supports or contradicts a hypothesis. However, the main difference is that this judgment requires a rich set of commonsense knowledge beyond the given premise.Consider the fifth problem in Table 5 : by supplementing the given fact "An Indian woman is dancing with her partner" with the commonsense knowledge "If someone is dancing, then he/she is moving.", we can derive the hypothesis "A woman is moving." The sixth problem is more challenging as we have to trace multiple logical steps while supplementing with sufficient commonsense knowledge as follows: "a church choir sings at a church," "baseball is often played at a baseball field," "a person cannot be in two or more places at the same time," "therefore, a church choir cannot sing for baseball."Since synthetic logic corpora only contain unknown facts, LLMs cannot acquire new knowledge from them. Therefore, the commonsense knowledge used to solve the above problems must have been acquired by the LLMs from pre-training. This suggests that LLMs can integrate their original knowledge with the logical reasoning capabilities newly acquired from ALT to solve problems. 4e ) demonstrate the broad benefits of the obtained reasoning capabilities beyond standard reasoning tasks; though the improvements were modest at up to 2 percentage points, which may be due to the following reasons. First, these benchmarks include problems that purely test knowledge, such as the first one in Table 6 . Since ALT does not aim to provide new knowledge, the ability to solve such problems does not improve by nature. Next, some problems may require knowledge that is too advanced for LLMs, so potential improvements by the enhanced reasoning capabilities may be bottlenecked. For example, the second problem does involve reasoning but requires sufficient quantum mechanics knowledge as a prerequisite. However, these knowledge-related issues should be solved by improving the quantity and quality of pre-training.Finally, LLMs may not be able to fully utilize the potential of enhanced reasoning capabilities for problems that require complex procedures. To solve the third problem, LLMs first must attempt reasoning related to each choice as follows: "To build homes in an aquatic environment, one needs to maintain body heat and insulation despite being frequently submerged in cold water. Therefore, the waterproof fur of (A) is essential", and "To build . . . , one must gather and process natural materials like wood. Large, sharp teeth of (C) are critical as they allow beavers to cut down trees and shape branches." Next, while reasoning traces on (A) to (D) all seem reasonable, LLMs must choose the single best answer, considering the subtle nuance of the question context, as follows: "Since the question emphasizes the aquatic environment, the least related reasoning trace should be (C)." This complex procedure contrasts with logical reasoning and NLI problems, where LLMs can directly obtain an answer from a single reasoning trace. Previous studies also observed that such procedure on multiple-choice QA problems are challenging for LLMs (Robinson and Wingate, 2023; Zheng et al., 2024; Wang et al., 2024a) . Since ALT alone does not teach LLMs such task-specific procedures, additional training on these procedures should be necessary to solve these problems.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Section 4, appendix E.principles:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 Towards versatile artificial intelligence with reasoning capabilities, we proposed Additional Logic Training on synthetic logic samples. We established systematic design principles well-grounded on symbolic logic theory and previous empirical findings. We constructed a corpus named Formal Logic Deduction Diverse (FLD ×2 ) based on the design principles. We empirically showed that ALT on FLD ×2 substantially enhances the capabilities of state-of-the-art LLMs.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: As stated in Appendix E.principles:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [No] Justification:principles:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage principles or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: All the corpora and benchmarks used in the experiments properly state their licenses. principles:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We will release our corpus. principles:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? 
 
 Appendix: Many studies examine LLMs' reasoning capabilities (Askell, 2020; Rae et al., 2021; Razeghi et al., 2022; Liu et al., 2023b; Turpin et al., 2023; Lanham et al., 2023; Wu et al., 2023; Hodel and West, 2023; Dziri et al., 2023; Dasgupta et al., 2023) . Patel et al. (2024) observed LLMs' performance significantly declines as reasoning steps increase in multi-step logical reasoning tasks. Dougrez-Lewis et al. ( 2024) revealed ChatGPT struggles with abductive reasoning when verifying claims by decomposing their evidence into atomic reasoning steps. Wang et al. (2024b) found that GPT-series models showed significant gaps compared to humans in dealing with inference rules. Parmar et al. (2024) introduced LogicBench and showed that existing LLMs struggle with instances involving complex reasoning and negations. Wan et al. (2024) introduced LogicAsker, which assesses whether LLMs can employ a set of atomic reasoning skills grounded in propositional and predicate logic and found significant gaps in LLMs' learning of logical rules. Bhuiya et al. (2024) proposed a challenging multi-hop reasoning benchmark with seemingly plausible but incorrect multi-hop reasoning chains and found that state-of-the-art LLMs' capabilities to perform multi-hop reasoning is affected by such chains. Mondorf and Plank (2024) introduced TruthQuest, which assesses LLMs' capabilities to conduct suppositional reasoning, i.e., reasoning where each statement can be false, and found that LLMs exhibit significant difficulties solving these tasks. Sprague et al. ( 2024) introduced a complex multi-step reasoning benchmark, MuSR, and characterized the gaps that remain for techniques like chain-of-thought to perform robust reasoning. (2024) found that LLMs exhibit human-like reasoning biases in syllogistic arguments. Jiang et al. (2024a) found that LLMs exibit "token-biases" in solving logical reasoning problems. Aoki et al. (2024) revealed that LMs rely heavily on heuristics, such as lexical overlap, in the earlier stages of reasoning. Zhao et al. (2024a) constructed a MATHTRAP with carefully designed logical traps into the problem descriptions of MATH and GSM8k and found that while LLMs possess the knowledge required to solve these traps, they do not spontaneously use such knowledge them to handle the problems. Han et al. (2024) found that LLMs exhibit A-Not-B errors similar to human infants, failing to suppress the previously established response pattern during ICL. Liu et al. (2024) found that LLMs often contradict themselves in reasoning tasks involving contextual information understanding or commonsense. Zhou et al. (2024b) found that subtle alterations in the surface form can significantly impact the answer distribution, suggesting that LLMs solve reasoning problems using surface cues. Chen et al. (2024) found that the reasoning performance of LLMs is affected by the order of the premises. Hong et al. (2024) ; Huang et al. (2024) found that LLMs struggle to identify fallacious reasoning steps accurately, suggesting challenges in self-verification methods. Zhao et al. (2024b) found that LLMs struggle with reasoning in uncommon situations. Zhu et al. (2024) introduced a framework to dynamically generate reasoning samples, and LLMs perform worse in those samples. Hu et al. (2024) found that while LLMs can conduct reasoning when relevant knowledge is given in context, they are not proficient at reasoning with knowledge embedded in the training data.Later studies (Saha et al., 2020; Dalvi et al., 2021; Tafjord et al., 2021; Sanyal et al., 2022b) showed that T5 can generate even the intermediate logical steps as well as the final answer. PARARULE-Plus (Bao et al., 2022) is the enhanced version of PARARULE (Clark et al., 2021) , a variation of RuleTaker, that includes more samples and more logical steps. RoBERTa (Liu et al., 2019) trained on PARARULE-Plus outperformed the models trained on RuleTaker.Artificial Argument Corpus (Betz et al., 2021) includes single-step deductive reasoning samples constructed from hand-selected deduction rules useful for critical thinking. They showed that the GPT-2 (Radford et al., 2019) trained on this corpus can generalize to solve NLI tasks. However, at the same time, they found that the LM does not generalize well to solve more challenging reasoning tasks such as ARC (Habernal et al., 2018) and LogiQA (Liu et al., 2020) .FLD by Morishita et al. (2023 Morishita et al. ( , 2024) ) is the first synthetic logic corpus based on formal logic theory. It includes multistep deductive reasoning samples constructed from the axioms of first-order predicate logic, which can express any deduction rule due to the completeness theorem. Due to this nature, T5 trained on FLD generalizes most effectively to other synthetic logic corpora, compared to models trained on other corpora. Gontier et al. (2020) investigated the deductive reasoning capabilities of LMs on a corpus composed of a specific type of multistep inference, kinship relationships on synthetic kinship graphs. They found that LMs can solve this task when there are relatively few proof steps, but it is difficult for them to generalize to solve proof steps longer than those shown in training data. Bostrom et al. (2021) studied how to create realistic natural language expressions that represent deduction rules. To this end, they scraped sentences from Wikipedia using a template-based method and paraphrased them. They showed that training on this corpus helps solve real-world deductive reasoning problems such as EntailmentBank (Dalvi et al., 2021) . Pi et al. (2022) used synthetic data from program executors, most notably SQL programs. They verified that this data can enhance numerical reasoning, logical reasoning, and multi-hop reasoning abilities. Trinh et al. ( 2024) generated 100 million geometry problems and verified that the capability of artificial intelligence can be enhanced to to pass the bronze medal threshold of the International Mathematics Olympiad. Saeed et al. (2021) ; Nafar et al. ( 2024) created soft reasoning rules involving with probabilistic logic, instead of hard-logic examined by the aformentioned studies. Sileo (2024) introduced a simpler and more general declarative framework for synthetic generation, and verified its effectiveness. Zhou et al. (2024a) synthetically generated a large dataset of mathematics, and gained over 12 points on GSM8k.While these studies partly examined the effect of synthetic logic corpora, whether this approach is promising remains an open question. It has been unexplored whether the capabilities obtained from synthetic logic corpora generalizes to solve various tasks beyond the original tasks in these corpora. Additionally, the effect of these corpora has only been examined for small LMs trained on small pre-training corpora such as T5 and RoBERTa; it has been highly questionable whether they can still benefit state-of-the-art LLMs trained on a huge pre-training corpus. Furthermore, even if their benefits were verified, it remains unclear which design of synthetic logic samples yields the largest benefits due to the lack of systematic discussions on sample designs and empirical verification of these designs. We aimed to answer these questions in this paper and demonstrate the potential of synthetic logic corpora.Recent approaches (Ho et al., 2023; Magister et al., 2023; Li et al., 2022 Li et al., , 2023;; Shridhar et al., 2023; Wang et al., 2023; Mitra et al., 2023; Liu et al., 2023c; Ben Allal et al., 2024; Lu et al., 2024) utilize very large LLMs, such as GPT-4, to prepare synthetic reasoning datasets to train smaller LLMs. A typical procedure is as follows: (i) prepare existing reasoning problems, (ii) prompt large LLMs to generate reasoning traces to solve these problems using techniques such as chain-of-thought prompting (Wei et al., 2022) , and (iii) train smaller LLMs on these reasoning traces.The distillation approach and the synthetic logic corpora approach examined in this paper have specific advantages and disadvantages, as follows.The advantage of the distillation approach is its immediate practical effect, as it directly teaches LLMs solutions to various existing problems. The disadvantages could be that (i) it is non-trivial for specific solutions to specific problems to generalize to other problems, (ii) the number of training samples is limited to existing problems in nature, (iii) the correctness and faithfulness of the reasoning traces are not guaranteed; indeed, some studies (Turpin et al., 2023; Lanham et al., 2023) suggest that large LLMs do not always faithfully follow the "reasoning traces" they themselves generate, and (iv) it cannot enhance the very large LLMs themselves by nature.The advantages of synthetic logic corpus approaches are that (i) since they teach the fundamentals of reasoning, such as deductive reasoning, they have the potential to generalize to various problems, (ii) they can generate an unlimited number of new samples, and (iii) the correctness of the reasoning traces is guaranteed by nature. The disadvantage of this approach is that, as it only teaches the basics of reasoning, additional training may be needed to solve more complex real-world problems, as suggested in Section 6.4.We hypothesize that integrating both approaches could be promising. That is, we first train LLMs using ALT to make them understand the fundamentals of reasoning through high-quality samples and then train them using more realistic reasoning traces to solve complex real-world problems.• We only used deductive reasoning samples for ALT. Future work should examine other reasoning samples, e.g., abductive and inductive reasoning.• We only examined the first-order predicate logic system. Future work should examine other logic systems, such as modal and linear logic.The ultimate goal of the direction of this study is to develop an AI capable of reasoning logically step by step. If AI can make a decision one logical step at a time, it would be highly explainable and transparent to users. Furthermore, the user would be able to trace the AI's errors. We believe that our study is a step towards such AI that will positively impact society. In addition to the logical steps, the samples of FLD ×2 and previous corpora include answer labels (Figure D .3): "proved" indicating that the hypothesis can be proved by the logical steps, "disproved" indicating that the hypothesis can be disproved, and "unknown" indicating that the given facts are insufficient for either proving or disproving the hypothesis. For samples with "unknown" labels, the logical steps are "None.". FLD ×2 have a uniform distribution over the labels.FLD ×2 includes 100k/5k/5k samples for train/valid/test splits.Our sample generator first randomly generates examples of multistep deduction by forward-and backward random deduction, using the deduction rules specified by a user.The forward random deduction is done as follows. The generator first chooses a deduction rule randomly and forms the initial tree where the root node is the conclusion of the chosen deduction rules and the child nodes are the premises of the chosen deduction rule. The generator next randomly chooses another deduction rule that can be "jointed" to the root note of the tree. A deduction rule can be jointed to the root node of a tree if one of the premises of that deduction rule can be identified with the root node. Then, the generator updates the tree by jointing this chosen deduction rule. The generator continues this step multiple times until the tree achieves the required depth.The backward random deduction is done as follows. For each step, the generator randomly chooses a leaf node of the tree. Then, the generator randomly chooses a deduction rule that can be jointed to the leaf node. Here, a deduction rule can be jointed to the leaf node if the deduction rule's conclusion can be identified with the leaf node. Then, the generator updates the tree by jointing this chosen deduction rule. The generator continues this step multiple times until the complexity of branches achieves the required level. We prepared linguistic templates for each logical formula, exemplified as follows:⟨(A ∧ B) → C⟩ : If ⟨(A ∧ B).predicate_phrase⟩, then ⟨C.predicate_phrase⟩.: ⟨(A ∧ B).noun_pharse⟩ ⟨cause_synonyms⟩ ⟨C.noun_phrase⟩.: (. . . )⟨(A ∧ B).predicate_phrase⟩ : A ⟨occur_synonyms⟩ and also B ⟨occur_synonyms⟩.: A and also B ⟨occur_synonyms⟩.: Both A and B ⟨occur_synonyms⟩. As can be seen, the templates can be nested deeply, yielding combinatorially diverse linguistic expressions.Expanding these templates beforehand is intractable due to the combinatorial explosion, so we expand these templates on the fly to randomly sample a single expression at a time. Estimating the exact number of expressions is intractable for the same reason.We manually crafted several additional English templates per logical formula (i.e., the left-hand sides of (D.1)) compared to those used in FLD, which yield combinatorially more diverse English expressions. We observed that at least dozens of expressions, including minor variations, are yielded for each formula.E Details of Experimental Setup E.0.1 Prevention of Knowledge Forgetting by Recall Adam OptimizerWe employed the Recall Adam (RecAdam) optimizer (Chen et al., 2020) , which regularizes parameter updates to prevent them from being too far from the pre-training parameters. Recall Adam stands out for LLM training as it does not require access to the pre-training corpus, which is often inaccessible or too huge to handle, nor does it require changes to the model architecture, and it has a proven track record of usage in language models such as BERT.Table E .7 details the benchmarks used in the experiments.We show the average and standard deviations over five seeds.The entire experiment, including preliminary ones, took about 1 week x 128 NVIDIA H100 GPUs of our own.Table F .8 shows the results of LLMs trained without using Recall Adam.Table E .7: 31 benchmarks used in the experiments. These benchmarks cover a wide range of tasks and are prominent for LLM evaluation. We also show the form of reasoning and the type of knowledge required to solve the problems in each benchmark.Required knowledge Logic bAbi deduction (Weston et al., 2015) , deduction -(not required) FOLIO (Han et al., 2022) LogicNLI (Tian et al., 2021) RobustLR (Sanyal et al., 2022a) AR-LSAT (Zhong et al., 2021) commonsense LogiQA2 (Liu et al., 2023a) ReClor (Yu et al., 2020) AbductionRules (Young et al., 2022) abduction ART (Bhagavatula et al., 2019) commonsense NLI HELP (Yanaka et al., 2019) validate a conclusion based on given premises commonsense MultiNLI (Williams et al., 2018) RTE (Dagan et al., 2005; Giampiccolo et al., 2007; Bentivogli et al., 2009) SNLI (Bowman et al.) Math GSM8k (Cobbe et al., 2021) Math Math MATH (Hendrycks et al., 2021b) MathQA (Amini et al., 2019) Coding HumanEval (Chen et al., 2021) Coding MBPP (Austin et al., 2021) MBPP+ (Liu et al., 2023d) MultiPL-E (cpp/go) (Cassano et al., 2023) Others CommonsenseQA (Talmor et al., 2018) complicated procedures commonsense HellaSWAG (Zellers et al., 2019) SQuAD2 (Rajpurkar et al., 2018) WinoGrande (Sakaguchi et al., 2021) ARC (easy/challenge) (Clark et al., 2018) science GPQA (Rein et al., 2023) OpenBookQA (Mihaylov et al., 2018) SciQ (Welbl et al., 2017) aggregated MMLU (Hendrycks et al., 2021a) various various MMLU-Pro (Wang et al., 2024c) BBH (Suzgun et al., 2022) Table F.8: 5-shot performance of LLMs before and after ALT. ⊕ALT-x denotes the LLM trained with ALT on the synthetic logic corpus x from Table 1 . Color shows the rank in each column (darker is better). "Logic", "Math", "Code", and "Others" each comprises various benchmarks (see Table E principles:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.https://github.com/hitachi-nlp/FLDhttps://github.com/hitachi-nlp/rec-adamhttps://anonymous.4open.science/r/ALT/README.md 


Decision: This is an interesting work to enhance reasoning ability of LLMs using synthetic rule-based data. By training on the additional data with as continual pretraining, LLMs show better performance in logical reasoning without forgetting the pretrained knowledge. Two of the three reviewers are convinced with the contribution of the paper, while the other reviewer who was against it, didn't participate in the rebuttal.

Reviews: review: paper_summary: paper_summary: The paper discusses a novel approach to improve the logical reasoning capabilities of large language models (LLMs). The authors propose a method called Additional Logic Pre-Training (ALPT), which involves training LLMs on a synthetic corpus named PureLogicDiverse. This corpus is designed to include high-quality, program-generated reasoning samples that adhere to strict logical principles.
strength_weakness: strength_weakness: There is a risk of models overfitting to the synthetic logic patterns, which could limit their generalizability to real-world applications. The success of ALPT heavily depends on the quality of the design principles used to create the synthetic corpus, and any deficiencies in these principles could reduce its effectiveness. Therefore, while ALPT shows great promise, it comes with challenges that need careful management to maximize its potential benefits.
questions: questions: Authors discussed about the performance on other tasks like CommonsenseQA and Hellaswag, the results show that there is no substantial improvement on these tasks. I am curious to see the performance on other tasks like math or coding.
limitations: limitations: The approach may not fully address tasks requiring complex procedural understanding or multiple-choice questions that involve nuanced reasoning. Further research is needed to integrate ALPT with other methods to enhance performance on these types of tasks.
he effectiveness of ALPT heavily depends on the quality and comprehensiveness of the design principles used to create the synthetic corpus. Any biases or limitations in these principles could affect the overall performance improvements and generalizability of the models.
paper_summary: paper_summary: This work proposes Additional Logic Pre-Training (ALPT) to enhance logical reasoning abilities using synthetic rule-based data. The paper first discusses the design principles for creating a logical corpus and subsequently builds PureLogicDiverse (PLD). By training on PLD with RecAdam, models demonstrate improved performance in logical reasoning across various tasks and datasets, effectively integrating enhanced logical reasoning capabilities with their inherent knowledge. The results confirm the effectiveness of the proposed dataset and the ALPT strategy.
strength_weakness: strength_weakness: The overall experiments are well-established. However, when comparing the ablation results with those of previous work (Table 2a and Figure 2), RuleTakers appears to perform well enough given the simplicity of its design. For example, comparing ‘ALPT-RT’(Table 2a) with ‘w/o DP2’(Figure 2), despite ‘w/o DP2’ containing much bigger vocabulary size and more extensive rules and expressions, these two settings gain similar performances.  

To clarify these comparisons, it would be helpful if the authors could provide statistics on the different dataset settings, such as the number of steps contained in the sampled training splits of each corpus and the average number of rules/steps per sample.
questions: questions: 1. In dataset creation (Line 188), when sampling from the predefined logical formulas, is there a prior distribution from which to sample, or are the formulas evenly distributed? Do you have statistics on the operators and formulas of the created dataset? 
2. Since the nouns and predicates are randomly composed to form logical expressions, they lack practical meanings. Intuitively, the coverage of rules and formulas plays a more important role than vocabulary coverage. Considering this, is covering a wide range of 15k vocabulary really necessary? As shown in Figure 2 (w/o DP1), restricting the vocabulary size appears to have minimal impact on the final performance and could also help decrease the dataset size, potentially improving data efficiency. Do you have any insights on this?
limitations: limitations: The authors have reasonably discussed the limitations and potential societal impacts of their research.
paper_summary: paper_summary: This paper aims to improve LLM's logical reasoning ability by constructing synthetic data used in continual training. This work is largely built upon FLD, and proposes four other design principles for the synthetic dataset. Namely, reasoning with unknown facts, illogical reasoning, diverse reasoning rules, and linguistic expressions.  By training llama-7b and 70b models on the synthetic dataset, the authors proved the effectiveness of their methods by achieving improvement on various benchmarks such as BBH.
strength_weakness: strength_weakness: 1.  I am mostly concerned about the contribution of this paper. It seems to me that the four design principles are incremental improvements based on [1] by increasing the vocab size and enriching the diversity of logical rules and linguistic expressions. The results in Table 2 seem incremental compared with FLD baselines either. Also, the writing of some parts of the paper is very similar to [1], with even the same examples.
2. The current writing makes it hard for non-experts to understand. Specifically, the description of how the generator works is listed in the appendix. If one hasn't read [1] or the appendix, it would be very hard to understand the overall workflow.
3. It would strengthen the paper if the authors could also provide the results for FLD in Table 3 as a baseline for comparison.


[1] Morishita, Terufumi, et al. "Learning deductive reasoning from synthetic corpus based on formal logic." International Conference on Machine Learning. ICML, 2023.
questions: questions: 1. I think the training process using the constructed synthetic data is more like a continual training or SFT setting. Calling it additional "pre-training" could be a little misleading.
limitations: limitations: There is a limitation section in the appendix.

Comments: comment: Thank you for your response. I believe this work is solid as it stands. I will be maintaining my original score.