LogiNumSynth: Synthesizing Joint Logical-Numerical Reasoning Problems for Language Models

## Abstract

Abstract Joint logical-numerical reasoning remains a major challenge for language models, yet existing datasets rely on fixed rule sets and offer limited control over task complexity, constraining their generalizability for evaluation and training. We present LogiNumSynth , a flexible natural language problem synthesizer that synthesizes tasks requiring proficiency in joint logical reasoning (e.g., rule-based reasoning) and numerical reasoning (e.g., arithmetic computation). LogiNumSynth supports fine-grained control over reasoning world richness, logical reasoning depth, and the complexity of numerical computations, enabling flexible data synthesis across difficulty levels. We demonstrate three key contributions: (1) Synthesizer —synthesizing fully controllable joint reasoning tasks over natural language; (2) Evaluation & Process Analysis —evaluating both process accuracy and answer accuracy; (3) Targeted Training —using synthesized data to enhance LLMs’ reasoning performance. Experiments with multiple LLMs highlight persistent weaknesses in logical-numerical reasoning, showing that LogiNumSynth can serve as both a diagnostic tool and a source of targeted supervision for advancing integrated reasoning skills.

## 1 Introduction

Comprehending the world and adeptly applying knowledge in practical scenarios pose fundamental challenges for natural language processing (NLP). At the core of this cognitive process lie *logical reasoning* and *numerical reasoning*, which jointly enable solving complex problems and deriving meaningful insights (Saxton et al., 2019; Clark et al., 2020; Liu et al., 2020; Hendrycks et al., 2021).

#### Motivation.

Although considerable efforts have been made to advance various aspects of reasoning (Lample & Charton, 2019; Li et al., 2022; Lu et al., 2022; Wei et al., 2022; Lightman et al., 2023a; Li et al., 2023; Xu et al., 2024), existing datasets typically focus on *either* logical reasoning (Clark et al., 2020; Morishita et al., 2023; 2024) *or* numerical reasoning (Cobbe et al., 2021; Chen et al., 2021b; Veeraboina, 2023; Gao et al., 2025) in isolation (see Appendix [C](https://arxiv.org/html/2510.11031v1#A3) for related work).
In practice, however, many real-world problems require these abilities to be integrated.
Table [1](https://arxiv.org/html/2510.11031v1#S1.T1) shows a biology example where solving the task demands both the application of genetic rules (logical reasoning) and probabilistic computation of phenotypic ratios (numerical reasoning).

RuleArena (Zhou et al., 2025) is, to our knowledge, the only existing dataset explicitly integrating logical and numerical reasoning, covering tasks such as taxation, airline luggage fees, and NBA trade validation.
However, it has the following limitations.
First, it offers limited control over logical and numerical complexity, and its reliance on fixed rules leads to repetitive task patterns, which limit its extensibility and generalizability for evaluation and targeted training.
Second, while the original work reports rule-level recall and precision (a coarse form of process evaluation), it does not provide fine-grained assessment of intermediate reasoning steps.
These limitations motivate the need for a flexible synthesizer that can (1) synthesize joint logical-numerical problems of adjustable complexity for evaluation and further serving as a targeted training resource to improve reasoning performance, (2) incorporate structured intermediate steps to enable fine-grained process-level diagnosis.

#### Our Work.

We introduce *LogiNumSynth*, a controllable natural language problem synthesizer that synthesizes tasks requiring integrated logical reasoning (e.g., rule-based reasoning) and numerical reasoning (e.g., arithmetic computation).
Our synthesizer allows *fine-grained control* over reasoning world richness (e.g., the number of rules and facts), logical reasoning depth, and the complexity of numerical computations (e.g., computation range), enabling flexible synthesis of customized datasets across difficulty levels.
It is also *highly extensible*: the rule language supports easy inclusion of new mathematical expressions and richer logical operators, allowing the construction of more diverse and challenging reasoning processes tailored to specific evaluation needs.
It operates *independently of domain-specific background knowledge*, ensuring that evaluation reflects a model’s inherent reasoning capability rather than its memorized factual knowledge.

To synthesize a problem instance, LogiNumSynth first constructs a formal world of automatically synthesized facts and rules, which incorporate numerical expressions, paired with a query, and then translates this structure into natural language descriptions.
Models must reason over the provided natural language facts and rules to answer the query.
In addition to answer accuracy, we evaluate *process accuracy* by comparing a model’s generated reasoning steps with the automatically derived gold-standard reasoning process.

#### Our Main Contributions

are:

- •
*Synthesizer:* A customizable and extensible synthesizer that synthesizes natural language problems requiring joint logical and numerical reasoning with controllable complexity.
- •
*Evaluation & Process Analysis:* A large-scale evaluation of 29 models, including process-level reasoning diagnostics.
- •
*Targeted Training:* Demonstrating that our synthetic data can enhance model performance on external reasoning tasks.

#### Code and Data

are available at [https://github.com/nju-websoft/LogiNumSynth](https://github.com/nju-websoft/LogiNumSynth).

Table: Table 1: Example of joint logical-numerical reasoning in biology.

## 2 Data Synthesization

Our synthesization starts from synthetic formal representations on which reasoning combines logical and numerical operations. Then we convert formal representations into diverse templated natural language descriptions and use an LLM to improve their fluency. We summarize the features of our synthesizer in Appendix [D](https://arxiv.org/html/2510.11031v1#A4). Now we elaborate on its design.

### 2.1 Formal Representation

To synthesize a problem instance (called a *sample* in the rest of the paper) that requires reasoning, we first define its underlying *formal representation* $\langle\texttt{Facts},\texttt{Rules},\texttt{Query}\rangle$. Facts and Rules jointly establish a *world model* where formal reasoning can be executed to derive the gold-standard answer for the given Query. Specifically, facts that match the condition of a rule will trigger this rule to infer new facts. An example is shown in the “Reasoning DAG” part of Figure [1](https://arxiv.org/html/2510.11031v1#S2.F1).

Figure: Figure 1: Overview of sample synthesization.
Refer to caption: x1.png

#### Entities, Attributes, and Relationships.

Each sample is constructed from elements that include a set of *entities* $\mathbf{E}=\{e_{1},e_{2},\cdots\}$ and a set of *attributes* $\mathbf{A}=\{a_{1},a_{2},\cdots\}$. Each entity is associated with a number of attributes. The *value* of an attribute is numerical and characterizes the degree of the attribute in practical scenarios. For example, when we declare that the cold attribute of the entity Dave has a value of 4, it signifies the level or intensity of coldness that he experiences, with 4 indicating a certain degree of severity.
We also include binary *relationships* between entities. The set of all relationships is denoted by $\mathbf{R}=\{r_{1},r_{2},\cdots\}$.

#### Facts

include all known values of the attributes of the entities and their relationships. They provide the foundational information needed for reasoning.
Facts come in two types:

- •
$r_{k}(e_{i},e_{j})$, denoting the existence of a directed relationship $r_{k}\in\mathbf{R}$ between two entities $e_{i},e_{j}\in\mathbf{E}$, and
- •
$\mathrm{is}(e_{i},a_{j},\mathrm{num})$, which specifies a numerical value $\mathrm{num}$ for a specific attribute $a_{j}\in\mathbf{A}$ of an entity $e_{i}\in\mathbf{E}$.

#### Rules

are expressed in an implication form, composed of a condition and a conclusion:

$$ $\displaystyle\texttt{Condition}\rightarrow\texttt{Conclusion}\,.$ $$

If condition is true, then conclusion is inferred to be true.

In our current implementation (which can be easily extended), the condition is formulated as a conjunction of one or more facts, while the conclusion is derived as a corresponding fact. Crucially, all entities within these facts are represented as anonymized placeholder variables, denoted using Greek letters as subscripts (e.g., $e_{\alpha}$, $e_{\beta}$). These placeholders are dynamically instantiated with concrete entities during rule application. To ensure logically valid inferences, any placeholder variable appearing in the conclusion must also appear in the corresponding condition. Formally, we define the condition using regular expression as:

$$ $\displaystyle\left[\mathrm{is}(e_{\alpha},a_{i},\mathrm{num})\;\middle|\;r_{j}(e_{\beta},e_{\gamma})\right]\;\big[\land\;\left[\mathrm{is}(e_{\epsilon},a_{k},\mathrm{num}^{\prime})\;\middle|\;r_{l}(e_{\zeta},e_{\iota})\right]\big]^{*}\,.$ $$

Similarly, the conclusion is implemented as a factual statement of the form:

- •
$r_{i}(e_{\alpha},e_{\beta})$, or
- •
$\mathrm{is}\left(e_{\gamma},a_{j},\mathrm{expr}(\dots)\right)$, where $\mathrm{expr}(\dots)$ is a function over placeholder variables defined in the condition.

Unlike facts, conclusions involving $\mathrm{expr(\dots)}$ introduce numerical computations.
In the current implementation, we support four types of expressions:

- •
Constant expression: a parameter-free expression representing a fixed constant value.
- •
Retrieval expression: parameterized by an entity $e_{\alpha}$ and an attribute $a_{i}$; its value is simply the attribute of the entity itself, denoted as $e_{\alpha}[a_{i}]$.
- •
Calculation expression: parameterized by an entity $e_{\alpha}$ and an attribute $a_{i}$; its value is computed as a linear form $k\times e_{\alpha}[a_{i}]+b$, where $k$ and $b$ are intrinsic parameters of the expression sampled from a specified range, as detailed in Appendix [E.1](https://arxiv.org/html/2510.11031v1#A5.SS1).
- •
Aggregation expression: defined over two sub-expressions (each being a constant, retrieval, or calculation expression).
Its parameters are inherited from the sub-expressions, and its value is obtained by aggregating the results of the two sub-expressions using one of the supported operators: max, min, addition, or subtraction (e.g. $\texttt{max}(k\times e_{\alpha}[a_{i}]+b,e_{\beta}[a_{j}])$).

#### Query

finally asks the model to determine the value of a specific attribute for a given entity and can be formulated as $\mathrm{is}(e_{i},a_{j},?)$.

A reasoning model capable of answering such queries requires proficiency in *joint rule-based logical reasoning and arithmetic-based numerical reasoning*.
Note that *the above implementation can be easily extended* by, for example, designing more diverse forms of rules and providing more expressive arithmetic operations. These extensions are incremental and are left for future work.

### 2.2 Sample Synthesization

We develop an initializer, a reasoning DAG constructor, a distraction synthesizer, and a natural language generator to synthesize samples described in natural language, as illustrated in Figure [1](https://arxiv.org/html/2510.11031v1#S2.F1). These four components equip our synthesizing process with diversity, purity (i.e., independent of background knowledge), and extensibility.

#### Initializer

randomly selects entities, attributes, and relationships from predefined pools according to the given configurations (i.e., the numbers of entities, attributes, and relationships), thereby constructing the world elements of the sample inference world. The selection process for $\mathbf{A}$ and $\mathbf{R}$ explicitly excludes synonyms, thereby mitigating the risk of semantic ambiguities that could confound the results. Specifically, we use an entity pool of 7,944 items from nltk.corpus.names, an attribute pool of 1,366 items from nltk.corpus.treebank, and a relationship pool of 976 items from nltk.corpus.wordnet. These resources are provided by the Natural Language Toolkit (NLTK) (Bird et al., 2009), a widely used library for natural language processing. After that, the initializer randomly selects one entity and one attribute as the query.

#### Reasoning DAG Constructor

takes as input the world elements, the query, and the configuration parameters (e.g., the reasoning depth). It then attempts to construct a reasoning directed acyclic graph (DAG) by working backward from the query as the target node, with each node representing either a fact or a rule along the reasoning path.
Specifically, the constructor first takes the query as the target and synthesizes a rule that can derive this target. The number of conditions and the form of the conclusion in this rule are randomly determined according to the configuration parameters. Next, it synthesizes the corresponding facts that satisfy the rule’s conditions and treats these facts as the subsequent targets. For each new target, the constructor either directly produces a supporting fact or synthesizes another rule together with additional targets as conditions to support it. This process continues until the reasoning depth specified in the configuration is reached. During the generation of each fact or rule, the constructor ensures that no conflicts occur, i.e., respecting attribute value uniqueness, thus guaranteeing the uniqueness of the query’s answer. Finally, it performs forward reasoning across the DAG to derive both the gold-standard reasoning process and final answer. Further implementation details are provided in Appendix [E.1](https://arxiv.org/html/2510.11031v1#A5.SS1).

#### Distraction Synthesizer

produces the remaining distracting facts and rules based on the constructed reasoning DAG and the configuration parameters (e.g., the numbers of facts and rules). Similar to the construction of reasoning DAG, the facts and rules are randomly synthesized according to the configurations, while ensuring that no conflicts arise with the existing facts and rules.

#### Natural Language Generator

operates in two steps. First, templates are used to convert the formal representation of the synthesized sample into diverse, template-based natural language descriptions, laying the foundation for the subsequent natural language optimization.
The number of predefined templates is summarized in Table [2](https://arxiv.org/html/2510.11031v1#S2.T2).
In particular, to describe a rule, we randomly select and combine templates for its components. For example, describing the rule $r_{i}(e_{\alpha},e_{\beta})\rightarrow\mathrm{is}(e_{\beta},a_{j},3\times e_{\alpha}[a_{k}]-9)$ requires one template for describing the implication, two templates to describe the two facts, one template to describe the calculation expression, and one template to describe the retrieval expression.
In the second step, it takes the template-based descriptions and further refines them using an LLM, making the text more fluent and natural. This refinement ensures that the generated text is syntactically accurate and semantically coherent, making it suitable for processing by both humans and language models.
More examples of templates and detailed settings are provided in Appendix [E.2](https://arxiv.org/html/2510.11031v1#A5.SS2).

Table: Table 2: Examples and numbers of templates used for converting formal representation into natural language.

## 3 Experiment Setup

We synthesized datasets of varying difficulty (detailed in Section [3.1](https://arxiv.org/html/2510.11031v1#S3.SS1)) to evaluate language models of different architectures and scales (detailed in Section [3.2](https://arxiv.org/html/2510.11031v1#S3.SS2)), with evaluation metrics described in Section [3.3](https://arxiv.org/html/2510.11031v1#S3.SS3), as well as to serve as training resources (detailed in Section [3.4](https://arxiv.org/html/2510.11031v1#S3.SS4)).

### 3.1 Synthetic Datasets

We first synthesized four datasets using our data synthesizer, named EL-EN, EL-HN, HL-EN and HL-HN. The first part of the name represents the logical reasoning difficulty (Easy or Hard Logical), while the second part indicates the numerical reasoning difficulty (Easy or Hard Numerical). To further assess the reasoning capabilities of state-of-the-art models, we also synthesized exHL-HN (extremely Hard Logical-Hard Numerical). Lastly, for the training resources for reasoning, we synthesized two datasets: EN-Train and EL-Train. These are specifically designed to lower the difficulty in numerical (Easy Numerical) or logical (Easy Logical) reasoning, respectively, to aid model training toward the other reasoning capability.
We controlled the difficulty of the samples in each dataset by varying factors such as reasoning depth, the number of conditions in a logical rule, the range of numerical computations, and the types of expressions used.
More details on the synthesizing configurations of these datasets are provided in Appendix [E.3](https://arxiv.org/html/2510.11031v1#A5.SS3).

### 3.2 Tested Models

We evaluated 29 LLMs,
including models with long reasoning chains:
GPT5-mini (based on GPT5-mini-2025-08-07) (OpenAI, 2025),
DeepSeek-R1 (based on DeepSeek-R1-0528) (Guo et al., 2025);
models specialized for reasoning:
Phi4-mini-reasoning (Xu et al., 2025),
Phi4-reasoning and Phi4-reasoning-plus (Abdin et al., 2025);
hybrid reasoning models: Qwen3 series (0.6B, 1.7B, 4B, 8B, 14B, 32B) (Yang et al., 2025);
models with MoE (Mixture of Experts) architectures: Qwen3-30B-A3B (Yang et al., 2025);
and common instruct models:
Llama3.1-8B-instruct (Dubey et al., 2024), Llama3.2-1B-instruct, Llama3.2-3B-instruct (AI, 2024),
Qwen2.5-instruct series (0.5B, 1.5B, 3B, 7B, 14B, 32B) (Team, 2024),
GLM4-chat (GLM et al., 2024), GLM4-airx, -0520, -plus(^2^22[https://open.bigmodel.cn/](https://open.bigmodel.cn/)),
Phi4-mini-instruct (Abouelenin et al., 2025), Phi4 (Abdin et al., 2024),
DeepSeek-V3 (based on DeepSeek-V3-0324) (Liu et al., 2024),
and GPT-4o (based on GPT-4o-2024-11-20) (OpenAI, 2023),
with parameter sizes ranging from 0.5B to 671B.

Large models with long reasoning chains and significant parameter sizes, such as GPT5-mini and DeepSeek-R1, demonstrate the current upper bound of reasoning capabilities in LLMs. On the other hand, models with moderate parameter sizes, such as Qwen3-14B and Phi4-reasoning-plus (14B), represent a balanced scale, offering a trade-off between computational efficiency and reasoning capability. Smaller models, such as Qwen3-1.7B and Llama3.2-1B-instruct, further emphasize computational efficiency, but with a reduction in reasoning depth and complexity. We prompted them using Chain of Thought (CoT), in the zero-shot and few-shot settings (Wei et al., 2022).
Implementation details are provided in Appendix [F.1](https://arxiv.org/html/2510.11031v1#A6.SS1).

### 3.3 Evaluation Metrics

Due to the nature of our synthesized datasets, each question’s reasoning can be decomposed into multiple steps, enabling a fine-grained evaluation of a model’s *process accuracy*—the extent to which its reasoning steps match the gold-standard process—and *answer accuracy*—the correctness of the final result. To compute process accuracy, we prompted models to explicitly summarize their reasoning at the end and used the structured output mode of Qwen3-8B to extract structured reasoning steps from their output. Both the raw and structured outputs were compared with the gold-standard reasoning process, allowing partial credit for intermediate steps that were correct. In cases where the model outputs a different reasoning path but still arrives at the correct final answer, the process is considered correct. This step-level evaluation is particularly important in scenarios such as automated theorem proving, multi-step medical diagnosis, and financial auditing, where the transparency, rigor, and correctness of each step are as critical as the final result. The implementation of process accuracy is detailed in Appendix [F.2](https://arxiv.org/html/2510.11031v1#A6.SS2).

### 3.4 Training Setup

To demonstrate the potential of our synthetic data as an additional training resource for enhancing model reasoning ability, we conducted supervised fine-tuning on Qwen3-1.7B and Llama3.2-1B-instruct using LoRA (Hu et al., 2021), and, in some experiments, the Recall Adam Optimizer Chen et al. (2020). The models were evaluated on numerical reasoning benchmarks:
GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), MATHQA (Amini et al., 2019), SVAMP (Patel et al., 2021), MAWPS (Koncel-Kedziorski et al., 2016), AIME (Veeraboina, 2023);
formal deductive logical reasoning benchmarks:
RuleTaker (Clark et al., 2020), ProofWriter (Tafjord et al., 2020), FOLIO (Han et al., 2024a), FLD (Morishita et al., 2023);
complex logical and joint logical-numerical reasoning benchmarks:
LogiQA (Liu et al., 2020), ReClor (Yu et al., 2020), AbductionR (Young et al., 2022), RuleArena (Zhou et al., 2025).
Detailed training configurations are provided in Appendix [F.3](https://arxiv.org/html/2510.11031v1#A6.SS3).

Table: Table 3: Performance of LLMs on datasets of different difficulty levels. ‘Proc’ refers to process accuracy and ‘Ans’ refers to answer accuracy.

## 4 Experiment Results

### 4.1 Evaluation Results

#### Most Models Perform Poorly.

The evaluation results of LLMs are shown in Table [3](https://arxiv.org/html/2510.11031v1#S3.T3).
Due to the inherent difficulty of the synthesized tasks, many commonly used models perform poorly. For instance, Llama3.1-8B-instruct achieves only 0.83 process accuracy and 5.00 answer accuracy in the 0-shot setting on HL-HN, while GPT-4o scores 5.12 and 38.80, respectively. In contrast, stronger reasoning models like DeepSeek-R1 and GPT5-mini achieve 76.35 and 92.60, and 79.72 and 94.40, respectively.

#### Analysis of Two Best Models.

For GPT5-mini and DeepSeek-R1, transferring from EN to HN results in a noticeable decrease in both process accuracy and answer accuracy, consistent with other models. However, transferring from EL to HL causes a more significant drop in process accuracy (e.g. from 91.93 on EL-HN to 76.35 on HL-HN for DeepSeek-R1 in the 0-shot setting), while answer accuracy remains relatively unchanged (e.g. from 92.60 on EL-HN to 92.60 on HL-HN for DeepSeek-R1 in the 0-shot setting). This phenomenon is observed only in these two models. This suggests that these two models are generally capable of handling the logical reasoning complexity of the tasks to produce correct answers, but struggle with presenting clear and logically sound reasoning processes.

Given the strong reasoning abilities demonstrated by the two top-performing models, we further stress-tested them using exHL-HN in the 0-shot setting. As shown in Table [4](https://arxiv.org/html/2510.11031v1#S4.T4), they fail to perform well (e.g., GPT5-mini only scores 23.23 and 54.75 as process and answer accuracy respectively), and the results highlight GPT5-mini’s stronger reasoning ability compared to DeepSeek-R1. This difference is less apparent in the previous four datasets, showcasing how our synthesizer can synthesize datasets of varying difficulty levels to evaluate the capabilities of different models.

Table: Table 4: Performance of two top-performing models on the exHL-HN dataset in the 0-shot setting. ‘Proc’ refers to process accuracy and ‘Ans’ refers to answer accuracy.

Figure: (a) EL-HN v.s. HL-EN
Refer to caption: x2.png

#### EL-HN v.s. HL-EN Comparison.

Figure [2(a)](https://arxiv.org/html/2510.11031v1#S4.F2.sf1) shows the process accuracy and answer accuracy of different models on EL-HN and HL-EN, indicating that while HL slightly increases the difficulty for answer accuracy, models perform significantly worse on process accuracy under HL, compared with HN. This is due to the larger reasoning depth and more complex logical rules, which significantly hinder the models’ ability to provide accurate reasoning processes.

#### 0-shot v.s. 3-shot Comparison.

Figure [2(b)](https://arxiv.org/html/2510.11031v1#S4.F2.sf2) shows the comparison of average accuracy across different datasets for 29 models in the 0-shot and 3-shot settings, indicating that, across all models, 3-shot learning generally has little effect, with some models showing improvement and others experiencing a decline in performance. This is mainly because the synthesized tasks are inherently complex reasoning problems. While the provided examples can help guide the model on how to approach the problem, they may also divert the model’s attention, leading to performance changes. We also conducted experiments with varying numbers of few-shot examples, as detailed in Appendix [G.1](https://arxiv.org/html/2510.11031v1#A7.SS1), and the results remain consistent.

#### Case Study and Error Analysis.

We conducted a case study that reveals several interesting observations.
We found instances where the answer is wrong but the process accuracy is non-zero, demonstrating that process accuracy can capture partial correctness in reasoning even when the final answer fails.
We also observed a few rare cases where a model discovers a shorter yet correct reasoning path that differs from the gold-standard reasoning path. This is possible and considered correct because our synthesizer guarantees logical consistency (i.e., no derived facts, including the query, contradict each other) while allowing the existence of multiple valid reasoning paths.
For cases where the answer accuracy equals 1 but the process accuracy is below 1, the errors in the reasoning process typically fall into three categories:
(1) incorrect application of rules: misapplication or omission of relevant facts or rules, yet resulting in a correct intermediate conclusion;
(2) incorrect numerical computations: errors in intermediate numerical calculations, e.g., the final answer is calculated as $\max(a,b)$ where $a$ is incorrect but $b$ is correct and larger;
(3) incorrect intermediate results: producing wrong intermediate conclusions—commonly caused by lexical errors in entities, attributes and relationships or by reversing the direction of relationships—but ultimately yielding the correct final answer.
Representative examples are provided in Appendix [G.2](https://arxiv.org/html/2510.11031v1#A7.SS2).

### 4.2 Training Results

**Table 5: Performance on external reasoning benchmarks before and after SFT on our synthetic data.**
|  |
| --- |
|  |
|  |

To further assess the utility of our synthetic data as an *additional training resource* for improving reasoning capabilities, we fine-tuned two models on the synthesized datasets and directly evaluated them on existing reasoning benchmarks. The results are reported in Table [5](https://arxiv.org/html/2510.11031v1#S4.T5.tab3).

We first present the pre- and post-fine-tuning performance of Llama3.2-1B-instruct and Qwem3-1.7B across multiple out-of-domain numerical and formal deductive logical reasoning benchmarks, along with their average scores. Results show consistent improvements across all tasks, ranging from relatively simple datasets such as GSM8k to more challenging benchmarks like AIME. Notably, Qwen3-1.7B demonstrates a substantial 10.20 point gain on FOLIO.

We also report results on datasets that involve more diverse reasoning types—LogiQA (diverse logical reasoning and reading comprehension), ReClor (natural language inference), AbductionR (abductive logical reasoning) and RuleArena (rule-guided numerical reasoning). These results illustrate the potential of our synthetic data to serve as an effective training resource for enhancing reasoning abilities in out-of-domain settings.

## 5 Conclusion

Recent NLP research has increasingly focused on equipping language models with robust logical and numerical reasoning capabilities. To contribute to this direction, we introduced *LogiNumSynth*, a data synthesizer designed to benchmark models’ integrated logical-numerical reasoning abilities. With four synthetic datasets of varying difficulty levels, we evaluated 29 models covering diverse architectures and scales, and conducted a detailed analysis of their joint reasoning performance.
In addition, we synthesized a more challenging dataset to assess the strongest two models, revealing that there remains substantial room for improvement. Beyond evaluation, LogiNumSynth also provides synthetic training resources that can effectively enhance models’ reasoning abilities.

The extensibility of LogiNumSynth makes it a *living benchmark*: researchers can easily extend it to new domains and reasoning settings by incorporating additional operations or logical constructs, thereby enabling continuous and targeted evaluation of ever-advancing large language models. We further discuss the limitations of our work in Appendix [B](https://arxiv.org/html/2510.11031v1#A2).

## Reproducibility Statement

To facilitate the reproducibility of our work, we provide a detailed description of the experimental settings in Section [3](https://arxiv.org/html/2510.11031v1#S3), Appendix [E.2](https://arxiv.org/html/2510.11031v1#A5.SS2), Appendix [E.3](https://arxiv.org/html/2510.11031v1#A5.SS3), and Appendix [F](https://arxiv.org/html/2510.11031v1#A6). In addition, our code and datasets are included at [https://github.com/nju-websoft/LogiNumSynth](https://github.com/nju-websoft/LogiNumSynth) to enable independent verification of our results.

## References

- Abdin et al. (2024)
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al.
Phi-4 technical report.
*arXiv preprint arXiv:2412.08905*, 2024.
- Abdin et al. (2025)
Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi, Neel Joshi, et al.
Phi-4-reasoning technical report.
*arXiv preprint arXiv:2504.21318*, 2025.
- Abouelenin et al. (2025)
Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al.
Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras.
*arXiv preprint arXiv:2503.01743*, 2025.
- AI (2024)
Meta AI.
Llama 3.2 model card, 2024.
URL [https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/MODEL_CARD.md](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/MODEL_CARD.md).
Accessed: 2025-09-13.
- Amini et al. (2019)
Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi.
Mathqa: Towards interpretable math word problem solving with operation-based formalisms.
*arXiv preprint arXiv:1905.13319*, 2019.
- Bird et al. (2009)
Steven Bird, Ewan Klein, and Edward Loper.
*Natural Language Processing with Python*.
O’Reilly Media Inc., 2009.
- Brown et al. (2020)
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.
Language models are few-shot learners.
*Advances in neural information processing systems*, 33:1877–1901, 2020.
- Chen et al. (2021a)
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al.
Evaluating large language models trained on code.
*arXiv preprint arXiv:2107.03374*, 2021a.
- Chen et al. (2020)
Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu.
Recall and learn: Fine-tuning deep pretrained language models with less forgetting.
In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pp. 7870–7881, Online, November 2020. Association for Computational Linguistics.
doi: 10.18653/v1/2020.emnlp-main.634.
URL [https://aclanthology.org/2020.emnlp-main.634/](https://aclanthology.org/2020.emnlp-main.634/).
- Chen et al. (2021b)
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema N Moussa, Matthew I. Beane, Ting-Hao ’Kenneth’ Huang, Bryan R. Routledge, and William Yang Wang.
Finqa: A dataset of numerical reasoning over financial data.
In *Conference on Empirical Methods in Natural Language Processing*, 2021b.
URL [https://api.semanticscholar.org/CorpusID:235399966](https://api.semanticscholar.org/CorpusID:235399966).
- Clark et al. (2020)
Peter Clark, Oyvind Tafjord, and Kyle Richardson.
Transformers as soft reasoners over language.
*arXiv preprint arXiv:2002.05867*, 2020.
- Cobbe et al. (2021)
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al.
Training verifiers to solve math word problems.
*arXiv preprint arXiv:2110.14168*, 2021.
- Dua et al. (2019)
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner.
Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs.
In *North American Chapter of the Association for Computational Linguistics*, 2019.
URL [https://api.semanticscholar.org/CorpusID:67855846](https://api.semanticscholar.org/CorpusID:67855846).
- Dubey et al. (2024)
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al.
The llama 3 herd of models.
*arXiv preprint arXiv:2407.21783*, 2024.
- Gao et al. (2025)
Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang.
Omni-MATH: A universal olympiad level mathematic benchmark for large language models.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=yaqPf0KAlN](https://openreview.net/forum?id=yaqPf0KAlN).
- GLM et al. (2024)
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang, Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang, Xixuan Song, Xunkai Zhang, Yifan An, Yifan Xu, Yilin Niu, Yuantao Yang, Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhen Yang, Zhengxiao Du, Zhenyu Hou, and Zihan Wang.
Chatglm: A family of large language models from glm-130b to glm-4 all tools, 2024.
- Guo et al. (2025)
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
*arXiv preprint arXiv:2501.12948*, 2025.
- Han et al. (2024a)
Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alexander Wardle-Solano, Hannah Szabó, Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Alexander Fabbri, Wojciech Maciej Kryscinski, Semih Yavuz, Ye Liu, Xi Victoria Lin, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Rex Ying, Arman Cohan, and Dragomir Radev.
FOLIO: Natural language reasoning with first-order logic.
In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pp. 22017–22031, Miami, Florida, USA, November 2024a. Association for Computational Linguistics.
doi: 10.18653/v1/2024.emnlp-main.1229.
URL [https://aclanthology.org/2024.emnlp-main.1229/](https://aclanthology.org/2024.emnlp-main.1229/).
- Han et al. (2024b)
Simeng Han, Aaron Yu, Rui Shen, Zhenting Qi, Martin Riddell, Wenfei Zhou, Yujie Qiao, Yilun Zhao, Semih Yavuz, Ye Liu, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Dragomir Radev, Rex Ying, and Arman Cohan.
P-FOLIO: Evaluating and improving logical reasoning with abundant human-written reasoning chains.
In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), *Findings of the Association for Computational Linguistics: EMNLP 2024*, pp. 16553–16565, Miami, Florida, USA, November 2024b. Association for Computational Linguistics.
doi: 10.18653/v1/2024.findings-emnlp.966.
URL [https://aclanthology.org/2024.findings-emnlp.966/](https://aclanthology.org/2024.findings-emnlp.966/).
- Hendrycks et al. (2020)
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.
Measuring massive multitask language understanding.
*arXiv preprint arXiv:2009.03300*, 2020.
- Hendrycks et al. (2021)
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.
Measuring mathematical problem solving with the math dataset.
*arXiv preprint arXiv:2103.03874*, 2021.
- Hu et al. (2021)
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.
Lora: Low-rank adaptation of large language models.
*arXiv preprint arXiv:2106.09685*, 2021.
- Koncel-Kedziorski et al. (2016)
Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi.
Mawps: A math word problem repository.
In *North American Chapter of the Association for Computational Linguistics*, 2016.
URL [https://api.semanticscholar.org/CorpusID:2228719](https://api.semanticscholar.org/CorpusID:2228719).
- Kwon et al. (2023)
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica.
Efficient memory management for large language model serving with pagedattention.
In *Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles*, 2023.
- Lample & Charton (2019)
Guillaume Lample and François Charton.
Deep learning for symbolic mathematics.
*arXiv preprint arXiv:1912.01412*, 2019.
- Li et al. (2022)
Xiao Li, Gong Cheng, Ziheng Chen, Yawei Sun, and Yuzhong Qu.
Adalogn: Adaptive logic graph network for reasoning-based machine reading comprehension.
In *Annual Meeting of the Association for Computational Linguistics*, 2022.
URL [https://api.semanticscholar.org/CorpusID:247518855](https://api.semanticscholar.org/CorpusID:247518855).
- Li et al. (2023)
Xiao Li, Yin Zhu, Sichen Liu, Jiangzhou Ju, Yuzhong Qu, and Gong Cheng.
Dyrren: A dynamic retriever-reranker-generator model for numerical reasoning over tabular and textual data.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 37, pp. 13139–13147, 2023.
- Lightman et al. (2023a)
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe.
Let’s verify step by step, 2023a.
- Lightman et al. (2023b)
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe.
Let’s verify step by step.
In *The Twelfth International Conference on Learning Representations*, 2023b.
- Liu et al. (2024)
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al.
Deepseek-v3 technical report.
*arXiv preprint arXiv:2412.19437*, 2024.
- Liu et al. (2023)
Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang.
Evaluating the logical reasoning ability of chatgpt and gpt-4.
*arXiv preprint arXiv:2304.03439*, 2023.
- Liu et al. (2020)
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang.
Logiqa: A challenge dataset for machine reading comprehension with logical reasoning.
In Christian Bessiere (ed.), *Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020*, pp. 3622–3628. ijcai.org, 2020.
doi: 10.24963/IJCAI.2020/501.
URL [https://doi.org/10.24963/ijcai.2020/501](https://doi.org/10.24963/ijcai.2020/501).
- Liu et al. (2025)
Shudong Liu, Hongwei Liu, Junnan Liu, Linchen Xiao, Songyang Gao, Chengqi Lyu, Yuzhe Gu, Wenwei Zhang, Derek F Wong, Songyang Zhang, et al.
Compassverifier: A unified and robust verifier for llms evaluation and outcome reward.
*arXiv preprint arXiv:2508.03686*, 2025.
- Liu et al. (2019)
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.
Roberta: A robustly optimized bert pretraining approach.
*arXiv preprint arXiv:1907.11692*, 2019.
- Lu et al. (2024)
Jianqiao Lu, Zhiyang Dou, Hongru Wang, Zeyu Cao, Jianbo Dai, Yunlong Feng, and Zhijiang Guo.
Autopsv: Automated process-supervised verifier.
*Advances in Neural Information Processing Systems*, 37:79935–79962, 2024.
- Lu et al. (2022)
Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai-Wei Chang.
A survey of deep learning for mathematical reasoning.
*arXiv preprint arXiv:2212.10535*, 2022.
- Luo et al. (2023)
Man Luo, Shrinidhi Kumbhar, Mihir Parmar, Neeraj Varshney, Pratyay Banerjee, Somak Aditya, Chitta Baral, et al.
Towards logiglue: A brief survey and a benchmark for analyzing logical reasoning capabilities of language models.
*arXiv preprint arXiv:2310.00836*, 2023.
- Miao et al. (2020)
Shen-yun Miao, Chao-Chun Liang, and Keh-Yih Su.
A diverse corpus for evaluating and developing English math word problem solvers.
In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pp. 975–984, Online, July 2020. Association for Computational Linguistics.
doi: 10.18653/v1/2020.acl-main.92.
URL [https://aclanthology.org/2020.acl-main.92/](https://aclanthology.org/2020.acl-main.92/).
- Morishita et al. (2023)
Terufumi Morishita, Gaku Morio, Atsuki Yamaguchi, and Yasuhiro Sogawa.
Learning deductive reasoning from synthetic corpus based on formal logic.
In *International Conference on Machine Learning*, pp. 25254–25274. PMLR, 2023.
- Morishita et al. (2024)
Terufumi Morishita, Gaku Morio, Atsuki Yamaguchi, and Yasuhiro Sogawa.
Enhancing reasoning capabilities of llms via principled synthetic logic corpus.
In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), *Advances in Neural Information Processing Systems*, volume 37, pp. 73572–73604. Curran Associates, Inc., 2024.
URL [https://proceedings.neurips.cc/paper_files/paper/2024/file/8678da90126aa58326b2fc0254b33a8c-Paper-Conference.pdf](https://proceedings.neurips.cc/paper_files/paper/2024/file/8678da90126aa58326b2fc0254b33a8c-Paper-Conference.pdf).
- OpenAI (2023)
OpenAI.
Gpt-4 technical report.
*ArXiv*, abs/2303.08774, 2023.
URL [https://api.semanticscholar.org/CorpusID:257532815](https://api.semanticscholar.org/CorpusID:257532815).
- OpenAI (2025)
OpenAI.
Gpt-5 system card, 2025.
URL [https://cdn.openai.com/gpt-5-system-card.pdf](https://cdn.openai.com/gpt-5-system-card.pdf).
- Patel et al. (2021)
Arkil Patel, Satwik Bhattamishra, and Navin Goyal.
Are nlp models really able to solve simple math word problems?
In *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pp. 2080–2094, 2021.
- Saxton et al. (2019)
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli.
Analysing mathematical reasoning abilities of neural models.
*arXiv preprint arXiv:1904.01557*, 2019.
- Srivastava et al. (2022)
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al.
Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.
*arXiv preprint arXiv:2206.04615*, 2022.
- Tafjord et al. (2020)
Oyvind Tafjord, Bhavana Dalvi, and Peter Clark.
Proofwriter: Generating implications, proofs, and abductive statements over natural language.
In *Findings*, 2020.
URL [https://api.semanticscholar.org/CorpusID:229371222](https://api.semanticscholar.org/CorpusID:229371222).
- Team (2024)
Qwen Team.
Qwen2.5: A party of foundation models, September 2024.
URL [https://qwenlm.github.io/blog/qwen2.5/](https://qwenlm.github.io/blog/qwen2.5/).
- Tian et al. (2021)
Jidong Tian, Yitian Li, Wenqing Chen, Liqiang Xiao, Hao He, and Yaohui Jin.
Diagnosing the first-order logical reasoning ability through logicnli.
In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pp. 3738–3747, 2021.
- Veeraboina (2023)
Hemish Veeraboina.
Aime problem set 1983-2024, 2023.
URL [https://www.kaggle.com/datasets/hemishveeraboina/aime-problem-set-1983-2024](https://www.kaggle.com/datasets/hemishveeraboina/aime-problem-set-1983-2024).
- Wang et al. (2022)
Siyuan Wang, Zhongkun Liu, Wanjun Zhong, Ming Zhou, Zhongyu Wei, Zhumin Chen, and Nan Duan.
From lsat: The progress and challenges of complex reasoning.
*IEEE/ACM Transactions on Audio, Speech, and Language Processing*, 30:2201–2216, 2022.
- Wei et al. (2022)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.
Chain-of-thought prompting elicits reasoning in large language models.
*Advances in neural information processing systems*, 35:24824–24837, 2022.
- Wolf et al. (2020)
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush.
Transformers: State-of-the-art natural language processing.
In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations*, pp. 38–45, Online, October 2020. Association for Computational Linguistics.
URL [https://www.aclweb.org/anthology/2020.emnlp-demos.6](https://www.aclweb.org/anthology/2020.emnlp-demos.6).
- Xu et al. (2025)
Haoran Xu, Baolin Peng, Hany Awadalla, Dongdong Chen, Yen-Chun Chen, Mei Gao, Young Jin Kim, Yunsheng Li, Liliang Ren, Yelong Shen, et al.
Phi-4-mini-reasoning: Exploring the limits of small reasoning language models in math.
*arXiv preprint arXiv:2504.21233*, 2025.
- Xu et al. (2024)
Jundong Xu, Hao Fei, Liangming Pan, Qian Liu, Mong-Li Lee, and Wynne Hsu.
Faithful logical reasoning via symbolic chain-of-thought.
In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024*, pp. 13326–13365. Association for Computational Linguistics, 2024.
doi: 10.18653/V1/2024.ACL-LONG.720.
URL [https://doi.org/10.18653/v1/2024.acl-long.720](https://doi.org/10.18653/v1/2024.acl-long.720).
- Yang et al. (2025)
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al.
Qwen3 technical report.
*arXiv preprint arXiv:2505.09388*, 2025.
- Young et al. (2022)
Nathan Young, Qiming Bao, Joshua Bensemann, and Michael Witbrock.
AbductionRules: Training transformers to explain unexpected inputs.
In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), *Findings of the Association for Computational Linguistics: ACL 2022*, pp. 218–227, Dublin, Ireland, May 2022. Association for Computational Linguistics.
doi: 10.18653/v1/2022.findings-acl.19.
URL [https://aclanthology.org/2022.findings-acl.19/](https://aclanthology.org/2022.findings-acl.19/).
- Yu et al. (2023)
Fei Yu, Hongbo Zhang, and Benyou Wang.
Natural language reasoning, a survey.
2023.
URL [https://api.semanticscholar.org/CorpusID:257766470](https://api.semanticscholar.org/CorpusID:257766470).
- Yu et al. (2020)
Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng.
Reclor: A reading comprehension dataset requiring logical reasoning.
*arXiv preprint arXiv:2002.04326*, 2020.
- Zheng et al. (2024)
Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin.
Processbench: Identifying process errors in mathematical reasoning.
*arXiv preprint arXiv:2412.06559*, 2024.
- Zhong et al. (2021)
Wanjun Zhong, Siyuan Wang, Duyu Tang, Zenan Xu, Daya Guo, Jiahai Wang, Jian Yin, Ming Zhou, and Nan Duan.
Ar-lsat: Investigating analytical reasoning of text, 2021.
- Zhou et al. (2025)
Ruiwen Zhou, Wenyue Hua, Liangming Pan, Sitao Cheng, Xiaobao Wu, En Yu, and William Yang Wang.
RuleArena: A benchmark for rule-guided reasoning with LLMs in real-world scenarios.
In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), *Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 550–572, Vienna, Austria, July 2025. Association for Computational Linguistics.
ISBN 979-8-89176-251-0.
doi: 10.18653/v1/2025.acl-long.27.
URL [https://aclanthology.org/2025.acl-long.27/](https://aclanthology.org/2025.acl-long.27/).

## Appendix A The Use of Large Language Models

We used large language models to polish the writing.

## Appendix B Limitations

The limitations of our work are as follows.

- •
*Better process evaluation*: Although we take care to minimize errors in process evaluation, it remains an open challenge (Lightman et al., 2023b; Zheng et al., 2024; Lu et al., 2024; Liu et al., 2025; Zhou et al., 2025). Current process evaluation methods are still imperfect, and a small number of such errors are inevitable, especially when intermediate reasoning steps are lengthy, interleaved, or involve intricate cross-references that disrupt the logical flow.
- •
*Training for more models and benchmarks*: Due to limitations in computational resources, we only experimented with fine-tuning Llama3.2-1B-instruct and Qwen3-1.7B using our synthetic data, and evaluated them primarily on logical and numerical reasoning benchmarks. Extending to a wider range of models, including larger-scale LLMs and diverse architectures, as well as more benchmarks such as standard general-purpose ones, would further validate the generality of our synthesizer and potentially yield more substantial improvements in reasoning performance.
- •
*Data-related limitations*: First, the relationships in our synthesized worlds (e.g., “$a$ visits $b$”) do not carry real-world semantic meaning, a limitation also common in prior work on logical synthetic datasets (Clark et al., 2020; Tafjord et al., 2020; Morishita et al., 2023; 2024). Currently, there is no effective solution to this issue other than relying on expert annotation (Han et al., 2024a; b). To mitigate ambiguity, we avoid introducing synonyms within the same sample, though deeper semantic grounding remains an open challenge. Second, our current synthesizer includes only conjunction and implication as logical structures, and four basic types of numerical expressions. While the synthesizer is inherently extensible, incorporating more complex logical constructs and numerical operations is left to future work.

## Appendix C Related Work

### C.1 Logical and Numerical Reasoning Datasets

Logical reasoning and numerical reasoning represent two of the most fundamental reasoning types in NLP (Wang et al., 2022; Yu et al., 2023), and considerable effort has been made in logical reasoning (Clark et al., 2020; Li et al., 2022) and numerical reasoning (Wei et al., 2022; Li et al., 2023).

Datasets involving a single type of reasoning, such as logical reasoning, including Ruletaker (Clark et al., 2020),
ProofWriter (Tafjord et al., 2020),
ReClor (Yu et al., 2020), AR-LSAT (Zhong et al., 2021), FOLIO (Han et al., 2024a), PFOLIO (Han et al., 2024b), LOGIGLUE (Luo et al., 2023), FLD (Morishita et al., 2023), FLDx2 (Morishita et al., 2024) and LogicNLI (Tian et al., 2021), as well as numerical reasoning, including MAWPS (Koncel-Kedziorski et al., 2016), DROP (Dua et al., 2019), MathQA (Amini et al., 2019), GSM8k (Cobbe et al., 2021), FinQA (Chen et al., 2021b), SVAMP (Patel et al., 2021), MATH (Hendrycks et al., 2021), AIME (Veeraboina, 2023), and Omni-MATH (Gao et al., 2025) require models to identify logical or numerical structures within text and to perform logical or numerical operations. These datasets do not provide scenarios that simultaneously require logical and numerical reasoning.

*A notable exception is RuleArena* (Zhou et al., 2025), which, to the best of our knowledge, is the only existing benchmark that explicitly combines logical and numerical reasoning in natural language.
It includes three real-world inspired tasks (taxation, airline luggage fee calculation, and NBA transaction validation) that require applying logical rules together with numerical computations.
While RuleArena makes an important contribution by integrating these two forms of reasoning, it adopts a fixed set of rules and scenarios with limited linguistic variation, resulting in repetitive task patterns and offering little control over the logical and numerical complexity of problem instances. This limits its suitability for probing models at varying levels of difficulty or for covering a broader diversity of reasoning scenarios.
Furthermore, although the original work reports rule-level recall and precision (providing a coarse form of process evaluation), it does not perform fine-grained, step-by-step assessment of intermediate reasoning stages.
Finally, RuleArena is positioned purely as an evaluation benchmark, with no intended support for targeted training data generation, which further constrains its extensibility.

These limitations motivate the development of more *flexible* and *scalable* synthesis frameworks. Our proposed LogiNumSynth addresses these gaps by supporting precise control over reasoning complexity, synthesizing richer and more varied natural language formulations, and incorporating structured intermediate reasoning steps for fine-grained process-level diagnosis. The dual applicability of LogiNumSynth for evaluation and targeted training enables systematic improvement of models’ joint logical-numerical reasoning capabilities.

### C.2 Benchmarking Reasoning Capabilities

RuleTaker (Clark et al., 2020) evaluated RoBERTa (Liu et al., 2019) using synthetic logical reasoning data. Based on this, FLD (Morishita et al., 2023) and LogicNLI (Tian et al., 2021) extended the expressiveness of logical reasoning by introducing more deduction rules and evaluated more models.
There are also works that evaluated LLMs such as ChatGPT (Brown et al., 2020) and GPT-4 (OpenAI, 2023) using logical reasoning problems from standard exams (Liu et al., 2023; 2020; Yu et al., 2020; Wang et al., 2022).
HumanEval (Chen et al., 2021a), MMLU (Hendrycks et al., 2020), BIG-bench (Srivastava et al., 2022), and AIME (Veeraboina, 2023) have become standard benchmarks to evaluate the foundational language and reasoning capabilities of language models.

In contrast, we *incorporate joint logical-numerical reasoning while minimizing confounding factors such as background knowledge*, to ensure that the evaluation results more purely reflect models’ reasoning capabilities. Furthermore, we conduct an evaluation of 29 models, with parameters from 0.5B to 685B, to observe the differences in their reasoning capabilities.
In addition, unlike existing synthetic datasets (Clark et al., 2020; Morishita et al., 2023; Tian et al., 2021; Morishita et al., 2023; 2024), which used a limited set of templates for natural language generation, we *employ hundreds of templates and an LLM to generate diverse, syntactically accurate, and semantically coherent natural language descriptions* for problem instances.

## Appendix D Features of the Data Synthesizer

Our data synthesizer exhibits diversity, purity, and extensibility, as described below.

### D.1 Diversity

Our data synthesizer achieves diversity through controllable scale and difficulty, specifically in world richness, depth of reasoning, and arithmetic computation.

#### World Richness

has two aspects: scale and density. Scale describes the size of the world through counts of entities, attributes, and relationships. Density reflects the complexity of interconnections among entities’ attribute values and relationships, as established by the numbers of facts and rules. In worlds with higher density, these elements are more tightly intertwined, enabling richer and more extensive inference chains. By varying scale and density, our synthesizer can produce knowledge bases ranging from simple, sparsely connected worlds to highly interwoven reasoning environments.

#### Depth of Reasoning.

The difficulty of reasoning can be controlled by specifying the required depth of logical reasoning.

#### Arithmetic Computation.

The difficulty of arithmetic computation is tuned through types of expressions (from constant expression to nested aggregation expression), their sampling distribution ratios, and ranges of operands.

### D.2 Purity

Unlike existing datasets derived from real applications which essentially evaluate a hybrid of reasoning capability and knowledge capacity of a reasoning model, we seek to decouple reasoning from knowledge so that our evaluation can be focused on reasoning capability. This is reflected in our synthesizing process, where world elements are combined randomly so that the knowledge stored in a reasoning model (e.g., commonsense knowledge learned during pre-training) is unlikely to help. Only the facts and rules provided should be useful. It helps to support less biased evaluation and offer a clearer view of the reasoning performance of models, representing an aspect that is pivotal for advances in areas where reasoning is prioritized over knowledge retention and retrieval.

### D.3 Extensibility

Our data synthesizer has a high degree of extensibility.
In particular, the expressions in the rules are designed to be extensible. As mentioned in Section [2.1](https://arxiv.org/html/2510.11031v1#S2.SS1), we currently incorporated only four types of expressions. However, our synthesizer is adaptable to the inclusion of additional mathematical expressions such as exponentiation and trigonometric functions. We recommend selecting different expressions based on the specific requirements to evaluate the model’s capabilities in the corresponding scenarios.

Also, as mentioned in Section [2.1](https://arxiv.org/html/2510.11031v1#S2.SS1), our rule condition is a conjunction of one or more atoms. It can be extended to support richer logical operators (e.g., disjunctions $\lor$ and negations $\neg$), thereby inducing more complex reasoning process.

## Appendix E Details on Data Synthesization

We present the method for synthesizing formal representations in Appendix [E.1](https://arxiv.org/html/2510.11031v1#A5.SS1), additional templates and details in Appendix [E.2](https://arxiv.org/html/2510.11031v1#A5.SS2), and the configuration settings for dataset synthesis in Appendix [E.3](https://arxiv.org/html/2510.11031v1#A5.SS3).

### E.1 Details on Formal Representation Synthesization

The sample synthesization process outlined in Section [2.2](https://arxiv.org/html/2510.11031v1#S2.SS2) consists of four sequential stages. Following the initialization of world elements and query, the subsequent two stages construct the formal representation $\langle\texttt{Facts},\texttt{Rules},\texttt{Query}\rangle$ as defined in Section [2.1](https://arxiv.org/html/2510.11031v1#S2.SS1), which will be illustrated in the following. The final stage generates corresponding natural language descriptions, with implementation details provided in Section [2.2](https://arxiv.org/html/2510.11031v1#S2.SS2) and Appendix [E.2](https://arxiv.org/html/2510.11031v1#A5.SS2).

Figure: Algorithm 1 Reasoning DAG Construction (Part 1/2)

Figure: Algorithm 2 Reasoning DAG Construction (Part 2/2)

#### Overall DAG Construction.

Starting from world elements, query and configurations, we first construct a reasoning directed acyclic graph (DAG) to represent the reasoning process required to answer the query. The DAG is built in a backward, top-down, and recursive manner. The root node is the query, and each non-leaf node represents an intermediate conclusion that can be inferred by applying a rule to its child nodes. Each leaf node corresponds to a fact or a rule that can be directly used to infer its parent node. The depth of the reasoning process is controlled by specified configuration, and the DAG is constructed until the desired depth is reached.

#### Rule Synthesis.

To synthesize a rule for inferring an intermediate conclusion or the query, we first randomly decide the number of atoms in the rule condition, which is sampled from a range predefined by configuration. Each atom can be either an attribute fact or a relationship fact, where the entities are anonymized as a variable and can be instantiated with concrete entities during rule application. For an attribute fact, we randomly select an attribute from the attribute set $\mathbf{A}$ of world elements. For a relationship fact, we randomly select a relationship from the relationship set $\mathbf{R}$. The conclusion of the rule is anonymized by the target intermediate conclusion or the query, so that it can be inferred when the rule is applied. Specially, for an attribute fact in the rule conclusion, an expression is needed to compute the attribute value.

To synthesize an expression, we first randomly select the type from the supported expression types with probabilities predefined by configuration, and then sample the parameters or other components required by the selected expression type. For calculation expressions, we randomly select the parameters $k$ and $b$ from the ranges specified in the configuration. For aggregation expressions, we uniformly select the aggregation operator from the supported set and recursively synthesize the two sub-expressions based on the predefined probabilities of expression types in the configuration.

It is important to ensure that the anonymized entities in the rule conclusion also appear in the rule condition, to avoid introducing new entities during rule application.

#### Recursive Node Building.

After synthesizing a rule to infer an intermediate conclusion or the query, more intermediate conclusions may be needed to infer the atoms in the rule condition. We recursively synthesize rules or direct facts for these intermediate conclusions until the desired depth is reached. The decision to synthesize a rule or a fact is made uniformly at random, except at the maximum depth (where a fact must be synthesized) or for the last node at a given level (where a rule is required to ensure progression toward the desired depth).

The algorithm for constructing the reasoning DAG is shown in Algorithm [1](https://arxiv.org/html/2510.11031v1#alg1) and Algorithm [2](https://arxiv.org/html/2510.11031v1#alg2). The main procedure begins with the ConstructDAG function, which initializes an empty state comprising the DAG $\mathcal{G}$, facts Facts, rules Rules, and inferable conclusions $\mathcal{C}$. It then recursively builds the DAG starting from the query node via BuildNode. In BuildNode, the synthesis strategy depends on the current depth: at the maximum depth, a fact is synthesized; otherwise, a rule is mandatory for the last node at that level to extend the depth, while for others, the choice between rule and fact is randomized with equal probability.

#### Conflict Detection and Validation.

Every time we synthesize a rule or a fact, we ensure that it does not lead to multiple, inconsistent derivations when combined with the existing rules and facts, thereby maintaining uniqueness of the inferred results. Specifically, we maintain a set of all conclusions that can be inferred by the existing rules and facts, and ensure that the new rule or fact, along with its triggered conclusions, does not conflict with them. Additionally, we also ensure that the reasoning DAG remains acyclic, i.e., no intermediate conclusion needs to be used to infer itself. If a conflict is detected or a cycle is introduced, we discard the newly synthesized rule or fact and re-synthesize it until no conflict or cycle is found.

The SynthesizeFact function generates a fact for the node and iteratively checks for conflicts by temporarily updating the conclusions using UpdateConclusions. Once a valid fact is found, it updates the state accordingly. Similarly, SynthesizeRule generates a rule, verifies it for conflicts and cycles, and upon validation, recursively calls BuildNode for each atom in the rule’s condition to build prerequisite substructures.

#### Termination of UpdateConclusions .

UpdateConclusions computes the iterative closure of a monotone consequence operator $T:2^{\mathcal{U}}\to 2^{\mathcal{U}}$, where one step updates the current conclusion set $\mathcal{C}$ by
$T(\mathcal{C})=\mathcal{C}\cup\mathrm{Infer}(\mathcal{C},\texttt{Rules})$.
The universe $\mathcal{U}$ consists of (i) attribute assignments $\text{is}(e,a,v)$ and (ii) binary relations $r(e_{i},e_{j})$. We impose the restriction that for any fixed entity-attribute pair $(e,a)$, at most one value $v$ is admissible. Given finite sets of entities $\mathbf{E}$, attributes $\mathbf{A}$, relations $\mathbf{R}$, and numeric values $\mathbb{V}$ (bounded by current facts and rule parameters), $\mathcal{U}$ is finite, with
$\mathcal{U}=\{\text{is}(e,a,v)\mid e\in\mathbf{E},a\in\mathbf{A},v\in\mathbb{V}\}\cup\{r(e_{i},e_{j})\mid r\in\mathbf{R},e_{i},e_{j}\in\mathbf{E}\}$
and cardinality at most $|\mathbf{E}||\mathbf{A}|+|\mathbf{R}||\mathbf{E}|^{2}$ for binary $r$.
Since $(2^{\mathcal{U}},\subseteq)$ is a finite lattice and $T$ is monotone, the ascending chain $\mathcal{C}_{0}\subseteq\mathcal{C}_{1}\subseteq\cdots$ must stabilize after finitely many steps, yielding the least fixed point reachable from the initial state. Conflicts are detected during closure and rejected, ensuring the result is both comprehensive for conflict checks and consistent.

#### Distraction Synthesization.

We synthesize additional, potentially irrelevant, facts or rules to distract the reasoning process.
These are randomly generated and added to the number specified in the configuration, while ensuring they do not conflict with existing knowledge.
This increases the complexity of the knowledge base, making it harder for models to locate and use the information needed to answer the query.

Table: Table 6: Prompt template and examples we used in the second step of natural language generator.

### E.2 Details on Natural Language Generator

Example templates used in the first step of natural language generator for different formal representations are as follows:

- 1.
Attribute fact (i.e. $\text{is}(e_{i},a_{j},num)$):
•
the value of $\{a_{j}\}$ for $\{e_{i}\}$ is $\{num\}$.
•
$\{num\}$ is recorded for $\{e_{i}\}$ in the $\{a_{j}\}$ attribute.
•
the $\{a_{j}\}$ property for $\{e_{i}\}$ is given by $\{num\}$.
•
the $\{a_{j}\}$ field of $\{e_{i}\}$ is represented by $\{num\}$.
- 2.
Relationship fact (i.e., $r_{k}(e_{i},e_{j})$):
•
it can be said that $\{e_{i}\}$ $\{r_{k}\}$ $\{e_{j}\}$.
•
the $\{r_{k}\}$ correlation is present between $\{e_{i}\}$ and $\{e_{j}\}$.
•
in the context of $\{r_{k}\}$, $\{e_{i}\}$ and $\{e_{j}\}$ share a connection.
•
the relationship $\{r_{k}\}$ defines a connection between $\{e_{i}\}$ and $\{e_{j}\}$.
- 3.
Implication (i.e., $condition$ $\rightarrow$ $conclusion$):
•
{$conclusion$} can be deduced from {$condition$}.
•
{$conclusion$} is a natural consequence of {$condition$} being true.
•
given {$condition$}, {$conclusion$} follows.
•
if {$condition$}, {$conclusion$}.
- 4.
Retrieval expression (i.e., $e_{\alpha}[a_{i}]$):
•
the value of {$a_{i}$} for {$e_{\alpha}$}.
•
property {$a_{i}$} of {$e_{\alpha}$}.
•
the value associated with the attribute {$a_{i}$} of {$e_{\alpha}$}.
•
the value corresponding to {$a_{i}$} within {$e_{\alpha}$}.
- 5.
Calculation expression (addition, i.e., $k\times e_{\alpha}[a_{i}]+b$):
•
multiplying {$e_{\alpha}[a_{i}]$} by {k} and adding {$b$}.
•
multiplying {$e_{\alpha}[a_{i}]$} by {k} with an addition of {$b$}.
•
scaling {$e_{\alpha}[a_{i}]$} by {k}, then adding {$b$}.
•
adding {$b$} to {k} times {$e_{\alpha}[a_{i}]$}.
- 6.
Aggregation expression (max, i.e., $\texttt{max}(\mathrm{expr1},\mathrm{expr2})$):
•
the larger of {$\mathrm{expr1}$} and {$\mathrm{expr2}$}.
•
the maximum value between {$\mathrm{expr1}$} and {$\mathrm{expr2}$}.
•
the greater of {$\mathrm{expr1}$} and {$\mathrm{expr2}$}.
•
the maximum of {$\mathrm{expr1}$} and {$\mathrm{expr2}$}.

In the second step, we used Qwen3-8B with thinking mode disabled, running on NVIDIA RTX 3090 and RTX 4090 GPUs with the vLLM framework (Kwon et al., 2023) and PyTorch 2.x. The sampling parameters were: temperature = 0.7, top p = 0.8, and top k = 2. The prompt template and few-shot examples are shown in Table [6](https://arxiv.org/html/2510.11031v1#A5.T6).

Table: Table 7: Detailed configurations of the synthetic datasets in our experiments.

### E.3 Detail on Synthesization Configurations

We synthesized 7 datasets for distinct purposes; their detailed configurations are summarized in Table [7](https://arxiv.org/html/2510.11031v1#A5.T7). The first four (EL-EN, EL-HN, HL-EN, HL-HN) target the evaluation of general model reasoning. The exHL-HN dataset is designed to stress‑test the best two LLMs: GPT5-mini and DeepSeek-R1; for this dataset, we explicitly synthesized four subsets with reasoning depths from 7 to 10, each containing 100 samples, rather than sampling depths uniformly. The last two (EL-Train, EN-Train) are training resources for strengthening reasoning capabilities of smaller models.

The sizes of the world element sets $\mathbf{E},\mathbf{A},\mathbf{R}$ are governed by #Entities, #Attributes, and #Relationships. The distraction module expands the knowledge base until the specified #Facts and #Rules are reached. World richness (scale and density) is thus jointly shaped by world sizes and the counts of facts and rules. The Reasoning Depth parameter defines the sampling range for the target inference depth that the DAG constructor is required to realize. #Condition gives either a fixed number or a uniform range for the atoms in a rule body (e.g., [2,3] means sample uniformly from $\{2,3\}$).
Expression Weights define the sampling distribution over expression types (constant, retrieval, calculation, aggregation). For example, “0 0 1 1” permits only calculation and aggregation, equally likely. Aggregation Weights analogously define the distribution of sub‑expression types (constant, retrieval, calculation) inside an aggregation (e.g., “1 1 1” = uniform). The intrinsic parameters of calculation expressions ($k,b$ in $expr(e_{\alpha},a_{i})=k\times e_{\alpha}[a_{i}]+b$) and the attribute value ranges in facts are controlled by Oprand Range. For instance, “[-100, 100]” means uniform sampling from $\{-100,-99,\dots,99,100\}$; negative $b$ implicitly yields a subtraction form. Finally, Size is the number of synthesized samples in the dataset.

Our synthesizer offers independent control over structure, interaction density, reasoning depth, rule complexity, numeric difficulty, and dataset size, which supports precise diagnosis during evaluation and serves as a targeted training resource to enhance reasoning capabilities.

Table: Table 8: Prompt template we used for evaluating LLMs.

## Appendix F Implementation Details

We summarize additional implementation details, including the evaluation settings (Appendix [F.1](https://arxiv.org/html/2510.11031v1#A6.SS1)), the process accuracy implementation (Appendix [F.2](https://arxiv.org/html/2510.11031v1#A6.SS2)), and the training settings (Appendix [F.3](https://arxiv.org/html/2510.11031v1#A6.SS3)).

### F.1 Evaluation Settings

We accessed the GPT series (GPT-4o and GPT5-mini), DeepSeek series (DeepSeek-V3 and -R1), and three models from the GLM4 series (GLM4-airx, -0520, and -plus) via API calls with default hyperparameters. For other LLMs, experiments were conducted on NVIDIA RTX 3090 and 4090 GPUs using vLLM (Kwon et al., 2023) and PyTorch 2.x. The sampling parameters were set to temperature = 0.3, top p = 0.8, and top k = 20. The prompt used is listed in Table [8](https://arxiv.org/html/2510.11031v1#A5.T8), and the few-shot examples, synthesized along with the corresponding datasets, can be found at [https://github.com/nju-websoft/LogiNumSynth](https://github.com/nju-websoft/LogiNumSynth).

Table: Table 9: Prompt template for extracting structured reasoning outputs, where {attributes_list} and {relations_list} denote the attribute set $\mathbf{A}$ and relationship set $\mathbf{R}$, respectively, provided as guidance for the extraction.

### F.2 Implementation of Process Accuracy

Our data synthesizer synthesizes tasks whose reasoning processes can naturally be represented as a reasoning DAG, where each node denotes a known fact, a rule, or an intermediate conclusion (also treated as a fact). As shown in Figure [1](https://arxiv.org/html/2510.11031v1#S2.F1), a non-leaf node is derived jointly from all the nodes pointing to it.
With the gold-standard reasoning steps produced during synthesis, our dataset is inherently suitable for evaluating a model’s reasoning process and gaining deeper insights.

To achieve this, we adopt a dual-extraction approach that combines a text-based parser with an LLM-based structured extractor. Both components operate in parallel and cross-reference each other to accurately reconstruct the reasoning process in a structured form.
For the LLM-based extraction, we employ vLLM (Kwon et al., 2023)’s structured output mode, which enforces JSON-formatted outputs during decoding. Specifically, we use Qwen3-8B with the temperature set to 0 and the thinking mode disabled. The prompt used for this structured extraction is provided in Table [9](https://arxiv.org/html/2510.11031v1#A6.T9).

After extracting the structured reasoning process of the tested model, we verify each reasoning step by checking:
(1) whether the facts and intermediate conclusions used satisfy the conditions of the corresponding rule, and
(2) whether the derived conclusion is correct.
We then compare all verified correct steps with the gold-standard reasoning steps.

If the model fails to produce the correct final answer, its score is determined by the proportion of intermediate conclusions from the gold-standard reasoning that it correctly derives (i.e., the reasoning steps are verified), regardless of whether it follows the exact same reasoning path.
If the model produces a fully correct reasoning chain that leads to the correct final answer, even if its steps differ from the gold-standard sequence, we assign the maximum score.

Process accuracy requires a model not only to produce the correct final answer, but also to present a clear, logically sound, and verifiable reasoning process. This is particularly important in scenarios such as automated theorem proving, multi-step medical diagnosis reasoning, and financial auditing, where the transparency, rigor, and correctness of each step are as critical as the final result. Unlike answer accuracy, which is either 0 or 1, process accuracy can award partial credit based on the correctness of intermediate steps. Thus, even if the final answer is wrong, models demonstrating partially correct reasoning can still receive proportionate scores.

**Table 10: Benchmark-specific learning rates applied during fine-tuning on our dataset for evaluation purposes.**
|  |
| --- |
|  |

### F.3 Training Settings

We fine-tuned the models using PyTorch 2.x, Transformers and Peft (Wolf et al., 2020), adopting the LoRA configuration of r = 32, lora_alpha = 64, and lora_dropout = 0.05. The target modules included
q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj.
The batch size was set to 8, and the learning rates for different benchmarks are listed in Table [10](https://arxiv.org/html/2510.11031v1#A6.T10.tab2).
All experiments were conducted on NVIDIA RTX 3090 and RTX 4090 GPUs.

For the Llama model, we utilized the Recall Adam Optimizer (Chen et al., 2020) with the following hyperparameters: $\beta_{1}$ = 0.9, $\beta_{2}$ = 0.999, $\epsilon$ = $1\times 10^{-8}$, anneal_fun = sigmoid, anneal_k = 0.2, anneal_t0 = 100, anneal_w = 1.0, and pretrain_cof = 3000 for the benchmarks: GSM8k, MATH, MATHQA, SVAMP, MAWPS, AIME, RuleTaker, ProofWriter, LogiQA, ReClor and AbductionR. For all other benchmarks (FOLIO and FLD), the standard Adam Optimizer, with $\beta_{1}$ = 0.9, $\beta_{2}$ = 0.999, and $\epsilon$ = $1\times 10^{-8}$, was employed.

For the Qwen model, the Recall Adam Optimizer was applied to the benchmarks: GSM8k, MATH, MATHQA, SVAMP, MAWPS, AIME, LogiQA, and RuleArena with identical hyperparameters. The standard Adam Optimizer was used for the remaining benchmarks, including RuleTaker, ProofWriter, FOLIO, FLD and AbductionR.

We used different synthesized training datasets depending on the benchmark category: (1) models evaluated on mathematical reasoning benchmarks were fine-tuned on the synthesized EL-Train dataset to enhance numerical reasoning skills, whereas (2) models evaluated on logical reasoning benchmarks (including RuleArena) were fine-tuned on the synthesized EN-Train dataset to enhance logical reasoning skills.

During fine-tuning, we used the validation set provided with each benchmark. If no validation set was available, we randomly sampled a subset from the train split. For the AIME and MAWPS benchmarks, which do not include official validation and train splits, we instead used the test set of ASDiv (Miao et al., 2020) as the proxy validation set. Similarly, for the RuleArena benchmark, which does not include official validation and train splits, we utilized the test set of AR-LSAT (Zhong et al., 2021) as the proxy validation set.

For evaluation, we used vLLM (Kwon et al., 2023) with sampling parameters set to temperature = 0.3, top_p = 0.8, and top_k = 20.

Table: Table 11: Performance of LLMs under varying few-shot settings. ‘Proc’ refers to process accuracy and ‘Ans’ refers to answer accuracy.

## Appendix G Additional Experiment Results

### G.1 Effect of Few-Shot Example Quantity Compared to Zero-Shot

To further investigate how varying the number of in-context examples affects model accuracy, we conducted additional experiments with $k\in\{0,1,2,3,4,5\}$ shot settings across 6 selected evaluated models. The results are summarized in Table [11](https://arxiv.org/html/2510.11031v1#A6.T11). Consistent with the observations reported in Section [4.1](https://arxiv.org/html/2510.11031v1#S4.SS1), the performance differences across different $k$ do not exhibit a consistent trend: in some cases, adding examples yields notable improvements, whereas in others it leads to performance drops. This indicates that, for the highly complex reasoning tasks, increasing the number of in-context examples does not guarantee better performance and may sometimes even be harmful.

### G.2 Representative Cases

As described in Section [4.1](https://arxiv.org/html/2510.11031v1#S4.SS1), we conducted a case study. Below, we present representative examples illustrating both the robustness and advantage of process accuracy, as well as three typical types of process errors.

#### Robustness and Advantage of Process Accuracy.

Case [1](https://arxiv.org/html/2510.11031v1#A7.SS2.SSS0.Px2) and Case [2](https://arxiv.org/html/2510.11031v1#A7.SS2.SSS0.Px2) demonstrate situations where the model’s reasoning process followed a different order from the gold-standard reasoning process, yet process accuracy was correctly awarded. Case [3](https://arxiv.org/html/2510.11031v1#A7.SS2.SSS0.Px2) illustrates that when the model produced the relationship *hampers*, which differs from the gold-standard relationship *hamper* but shares the same lemma, the evaluation correctly handled this case. Case [4](https://arxiv.org/html/2510.11031v1#A7.SS2.SSS0.Px2) presents an example where the model derived the correct answer through an entirely different reasoning path than the gold-standard one, essentially finding a shortcut. This is possible and considered correct because our synthesizer guarantees logical consistency (i.e., no derived facts, including the query, contradict each other), while allowing the existence of multiple valid reasoning paths. These cases collectively highlight the robustness of our process accuracy evaluation.

Moreover, in Case [2](https://arxiv.org/html/2510.11031v1#A7.SS2.SSS0.Px2), the model made an error only in the final computation step. As a result, answer accuracy assigned a score of 0, whereas process accuracy awarded $\tfrac{5}{6}$. Similarly, in Case [5](https://arxiv.org/html/2510.11031v1#A7.SS2.SSS0.Px2), the model correctly derived an intermediate step that contributed toward the final answer, and process accuracy appropriately gave partial credit. In Case [1](https://arxiv.org/html/2510.11031v1#A7.SS2.SSS0.Px2), however, the model miscalculated $25913$ as $26047$, but due to the presence of an expression $\min(88\times 26047-96,-69)=-69$, the final answer happened to be correct. In this case, answer accuracy assigned a score of $1$, while process accuracy awarded $0.6$, successfully identifying the error.
These cases collectively highlight the advantage of process accuracy over answer accuracy. Whereas answer accuracy provides a coarse, binary evaluation, process accuracy offers a more fine-grained and informative assessment: it not only recognizes partially correct reasoning steps but also distinguishes between coincidentally correct answers and genuinely correct reasoning, thus offering a more faithful measure of a model’s reasoning ability.

#### Cases of Typical Process Errors.

Case [6](https://arxiv.org/html/2510.11031v1#A7.SS2.SSS0.Px2) and Case [7](https://arxiv.org/html/2510.11031v1#A7.SS2.SSS0.Px2) illustrate *incorrect application of rules*. In Case [6](https://arxiv.org/html/2510.11031v1#A7.SS2.SSS0.Px2), the model mistakenly applied an irrelevant fact, while in Case [7](https://arxiv.org/html/2510.11031v1#A7.SS2.SSS0.Px2) it omitted a relevant fact; nevertheless, both cases still led to the correct intermediate or final conclusion.
Case [1](https://arxiv.org/html/2510.11031v1#A7.SS2.SSS0.Px2) is an example of *incorrect numerical computations*, where the model made an error in an intermediate calculation, but the mistake was masked by a subsequent $\min$ operation, resulting in the correct final answer.
Additionally, Case [8](https://arxiv.org/html/2510.11031v1#A7.SS2.SSS0.Px2) and Case [9](https://arxiv.org/html/2510.11031v1#A7.SS2.SSS0.Px2) demonstrate *incorrect intermediate results*. In Case [8](https://arxiv.org/html/2510.11031v1#A7.SS2.SSS0.Px2), the error arose from a lexical mistake in an entity, while in Case [9](https://arxiv.org/html/2510.11031v1#A7.SS2.SSS0.Px2), it resulted from reversing the direction of a relationship.

## Peer Reviews

confidence: 3
contribution: 1
initial_rating: 2
presentation: 2
questions: While this paper, with its proposed methods, is clearly scientifically inspired, it at least needs a discussion of its implicit assumptions, expectations, and potential limitations to function as a stand-alone conference publication.
soundness: 2
strengths: Overall, the paper is clearly written and easy to follow. The experiments and analyses appear to be fully documented.
summary: This paper introduces LogiNumSynth, a natural language problem synthesizer for logical and numerical reasoning tasks. It demonstrates its usage through the creation of four datasets, based on different combinations of both easy and hard numerical and logical tasks, which are then evaluated on several LLM. It is shown that fine-tuning LLM on the synthetic data can improve its performance on several reasoning benchmarks.
weaknesses: Unfortunately, the paper lacks innovative ideas, impactful contributions, and analytical depth. It is unclear what purpose the proposed method and analysis can serve meaningfully.

1.  The abstract states that LogiNumSynth should serve as a diagnostic tool and a source of datasets for fine-tuning, but it provides no explanation or insight into why this should significantly improve LLM performance or reduce systematic wrong behavior. The slight to moderate performance gains could be attributed to fine-tuning on a somewhat related domain with respect to the reasoning benchmarks.
2.  The diagnostic possibilities are limited to CoT (process) and answer evaluation on logical and numerical tasks of varying complexity, with no theory on how to explain particular good or bad performance or an indication of how to categorize or assess different failure modes. 
3.  There is also no theoretical discussion on why current LLM struggle with numerical and logical reasoning or on the possible limitations of fine-tuning for solving these problems. This makes it difficult to place this work in a scientific context, which is necessary for an impactful publication.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See the weaknesses
soundness: 3
strengths: It introduces LogiNumSynth, a novel tool that can generate a wide range of joint logical-numerical reasoning problems with fine-grained control over complexity, moving beyond fixed and limited existing datasets.

The framework is designed not just for evaluation but also for targeted training, demonstrating its value as both a diagnostic tool to identify model weaknesses and a means to improve them.

This paper presents a large-scale evaluation of 29 models, demonstrating that training with synthetically generated data enhances model performance on external reasoning benchmarks.
summary: This paper introduces LogiNumSynth, a synthesizer for creating natural language tasks that require combined logical and numerical reasoning, with fine-grained control over difficulty. The tool is used to evaluate and diagnose specific weaknesses in large language models' joint reasoning capabilities. Furthermore, the synthesized data is shown to be effective for targeted training, improving the models' performance on these challenging reasoning tasks.
weaknesses: The performance improvement from SFT on the synthetic data appears to be marginal (approximately 1%).

Experimental results for SFT on larger-scale models (>=7B) are lacking.

How was the correctness and quality of the synthetic data validated? It is recommended to conduct a small-scale human evaluation.

How diverse is the synthetic data? Is there a trade-off between the diversity of the data and its correctness?

---

confidence: 4
contribution: 1
initial_rating: 0
presentation: 3
questions: ## Revise Empirical results
* Clarify in the main body that the model is also fine-tuned on the downstream dataset.
* Just train your model only on the LogiNumSynth train set (EN-train+EL-train) and check for downstream improvements on the out-of-domain datasets
* If your domain scope is to narrow and the model overfits, you can compare your extensive training run with a model trained only on the downstream benchmarks (excluding LogiNumSynth).
* Also evaluate on MMLU and see whether you can maintain performance or the degradation is not too large, and on inductive logical reasoning to see whether you generalize OOD (e.g., SLR-Bench or CLUTRR)?
* You might also want to compare your tuned model to other logic-tuned models (e.g. 'AIML-TUDA/Llama-3.1-8B-SLR'). If you do so, remeber to also evaluate on OOD dataset, both models have not been trained on.
* Also, add how well the tuned model performs on the test set of LogiNumSynth.

## Add clarifications
* How is the overlap between train/test and within the dataset prevented?
* Specify number of samples of LogiNumSynth datasets in the main body of the paper?
* How is correctness of a solution verified? exact match?
* Also report the selected “reasoning effort” parameter for GPT-5-mini?
* Have you considered using RL-based reasoning training (e.g., GRPO, GSPO, ...) on LogiNumSynth?

While I remain unconvinced by some technical aspects, I find the general idea promising and the problem important. If the authors can address my concerns, I would be open to reconsidering my score.
soundness: 1
strengths: * **Relevance and Motivation:** The focus on combining logical and numerical reasoning is well-motivated. It tackles a structured synthesis approach that could help evaluate or improve models’ reasoning abilities.
* **Control over Task Complexity:** Allowing adjustable difficulty and reasoning depth is a valuable feature for both evaluation and curriculum-based training.
* **Process and Answer Evaluation:** Assessing reasoning steps, not just final outputs, aligns with current trends in verifiable reasoning.
summary: The paper introduces LogiNumSynth, a synthetic data generator designed to create logical and numerical reasoning tasks for language models. Its main goal is to bridge the gap between these two types of reasoning. The framework offers fine-grained control over logical depth and numerical complexity, allowing the generation of tasks across multiple difficulty levels.

The authors present a dataset composed of 4+1 subsets covering easy/hard logical and easy/hard numerical reasoning, along with an additional “extra hard” subset that combines both. They evaluate language models trained on LogiNumSynth to measure their logical and numerical reasoning capabilities. Furthermore, they train models on both downstream benchmarks and their own dataset, demonstrating improved performance on the downstream benchmarks they have trained on.
weaknesses: * **Lack of Clarity in Training Setup:**
  The description of the training and evaluation procedure is inconsistent. While the paper claims *out-of-domain evaluation* after fine-tuning on LogiNumSynth, Appendix F.3 suggests that the models were directly trained on downstream benchmarks (e.g., GSM8K, LogiQA). This critical detail is omitted from the main text, giving the impression that improvements stem from LogiNumSynth training, when in fact they may result from training on the downstream datasets themselves. Table 5 reinforces this concern. The current wording misleads readers about the dataset’s generalization effects and must be corrected. 

If this interpretation is incorrect, I invite the authors to clarify. However, if my understanding is accurate, the current presentation seriously misleads readers. This is a major concern and I therefore must strongly recommend rejection unless this issue is convincingly addressed.

* **Unclear Novelty Claim:**
  The claim of being the *first dataset* to integrate logical and numerical reasoning is questionable. Numerical operators are already intrinsic to many logical reasoning tasks [1, 2, 3]. The authors should explicitly compare LogiNumSynth to prior datasets that also integrate logical and numerical reasoning.

* **Limited Training Scope:**
  The paper does not evaluate whether fine-tuning on LogiNumSynth improves or maintains performance on downstream reasoning benchmarks not included at training timesuch as ProntoQA, CLUTRR, or SLR. Such experiments would be essential to demonstrate the dataset’s real-world usefulness beyond its synthetic scope.

* **Missing Rationale for Separate Training:**
  The authors train logical and numerical subsets separately, despite positioning their work as an integration of the two. A joint training setup would be a more natural test of the dataset’s stated purpose and could reveal cross-domain reasoning benefits.

* **Insufficient Explanation of Data Quality:**
  The paper provides little evidence regarding the semantic coherence or plausibility of the synthesized samples. If tasks are generated purely through random sampling, they risk being logically inconsistent or meaningless. Including representative examples and a qualitative analysis of data quality would strengthen the paper’s credibility.

---

**References**

[1] Helff et al., *SLR: Automated Synthesis for Scalable Logical Reasoning*

[2] Ma et al., *KOR-Bench: Benchmarking Language Models on Knowledge-Orthogonal Reasoning Tasks*

[3] Liu et al., *SynLogic: Synthesizing Verifiable Reasoning Data at Scale for Learning Logical Reasoning and Beyond*


Decision: Reject