Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thought
Authors: Qiguang Chen
Libo Qin
Jiaqi WANG
Jingxuan Zhou
Wanxiang Che
Venue: NeurIPS 2024 oral (2024)
Keywords: Chain-of-Thought
Reasoning Granularity
Reasoning Boundary
Primary area: natural_language_processing

Abstract: Chain-of-Thought (CoT) reasoning has emerged as a promising approach for enhancing the performance of large language models (LLMs) on complex reasoning tasks. Recently, a series of studies attempt to explain the mechanisms underlying CoT, aiming to deepen the understanding of its efficacy. Nevertheless, the existing research faces two major challenges: (1) a lack of quantitative metrics to assess CoT capabilities and (2) a dearth of guidance on optimizing CoT performance. Motivated by this, in this work, we introduce a novel reasoning boundary framework (RBF) to address these challenges. To solve the lack of quantification, we first define a reasoning boundary (RB) to quantify the upper-bound of CoT and establish a combination law for RB, enabling a practical quantitative approach applicable to various real-world CoT tasks. To address the lack of optimization, we propose three categories of RBs. We further optimize these categories with combination laws focused on RB promotion and reasoning path optimization for CoT improvement. Through extensive experiments on 27 models and 5 tasks, the study validates the existence and rationality of the proposed framework. Furthermore, it explains the effectiveness of 10 CoT strategies and guides optimization from two perspectives. We hope this work can provide a comprehensive understanding of the boundaries and optimization strategies for reasoning in LLMs. Our code and data are available at https://github.com/LightChen233/reasoning-boundary.

Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thought
Qiguang Chen
Libo Qin Jiaqi Wang Jinxuan Zhou Wanxiang Chel-
i Research Center for Social Computing and Information Retrieval
Harbin Institute of Technology
School of Computer Science and Engineering, Central South University & The Chinese University of Hong Kong
carr (irr hit. edu. cn, lbqin@csu. wwwllcsuuedu.coom cn
Abstract
Chain-of-Thon Thought (CoT) reasoning has emerged as a promising approach for enhancing the performance of large language models (LLMs) complex reasoning tasks. Recently, a series of studies attempt to explain the mechanisms underlying COT,, aiming to deepen the understanding of its efficacy. Nevertheless, the existing research faces two major challenges: (1) a lack of quantitative metrics to assess CoT capabilities and (2) dearth of guidance on optimizing CoT performance. Motivated by this, in this work, we introduce a novel reasoning boundary framework (RBF) to address these challenges. To solve the lack of quantification, we first define a reasoning boundary (RB) to quantify the upper-bound of CoT and establish a combination law for RB, enabling a practical quantitative approach applicable various
CoT tasks. To address the lack of optimization, we propose three categories of RBs. We further optimize these categories with combination laws focused on RB promotion and reasoning path optimization for CoT improvement. Through extensive experiments on 27 models and 5 tasks, the study validates the existence and rationality of the proposed framework. Furthermore, it explains the effectiveness of 10 CoT strategies and guides optimization from two perspectives. We hope this work can provide comprehensive understanding of the boundaries and optimization strategies for reasoning in LLMs. Our code and data are available at https: / thub. http://iggtcche223/raaaaoo bwwna//ononnonnnnnnbbbrrrrrrrrr
Introduction
In recent years, Large Language Models (LLMs) have demonstrated increasing capabilities and applications across various tasks [Zhao et al., 2023, Chang et al., 2023, Pan et al., 2023, Qin et al., 2024a]. Notably, advanced LLMs, such as GPT [Brown al., 2020, OpenAI, 2022, 2023], PaLM [Anil et al., 2023] and LlaMa [Touvron et al., 2023a,b, Meta, 2024] series have demonstrated emergent capabilities, particularly like Chain-of-Thouseee
(CoT) [Nye et al., 2022, Wei et al., 2022]. This methodology enables models to verbalize step-by-step reasoning, thereby enhancing prediction accuracy by basing decisions the logical rationale (Wei al., 2022, Kojima al., 2022, Hu et al., 2024, Qin et al., 2023, Zhuang et al., 2023, Chen et al., 2024a].
Recently, some research in the literature has begun to investigate the mechanism of CoT to enhance the understanding of its operational nature. To this end, Madaan et al. [2023] and Wang et al. [2023a]
Coorresponiing Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

first give qualitative boundary conclusion through a large number of experiments on the natural language planning capability: The CoT limited by the reasoning logic the context demonstrations. Bi et al. [2024] investigate these boundaries on the code planning capability, by training L.Mss on CoT samples of varying difficulties. demonstrates L.MSs are unable to learn or effectively manage tasks that exceed certain complexity upper-bound. To delve deeper into potential constraints of COT, Feng et al. [2024] develop a theoretical framework on the single-step calculation capability, suggesting that there is an upper-bound of model performance dependent on the length of input in single-step reasoning processes. Although existing research has made some progress, where the boundaries of CoT lie and how these boundaries affect the performance of CoT are still unresolved questions. Specifically, the existing work still faces two major challenges:
Lacking quantification metries for CoT: Current research primarily relies on qualitative assessments of CoT performance, which leads to the absence of quantitative metrics. It hinders the ability to objectively compare different CoT approaches and establish definitive upper-bound for CoT capabilities.
Lacking optimization guidance for CoT: While current research prioritizes understanding the mechanisms underlying CoT reasoning, there is a dearth of guidance on optimizing CoT performance. This gap hinders the transformation of CoT research into actionable strategies for enhancing model capabilities.
Motivated by this, in this work, we introduce a reasoning boundary framework (RBF) to thoroughly examine and optimize the boundaries of current LLMs. Specifically, to address the quantification challenge, we propose a new concept, named reasoning boundary (RB) to quantify the upper-bound on task-specific reasoning complexity within model. Furthermore, scenarios, we present the combination law of RBs to generalize the RB for quantification in more real and complex scenarios. To address the CoT optimization challenge, we propose and analyze three reasoning boundary intervals, guiding optimization through improved RB and optimized reasoning paths based on the combination law, which achieves state-of-the-art performance in our proposed benchmark. We extensively validate the efficacy of our framework across 27 models and 5 tasks: arithmetic computing, mathematical reasoning, multi-hop question answering, and multilingual mathematical reasoning.
explore more practical
Our main contributions are as follows:
To the best of our knowledge, this is the first work to propose a reasoning boundary framework (RBF) to quantify the upper-bound of COT. Furthermore, we establish the combination law of RB as the weighted harmonic mean of fundamental RBs to address practical CoT tasks.
To solve the lack of CoT optimization, we define three categories of RBs. Based on the combination law and the nature of these RBs, we effectively improve the existing CoT strategies by RB promotion and reasoning path optimization.
We validate the existence and rationality of our framework on 27 models and 5 CoT tasks. Fur- thermore, we explain the optimal performance from two optimization perspectives in numerous CoT strategies. We consider both optimal perspectives and propose a minimum acceptable reasoning path (MARP) prompting to achieve state-of-the-an performance.
2
Quantification Methodology
2.1 Reasoning Boundary
In order to quantify the capacity for complex reasoning in LLMs, we introduce an upper-bound concept termed reasoning boundary (RB), which formally defines the degree of ease that an LLM can handle within a specific reasoning process. In simpler terms, as shown in Figure 1 (a), RB reflects the limit beyond which a model's accuracy significantly degrades. Mathematically, RB is defined for model m and a task t as the maximum of problem difficulty d at which the model's 's accuracy reaches a predefined threshold K1:
B Ar K, (1)m = suc(d] m) < Ki). d
where Acc(L]d, m) represents the accuracy of the model's accuracy on task with difficulty d. Difficulty can be measured by factors like the number of reasoning steps or computational complexity. For brevity, we denote RB as B(t]m) in subsequent sections.

---

(a) Reasoning Boundary ($2.1)
(b) Combination Law of Reasoning Boundary ($2.2)
Arithmeticnl Calculation Planning + Markemazical Reasowing
B Accesox
Acc=90%
(1+2)*3 4--
V.M
Arithmetical Cal- (B(o))
Weighted Harmonic Mean
Reasoning
Mathematical
p))
B Accesox
5cc=50%
1+2=7; 3+3=
calculltte that Joan is 1+2-3
Step
you
(112-21)'32
399
Natural Language Planning RB (B(p))
years old.
Step Then you need
B Acc=sox
Acc=10%
calculltte that Jack is 3*3-9
Stepl:
First you need to...
years
(1231231))332-99
1,209,653
Step2: Then you need
(c) Categories of Reasoning Boundary ($2.3)
Acce90%
B 10%sAcc=x0x
B Accs10%
Reasoning Boundary
Completely Feasible Reasoning Boundary (CFRB)
Partially Feasible Reasoning Boundary (PFRB)
Completely Infeasible Reasoning Boundary (CIRB)
Figure I: Overview of the introduced concepts.
Conclusion:
The reasoning boundary for a model defined by its ability to achieve specific accuracy for given task difficulty.
2.2 Combination Law of Reasoning Boundary
In practical scenarios, models often require the integration multiple capabilities to address single task effectively. To quantify how a large language model can be boosted by the cooperation of multiple capabilities through the CoT mechanism, we introduce the "Combination Law of RB", giving concrete formula of the upper-bound of the COT. The law estimates the unified reasoning boundary BACc=K, (t1, .2............................ to (m) for n tasks within a model m, which is formulated as:
B BAC==K K, (f1, .2...., tn (m) R
(2)
BAce- (Pa m/) (m) b;
where BA (f) denotes the reasoning boundary of model m for task f.. N,, and b; are scaling factors, which are only affected by the related task. As shown in Figure I (b), Equation (2) provides a mathematical formula estimate the combined RBs from the independent ones, enabling deeper insights into model behavior for intricate tasks. See Appendix A for detailed mathematical analysis.
Furthermore, the combination law for reasoning boundary demonstrates favorable theoretical prop- erties, with broad upplicability across diverse scenarios and flexibility in accommodating various boundary segmentation methods. For detailed practical application, please refer to Appendix B.
The combination law of reasoning boundary satisfies the weighted harmonic average of each basic reasoning boundary.
Conclusion:
2.3 Categories of Reasoning Boundary
Furthermore, in order to guide the optimization of CoT and more convenient expression, as shown in Figure (c), we define the following three categories of RBs based on their empirical accuracy:
Completely Feasible Reasoning Boundary: We define that the part with an accuracy greater than 90% is a completely feasible reasoning boundary (CFRB = Ba (t1, ,,, (m)m) which means that LLMs can effectively grasp the performance of this part.
Completely Infeasible Reasoning Boundary: We believe that the part with an accuracy less than 10% is completely infeasible reasoning boundary (CIRB BACe 10% (t1, f2...., ftmmm) which means that the model can never effectively grasp the performance of this part.
Partially Feasible Reasoning Boundary: We define the RB in the rest part except CFRB and CIRB as a partially feasible reasoning boundary (PFRB = B10% < 90%% (41, (1,,2,,.. /2,..., the model to repeat thinking more clear information to solve the problem.
.,, In, [m)). which requires
3

---

100
Correct
Incorrect
0 CFRB
PFRB
O CIRB
80
2000.
Accuracy (%)
60
60
1500.
by 1000
-2e6
x²y
40
500-
20
x-y
-2:2e5
500
1000
1500
2000
9
11
13
15
11 13 15
The number of planning step (B(p))
The number of planning step (B(p)) (e) Distribution of cenect predictions for code plamning.
(a) Distribution of correct predictions for x*y samples.
(b) Distribution of correct predictions for nature language planning.
Figure 2: Existence Verification for Reasoning Boundary. Figures (b, c) present evaluations performed on BigGSM, where the reasoning paths are manually analyzed to identify the specific steps at which errors occur, without considering whether the final conclusions are correct.
We analyze the nature these three categories of RB in detail (in Section 4.3), and further utilize the combination law to optimize these three reasoning boundaries (in Section 5), so to provide effective suggestions and guidance to support future CoT optimization.
Experimental Setup
Benchmark Settings To assess the reasoning boundaries of LLMs, we require a dataset rich in RB. This necessitates tasks with evenly distributed complexities and reasoning steps that challenge the models' upper-bounds. To meet these requirements, we introduce BIGGSM, a new dataset offering greater calculation complexity and longer reasoning chains. The detailed construction process for BIGGSM is provided in Appendix C.
Model Settings Except for model expansion experiments, all experiments are conducted GPT- 3.5-Turbo. Following the setting of Wei et al. [2022], in our CoT experiment, all multi-step reasoning tasks utilize three manually constructed demonstrations. In addition, for all the experiments, top-p selected from (0.95, 1}. Temperature selected from [o, 1] and serves as the main error variable.
Empirical Analysis of Reasoning Boundary
4.1 Existence Verification for Reasoning Boundary
In this study, we investigate the hypothesis that an LLM exhibits varying levels of reasoning boundary across various tasks. To this end, we will verify whether the model has widespread reasoning boundary in various tasks in the following three tasks:
Basic Arithmetic Calculation First, to investigate the existence of RB, we first examine basic arithmetic operations (including addition, subtraction, multiplication, and division). As illustrated in Figure 2 (a), the results reveal significant performance variations across three distinct regions. For multiplication, accuracy surpasses 90% for results up 2.2e5. Conversely, accuracy falls below 10% for products exceeding 2e6. Similar presences of varying RBs are observed for other operations, which verifies the existence of reasoning boundary in basic arithmetic calculation tasks. Further results and implementation details are provided in Appendix D.
Nature Language Planning We further investigate RB in natural language planning tasks for mathematical reasoning. We prompt the model to generate plans and assess their accuracy through manual evaluation. There is a strong correlation between the number of reasoning steps and LLMs' performance in Figure 2 (b). When the model meets the question with fewer than reasoning steps, accuracy surpasses 90%. Conversely, when reasoning steps exceed 4, accuracy falls below 10%. This finding suggests that there are also three different RB categories in natural language planning tasks.
Code Planning For further extensive exploration, we further prompt LLMSs by PAL [Gao et al., 2023] to generate code-format plans and evaluate them by manual annotation. As shown Figure (c), the code planning task is similar to natural language planning, which is also an obvious division

---

Correct sample
* Incorrect sample O CFRB
O PORB
O CIRB
Maximam multipiication
calcalation value (m))
1eg.
Maximum multip liaation
calcalation value (2R)))
The number of entitt (1B(0))
Bc4
2.545
6e4
245
ge4-
1,585
38000
2e4
10
20
16
10 20
30
40
The number of calculation step (B(s))
The number of planning step (B(p))
The number of hop (B(b)
(4) The combination law of different reasoning bounduries in complex calculation task.
(b) The combination law of different
(c) The combination law of different reasoning boundaries mulli--hp
reasoning boundaries in muthematical reasoning task.
questio--onswering task.
Figure 3: Combination law verification of RB on different tasks. More verification results on other tasks are shown in Figure 12.
and different categories of RBs. Notably, since code planning utilizes code for clearer logic and reduced expression complexity, its planning accuracy surpasses that of natural language planning.
4.2 Combination Law Verification on Different Tasks
Combination Law in Complex Arithmetic Calculation Building on the proof of Equation (13), we hypothesize that the combination law for RB in the complex arithmetic calculation is the harmonic average of the arithmetic calculation RB and calculation planning RB. To verify this, we designed an experiment focusing on formulas containing addition, subtraction, and multiplication, like "(1 + 2) x 3 - 4", Since addition and subtraction complexities are assumed to be around 1e15 (as shown in Figure 13), the arithmetic calculation RB primarily depends on the multiplication RB and calculation planning RB. Therefore, shown in Figure 3 (a), there two obvious RB lines, namely B A0cc900% and B 10% which are completely consistent with the combination law of these basic RB based on the Equation (2). Besides, these two lines also clearly divide the RBs into three categories.
Combination Law in Mathematical Reasoning Inspired by Tan [2023b], Xiao and Liu [2024], we posit that the natural language mathematical CoT task is determined by two sub-tasks: step planning task and step calculation task for global logic planning and local mathematical calculation. Furthermore, each model output step requires a single basic operation, resulting in a step calculation boundary close to the maximum number of multiplications, denoted by B(c) R B(m). Formally, with step planning RB denoted by (B(p)) and the step calculation RB by (B(c)), then the combined RB satisfies the following law:
BC (c,
N
(B(c)-b1)
(B(p)-b2)
As illustrated in Figure 3 (b), the actual performance distribution of RB (including B Acc=90%% and Bocc-10% 10%) in natural language mathematical reasoning task fully aligns with the proposed combination law in Equation (3). Additionally, there are also obviously three RBs in Figure 3 (b).
Combination Law in Multi-hop Reasoning Beyond the realm of mathematics, we further extend our exploration of the combination law to the field of multi-hop question answering. Specifically, we validate our law on HotpotQA [Yang et al., 2018], where we define the reasoning boundary as the combination of global hop- plaanninn RB and local knowledge entity reasoning RB. As shown in Figure? (c), B 90% and B Aco 10% also satisfy the weighted harmonic mean of these two sub-reasoning boundaries. It is also proved that, in addition to answering also satisfies our proposed combined law and also exhibits three distinct RBs. We will describe in detail how to calculate the combination law on multi-hop reasoning in Appendix E.
tasks, multi-hop question
4.3 Nature Analysis for different Reasoning Boundary
According to the definition of different RBs, we have divided the problem into three parts for LLMs. In this section, we will verify whether the defined RB adheres to the intrinsic nature of the model itself. We will discuss the natures of these RBs in detail:
CFRB means complete mastery of the model even without demonstration. According to the definition, we assume that question within CFRB implics comprchensive understanding of the
5

---

CARR
PARR
C
[1 CHRB [1] P9RR CICI
ACFR
APORER
ACIRR
95
84.1
100
90
89.7
90.7
91.5
Accuracy (%))
75
Accuracy (%6
80
Accuracy (%)
Accuracy (%)
65
65
54-9
$3.0
$4.8
50.5
55
50.0
50
12.7
9-5
9-5
0.5
9.5
CFRB PERB CIRB
6y%
Reasoning Boundary Type
+4
(a) The accuracy distribution of
The size of self-consistency
(e) The accurncy (top) and quantity (bottom) distribution synthetic samples Synthetic.CoT.
generated rationales based on Auto-
(b) Model self-consistency integrated porformance dif-
ferent Reasoning Boundary areas.
Figure 4: Nature analysis at different reasoning boundaries. For Figure (c), all demosntrations in CIRB special value points obtained by calculation methods similar to 25000 x 1000. In fact, real CIRB demosntrations are obtained.
associated issue for certain LLM. To verify this, following Zhang et al. [2022] and Wei et al. [2022], we formulate a mathematical request and generate thain-of-thoughe rationale and answer through zero-shot prompting without any demonstration. As shown in Figure 4 (a), it still achieves 29.2% improvement in CFRB on generating the correct rationale compared to other RBs. This also proves that the model can indeed master tasks well on the questions in CFRB.
PFRB means moderate confidence in its solution and needs consensus building process. To gauge the level of performance and confidence, we draw parallels to human becision-making.. where moderate confidence often necessitates multiple times consensus building. Inspired by this, we investigate it on Self-Const we--oCnnsseeene [Wang et al., 2022], which integrates results from various reasoning answers to reach a conclusive answer. Figure (b) demonstrates that as the integration of reasoning paths increases, the accuracy improves significantly within PFRB compared with other RBs. This suggests that within PFRB, the LLM exhibits moderate confidence in solving problems, which needs multiple consensus building.
CIRB exhibits poor reasoning performance even with consensus building. As illustrated in Figure 4 (a), questions in CIRB display extremely low accuracy (around 9.5%). And the model shows consistently poor performance and no improvement on Figure 4. It signifies that the model exhibits poor reasoning performance.
in this boundary in
LLM has self-awareness of its own RBs. parallel, natural question arises: Is the model capable of discerning its inherent RBs? To investigate this, we employ the Synthetic-CoT [Shao et al., 2023] to prompt LLM generate CoT data. As depicted in Figure (c), the results demonstrated that there are over 65% of generated samples within CFRB, which achieves much higher percentage and performance than other RBs. This suggests that LLMs possess an intrinsic understanding of their RBs and constraints to generate the task they grasp. indicative of a potential for self assesmetttttt
Takeaways:
Reasoning boundary (RB) and the combination law of RB both widespread across series of tasks. (2) Different categories of RB can reflect the corresponding performance, and the model can also have a self-understanding its own RB.
5 RB-based CoT Optimization
5.1 How can improve CoT by optimizing RB?
Based on our framework, the reasoning boundary limits the performance the model. The simplest approach to improve CoT is to optimize the step calculation RB B(c) to promote the value of RB. Specifically, Tool-Usage [Paranjape al., 2023) and Program-of-Thoughl (PoT) [Chen et al., 2024b) have shown significant success in CoT optimization. We explain the rationale behind their effectiveness, why PoT consistently outperforms direct Tool Usage [aao al., 2023, Chen al., 2023], and take them examples to demonstrate how to improve CoT by promoting RB.

---

BIGGSM
Theoretical interval
18
Model
Acc. (1)
Input Token (1)
Output Token (
Practical value - B
18
CoT
57.00 $0.03
780.43
96.76 #8.22
0.8
RB- AB-Oppimezed Methods
B(c, p)
Tool Usage
71.64 :0.00
688.43
129.53 =3.82
PoT
78.25 #1.09
657.43
78.25 #1.00
0.4
Reasoning-Patte Optimized Methods
Least-to-moat
58.25 #3.28
679,59
176.09
1: 22 131,82
Complex- CoT
59,78 $0.00
1111.43
COT+MARP
64.37 #2:24
614.43
95.12 60.77
B 2
B260%
B Be20%
B B-10%
PoT+MARP
80.55 #2.40
576.43
76.34 #2.84
Figure 5: Analysis of the impact of Tool- Usage and PoT on reasoning boundary (c.,p).
Table 1: Main experimental results on GPT-3.5-Turbo.
Results on different benchmarks are shown in Table 2.
Tool Usage can boost the value of RB for an LLM. When the model uses tools [Paranjape et al., 2023], we can simply think that the model can perform calculations with infinite precision, so that the RB of mathematical calculations tends to infinity, viz B(c) > +00. It is obvious that the combined RB of the model can be calculated as:
B(p) - b2 N₂
BToo1 (c.p) =
lim N -(c)-+ (B(c)-b))
(4)
(B(p)-b)
Easy to get, [TToo (c. B (c, this shows that Tool Usage can improve the boundary of reasoning. This explains why Tool Usage can have better performance than vanilla CoT (as shown in Table 1). Furthermore, shown in Figure 5, the distribution of theoretical RB and the actual one almost perfectly coincide. This also demonstrates the reliability and applicability of our theory.
Program-of-Thouggee can further enhance the value of LLM's RB. Equation (4) reveals that an LLM's RB hinges entirely on its planning capability. Since natural language can be verbose, hinders the planning capability of LLM [Gao et al., 2023, Hu et al., 2023, Puerto et al., 2024, Chen et al., 2024b]. PoT [Chen al., 2023] offers clearer representation of logic using code, allowing for clearer planning (as shown in Figure 2 (b, c)). This leads to finer- mner-grained planning reasoning 1 (p) B(p). Then the PoT reasoning boundary 5 (c. p) > gTool (). aligning with the observed performance gains of PoT over Tool Usage (sce Table 1). Furthermore, Figure 5 visually demonstrates that PoT's theoretical and practical reasoning boundaries consistently outperform Tool Usage. This reinforces the theoretical advantage of PoT and its empirical effectiveness.
5.2 How can we improve CoT based on a certain RB?
Enhancing RB is crucial for optimizing CoT, but requires changes to the model or its reasoning architecture to be effective. Therefore, we need to consider how to optimize the reasoning path so that the difficulty satisfies the RB (d" Acc= ) instead of the original RB (d = B Bece= K, where K₂ < K₁. According to Equation (3), B is affected by both arithmetical RB and planning RB. Given B, we consider optimizing reasoning ability from the following two strategies examples" 2:
Complex CoT (CCoT): By increasing the boundary of planning to reduce the pressure of single-step calculation, reduce the arithmetical RB, and then get smaller d; However, it introduces more planning steps, which adds the planning pressure. As shown in Figure 6, the model performance first increases and then decreases with the increasing number of CCoT steps.
Least-to-Moss (LtM): By dividing multiple sub-questions reduce the pressure of local planning within sub-question, reduce the boundary of local planning, and then get smaller d. However, even though it can release local planning pressure (as demonstrated in Figure 7), this approach simultancously intensifies global planning pressure by generating an excessive number of sub- questions (as depicted in Figure 15).
Limitation: (1) CCoT needs to keep balance in the number of reasoning steps and calculation pressure. (2) Although the pressure of local planning has been reduced, LtM has not effectively reduced the pressure of global planning, nor the pressure of optimization calculations.
"See detailed analysis for these strategies in Appendix F.
7

---

GP1
Manual Complex GoT
Inl-aawinn--0g + GPT35
... AnmerGee Complex OIT
test dont--oonn-
Jhatitoff-Tooghh Prompinng Lecet-ito-toet Prompting
90
05.8
80
Accuracy (%)
70
51-9
Accuracy
60
93
Accuracy
50
42.5
42.4
46-1
30
40
92
30.0
30
91
20
90
6
The number steps
The number of steps
0-100
6eg-1.205
1.865-1.8e5
3-453-305
3.5e5 4-225
multiplication calculation (2())
Figure 6: Correlation between the number of steps and performance of Complex-CoT on GSM8K (left) and SingleEq (right). See Appendix for more meta eecaa-analysis results.
Figure 7: The performance distribution of Least-to-Most prompting on different calcula- tion amounts.
Minimum acceptable reasoning paths prompting can further achieve better CoT within spe- cific RB. To address the aforementioned two issues, we proposed Minimum Acceptable Reasoning Paths (MARP). Our first objective is to alleviate the computational burden of the model. We achieve this by introducing instructions that set an upper limit on its single-step computational capacity, thereby optimizing the boundary of its computational reasoning. Secondly, aim enhance the model's acceptability. Within the calculation and planning boundary, we increase the amount of much possible while simultaneously
computation performed in each step in demonstrations reducing the number of global planning steps, which effectively mitigates planning pressure. As shown in Table MARP demonstrably improves model performance and effectively reduces the token consumption. By maximizing operations per step, MARP leads to a more streamlined and efficient problen-solving process. Detailed descriptions of this strategy are shown in Appendix G.3.
(1) Tool-Usage and PoT can be utilized to optimize CoT by the calculation and planning reasoning boundary optimization. (2) MARP can well lessen planning and calcula- tion pressure by problem optimization in certain RB (3) Users can effectively optimize CoT performance by optimizing the reasoning boundary and the problem.
Takeaways:
Expansion Verification & Exploration
RB can be extended to various models. To extend our mechanism's 's applicability, we verify the mechanism on 25 diverse models (details in Table 3). As shown in Figure 8 (a), we observe positive correlation between reasoning boundary and model accuracy on mathematical benchmarks.
* General LLM @ Math LLM
Closed L1M O f Open LLM
BigGSM
+ GSMBBI
BigGSM
MATH
BigOSM
95
80
Accuracy (%))
Accuracy (%)
Accuracy (%))
65
50
50
35
35
20
0.4
0.2
0.4
0.6
0.8
0.102 0.104 0.106 0.188
Acee10%
B AcC-1U
B Acca900%
Concelation values Bace-1on for different general LLMs and performance real benchmarks.
(b) Correlation Bacecios for different math LLMs and performance real benchmarks.
values CIRB
Correlation between
2 Raccoos for different closed and open L. and performance benchmarkss.
Figure Correlation between the values of RB for different models and performance real benchmarks. See Appendix H for more empirical details.

---

LLaMA Code-LLaMA
0.14
0.12
(ooa Sep.. sourragee Seteemee somejaryy
BACCC10
0.10
300000
0.08
400000
0.06
0.04
0.02
The mumber plaming Happ (Rip))
0.00
30
40
50
70
Language Performance (8(1)
3o
50
Parameters (B)
Figure 9: Scaling law correlation between model parameters and CIRB.
Figure 10: Different boundaries MGSM.
Moreover, the models that use mathematical data such as MathInstruct for SFT, often have interesting outliers that are different from the general LLMs' area, but they also satisfy positive correlation with our RBs (as shown in Figure 8 (b)), which helps determine if the model underwent mathematically targeted training.
However, as shown in Figure 8 (c), we find some interesting phenomena. For example, the main difference between the current open-source model and the closed-source model is still CFRB. Except for the closed source model, the CFRB of all models is O. shows the potential and the direction of the model optimization. Furthermore, a scaling law of RB can also emerge shown in Figure 9): reasoning boundary increased with model parameter count and data quality.
RB be extended to more tasks. To assess the RB in multilingual mathematical reasoning task. Inspired by Qin et al. [2024b], we hypothesize that multilingual RB, assessed through direct answer accuracy across different languages, mathematical computation RB, represented by the maximum product result, and reasoning planning RB, indicated by the planning steps, are orthogonal dimensions of performance. We propose that these RBs be effectively combined using a weighted harmonic mean. As illustrated in Figure 10 confirms that the combined RB maintains the expected three different RBs. Detailed implementation description is shown in Appendix I.
tasks, evaluate them
7 Related Work
In this section, review recent literature related to Chain-of-Thought (CoT) prompting, focusing theoretical and empirical investigations. Madaan et al. [2023]. Wang et al. [2023a], Saparov and He [2023], He- Yueya et al. [2023], Zhang et al. [2024], Wang et al. [2024] and Prystawski et al. [2024] qualitatively show that the LLMs learn the reasoning chain based on the demonstrations the context. Besides, Lampinen et al. [2022] and Tan [2023a] find causal link between generated intermediate steps and the final answers during a series of qualitative experiments. Wang et al. [2023c], Hanna et al. [2024] and Dutta et al. [2024] study neural substructure within the LLMs, embodying CoT reasoning from a white-box mechanism perspective, demonstrating that LLMs deploy multiple parallel answer generation paths internally.
Recently, large amount of work has demonstrated the upper-bounds and limitations of LLM in various CoT tasks [Qin et al., 2023, Imani al., 2023, Huang et al., 2024, Sprague et al., 2024]. Bi et al. [2024] investigate these bounds on planning capability in code generation by training LLM on CoT samples of varying difficulties. Their findings suggest that LLMs have limited capacity to learn or manage tasks exceeding a certain complexity threshold. Further understanding of the CoT upper-be soper-bound, Merrill and Sabharwal [2023], Li et al. [2023] and Feng al. [2024] analyze single-step arithmetic capability, which suggests an upper bound on model performance related to input length in single-step reasoning processes.
Despite advancements in CoT explanation for L.MMs, significant challenges remain, including the absence of quantifiable metrics for CoT's upper-bounds and the deficiency in optimization guidelines.

---

Correct sample
Incurrect sample O CRRB
C PERB
O CIRB
Masiiaum moltipicction calcalatioo value (B(cn)
3e5
Maximmu multiplication
calcslation (88ta)))
365
1800
Maximum multipiicttion
calcolation valuve (2) (m))
345
1900.
2.345
1000
2.545
6:00 ae
2-945
245
8x
3e5
640
245
4.585
crass vaare
ceose 30000
aI 4
1,945
20084
1.595
385
165
5e4
3C0 MK
5e1
13
16
number planning step (R()))
The number planning stop (B(p))
The number of plansing step (1()))
(a) G-r-b.--tubo
(b) GPT-4o
(c) O1-preview
Figure 1I: Combination law verification of reasoning boundaries GPT-series models.
To tackle this, we propose a reasoning boundaries framework (RBF) to systematically quantify and optimize various CoT approaches. This framework offers transferable and user-friendly methodology to enhance model performance from a mechanistic perspective. We anticipate that will furnish systematic insights for ongoing research and inform future developments in the field.
Discussion
Discussion on the Boundaries Improvements Furthermore, in order to better understand the best existing LLMs, we utilize RBF to test the current most advanced GPT-series models. As shown in Figure 11, all reasoning boundaries improve lot compared to the last version which also achieves performance enhancement. Notably, the CFRB increases slightly compared with the improvement of CIRB between GPT-3.5 and GPT-4o. But ol significantly improves the CFRB. Furthermore, as shown in Figure 14 in Appendix, ol shows extremely significant improvements on CFRB, which is almost three times of other models. We attribute it to the fact that the advanced Reinforce-Learinng and Inference Scaling strategies play key role in improving this part of the ability compared with the normal improvements in CFRB, which might trigger more in-depth research.
Broader impacts. Our framework is the first work to quantify the reasoning upper-bound of LLMs. This enables the explanation for huge part of the valid CoT framework. We hope that our work can provide new insights and more systematic guidance for future interpretability analysis of COT. For social impact, this work may have a certain impact on the controllable and explainable AGI.
Limitations & Future. Due to the cost and time constraints, this work does not discuss the complex relationships such as causal conditions among the basic RBs. addition, evaluating the robustness and applicability of CoT reasoning boundaries-related techniques in dynamic scenarios will be crucial for future research.
Conclusion
This study introduces a novel reasoning boundaries framework (RBF) to quantify and optimize the limitations of LLMs in CoT tasks. Specifically, we propose the concept of reasoning boundaries (RBs) and the combination law of RBs in more complex scenarios for quantitative metrics. We further introduce three categories of RB for CoT optimizations. The framework is validated through extensive experiments across 27 models and 5 tasks. Furthermore, we improve the CoT in both RB and question optimization perspectives to achieve state-of-the-re of-the--rt art performance in BIGGSM. We hope that this framework paves the way for further research on understanding and enhancing LLMs' reasoning capabilities.
Acknowledgments
This work was supported by the National Natural Science Foundation of China (NSFC) via grant 62236004, 62441603, 62476073 and 62306342. This work was also sponsored by the Excellent Young Scientists Fund in Hunan Province (2024JJ4070), the Science and Technology Innovation Program of Hunan Province under Grant 2024RC3024, and the CCF-Zhipu.A Large Model Innovation Fund
10

---

References
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. Palm 2 technical report. arXiv preprint arXi::2305. 10403, 2023.
Anthropic. The claude 3 model family: Opus, sonnet, haiku. //wwu- odn.cddnnnn aath.nnttopiic...coo http:///cw.............a.aaaaoaa /Model_ Ward_Claude_3 ...add_3.pp. pdf.
2024. URL https:
Zhen Bi, Ningyu Zhang, Yinuo Jiang, Shumin Deng, Guozhou Zheng, and Huajun Chen. When do program-of-thought works for reasoning? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17691-17699, 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Yupeng Chang, Xu Wang, Jindong Wang. Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 2023.
Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. C A novel benchmark for multi-domain multi-step multi-modal chain-of-thought. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 8199-8221, Bangkok, Thailand, August 2024a. Association for Computational Linguistics. doi: 10. 10.18655//1/2022.all-oo URL https::///// htR.htts:://aclaaaa haap::::///aaaaholooooyy wwaellooyyoogg//// 2rgg/022. acl- -oogg. 446.
Weize Chen, Chenfei Yuan, Jiarui Yuan, Yusheng Su, Chen Qian, Cheng Yang, Ruobing Xie, Zhiyuan Liu, and Maosong Sun. Beyond natural language: Lims leveraging alternative formats for enhanced reasoning and communication. arXiv preprint arXiv:2402. 18439, 2024b.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research, 2023.
Daixuan Cheng, Shaohan Huang, and Furu Wei. Adapting large language models via reading comprehension. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreevie. httpeaarreieeeeeeeeeeeeeeeeeeeee http://rrrrnndyyyddememeae
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchir Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Subhabrata Dutta, Joykirat Singh, Soumen Chakrabarti, and Tanmoy Chakraborty. How to think step-by-step: A mechanistic understanding of thain-of-thought reasoning. arXiv preprini arXiv:2402. 18312, 2024.
Guhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, and Liwei Wang. Towards revealing the mystery behind chain of thought: theoretical perspective. Advances in Neural Information Processing Systems, 36, 2024.
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity- prompting for multi-step reasoning. In The Eleventh International Conference on Learning Representations. 2023. URL hww.t:ttt////seeeeeee
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. PAL: Program aided language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 10764-10799. PMIR, 23-29 Jul 2023. URL https: htteee//pooopeeeeeee mlr, ppess/v202/ ggt.2f.......m// html.

---

Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. Transactions of the Association for Computational Linguistics, 2,146-361, 04 2021. ISSN 2307-387X. doi: 10.1162/tacl ___00370. URL wtw.htttss///ii..eo http///i..........// http:/.....1aaaaaleeeee a_0070.
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?: Inter- preting mathematical abilities in a pre-trained language model. Advances in Neural Information Processing Systems, 36, 2024.
Joy He- Yueya, Gabriel Poesia, Rose Wang, and Noah Goodman. Solving math word problems by combining language models with symbolic solvers. In The 3rd Workshop on Mathematical Reasoning and Al at NeurlPS 23, 2023.
Mengkang Hu, Yao Mu, Xinmiao Chelsey Yu, Mingyu Ding, Shiguang Wu, Wenqi Shao, Qiguang Chen, Bin Wang, Yu Qiao, and Ping Luo. Tree-planner Efficient close-loop task planning with large language models. In The Twelfth International Conference on Learning Representations. 2024.
Yi Hu, Haotong Yang, Zhouchen Lin, and Muhan Zhang. Code prompting: a neural symbolic method for complex reasoning in large language models. arXiv preprint arXiv:2305.18557, 2023.
Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei. Not all languages are created equal in Ilms: Improving multilingual capability by cross- lingual-thought prompting. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12365-12394, 2023.
Jen-tse Huang, Eric John Li, Man Ho Lam, Tian Liang, Wenxuan Wang, Youliang Yuan, Wenxiang Jiao, Xing Wang. Zhaopeng Tu, and Michael R Lyu. How far are we on the decision-making of llss? evaluating Ilms' gaming ability in multi-agent environments, arXiv preprint arXiv:2403.1188 2024.
Shima Imani, Liang Du, and Harsh Shrivastava. Mathprompter: Mathematical reasoning using large language models. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pages 37 42, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.0682, 2023a.
Song Jiang, Zahra Shakeri, Aaron Chan, Maziar Sanjabi, Hamed Firooz, Yinglong Xia, Bugra Akyildiz, Yizhou Sun, Jinchao Li, Qifan Wang, et al. Resprompt: Residual connection prompting advances multi-step reasoning in large language models. arXiv preprint arXiv:2310.04743, 2023b.
Mingyu Jin, Qinkai Yu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, Mengnan Du, et al. The impact of reasoning step length on large language models. arXiv preprint 240442240..04925, 2024.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 2022.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Pruceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Andrew Lampinen, Ishita Dasgupta, Stephanie Chan, Kory Mathewson, Mh Tessler, Antonia Creswell, James McClelland, Jane Wang, and Felix Hill. Can language models learn from explanations in context? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Findings of the Association for Computational Linguistics: EMNLP 2022, pages 537-563, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/ 2022. 2022.findings-emen findings-emmlp.38. URL httpt:://ccaantaaih hcl.t/oooonhhoogg...oooo org/2022 findingseemnlp.e
12

---

Zhiyuan Li, Hong Liu, Denny Zhou, and Tengyu Ma. Chain of thought empowers transform- ers to solve inherently serial problems. In The Tweifth International Conference on Learning Representations, 2023.
Aman Madaan, Katherine Hermann, and Amir Yazdanbakhsh. What makes chain-of-thought prompt- ing effective? a counterfactual study. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings the Association for Computational Linguistics: EMNLP 2023, pages Singapore, December 2023. Association for Computational Linguistics. doi: 18.18//3/v 1/2023. findings-emnlp.10 101. URL https: acl aclaachoooggyy org/2023..2. w wewaddingg-emlpp 101.
William Merrill and Ashish Sabharwal. The expressive power of transformers with chain of thought. In The Twelfth International Conference on Learning Representations, 2023.
Meta. Llama 3. 2024.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski. Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. In Deep Learning for Code Workshop, 2022.
OpenAl. Introducing chatgpt. 2022.
OpenAl. Gpt-4 technical report, 2023.
Wenbo Pan, Qiguang Chen, Xiao Xu, Wanxiang Che, and Libo Qin. A preliminary evaluation of chatgpt for zero-shot dialogue understanding. arXiv preprint arXiv:2304.04226, 2023.
Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannanch Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. Art: Automatic multi-step reasoning and tool-use for large language models. arXiv preprint arXiv:2303.09014 2023.
Ben Prystawski, Michael Li, and Noah Goodman. Why think step by step? reasoning emerges from the locality of experience. Advances in Neural Information Processing Systems, 36, 2024.
Haritz Puerto, Martin Tutek, Somak Aditya, Xiaodan Zhu, and Iryna Gurevych. Code prompting elicits conditional reasoning abilities in text+ code 1lms. arXiv preprint wrXiv:2401110063 2024.
Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che. Cross-lingual prompting: Improving zero-shot chain-of-though reasoning across languages. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pages 2695-2709, 2023.
Libo Qin, Qiguang Chen, Xiachong Feng, Yang Wu, Yongheng Zhang, Yinghui Li, Min Li, Wanxiang Che, and Philip Yu. Large language models meet nlp: A survey. arXiv preprint arXiv:2405. 12811, 2024a.
Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and Philip S Yu. Multilingual large language model: A survey of resources, taxonomy and frontiers. arXiv preprint arXiv:2404. 04995, 2024b.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of chain-of-thouuht. In The Eleventh International Conference Learning Representations, 2023. URL https:t/////. http::///ppeeeeeeeeee hett/forrmm..ddqq/VBaaaae
Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. Syn- thetic prompting: Generating chain-of -f-thughtt demonstrations for large language models. In International Conference on Machine Learning, pages 30706-30775. PMLR, 2023.
Kashun Shum, Shizhe Diao, and Tong Zhang. Automatic prompt augmentation and selection with thain-of-thought from labeled data. Findings of the Association for Computational Linguisties: EMNLP 2023, pages 12113-12139, 2023.
13

---

Zayne Rea Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. MuSR: Testing the limits of chain-of- tfthouhh with multistep soft reasoning. In The Twelfth International Conference on Learning Representations, 2024. URL. https://openrevierr net/forum'dd--jnn//
Jiashuo Sun, Yi Luo, Yeyun Gong. Chen Lin, Yelong Shen, Jian Guo, and Nan Duan. Enhancing ohain-of-thoughts prompting with iterative bootstrapping in large language models. arXiv preprint arXiv:2304. 1657, 2023.
Juanhe (TJ) Tan. Causal abstraction for chain-of-thought reasoning in arithmetic word problems, In Yonatan Belinkov, Sophie Hao, Jaap Jumelet, Najoung Kim, Arya McCarthy, and Hosein Mohebbi, editors, Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Nerworks for NLP, pages 155-168, Singapore, December 2023a. Association for Computational Linguistics. doi: 10. 10.18653/v1/2000 1/t.:2222.2.2..0om11. 1.12. URL 2023. blackboxnlp-1112
org/
Juanhe TJ Tan. Causal abstraction for Shain--ff-thoulhe reasoning in arithmetic word problems. In Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 155 168, 2023b.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 2023.
Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Git- man. Openmathinstruct-II: A 1.8 million math instruction tuning dataset. arXiv preprint arXiv:2402.10176. 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307. 092888, 2023b.
Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models-a critical investigation. Advances in Neural Information Processing Systems, (16775993-70005. 2023.
Boshi Wang. Sewon Min, Xiang Deng. Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. Towards understanding ofain-of-thought prompting: An empirical study of what matters. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2717-2739, Toronto, Canada, July 2023a. Association for Computational Linguistics. doi: 10.18653/v 1/20231 acc.o//a.ooo....gg....o...
URL https: htwp:: http://taaaaaacoooo/o///oo org/2023 acl-long 153.
Haoyu Wang, Hongming Zhang, Yueguan Wang. Yuqian Deng. Muhao Chen, and Dan Roth. Are all steps equally important? benchmarking essentiality detection in event processes. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4048-4056, 2023b.
Qineng Wang, Zihao Wang, Ying Su, Hanghang Tong, and Yangqiu Song. Rethinking the bounds of Ilm reasoning: Are multi-agent discussions the key? arXiv preprins arXiv:2402.18272. 2024.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203. 111111,, 2022.
Yiqun Wang, Sile Hu, Yonggang Zhang, Xiang Tian, Xuesong Liu, Yaowu Chen, Xu Shen, and Jieping Ye. How large language models implement tfhin-of-thought? 2023c.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models, Advances in neural information processing systems, 25:24824-24837 2022.
14

---

Changnan Xiao and Bing Liu. A theory for length generalization in learning to reason. arXiv preprint 005600 2024.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpoiqa: A dataset for diverse, explainable multi-hop question answering. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369-2380, 2018.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. In The Twelfth International Conference on Learning Representations, 2023.
Yufeng Zhang, Xuepeng Wang, Lingxiang Wu, and Jinqiao Wang. Pattem-aware chain-of-thought prompting in large language models. arXiv preprint arXiv:2404.1481 2024.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. In The Eleventh International Conference on Learning Representations, 2022.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. survey of large language models. arXiv preprint arXiv:2303.1822. 2023.
Ziyu Zhuang. Qiguang Chen, Longxuan Ma, Mingda Li, Yi Han, Yushan Qian, Haopeng Bai, Zixian Feng, Weinan Zhang, and Ting Liu. Through the lens of core competency: Survey on evaluation of large language models. arXiv preprint arXiv:2308 07902, 2023.
15

---

Appendix
Mathematical Analysis Proof
A.1
Definitions & Assumptions
In order to further quantify and analyze the combination law of RB, we will define the concept of difficulties for different tasks:
Definition The difficulty of solving certain problem during model reasoning is an independent constant.
That is, the difficulty ((1,, satisfies:
D(t,, (2/m) = ((tilm) D(t2/m) = Ki + K2,
(5)
where, K1, K2 denotes the relevant constants. Therefore, the combined difficulty formally satisfies:
tn/mm = f..............111..............
+ ((ta(m) = ((t)lm))
(6)
Definition The RB is defined as the reciprocal of the difficulty of solving the problem. The greater the difficulty of solving the problem, the lower the RB and the smaller the feasible area.
Therefore, the combination law of RB satisfies:
1
B(b,, (ll,,te.....,//// x D(1,, l2,..., (m)m))
Definition If all basic RBs are infinite, means that all the difficulties approach to the zero and the model is omnipotent. Therefore, the combined RB is also infinite.
Formally, the combination law satisfies that:
B(+00, +o0...,, +00|m) = +
(8)
Assumption The combination law function is continuously difserentiable everywhere.
Assumption All basic reasoning boundary for combined reasoning boundary are mutually inde- pendent.
A.2 The Proof Combination Law
Based on the above definitions and assumptions, we need to prove that the combination law is combined RB and is the weighted harmonic average of two basic RBs.
Proof. Following Equation (7), we can get the D(I1, #2,1,,2....... .......,,..,... fm)m) as:
D(11, .2,.... En mm)
Σ D(O. Ii. 0mm)
(9)
According to the Taylor expansion formula, we expand this formula at x; - kis we can get:
22, #m/m) = Nuçl; -.)
(10)
Na (x; k.)) O(xs)
(11)
Na(2; kz),
(12)
16

---

where N₁ (1( _i (. We set t; = 1 + b; and
Biti.aammtt ml & D(21, 22....,. ..,,..,,,,,,,,,... mm).
Then the original formula is expressed as:
No + ku
B(t tn
ku:
(13)
No K;
1i=1
where 4 represents the specific task measurement value, and No and kg denote the linear parameters. Given the minimal change the derivative within the observable range. N₁ Nu is treated as constant N, in this task for simplicity. Experimental results show that, if sub-RBs are separated basic
No
&s and ko is typically O. Since cannot be directly quantified, independently, K; No form of [m) as quantized substitute, thus simplifying the combination law as:
B(1,, : :.lm...)..mm.)))....)... R
(14)
B(b m)
A.3 Calculation of RB in Practical Process
14
To determine the constants, we first fit parameters to a model using development dataset (or 20% of the test dataset if the development dataset is not available). This fitting process
The number of medical
entity (B()))
10
yields the corresponding constants. For given task and prompt strategy, these constants remain fixed. Additionally, the combination law constants are established, differ- ent reasoning boundaries are determined through a binary search on performance in standard setting (3-shot CoT). For instance, we use binary search to identify a reasoning boundary that ensures the accuracy of all problems below that boundary approaches 90%, achieving B Ace-90%. For one model, one task, and one prompt type, the reasoning boundary remains fixed. Zero-shot and few-shot settings share the same set of reasoning boundaries.
6
10
12
14
16
The number of planning step (B(p))
Figure 12: Extended verification of combination law Medical Knowl- edge Probing [Cheng et al., 2024] tasks.
The Application Tutorial of Reasoning Boundary
From a practical standpoint, our mechanism framework exhibits universal adaptability, making suitable for application in a wide range of scenarios. When confronted with new problem context, the framework enables a systematic approach to problem-solving A key feature of the framework is its reliance on the weighted harmonic mean, which imparts advantageous mathematical properties to its structure. Specifically, the framework operates effectively if the reasoning process can be segmented into relatively independent boundaries. This segmentation allows the framework to be fully leveraged in addressing diverse problems.
Reasoning Boundary Application. In the case of a vertical domain problem based CoT reasoning, the process can be divided into two key boundary levels: task planning and domain- specific reasoning. These can be modeled as follows:
B
(15)
where: B,, represents the task planning boundary, B,, represents the vertical domain boundary, and k a constant reflecting the degree of boundary independence.
Reasoning Boundary Definition Segmentation. Neglecting any of these boundaries only results in an increase in k, but keeping the overall efficiency of the framework. If the reasoning boundary is well-defined and independent, the value approaches zero, showcasing the effectiveness of our mechanism framework.
Further Reasoning Boundary Segmentation. Further refinement of the vertical domain boundary, B., into B, and B,g is straightforward. No additional complexity is introduced, the following
17

---

relationship holds:
B.
(16)
K2
B
Buz
Thus, the overall boundary equation can be extended to:
B
(17)
k1 k2
This formulation allows for flexible and systematic boundary division at multiple levels, enhancing the framework's practical utility across various problem domains.
Challenging Reasoning Boundary Measurement. In addition, propose alternative method to measure the reasoning boundaries. This approach allows the model to provide direct answers without relying CoT reasoning steps. By doing so, the model's reasoning process for specific task depends solely on a single reasoning boundary, which can be represented as follows:
B
(18)
K1
B
For instance, in the MGSM task, assessing multilingual reasoning boundary is particularly challenging. To address this, we directly evaluate the model's 's performance using a direct prompting strategy without CoT outputs and this performance define the multilingual reasoning boundary. which in turn helps determine the corresponding normalization constant. Subsequently, we apply multilingual CoT reasoning to the MGSM task to calculate the combined boundary using the framework's combination law. This approach provides a more generalized solution and may be more adaptable to specific needs.
Details of Dataset
C.1 Dataset Construction
To adequately assess the reasoning boundary of LLMs, it is essential to develop dataset that encompasses range of complexities and reasoning boundaries. To address these challenges, we propose novel approach to constructing mathematical reasoning dataset using manual synthesis and annotation which finally leads to the BIGGSM benchmark. Specifically, our proposed method involves the manual synthesis and annotation of mathematical reasoning dataset. The construction process includes the following steps:
Step 1: Domain Template Generation Initially, employ prompt-driven LLM (GPT-4) to generate complex scenarios necessitating multi-step calculations. This process also yields initial example templates. Specifically, the prompt given the large model is as follows:
Generate a scenario-related template involving multiple mathematical steps solve real- world problem. Ensure the scenario requires the application of different mathematical concepts. Please use "[VAR]" as variable to mark the template of the question.
Step 2: Natural Language Template Creation Recognizing that LLMs produce errors and logical inconsistencies, we refine these initial templates to improve their accuracy and add mathematical caleulations. To facilitate the generation of extended sequences, we decompose the templates into smaller, loopable segments that incrementally meet the multi-step reasoning demands.
Step Domain Template Augmentation To address the limited diversity in individual samples and provide a broader evaluation of LLMs' mathematical abilities, we use an LLM (GPT-4) to generate at least three alternative augmented templates for each original template and step. The generation prompt we use is as follows:
Create three alternative versions of the following template that introduce different complexities variables, ensuring each version demands an equivalent level of reasoning.
18

---

Correct sample * Incorrect sample O CERB
OPRB CIRB
1e16
1e16
200
1.0
2.0
150
0.8
x/y
0.6
100
=50
0.4
50
0.5
0.2
x/y
-10
0.5e5
105
1.505
2c5
0.25
0.50
0.75
1.00
0.2
0.4
0.6
0.8
1.00
1e16
1e16
(a) Distribution of correct predictions for x/y samples.
(b) Distribution of correct predictions for xty samples.
(c) Distribution of correct predictions for samples.
Figure 13: Existence verification for reasoning boundaries on basic arithmetic calculation tasks, including division, addition, and subtraction operations.
Generall LLM
Math 11M
Closed
GSM8K
RigGSM
MATH
GSMBK
RigGSM -8 MATH
95
95
80
F1-preview
Accuracy (%))
80
O1-preview
Accuracy (%)
65
65
CPT-40
P3.5
50
G3.5
5
0.2
0.4
0.6
0.8
0.1
0.2
0.3
B
Bccec10%
B BAcce90%
Correlation the values of CIRB BAnocio% for different general LLMs and performance on real benchmarks.
(b) Correlation between values CFRB Baccsson, for different closed and I.I.Ms and performance on real benchmarks.
Figure 14: Correlation between the values of RB for different models and performance on real benchmarks.
Step 4: Numeric Filling Once all templates are prepared, we aim to test the upper-bound of the LLMs' computational reasoning boundary by introducing numerical values ranging from to le5 in multiplication tasks. This step is designed to thoroughly assess the models' performance across spectrum of numerical challenges.
Step Manual Annotation ensure the quality and logical coherence of our synthetic samples, manually review them to correct any errrr introduced during the automated generation process. Finally, we hired three experts to mark whether the samples in the data set were correct. Only for those samples where more than two experts agreed did we retain the corresponding samples. The Cohen's 's kappa value marked by the experts was 0.97, which indicates the perfect agreement.
C.2 Dataset Analysis
Our dataset comprises 610 test samples, which is extensive when compared to the GSM8K dataset. It features broader range of procedural steps, varying from to 16 steps. Additionally, our dataset encompasses wider spectrum of computational efforts, ranging from 6 to 3e5.
19

---

D The Implementation Details of Basic Arithmetic Calculation
D.1 Data Construction
This section outlines the process used to construct datasets for examining the existence of reasoning boundaries (RB) in basic arithmetic calculations. Initially, we identify the operations for investigation, namely addition, subtraction, multiplication, and division. We then determine the range of integer operands (z and y). starting from to 1c10, subsequently extending to 1c20. A random number and y within the specified
generator is employed to produce independent and unbiased pairs range. For each pair, we compute the expected correct outcome of the chosen operation using standard arithmetic procedures. In addition, order ensure that decimals not affect the computational complexity, restrict our analysis to integer operands and outcomes to control for complexity and randomly generate numerical values of z and y.
D.2 Prompt Construction
The prompt configuration in our study involves inputting the structured data into computational model to analyze the arithmetic accuracy. The following prompting is used for LLMs' input:
Please calculate the formula given below: x op y=
where op denotes the arithmetic operation (selected from addition, subtraction, multiplication, division). And and y values are generated from Section D.I. The final experimental results are shown in Figure 13.
E
The Implementation Details of Multi-hop Reasoning
We propose that the natural language multi-hop CoT task comprises two sub-tasks: multi-hop planning and knowledge step reasoning for multi-hop question answering. To address the challenge of measuring knowledge difficulty, we utilize NER model to identify the number of knowledge entities in each hop, thus marking the knowledge step reasoning RB in the single-step task. Formally, let B(h) represent the RB of multi-hop planning and B(e) denote the RB of knowledge step reasoning. The combined RB satisfies the following combination law:
BoT BooT(e, hc
(19)
N (B(e)-ba) b.)
(2())-b2)
Analysis for Complex-CoT and Least-to-Most within Reasoning Path Optimization Perspective
Complex CoT Prompting can achieve better CoT within a specific RB by simplifying the calculation reasoning step. We believe that Complex CoT optimizes the performance of the model by allowing the model to reach its computational limit as much as possible in single-step reasoning. Therefore, the combined RB for Complex-Col can be expressed as:
Complox (p. c) =
lim B(c)+B 100% (c)
(20)
N (B(c)
+
N (B)()) b3)
b
Assuming the premises of RB remain unchanged Compaxx (p. c) = B (p, c)), it can obviously yield the solution (p(p) > B(p). Therefore, the model can accept more steps of reasoning boundary, that is, the planning difficulty dp is less than reasoning capability 5 (z). the accuracy is higher. In order analyze this problem, we adopted meta-analysis method. We count the performance of the work of Jin et al. [2024], Fu et al. [2023] using Complex COT. The relationship between the performance label and the number of steps shown Figure (left). For most multi-step reasoning tasks, generally speaking, within certain range, as the number of steps increases, the computational pressure of the model is relieved and the performance improved, which is consistent with the theory and exploration of Feng et al. [2024], Wang et al. [2023b], Valmeekam et al. [2023].
"baps://huggingftcc..///siii/// NER
20

---

3%
14A
70
64:32
61.9
9%
2/%
O P
+3
9%
R5
Accurasy
40
36.84
29%
46
8.5
3196
14%
64%
88
P27
1796
10
12
0.00
#210
(a) Distribution of the number of CoT steps performed for each sub question.
(b) The Accuracy Distribution on the CoT steps
(c) Distribution of the number of sub-question.
performed for each sub-question.
Figure 15: Analysis of output results for Least-to-Mon Prompting.
However, We can also clearly recognize the flaws of Complex COT. Once the difficulty of planning dp (that is, the number of planning steps) is greater than B' (p). it exceeds the capabilities of the model and the performance will decline. We can observe that for single-step calculation reasoning, as shown in Figure 6 (right), the performance of using Complex CoT will gradually decrease. The rest of mathematical reasoning will also decrease when the number of steps is greater than a certain threshold. This phenomenon calculation is reduced, the number of reasoning steps is also increasing. If the acceptable number of reasoning steps exceeded, the reasoning boundary is exceeded, and the model performance will decline, which demonstrates that is necessary to keep a balance between the number of reasoning steps and computational pressure (see Appendix for detailed meta-analysis process).
also be explained by combination law. While the amount of
Limitation: Need keep balance in the number of reasoning steps and calculation pressure.
Least-to-Most Prompting can achieve better CoT within a specific RB by simplifying the planning reasoning paths. Least-to-most prompting structures problem-solving hierarchically, by breaking questions into smaller sub-questions and further solving them one-by-one, Accordingly, Least-to-most RB be divided into three sub-RBs, namely, the problem decomposition RB ((d), the problem planning RB B(p), and the single-step calculation RB B(c). Therefore, the combined RB for least-to-most can be expressed as:
BLt (d.p.c)
(21)
Ng (B(d) ba)
(B(())-b)
(6(p)
Ideally, the problem decomposition ability of the model is excellent (B(d) + +x0), it can decompose the problem into sub-problems that be solved in step every time B(c) therefore the least-to-most RB can be expressed as:
(c(c) by BLAN (d,p,c) (B' by) N₂
lim BLtM (d.p. B())+11BB))+
(22)
Assuming the premises of RB remain unchanged (1, P. = 13oT c)). it can obviously yield the solution (c) > B(c). On the contrary, the model can accept larger difficulty d, which also shows that using least-to-most prompting
effectively increase the maximum of acceptable calculation RB under given RB (as shown in Figure 7), thereby improving model performance. As shown in Table we find that LLM can be optimized by Least-to-most from vanilla COT.
However, the performance improvement of the model is not significant, which we attribute to the fact that the current model cannot push its performance to the ideal limit. As shown in Figure 15 (a), the reasoning boundary of the model cannot make each reasoning step completely tend to which also leads to the difference in reasoning performance in Figure 15 (b). In the meantime, the model's ability divide problems is also limited. What's more, as shown in Figure (c), in around 90% of cases, the model will only divide less than 6 problems, which also limits the performance.
Limitation: Although the pressure of local planning has been reduced, it has not actually effectively reduced the pressure of global planning, nor the pressure of optimization calculations.
21

---

GPT4
text text-davinc-
GPT3-5
Manual Complex GenerGeeeetted Complex
98
90
Accuracy (96)
Accuracy (96)
94
Accuracy (%)
96
70
93
94
92
30
91
92
10
90
3
5
5
6
3
4
The number steps
The number of steps
number steps
(a) GSM8K
(b) SingleEq
(c) MultiArith
86
71
75
Accuracy (%%
Accuracy (%)
70
Accuracy (%)
70
69
84
65
68
83
60
67
82
66
55
65
50
1
2
3
4
5
6
3
4
5
6
0
5 6
The number of steps (d) SAVMP
The number of steps
The number of steps
(e) StrategyQA
(f) AQuA
Figure 16: The effectiveness on average step length in demonstrations for CoT performance.
G The Meta-Analysis for Complex-CoT Prompting
In order to diseuss the advantages and limitations of Complex-Coll conduct two detailed meta- analyses through two distinct perspectives. The first one assesses the influence of the reasoning steps demonstrated in In-Context Learning (ICL) across various tasks, while the second evaluates the effects of employing fixed number of reasoning steps in ICL on questions with different reasoning steps. These meta-analyses aim compare the efficacy of these methods against prior studies systematically. Specifically, we conducted a systematic search for relevant studies addressing the same problem tackled by Jin al. [2024], Fu al. [2023], Shum et al. [2023], Sun et al. [2023], and Jiang et al. [2023b]. We ensured that retrieved studies were pertinent, focusing on studies that addressed the same problem and used similar evaluation metrics.
G.1 The effectiveness of step length in demonstrations
From each selected study, including Jin et al. [2024] and Fu et al. [2023], we evaluate the performance using Complex COT. The relationship between performance and number of Complex CoT's steps is shown in Figure 16. For most multi-step reasoning tasks, as the number of steps increases within certain range, the computational load decreases, and performance improves.
However, as described in Appendix F, the flaws of Complex CoT are apparent. When the difficulty of planning (d,)). defined as the number of planning steps, exceeds B' (p),.. the model's capabilities surpassed, leading to a performance decline. This is evident in single-step calculation reasoning, as shown in Figure 16 (c, e), where performance using Complex CoT gradually decreases. Similarly, for other mathematical reasoning tasks, performance decreases when the number of steps exceeds certain threshold. This phenomenon aligns with our combination law: while reducing the amount of calculation, the number of reasoning steps increases. Exceeding the acceptable number of reasoning steps surpasses the reasoning boundary, causing a decline in model performance. Therefore, maintaining balance between the number of reasoning steps and computational pressure crucial.
22

---

GPT3-5 = text davince 002
text-Gavince-003
LLaMA2 LLMA
Vanilla OUT
Manual Complex COT
60
90
90
Accuracy (%6)
Accuracy (%()
Accuracy (%)
50
70
40
50
30
30
30
10
10
20
2
5
8
3
4
6
7
8
The number of steps
The number of steps
The number of steps
(a) GSMBK
(b) GSMBK
(c) AQuA
Figure 17: The effectiveness of step length in golden samples with fixed step length in demonstra- tions for CoT performance.
G.2 The effectiveness on step length in golden samples
Furthermore, to gain a nuanced understanding of the impact of Complex CoT when the number of steps exceeds the golden step number, conduct further meta-analysis from Fu et al. [2023], Shum et al. [2023], Sun et al. [2023], Jiang et al. [2023b]. Specifically, as illustrated Figure 17 (a, b), analysis reveals that for problems of low complexity and with smaller golden step numbers, Complex CoT tends to underperform compared to Vanilla CoT. Notably, it is only when the reasoning steps exceed two that Complex CoT outperforms Vanilla CoT. This suggests that Complex CoT effectively optimizes single-step computations and enhances model performance for complex problems. However, increases the cognitive load for simple problems, resulting performance decline.
Interestingly, this phenomenon is also observed with the simpler Vanilla CoT, shown in Figure 17 (c). The model achieves significant performance gains only when the number of reasoning steps aligns with the target output steps. If the complexity of the planned steps exceeds the necessary reasoning boundary, or there no effective optimization for reasoning boundary, the performance deteriorates.
G.3
The Implementation Details of Minimum Acceptable Reasoning Paths
To address the two aforementioned limitations, we propose Minimum Acceptable Reasoning Paths (MARP). Firstly, to reduce the model's computational load, we introduce instructions that limit its single-step computing power, thereby optimizing its reasoning boundary. Secondly, to enhance the model's 's acceptability, we increase the computation amount per step within this boundary and reduce the number of global planning steps, thus alleviating planning pressure.
To control variables effectively, we make only the simplest modifications to the prompt to achieve the desired CoT optimization.
Minimum Reasoning Path Prompting alleviate the cognitive load associated with planning, it is essential to have the model respond to the question as succinetly as possible. This approach ensures that the focus remains on providing a short, clear and direct reasoning path. The following prompt is designed to achieve this objective:
You need to perform multi-step reasoning, with each step carrying out as many basic operations possible.
Acceptable Reasoning Prompting To effectively utilize the model, it is crucial to define the upper-bound of reasoning boundary. This ensures that the complexity of the reasoning process manageable and within acceptable bounds. The specific prompt achieve this is as follows:
23

---

[Minimum Reasoning Path Prompting]
You need to perform multi-step reasoning, with each step carrying out as many basic operations possible.
[Acceptable Reasoning Prompting] Remember, you can only complete tasks that contain up to 5 basic operations per step, and multiplication operations must less than 1.5e5. The upper limit the multiplication operations decreases as the number of operations per step increases.
[EXAMPLE]
Question: Leo's assignment was divided into three parts. He finished the first part of his assignment 25 minutes. It took him twice as long finish the second part. If he was able finish his assigmment 2 hours, how many minutes did Leo finish the third part of the assiqnment?
Answer: Leo finished the first and second parts of the assignment in 25 25*2 <=25+25+==75>>7 minutes.
Therefore, it took Leo 60 x 75 = 2 minutes to finish the third part of the assignment.
####
[REQUEST]
Question [Question]
Figure Minimum acceptable reasoning path prompting for natural language chain-of- ofaiinof-thouugt. examples given in the context transform from Wei et al. [2022].
Model
Acc. (*)
Input Token (4)
Output Token (4)
HotpotQA [Yang al., 2018]
CoT
289.50
67.27
26.50
COT+MARP
309.51
68.39
28.73
Medical Probing [Cheng et al., 2024]
CoT
636.11
249.78 86.52
48.9
CRT-MRP
476.11
69.41
StrategyQA [Geva et al., 2021]
CoT
1046.28
225.35
63.90
COT+MARP
649.28
167.40
74.09
Table 2: Extended experimental results on GPT-3,5-Turbo.
Remember, you can only complete tasks that contain up to basic operations per step, and multiplication operations must be less than 1.5e5. The upper limit of the multiplication operations decreases as the number of operations per step increases.
This prompt designed clear boundaries for the model's operations, thereby optimizing its performance and accuracy.
Furthermore, it is necessary to enhance the demonstration within the corresponding in-context learning framework to meet the specific needs of our Model-Agnostic Reasoning Protocol (MARP). This involves refining the examples and instructions provided to ensure they align perfectly with the MARP requirements. Figures 18 and Figure 19 illustrate our MARP prompt, showcasing how to structure the demonstrations to facilitate effective learning and reasoning in natural language CoT and program-of-thone -fo-hhgghttthtttthh setting. By adhering to these guidelines, we can ensure that the model operates efficiently and produces reliable results.
In summary, setting precise boundaries for reasoning boundary and optimizing in-context learning demonstrations essential steps in enhancing the model's performance. By following the specified

---

[Minimum Reasoning Path Prompting]
You need to perform multi-step reasoning, with each step carrying out as many basic operations possible.
[Acceptable Reasoning Prompting] Remember, you can only complete tasks that contain up to 5 basic operations per step, and multiplication operations must be less than 1.5e5. The upper limit of the multiplication operations decreases as the number of operations per step increases.
[EXAMPLE]
Question Leo's assignment was divided into three parts. He finished the first part of his assignment 25 minutes. It took him twice as long finish the second part. If he was able finish his assigmment 2 hours, how many minutes did Leo finish the third part of the assiqument?
Answer: Leo finished the first and second parts of the assignment in 25 25*2 <=25+25+==75>>7 minutes.
Therefore, it took Leo 60 x 75 = 2 minutes to finish the third part of the assignment.
####
[REQUEST]
Question [Question]
Minimum acceptable reasoning path prompting for program-of-thought. All examples given in the context transform from Wei et al. [2022].
Figure 19:
prompt and refining the MARP examples, we can achieve high level of accuracy and efficiency in the model's reasoning processes.
H The Implementation Details in various LLMs
We employ 25 commonly used models to evaluate the extensibility of our framework to a broader range of models. The specific models are listed in Table 3. For each model, we utilize the chat/instruct version whenever available to maximize their ability to follow instructions. Additionally, we deploy ensure fair comparison. Except for
all models on the vLLM [Kwon et al., 2023] framework model Optheaatssseiess [Toshniwal et al., 2024] which does not conform to the vLLM format, all other models are deployed on vLLM for testing. All experiments on open-source models were conducted on two A100 80G. Following the setting of Wei et al. [2022]. in our CoT experiment, all multi-step reasoning tasks are with three manually constructed demonstrations, In addition, for all the experiments, our top-p is selected from {0.95, 1 }. and temperature is selected from [0, 1].
In addition, the only difference in the prompt is that use different dialogue delimiters to make it conform to the format the LLM instruction fine-tuning thereby avoiding the bias caused by the gap between training and inference.
I The Implementation of Combination Law in MGSM
Inspired by Qin et al. [2023] and Huang et al. [2023], we propose that the multilingual mathematical CoT task comprises three sub-tasks: step planning, step calculation, and multi-modal expression. We evaluate the model's mathematical expression ability in different languages based on its zero-shot direct performance on MGSM, as reported by Qin al. [2023]. For relevant parameter calculations, please see the "Challenging Reasoning Boundary Measurement" part of Appendix B. Formally, let step planning RB be denoted by (pp), step calculation RB by B(c), and multilingual expression RB by B((). The combined RB satisfies the following law:
BCTT (c.p.
(23)
[1C)-bb
N
+
(B(p) b3
N₂
N2
25

---

Model
Base Model
Parameters (B)
Open-source General LLM
LLaMA [Touvron et al.. 2023a]
13, 33,
LLaMA-2 [Touvron et al., 20236]
13, 70
LLaMA-3 [Meta, 2024]
8. 70
Code-LLaMA [Roziere al., 2023]
LLaMA-2 Touvron al., 2023b]
7. 13, 34, 70
Mistral [Jiang al., 2023a]
Close-source General LLM
Gemini- 1.0-Pro [Team et al., 2023)
GPT3.5-Turbe [OpenAI, 2022]
Claude-3-Haiku [Anthropic, 2024]
Claude-3-Sonne (Anthropic, 2024]
Claude-3-Oput [Anthropic, 2024]
GPT4 [OpenAI, 2023]
Open-source Math LLM
MAmmoTT Yue et al., 2023]
LLaMA-2 [Touvron et al., 2023b]
7,13
MH (Yue ct al., 2023]
Mistral [Jiang et al., 2023a]
OpenMATTH-Insee I-sstruct (TTosnniwal al., 2024]
LLaMA-2 [Touvron et al., 20236]
70
OpenMAATH-Isttrrn [Toshniwal et al., 2024]
Mistral [Jiagg et al., 2023a]
Table 3: Model list. In order to ensure a certain ability to follow instructions, we use the Instruct version of the model as much as possible (if available).
As shown in Figure 10, the performance distribution of RB (including Acc=90% and Acc==00%) in the multilingual mathematical reasoning task aligns with the proposed combination law Equation (23). Moreover, three distinct RBs evident in Figure 10.
J Ethical Considerations
Data Access. Our data adapted from GSM8K [Cobbe et al., 2021] and supplemented with manually created samples. GSM8K is an open-source dataset available for academic research.
Dataset Collection Process. We began with an introductory task interview using example questions, compensating participants $20 each to familiarize themselves with the task. During the annotation process, annotators were paid $15 per hour, totaling approximately 60 hours of work.
The Rest of Data Annotation Process. For the remaining data annotation, we hired graduate student with CET-6 proficiency Chinese and English and strong mathematical knowledge. The student was compensated $15 per hour, which is above the local average salary. The instructions for annotation are as follows:
You need to annotate the generated number of steps, maximum computation amount, correctness of the generation steps, correctness of the calculations, and correctness of the model output:
Number of generated steps: This refers how many reasoning steps the model generated. Maximum computation amount: This indicates the largest product of operations in the model's s reasoning steps.
Correctness of generation steps: This assesses the accuracy of the model's planning. If all steps operators are planned correctly, and the operand values are logically correct, it is considered correct, regardless of calculation accuracy.
Correctness of calculations: This considers only whether the calculations correct, ignoring planning factors.
Correctness of the output: This checks whether the model's final answer is correct.
26

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: As shown in lines 5-18 of the Abstract and 59-69 of the Introduction, present our main claims and outline the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our work in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
27

---

Our work is not strictly purely theoretical work, we provide more of an empirical formula. addition, we analyze the source of our empirical formula in Appendix and provide the corresponding assumption and proof.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: As shown in Appendix to Appendix I, we have provided detailed descriptions and analyses the experimental setups for all our investigations.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
28

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We will release our code in the official version of the subsequent paper to provide reproduction and provide more help to the future community.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
Seaedddde/ooeeSummissseeee wwwp:/uwwwsssioooooooooooooo onpolicy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /nips. http/uulic//geeee weaiiccgguee//eeeee Cot////beebbieeec celoobmmssioooooliiiiiioo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: shown Section C to Section H. we have provided detailed descriptions and analyses of the experimental setups for all our investigations.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report error bars in Table I and Figure 4, and explain the error variables in Section However, error bars are not reported for all tasks because it would be too expensive for human annotation and computational resource consumption.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
29

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: As shown in Section and Appendix H, we provide detailed model compute resources under different settings.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http://neerppppeeeee wwtppctcccccccccccccctcccccccltt Ceabblic/Ethissteee ecchhiisciiiellleeee
Answer: [Yes]
Justification:
We convinced that comply with NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction)
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed the broader impacts of our work in Section 8. In addition, since our work is more like providing an empirical formula and has no additional social harmfulness, we do not discuss this part.
Guidelines:
30

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications,
authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification:
The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
31

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: shown Section J, We describe and analyze the details and ethical considerations of our crowdsourcing in detail.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
supplemental material is fine, but if the main contribu-
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [No]
Justification: Since our region and institution are not required to provide IRB approval, we do not describe this section. We are convinced that our work complies with the NeurIPS Code of Ethics and the guidelines.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
32

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "gnDYFz5BDR", "reviewer": "Reviewer_fHpa", "summary": "The paper introduces a novel reasoning granularity (RG) framework to quantify and optimize CoT capabilities in LLMs. The authors define RG to measure the upper bounds of CoT and establish a combination law for RG, enabling a practical quantitative approach. They categorize tasks into three categories based on accuracy and propose methods to optimize CoT for improvement. Extensive experiments across models and tasks demonstrate the framework's efficacy in explaining and optimizing CoT performance.", "questions": "1. When evaluating RG on multi-hop question answering, the difficulty of sub-questions in each hop is measured by the number of entities, which may not necessarily be the case. Can you justify this choice?\n\n2. Why does the categorization use 10% and 90% as the cut-off points? Is there statistical support for this categorization? Since most tasks fall in the difficult range of 10% to 90%, what insights does this framework offer for optimization within this range?\n\n3. How would the RG framework perform with different types of reasoning tasks beyond those covered in the experiments?", "limitations": "Yes, discussed in paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The introduction of RG provides a new way to quantify the upper bound of CoT capabilities in LLMs.\n2. The experiments conducted across 25 models and 4 tasks show the generalizability of the proposed evaluation.\n3. The framework offers optimization strategies to guide better CoT for complex tasks based on RG.", "weaknesses": "1. While the framework is validated on 4 tasks, broader evaluations across more diverse tasks would strengthen the generalizability of the findings.\n\n2. The evaluation requires the difficulty level of the task as input, which is not always available. The paper should discuss how to evaluate RG for a task without an explicit difficulty level.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback. We appreciate the opportunity to address the concerns raised. Below are our responses to the points mentioned:\n\n---\n**Q1:** When evaluating RG on multi-hop question answering, the difficulty of sub-questions in each hop is measured by the number of entities, which may not necessarily be the case. Can you justify this choice?\n\n**R1:** Thanks for your constructive comment. In fact, the reasoning path of the answer to this question is completely affected by several core entities and entity relations for multi-hop data construction, which are also mentioned in HotpotQA original paper[1]. Inspired by this, we measure the number of entities as the difficulty of sub-questions in each hop .\n\n---\n**Q2:** Why does the categorization use 10% and 90% as the cut-off points? Is there statistical support for this categorization? Since most tasks fall in the difficult range of 10% to 90%, what insights does this framework offer for optimization within this range?\n\n**R2:** Thanks for your insightful feedback. Our intuition for using this classification is that a model with 90% accuracy actually means almost complete mastery, while 10% accuracy means absolutely no ability to do it.\n\nIn addition, we have conducted preliminary experiments on multiple models and found that no matter how the prompt changes, the accuracy difference will not exceed 2%. The specific information is shown in Table 1, and we will provide more discussion in subsequent versions.\n\n| | Acc in CFRG | Acc in IFRG |\n| --: | :--: | :--: |\n| Prompt 1 | 90.65 | 8.62 |\n| Prompt 2 | 89.72 | 10.34 |\n| Prompt 3 | 88.79 | 8.62 |\n| Prompt 4 | 91.59 | 10.34 |\n||\n\nTable 1: The performance of different prompts at different reasoning granularities.\n\n---\n**Q3:** How would the RG framework perform different types of reasoning tasks beyond those covered in the experiments?\n\n**R3:** Thank you for your insightful comment. From an application perspective, our mechanism framework is universal and can quickly adapt to a variety of new scenarios.\n\nFor example, when encountering a new scenario, we will discuss how to utilize the framework and solve the problems effectively:\n- **Framework Utilization**: Since the combination law conforms to the weighted harmonic mean, it has excellent properties. You only need to be able to ensure relatively independent segmentation into several reasoning granularities, which can effectively utilize our framework.\n\n    Specifically, for any CoT vertical domain problem, two reasoning granularities can be divided into task-planning and vertical domain solution, which satisfy that:\n    $$\n    G=\\frac{1}{\\frac{1}{G_p} + \\frac{1}{G_v} +k_1}\n    $$\n    - If you ignore a certain reasoning granularity, it will only cause $k$ to increase.\n    - If your reasoning granularity is divided reasonably, it will make $k=0$.\n    - If you want to further divide $G_v$ into $G_{v1}$ and $G_{v2}$, it is also very convenient. There is no need to consider additional new formulas, because the following formula is satisfied:\n\n    $$\n    G_v=\\frac{1}{\\frac{1}{G_{v1}} + \\frac{1}{G_{v2}} +k_2}\n    $$\n    $$\n    G=\\frac{1}{\\frac{1}{G_p} + \\frac{1}{G_{v1}} + \\frac{1}{G_{v2}} +k_2 +k_1}\n    $$\n- **Framework Generalization:** As shown in Figure 3 (c) in the original article and Figure 2 in the supplementary material, we can also verify the existence of Combination Law on tasks such as HotpotQA and Medical Knowledge Probing. In addition, our proposed MARP strategy significant improves performance (2.23%-20.51%) and reduces token cost (-1.63%-188%) on these tasks and StrategyQA. Specifically, our observations are:\n    - **Planning RG Optimization:** The performance on Medical Knowledge Probing and StrategyQA has improved, and brought great token savings. It shows that our method effectively reduces the original planning RG and optimized the overall performance according to the combination law.\n    - **Entity RG Optimization:** For HotpotQA, MARP did not change token usage significantly but increased accuracy. This suggests that shorter demonstrations in HotpotQA benefit from a smaller planning RG, but instroduce the larger local entity RG.\nTherefore, according to the combination law, by appropriately keeping the planning RG, entity RG can be optimized based on MARP, which makes the problem difficulty less than the combined RG, thereby improving performance.\n\nWe will add more discussion in next version.\n\n|***HOTPOTQA[1]*** | |||\n| --: | :--: | :--: | :--: |\n| | **Input Token** | **Output Token** | **ACC** |\n| CoT | **289.50** | **67.27** | 26.50 |\n| CoT-MRP | 309.51 | 68.39 | **28.73** |\n| ***Med_Prob[2]*** | |||\n| CoT | 636.11 | 249.78 | 48.9 |\n| CoT-MRP | **476.11** | **86.52** | **69.41** |\n| ***StrategyQA[3]*** | |||\n| CoT | 1046.28 | 225.35 | 63.90 |\n| CoT-MRP | **649.28** | **167.40** | **74.09** |\n||\n\nTable 1: Effectiveness of MARP strategies on different tasks.\n\n[1] Yang et al. HOTPOTQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. EMNLP2018.\n\n[2] Cheng et al. Adapting Large Language Models via Reading Comprehension. ICLR 2024.\n\n[3] Geva et al. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. TACL 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mJCHP50e80", "reviewer": "Reviewer_oUcy", "summary": "This paper proposed a novel reasoning granularities (RG) methodological framework to quantitatively assess CoT capabilities and provide guidance on optimizing CoT performance. The experiement results show an upper bound of CoT, and the authors have proposed three catergories of RG to optimize CoT with combination laws focused on RG promotion and reasoning path optimization for CoT improvement.", "questions": "Check the writing and grammar. Some occassional typo or mis-used comma.", "limitations": "Yes.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The authors proposed a new concept, named reasoning granularity (RG) to quantify the upper-bound on task-specific reasoning complexity within a model. \nThe authors show that Tool Usage and Program-of-Thought can improve the value of LLM's RG.", "weaknesses": "No major weekness from my perspective", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your careful review and affirmation of our paper.\n\n**Q1:** Check the writing and grammar. Some occasional typos or misused commas.\n\n**R1:** Thank you for your constructive suggestions. We will correct these issues one by one in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MEE5u8BksE", "reviewer": "Reviewer_ynAv", "summary": "The article introduced a novel framework for quantifying and optimizing the reasoning capabilities of large language models (LLMs). The concept of Reasoning Granularity (RG) is innovative and may have the potential to significantly impact the field of natural language processing with LLMs.", "questions": "1. Can you provide some comparative results based on GPT4 or other models with stronger reasoning capabilities as baselines to demonstrate the performance improvement of RG in path planning?", "limitations": "Yes.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The Reasoning Granularity (RG) framework provided a novel perspective on quantifying and optimizing the chain-of-thought reasoning capabilities of large language models.\n2. The paper supported its claims through extensive experiments across 25 models and 4 tasks, demonstrating the broad applicability and robustness of the proposed RG framework.\n3. The paper provided a number of examples in the appendix, which makes it an engaging read and easy to follow.", "weaknesses": "1. Although the paper has demonstrated the effectiveness of the RG framework across several models and tasks, it could further strengthen its claims by discussing how these findings might generalize to other types of reasoning tasks or different domains beyond the ones tested.\n2. Compared to GPT4, the multi-step reasoning capability of GPT3.5 used in this article might be insufficient. It would be better to add experiments based on GPT4 to prove that the improvement comes from the stimulation of model capabilities, rather than the introduction of a priori frameworks for specific tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We appreciate the opportunity to address the concerns you have raised. Our responses to the specific points mentioned are as follows:\n\n---\n**Q1:** Although the paper has demonstrated the effectiveness of the RG framework across several models and tasks, it could further strengthen its claims by discussing how these findings might generalize to other types of reasoning tasks or different domains beyond the ones tested.\n\n**R1:** Thank you for your insightful feedback. We totally agree with your comment.\n\nIn fact, since the combination law conforms to the weighted harmonic mean, it has excellent properties. You only need to be able to ensure relatively independent segmentation into several reasoning granularities, which can effectively utilize our framework. Specifically, for any CoT vertical domain problem, two reasoning granularities can be divided into task-planning and vertical domain solution, which satisfy that:\n$$\nG=\\frac{1}{\\frac{1}{G_p} + \\frac{1}{G_v} +k_1}\n$$\n- If you ignore a certain reasoning granularity, it will only cause $k$ to increase.\n- If your reasoning granularity is divided reasonably, it will make $k=0$.\n- If you want to further divide $G_v$ into $G_{v1}$ and $G_{v2}$, it is also very convenient. There is no need to consider additional new formulas, because the following formula is satisfied:\n\n$$\nG_v=\\frac{1}{\\frac{1}{G_{v1}} + \\frac{1}{G_{v2}} +k_2}\n$$\n$$\nG=\\frac{1}{\\frac{1}{G_p} + \\frac{1}{G_{v1}} + \\frac{1}{G_{v2}} +k_2 +k_1}\n$$\n\n***Based on this, the granularity of different sizes can be easily divided, making it more practical.***\n\n---\n**Q2:** Can you provide some comparative results based on GPT4 or other models with stronger reasoning capabilities as baselines to demonstrate the performance improvement of RG in path planning?\n\n**R2:** Thank you for your suggestion. As shown in Figure 1 in the supplementary material, GPT4o also conforms to the combination law. Moreover, compared with GPT3.5, both CFRG and IFRG have significantly improved.\nHowever, due to the current powerful capabilities of GPT4o, it is difficult to measure the IFRG of the model in other benchmarks such as HotpotQA. Therefore, we did not include GPT4o into the scope of verification in the main experiment.\n\nIn addition,  as shown in Table 1 below, we found that on the GPT4o model, MARP strategy also achieved the SOTA effect.\n\nWe will add more discussion in the next version.\n\n| | Input Token | Output Token | ACC |\n| :--: | :--: | :--: | :--: |\n| CoT | 781.30 | 224.09 | 74.15 |\n| CoT-MRP | **615.30** | **222.90** | **78.84** |\n||\n\nTable 1: Effectiveness of MARP strategies on BigGSM on GPT4o.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "znr2IMpM0z", "reviewer": "Reviewer_MXRP", "summary": "The paper introduces a Reasoning Granularity (RG) framework that quantifies and optimizes Chain-of-Thought reasoning in large language models. Through extensive experiments, the authors validate the RG framework's effectiveness across various tasks and models, providing new insights into enhancing reasoning capabilities in LLM.", "questions": "See the Weakness section.", "limitations": "Authors have adequately discussed limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Innovative Framework**: The introduction of the Reasoning Granularity (RG) framework provides a novel approach to quantify and optimize complex reasoning in large language models.\n2. **Comprehensive Empirical Analysis**: This paper provides a thorough empirical analysis with extensive experiments across 25 models and 4 different tasks, demonstrating the robustness of the proposed framework.\n3. **Good Presentation**: This paper is well-organized, with a clear presentation of the methodology, experiments, and results, making it easy for readers to understand.", "weaknesses": "1. **Lack of Theoretical Analysis**: While the paper provides an empirical framework and experimental validation, it may not delve deeply enough into the theoretical understanding of the concept of Reasoning Granularity (RG). A more rigorous theoretical foundation, although difficult, could strengthen the arguments and enhance the contribution of the paper.\n2. **Limited Generalizability**: The RG framework, although validated across 25 models and 4 tasks, may not be fully generalizable to all types of large language models or reasoning tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our gratitude for your insightful feedback. We appreciate the opportunity to address the concerns presented. Below, we provide our detailed responses to each of the points raised:\n\n---\n**Q1:** **Lack of Theoretical Analysis**: While the paper provides an empirical framework and experimental validation, it may not delve deeply enough into the theoretical understanding of the concept of Reasoning Granularity (RG). A more rigorous theoretical foundation, although difficult, could strengthen the arguments and enhance the contribution of the paper.\n\n**R1:** Thank you for your suggestion. In fact, our paper has a theoretical analysis in Appendix A.1. Specifically, for the two core concepts of this article, our theoretical analysis is as follows:\n1. **Reasoning Granularity:** In fact, the existence of a universal RG upper limit has actually been done, so we don't need additional elaboration [1]. Based on the proof, it is also obvious that there are different upper bounds for different task conditions.\n2. **Combination Law:** In fact, we provide **a theoretical analysis of this formula in Appendix A.1** for combination law, and we prove that it is theoretically consistent with Combination Law under the condition that the RG is relatively independent.\nWe will add more discussion in the next version.\n\n[1] Towards revealing the mystery behind chain of thought: a theoretical perspective. NeurIPS 2024.\n\n---\n**Q2:** The RG framework, although validated across 25 models and 4 tasks, may not be fully generalizable to all types of large language models or reasoning tasks.\n\n**R2:** Thank you for your insightful comment. In fact, our method can be generalized to other tasks.\n\nSpecifically, when encountering a new scenario, we will discuss how to utilize the framework and solve the problems effectively:\n- **Framework Utilization**: Since the combination law conforms to the weighted harmonic mean, it has excellent properties. You only need to be able to ensure relatively independent segmentation into several reasoning granularities, which can effectively utilize our framework.\n\n    Specifically, for any CoT vertical domain problem, two reasoning granularities can be divided into task-planning and vertical domain solution, which satisfy that:\n    $$\n    G=\\frac{1}{\\frac{1}{G_p} + \\frac{1}{G_v} +k_1}\n    $$\n    - If you ignore a certain reasoning granularity, it will only cause $k$ to increase.\n    - If your reasoning granularity is divided reasonably, it will make $k=0$.\n    - If you want to further divide $G_v$ into $G_{v1}$ and $G_{v2}$, it is also very convenient. There is no need to consider additional new formulas, because the following formula is satisfied:\n\n    $$\n    G_v=\\frac{1}{\\frac{1}{G_{v1}} + \\frac{1}{G_{v2}} +k_2}\n    $$\n    $$\n    G=\\frac{1}{\\frac{1}{G_p} + \\frac{1}{G_{v1}} + \\frac{1}{G_{v2}} +k_2 +k_1}\n    $$\n- **Framework Generalization:** As shown in Figure 3 (c) in the original article and Figure 2 in the supplementary material, we can also verify the existence of Combination Law on tasks such as HotpotQA and Medical Knowledge Probing. In addition, our proposed MARP strategy significant improves performance (2.23%-20.51%) and reduces token cost (-1.63%-188%) on these tasks and StrategyQA. Specifically, our observations are:\n    - **Planning RG Optimization:** The performance on Medical Knowledge Probing and StrategyQA has improved, and brought great token savings. It shows that our method effectively reduces the original planning RG and optimized the overall performance according to the combination law.\n    - **Entity RG Optimization:** For HotpotQA, MARP did not change token usage significantly but increased accuracy. This suggests that shorter demonstrations in HotpotQA benefit from a smaller planning RG, but instroduce larger the local entity RG.\n    Therefore, according to the combination law, by appropriately keeping the planning RG, entity RG can be optimized based on MARP, which makes the problem difficulty less than the combined RG, thereby improving performance.\n\nWe will add more discussion in next version.\n\n|***HOTPOTQA[1]*** | |||\n| --: | :--: | :--: | :--: |\n| | **Input Token** | **Output Token** | **ACC** |\n| CoT | **289.50** | **67.27** | 26.50 |\n| CoT-MRP | 309.51 | 68.39 | **28.73** |\n| ***Med_Prob[2]*** | |||\n| CoT | 636.11 | 249.78 | 48.9 |\n| CoT-MRP | **476.11** | **86.52** | **69.41** |\n| ***StrategyQA[3]*** | |||\n| CoT | 1046.28 | 225.35 | 63.90 |\n| CoT-MRP | **649.28** | **167.40** | **74.09** |\n||\n\nTable 1: Effectiveness of MARP strategies on different tasks for GPT3.5.\n\n[1] Yang et al. HOTPOTQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. EMNLP2018.\n\n[2] Cheng et al. Adapting Large Language Models via Reading Comprehension. ICLR 2024.\n\n[3] Geva et al. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. TACL 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fL14sat50D", "reviewer": "Reviewer_z3Vj", "summary": "The paper presents a Reasoning Granularity (RG) framework designed to quantify and optimize the Chain-of-Thought (CoT) reasoning capabilities of large language models (LLMs). The framework introduces a new metric, RG, to measure the complexity of reasoning tasks that LLMs can handle. It also establishes a combination law to integrate multiple reasoning tasks and categorize RG into three distinct types. The study validates this framework through extensive experiments and demonstrates the effectiveness of various optimization strategies to enhance CoT performance.\n\nSpecifically, the authors: \n1. introduces the concept of RG to quantify the upper bound on task-specific reasoning complexity within a model.\nDefines a combination law for RG, using the weighted harmonic mean to integrate multiple reasoning tasks.\n2. proposes three categories of RG (Completely Feasible, Partially Feasible, Completely Infeasible) to guide the optimization of CoT performance.\n3. introduces Minimum Acceptable Reasoning Paths (MARP) to optimize reasoning paths and reduce computational load.\n\nTo validate the effectiveness of their work, they:\n1. validate the RG framework through extensive experiments on 25 models and 4 tasks, demonstrating its robustness and applicability.\n2. Explains the effectiveness of 10 CoT strategies and provides optimization techniques like Tool Usage and Program-of-Thought (PoT).\n3. They also establish a theoretical foundation for understanding the boundaries of CoT reasoning capabilities in LLMs. A combination law for RG is proposed to generalize the quantification in complex scenarios.\n\nOverall, the paper advances both theoretical understanding and practical optimization of CoT reasoning in LLMs, providing a robust framework and concrete metrics to enhance model performance on complex reasoning tasks.", "questions": "1. Can you provide more detailed examples of tasks that fall into each of the three RG categories (Completely Feasible, Partially Feasible, and Completely Infeasible)? How do these categories affect the model’s optimization process?\n2. How robust is the combination law of RG across different types of reasoning tasks? Are there specific scenarios where this law does not hold or requires adjustments?", "limitations": "1. P2 in weaknesses. \n2. As far as I understand, the benchmark that is used in the evaluation, BIGGSM focuses more on math problems, like in MATH and GSM8K. This may raise concerns about the generality of the solution on other types of reasonings, for example in StrategyQA, or planning benchmarks. Improve the diversity of benchmarks could provide a more holistic view of the framework's impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. An innovative framework is proposed targeting further optimization of CoT. (Section 2) The definition of Reasoning Granularity to quantify the upper bound of CoT is novel and reasonable. It then leads to a concrete metric to assess and compare CoT capabilities across several models and tasks. \n2. It's quite impressive that the definition of combination law of RG considers the requirement of integrating multiple capabilities for a single task, which is crucial for real-world benchmarks. \n3. For optimization guidance, defining three categories of RG (Completely Feasible, Partially Feasible, and Completely Infeasible) helps in systematically optimizing CoT performance based on the specific granularity. Then the introduction of MARP to optimize CoT within a specific RG leads to a practical solution to enhance CoT and reduce token consumption. \n4. Validation over 4 tasks and 25 models is quite solid. The study explains the effectiveness of 10 CoT strategies and introduces optimization techniques like Tool Usage and Program-of-Thought (PoT) that significantly improve CoT performance", "weaknesses": "1. Although the framework is validated on 25 models and 4 tasks, there may be concerns about how well these results generalize to other tasks or models not included in the study, as the solution for now are more task/RG-specific. \n2. The combination law for RG relies on certain assumptions that may not hold universally across all reasoning tasks or model architectures. Further empirical validation is needed to confirm these assumptions in diverse settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our sincere appreciation for your comprehensive feedback. We value the opportunity to address the concerns identified. Our responses to the enumerated points are as follows:\n\n---\n**Q1:** Can you provide more detailed examples of tasks that fall into each of the three RG categories? How do these categories affect the model’s optimization process?\n\n**R1:** Thank you for your enlightening advice. Examples of our different inference granularities are as follows:\n\n- **CFRG:**\n```\nA ship traverses the ocean waves. Below are entries from the ship's logbook:\n- The ship sailed 31 kilometers north.\n- Heading northward, the ship ventured 6 times further than it did yesterday.\n\nHow much distance has it covered from the beginning?\n```\n\n- **PFRG:**\n```\nA boat traverses the ocean waves. Below are entries from the boat's logbook:\n- The boat navigated 42 kilometers southward.\n- Navigating north, the boat traveled 570 times the distance it had managed yesterday.\n- The boat journeyed 165 kilometers in a southerly direction.\n- The boat navigated 339 kilometers northward.\n\nWhat is the distance from its origin?\n```\n\n- **CIRG:**\n```\nUpon the vast expanse of the sea sails a vessel. Herein are the chronicles from the vessel's diary:\n- The vessel traveled 564 kilometers towards the west.\n- The vessel navigated 856 kilometers eastward.\n- The vessel traveled 439 kilometers towards the west.\n- The vessel sailed 990 kilometers west.\n- The vessel journeyed 291 kilometers in a easterly direction.\n- Navigating east, the vessel traveled 490 times the distance it had managed yesterday.\n- The vessel navigated 161 kilometers westward.\n- The vessel sailed 914 kilometers west.\n- The vessel sailed 649 kilometers west.\n- The vessel traveled 6 kilometers towards the west.\n\nWhat is the extent of its travel from the starting location?\n```\n\nAs shown in the example, the calculation amount and calculation steps of different cases increase, but the core logic of the problem does not change in any way. In fact, as shown in Figure 4 in the paper, different granularities receive different benefits from self-consistency, and PFRG significantly affects the performance of the Self-consistency optimization strategy. In addition, as shown in Figures 5 and 8 in the paper, different models and different strategies actually improve model performance from different angles by optimizing PFRG and CIRG.\n\n---\n**Q2:** How robust is the combination law of RG across different types of reasoning tasks? Are there specific scenarios where this law does not hold or requires adjustments?\n\n**R2:** Since the combination law conforms to the weighted harmonic mean, it has excellent and robust properties for diverse scenarios. You only need to ensure relatively independent segmentation into several reasoning granularities, which can effectively utilize our framework. Specifically, for any CoT vertical domain problem, two reasoning granularities can be divided into task-planning and vertical domain solution, which satisfy that:\n\n$$\nG=\\frac{1}{\\frac{1}{G_p}+\\frac{1}{G_v}+k_1}\n$$\n\n- If you ignore a certain reasoning granularity, it will only cause $k$ to increase.\n- If your reasoning granularity is divided reasonably, it will make $k=0$.\n- If you want to further divide $G_v$ into $G_{v1}$ and $G_{v2}$, it is also very convenient. There is no need to consider additional new formulas, because the following formula is satisfied:\n\n$$\nG_v=\\frac{1}{\\frac{1}{G_{v1}}+\\frac{1}{G_{v2}}+k_2}\n$$\n$$\nG=\\frac{1}{\\frac{1}{G_p}+\\frac{1}{G_{v1}}+\\frac{1}{G_{v2}}+k_2+k_1}\n$$\n\n---\n**Q3:** This may raise concerns about the generality of the solution on other types of reasonings, for example in StrategyQA, or planning benchmarks.\n\n**R3:** Thank you for your recognition of our work. In fact, we have conducted non-mathematical experiments. As shown in Figure 3 (c) and Figure 10 in original paper, we have conducted an analysis of multi-hop QA and even multilingual scenarios.\nIn addition, in order to dispel your doubts, we have decomposed the Medical Knowledge Probing problem in detail according to the steps and related medical entities.  As shown in Figure 2 in the supplementary material, the combination law is also satisfied in this benchmark.\n\nIn addition, as shown in Table 1 below, the MARP we proposed can also work on this data set and can achieve SOTA results on Medical Knowledge Probing, StrategyQA, and HotpotQA. Specifically, our observations are:\n- **Planning RG Optimization:** The performance on Medical Knowledge Probing and StrategyQA has improved, and brought great token savings. It shows that our method effectively reduces the original planning RG and optimized the overall performance according to the combination law.\n- **Entity RG Optimization:** For HotpotQA, MARP did not change token usage significantly but increased accuracy. This suggests that shorter demonstrations in HotpotQA benefit from a smaller planning RG, but instroduce larger the local entity RG.\n    Therefore, according to the combination law, by appropriately keeping the planning RG, entity RG can be optimized based on MARP, which makes the problem difficulty less than the combined RG, thereby improving performance.\n\nWe will add more discussion in next version.\n\n|***HOTPOTQA[1]***||||\n|--:|:--:|:--:|:--:|\n||**Input Token**|**Output Token**|**ACC**|\n|CoT|**289.50**|**67.27**|26.50|\n|CoT-MRP|309.51|68.39|**28.73**|\n|***Med_Prob[2]***||||\n|CoT|636.11|249.78|48.9|\n|CoT-MRP|**476.11**|**86.52**|**69.41**|\n|***StrategyQA[3]***||||\n|CoT|1046.28|225.35|63.90|\n|CoT-MRP|**649.28**|**167.40**|**74.09**|\n||\n\nTable 1: Effectiveness of MARP strategies on different tasks.\n\n[1] Yang et al. HOTPOTQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. EMNLP2018.\n\n[2] Cheng et al. Adapting Large Language Models via Reading Comprehension. ICLR 2024.\n\n[3] Geva et al. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. TACL 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We extend our gratitude to all reviewers for their insightful and thoughtful feedback.

1. We are greatly encouraged that all reviewers observe that our work introduces an **innovative Reasoning Granularity** framework targeting further **optimization of CoT** (Reviewer #fHpa, Reviewer #oUcy, Reviewer #ynAv, Reviewer #MXRP, Reviewer #z3Vj).
2. We are pleased that reviewers found that our work provides **comprehensive empirical analysis**, demonstrating the robustness and generalizability of the proposed RG framework (Reviewer #fHpa, Reviewer #ynAv, Reviewer #MXRP, Reviewer #z3Vj).
3. We are also glad that all reviewers appreciated the presentation of our methodology, experiments, and results, noting that it makes our paper **well-organized and easy to follow** (Reviewer #ynAv, Reviewer #MXRP).

We will address all concerns to polish our work according to reviewers’ comments in the next version. Thanks once again for the valuable contributions of all the reviewers.

Meta-review: The aim of this paper is to improve the ability of Large Language Models in solving complex reasoning tasks by addressing two main drawbacks in the Chain-Of-Thought (CoT) approach: the lack of quantification metrics and the absence of optimization guidance. The authors introduce the concept of Reasoning Granularity (RG) and a combination law for assessing the upper bound of CoT. They propose three categories of RGs to guide the optimization of CoT strategies and conduct extensive experiments on 4 different tasks and 25 models to validate this approach, highlighting its generality.

All reviewers agree that this study makes a significant contribution to the CoT approach in LLMs. They commend the well-organized paper, innovative and well-founded framework, and convincing empirical analysis. In summary, this is a strong piece of work.

To further enhance the quality of this study, I suggest including in the paper or the appendix some responses to reviewers’ questions. Notably, incorporating information on extending this approach to new incoming tasks, addressing concerns about the robustness of the combination law, and summarizing the discussion about assessing the difficulty level of an incoming task could be valuable additions.
Decision: Accept (oral)