Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thought

Authors: qiguang  chen
libo  qin
jiaqi  wang
jinxuan  zhou
wanxiang  che
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 14:13:38

Abstract: Chain-of-Thought (CoT) reasoning has emerged as a promising approach for enhancing the performance of large language models (LLMs) on complex reasoning tasks. Recently, a series of studies attempt to explain the mechanisms underlying CoT, aiming to deepen the understanding of its efficacy. Nevertheless, the existing research faces two major challenges: (1) a lack of quantitative metrics to assess CoT capabilities and (2) a dearth of guidance on optimizing CoT performance. Motivated by this, in this work, we introduce a novel reasoning boundary framework (RBF) to address these challenges. To solve the lack of quantification, we first define a reasoning boundary (RB) to quantify the upper-bound of CoT and establish a combination law for RB, enabling a practical quantitative approach applicable to various real-world CoT tasks. To address the lack of optimization, we propose three categories of RBs. We further optimize these categories with combination laws focused on RB promotion and reasoning path optimization for CoT improvement. Through extensive experiments on 27 models and 5 tasks, the study validates the existence and rationality of the proposed framework. Furthermore, it explains the effectiveness of 10 CoT strategies and guides optimization from two perspectives. We hope this work can provide a comprehensive understanding of the boundaries and optimization strategies for reasoning in LLMs. Our code and data are available at https://github.com/LightChen233/reasoning-boundary.

Main paper: 
 introduction: 
 In recent years, Large Language Models (LLMs) have demonstrated increasing capabilities and applications across various tasks [Zhao et al., 2023 , Chang et al., 2023 , Pan et al., 2023 , Qin et al., 2024a] . Notably, advanced LLMs, such as GPT [Brown et al., 2020 , OpenAI, 2022 , 2023] , PaLM [Anil et al., 2023] and LlaMa [Touvron et al., 2023a ,b, Meta, 2024] series have demonstrated emergent capabilities, particularly like Chain-of-Thought (CoT) [Nye et al., 2022 , Wei et al., 2022] . This methodology enables models to verbalize step-by-step reasoning, thereby enhancing prediction accuracy by basing decisions on the logical rationale [Wei et al., 2022 , Kojima et al., 2022 , Hu et al., 2024 , Qin et al., 2023 , Zhuang et al., 2023 , Chen et al., 2024a] .Recently, some research in the literature has begun to investigate the mechanism of CoT to enhance the understanding of its operational nature. To this end, Madaan et al. [2023] and Wang et al. [2023a] Motivated by this, in this work, we introduce a reasoning boundary framework (RBF) to thoroughly examine and optimize the boundaries of current LLMs. Specifically, to address the quantification challenge, we propose a new concept, named reasoning boundary (RB) to quantify the upper-bound on task-specific reasoning complexity within a model. Furthermore, to explore more practical scenarios, we present the combination law of RBs to generalize the RB for quantification in more real and complex scenarios. To address the CoT optimization challenge, we propose and analyze three reasoning boundary intervals, guiding optimization through improved RB and optimized reasoning paths based on the combination law, which achieves state-of-the-art performance in our proposed benchmark. We extensively validate the efficacy of our framework across 27 models and 5 tasks: arithmetic computing, mathematical reasoning, multi-hop question answering, and multilingual mathematical reasoning.Our main contributions are as follows:• To the best of our knowledge, this is the first work to propose a reasoning boundary framework (RBF) to quantify the upper-bound of CoT. Furthermore, we establish the combination law of RB as the weighted harmonic mean of fundamental RBs to address practical CoT tasks.• To solve the lack of CoT optimization, we define three categories of RBs. Based on the combination law and the nature of these RBs, we effectively improve the existing CoT strategies by RB promotion and reasoning path optimization. • We validate the existence and rationality of our framework on 27 models and 5 CoT tasks. Furthermore, we explain the optimal performance from two optimization perspectives in numerous CoT strategies. We consider both optimal perspectives and propose a minimum acceptable reasoning path (MARP) prompting to achieve state-of-the-art performance.2 Quantification Methodology 
reasoning boundary: 
 In order to quantify the capacity for complex reasoning in LLMs, we introduce an upper-bound concept termed reasoning boundary (RB), which formally defines the degree of ease that an LLM can handle within a specific reasoning process. In simpler terms, as shown in Figure 1 (a), RB reflects the limit beyond which a model's accuracy significantly degrades. Mathematically, RB is defined for a model m and a task t as the maximum of problem difficulty d at which the model's accuracy reaches a predefined threshold K 1 :EQUATIONwhere Acc(t|d, m) represents the accuracy of the model's accuracy on task t with difficulty d. Difficulty can be measured by factors like the number of reasoning steps or computational complexity.For brevity, we denote RB as B(t|m) in subsequent sections.Figure 1 : Overview of the introduced concepts.The reasoning boundary for a model is defined by its ability to achieve a specific accuracy for a given task difficulty.In practical scenarios, models often require the integration of multiple capabilities to address a single task effectively. To quantify how a large language model can be boosted by the cooperation of multiple capabilities through the CoT mechanism, we introduce the "Combination Law of RB", giving a concrete formula of the upper-bound of the CoT. The law estimates the unified reasoning boundary B Acc=K1 (t 1 , t 2 , . . . , t n |m) for n tasks within a model m, which is formulated as:EQUATIONwhere B Acc=K1 (t i |m) denotes the reasoning boundary of model m for task t i . N i , and b i are scaling factors, which are only affected by the related task. As shown in Figure 1 (b), Equation (2) provides a mathematical formula to estimate the combined RBs from the independent ones, enabling deeper insights into model behavior for intricate tasks. See Appendix A for detailed mathematical analysis.Furthermore, the combination law for reasoning boundary demonstrates favorable theoretical properties, with broad applicability across diverse scenarios and flexibility in accommodating various boundary segmentation methods. For detailed practical application, please refer to Appendix B.The combination law of reasoning boundary satisfies the weighted harmonic average of each basic reasoning boundary.Furthermore, in order to guide the optimization of CoT and more convenient expression, as shown in Figure 1 (c), we define the following three categories of RBs based on their empirical accuracy:Completely Feasible Reasoning Boundary: We define that the part with an accuracy greater than 90% is a completely feasible reasoning boundary (CFRB = B Acc≥90% (t 1 , t 2 , . . . , t n |m)), which means that LLMs can effectively grasp the performance of this part.Completely Infeasible Reasoning Boundary: We believe that the part with an accuracy less than 10% is a completely infeasible reasoning boundary (CIRB = B Acc≤10% (t 1 , t 2 , . . . , t n |m)), which means that the model can never effectively grasp the performance of this part.Partially Feasible Reasoning Boundary: We define the RB in the rest part except CFRB and CIRB as a partially feasible reasoning boundary (PFRB = B 10%<Acc<90% (t 1 , t 2 , . . . , t n |m)), which requires the model to repeat thinking or more clear information to solve the problem. We analyze the nature of these three categories of RB in detail (in Section 4.3), and further utilize the combination law to optimize these three reasoning boundaries (in Section 5), so as to provide effective suggestions and guidance to support future CoT optimization. 
experimental setup: 
 Benchmark Settings To assess the reasoning boundaries of LLMs, we require a dataset rich in RB. This necessitates tasks with evenly distributed complexities and reasoning steps that challenge the models' upper-bounds. To meet these requirements, we introduce BIGGSM, a new dataset offering greater calculation complexity and longer reasoning chains. The detailed construction process for BIGGSM is provided in Appendix C.Model Settings Except for model expansion experiments, all experiments are conducted on GPT-3.5-Turbo. Following the setting of Wei et al. [2022] , in our CoT experiment, all multi-step reasoning tasks utilize three manually constructed demonstrations. In addition, for all the experiments, top-p is selected from {0.95, 1}. Temperature is selected from [0, 1] and serves as the main error variable.4 Empirical Analysis of Reasoning Boundary 
existence verification for reasoning boundary: 
 In this study, we investigate the hypothesis that an LLM exhibits varying levels of reasoning boundary across various tasks. To this end, we will verify whether the model has widespread reasoning boundary in various tasks in the following three tasks:Basic Arithmetic Calculation First, to investigate the existence of RB, we first examine basic arithmetic operations (including addition, subtraction, multiplication, and division). As illustrated in Figure 2 (a), the results reveal significant performance variations across three distinct regions. For multiplication, accuracy surpasses 90% for results up to 2.2e5. Conversely, accuracy falls below 10% for products exceeding 2e6. Similar presences of varying RBs are observed for other operations, which verifies the existence of reasoning boundary in basic arithmetic calculation tasks. Further results and implementation details are provided in Appendix D.Nature Language Planning We further investigate RB in natural language planning tasks for mathematical reasoning. We prompt the model to generate plans and assess their accuracy through manual evaluation. There is a strong correlation between the number of reasoning steps and LLMs' performance in Figure 2 (b). When the model meets the question with fewer than 2 reasoning steps, accuracy surpasses 90%. Conversely, when reasoning steps exceed 4, accuracy falls below 10%. This finding suggests that there are also three different RB categories in natural language planning tasks.Code Planning For further extensive exploration, we further prompt LLMs by PAL [Gao et al., 2023] to generate code-format plans and evaluate them by manual annotation. As shown in Figure 2 (c), the code planning task is similar to natural language planning, which is also an obvious division and different categories of RBs. Notably, since code planning utilizes code for clearer logic and reduced expression complexity, its planning accuracy surpasses that of natural language planning. The number of entity (𝓑(e))The number of hop (𝓑(h))(c) The combination law of different reasoning boundaries in multi-hop question-answering task. Combination Law in Complex Arithmetic Calculation Building on the proof of Equation ( 13), we hypothesize that the combination law for RB in the complex arithmetic calculation is the harmonic average of the arithmetic calculation RB and calculation planning RB. To verify this, we designed an experiment focusing on formulas containing addition, subtraction, and multiplication, like "(1 + 2) * 3 -4". Since addition and subtraction complexities are assumed to be around 1e15 (as shown in Figure 13 ), the arithmetic calculation RB primarily depends on the multiplication RB and calculation planning RB. Therefore, as shown in Figure 3 (a), there are two obvious RB lines, namely B Acc=90% and B Acc=10% , which are completely consistent with the combination law of these basic RB based on the Equation (2). Besides, these two lines also clearly divide the RBs into three categories.Combination Law in Mathematical Reasoning Inspired by Tan [2023b], Xiao and Liu [2024] , we posit that the natural language mathematical CoT task is determined by two sub-tasks: step planning task and step calculation task for global logic planning and local mathematical calculation. Furthermore, each model output step requires a single basic operation, resulting in a step calculation boundary close to the maximum number of multiplications, denoted by B(c) ≈ B(m). Formally, with step planning RB denoted by (B(p)) and the step calculation RB by (B(c)), then the combined RB satisfies the following law:B CoT (c, p) = 1 N1 (B(c)-b1) + N2 (B(p)-b2).(3)As illustrated in Figure 3 (b), the actual performance distribution of RB (including B Acc=90% and B Acc=10% ) in natural language mathematical reasoning task fully aligns with the proposed combination law in Equation (3). Additionally, there are also obviously three RBs in Figure 3 (b) .Combination Law in Multi-hop Reasoning Beyond the realm of mathematics, we further extend our exploration of the combination law to the field of multi-hop question answering. Specifically, we validate our law on HotpotQA [Yang et al., 2018] , where we define the reasoning boundary as the combination of global hop-planning RB and local knowledge entity reasoning RB. As shown in Figure 3 (c), B Acc=90% and B Acc=10% also satisfy the weighted harmonic mean of these two sub-reasoning boundaries. It is also proved that, in addition to math-related tasks, multi-hop question answering also satisfies our proposed combined law and also exhibits three distinct RBs. We will describe in detail how to calculate the combination law on multi-hop reasoning in Appendix E.According to the definition of different RBs, we have divided the problem into three parts for LLMs.In this section, we will verify whether the defined RB adheres to the intrinsic nature of the model itself. We will discuss the natures of these RBs in detail:CFRB means complete mastery of the model even without demonstration. According to the definition, we assume that a question within CFRB implies a comprehensive understanding of the associated issue for a certain LLM. To verify this, following Zhang et al. [2022] and Wei et al. [2022] , we formulate a mathematical request and generate chain-of-thought rationale and answer through Figure 4 : Nature analysis at different reasoning boundaries.zero-shot prompting without any demonstration. As shown in Figure 4 (a), it still achieves 29.2% improvement in CFRB on generating the correct rationale compared to other RBs. This also proves that the model can indeed master tasks well on the questions in CFRB.PFRB means moderate confidence in its solution and needs consensus building process. To gauge the level of performance and confidence, we draw parallels to human decision-making, where moderate confidence often necessitates multiple times of consensus building. Inspired by this, we investigate it on Self-Consistency [Wang et al., 2022] , which integrates results from various reasoning answers to reach a conclusive answer. Figure 4 (b) demonstrates that as the integration of reasoning paths increases, the accuracy improves significantly within PFRB compared with other RBs. This suggests that within PFRB, the LLM exhibits moderate confidence in solving problems, which needs multiple consensus building.CIRB exhibits poor reasoning performance even with consensus building. As illustrated in Figure 4 (a), questions in CIRB display extremely low accuracy (around 9.5%). And the model shows consistently poor performance and no improvement on Self-consistency in this boundary in Figure 4 . It signifies that the model exhibits poor reasoning performance.LLM has self-awareness of its own RBs. In parallel, a natural question arises: Is the model capable of discerning its inherent RBs? To investigate this, we employ the Synthetic-CoT [Shao et al., 2023] to prompt LLM to generate CoT data. As depicted in Figure 4 (c), the results demonstrated that there are over 65% of generated samples within CFRB, which achieves a much higher percentage and performance than other RBs. This suggests that LLMs possess an intrinsic understanding of their RBs and constraints to generate the task they grasp, indicative of a potential for self-assessment.Takeaways: (1) Reasoning boundary (RB) and the combination law of RB are both widespread across a series of tasks. (2) Different categories of RB can reflect the corresponding performance, and the model can also have a self-understanding of its own RB. 
rb-based cot optimization: 
 5.1 How can we improve CoT by optimizing RB?Based on our framework, the reasoning boundary limits the performance of the model. The simplest approach to improve CoT is to optimize the step calculation RB B(c) to promote the value of RB. Specifically, Tool-Usage [Paranjape et al., 2023] and Program-of-Thought (PoT) [Chen et al., 2024b] have shown significant success in CoT optimization. We explain the rationale behind their effectiveness, why PoT consistently outperforms direct Tool Usage [Yao et al., 2023 , Chen et al., 2023] , and take them as examples to demonstrate how to improve CoT by promoting RB.Tool Usage can boost the value of RB for an LLM. When the model uses tools [Paranjape et al., 2023] , we can simply think that the model can perform calculations with infinite precision, so that the RB of mathematical calculations tends to infinity, viz B(c) → +∞. It is obvious that the combined RB of the model can be calculated as:𝓑 "#$% 𝓑 "&$% 𝓑 '($% 𝓑 ')$% 0.8 0.6 0.4 0.2 0 𝓑 !"# 𝓑 #""$ 𝓑 %"# Practical value Theoretical interval 𝓑 !"# 𝓑 #""$EQUATION)Easy to get, B Tool (c, p) > B CoT (c, p), this shows that Tool Usage can improve the boundary of reasoning. This explains why Tool Usage can have better performance than vanilla CoT (as shown in Table 1 ). Furthermore, as shown in Figure 5 , the distribution of theoretical RB and the actual one almost perfectly coincide. This also demonstrates the reliability and applicability of our theory.Program-of-Thought can further enhance the value of LLM's RB. Equation ( 4) reveals that an LLM's RB hinges entirely on its planning capability. Since natural language can be verbose, it hinders the planning capability of LLM [Gao et al., 2023 , Hu et al., 2023 , Puerto et al., 2024 , Chen et al., 2024b] . PoT [Chen et al., 2023] offers a clearer representation of logic using code, allowing for clearer planning (as shown in Figure 2 (b, c )). This leads to finer-grained planning reasoning B * (p) > B(p).Then the PoT reasoning boundary B PoT (c, p) > B Tool (c, p), aligning with the observed performance gains of PoT over Tool Usage (see Table 1 ). Furthermore, Figure 5 visually demonstrates that PoT's theoretical and practical reasoning boundaries consistently outperform Tool Usage. This reinforces the theoretical advantage of PoT and its empirical effectiveness.Enhancing RB is crucial for optimizing CoT, but requires changes to the model or its reasoning architecture to be effective. Therefore, we need to consider how to optimize the reasoning path so that the difficulty satisfies the RB (d * = B Acc=K1 ) instead of the original RB (d = B Acc=K2 ), where K 2 < K 1 . According to Equation (3), B is affected by both arithmetical RB and planning RB. Given B, we consider optimizing reasoning ability from the following two strategies as examples2 :Complex CoT (CCoT): By increasing the boundary of planning to reduce the pressure of single-step calculation, reduce the arithmetical RB, and then get smaller d; However, it introduces more planning steps, which adds the planning pressure. As shown in Figure 6 , the model performance first increases and then decreases with the increasing number of CCoT steps.Least-to-Most (LtM): By dividing multiple sub-questions to reduce the pressure of local planning within a sub-question, reduce the boundary of local planning, and then get smaller d. However, even though it can release local planning pressure (as demonstrated in Figure 7 ), this approach simultaneously intensifies global planning pressure by generating an excessive number of subquestions (as depicted in Figure 15 ).(1) CCoT needs to keep balance in the number of reasoning steps and calculation pressure.(2) Although the pressure of local planning has been reduced, LtM has not effectively reduced the pressure of global planning, nor the pressure of optimization calculations.Minimum acceptable reasoning paths prompting can further achieve better CoT within a specific RB. To address the aforementioned two issues, we proposed Minimum Acceptable Reasoning Paths (MARP). Our first objective is to alleviate the computational burden of the model. We achieve this by introducing instructions that set an upper limit on its single-step computational capacity, thereby optimizing the boundary of its computational reasoning. Secondly, we aim to enhance the model's acceptability. Within the calculation and planning boundary, we increase the amount of computation performed in each step in demonstrations as much as possible while simultaneously reducing the number of global planning steps, which effectively mitigates planning pressure. As shown in Table 1 , MARP demonstrably improves model performance and effectively reduces the token consumption. By maximizing operations per step, MARP leads to a more streamlined and efficient problem-solving process. Detailed descriptions of this strategy are shown in Appendix G.  
expansion verification & exploration: 
 RB can be extended to various models. To extend our mechanism's applicability, we verify the mechanism on 25 diverse models (details in Table 3 ). As shown in Figure 8 (a), we observe a positive correlation between reasoning boundary and model accuracy on mathematical benchmarks. Moreover, the models that use mathematical data such as MathInstruct for SFT, often have interesting outliers that are different from the general LLMs' area, but they also satisfy a positive correlation with However, as shown in Figure 8 (c), we find some interesting phenomena. For example, the main difference between the current open-source model and the closed-source model is still CFRB. Except for the closed source model, the CFRB of all models is 0. It shows the potential and the direction of the model optimization. Furthermore, a scaling law of RB can also emerge (as shown in Figure 9 ): reasoning boundary increased with model parameter count and data quality.RB can be extended to more tasks. To assess the RB in more tasks, we evaluate them on a multilingual mathematical reasoning task. Inspired by Qin et al. [2024b] , we hypothesize that multilingual RB, assessed through direct answer accuracy across different languages, mathematical computation RB, represented by the maximum product result, and reasoning planning RB, indicated by the planning steps, are orthogonal dimensions of performance. We propose that these RBs can be effectively combined using a weighted harmonic mean. As illustrated in Figure 10 confirms that the combined RB maintains the expected three different RBs. Detailed implementation description is shown in Appendix I.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: As shown in Section C to Section H, we have provided detailed descriptions and analyses of the experimental setups for all our investigations.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
related work: 
 In this section, we review recent literature related to Chain-of-Thought (CoT) prompting, focusing on theoretical and empirical investigations. methodology to enhance model performance from a mechanistic perspective. We anticipate that it will furnish systematic insights for ongoing research and inform future developments in the field.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We report error bars in Table 1 and Figure 4 , and explain the error variables in Section 3. However, error bars are not reported for all tasks because it would be too expensive for human annotation and computational resource consumption.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
discussion: 
 Discussion on the Boundaries Improvements Furthermore, in order to better understand the best existing LLMs, we utilize RBF to test the current most advanced GPT-series models. As shown in Figure 11 , all reasoning boundaries improve a lot compared to the last version which also achieves performance enhancement. Notably, the CFRB increases slightly compared with the improvement of CIRB between GPT-3.5 and GPT-4o. But o1 significantly improves the CFRB. Furthermore, as shown in Figure 14 in Appendix, o1 shows extremely significant improvements on CFRB, which is almost three times of other models. We attribute it to the fact that the advanced Reinforce-Learning and Inference Scaling strategies play a key role in improving this part of the ability compared with the normal improvements in CFRB, which might trigger more in-depth research.Broader impacts. Our framework is the first work to quantify the reasoning upper-bound of LLMs. This enables the explanation for a huge part of the valid CoT framework. We hope that our work can provide new insights and more systematic guidance for future interpretability analysis of CoT. For social impact, this work may have a certain impact on the controllable and explainable AGI.Limitations & Future. Due to the cost and time constraints, this work does not discuss the complex relationships such as causal conditions among the basic RBs. In addition, evaluating the robustness and applicability of CoT reasoning boundaries-related techniques in dynamic scenarios will be crucial for future research. 
conclusion: 
 This study introduces a novel reasoning boundaries framework (RBF) to quantify and optimize the limitations of LLMs in CoT tasks. Specifically, we propose the concept of reasoning boundaries (RBs) and the combination law of RBs in more complex scenarios for quantitative metrics. We further introduce three categories of RB for CoT optimizations. The framework is validated through extensive experiments across 27 models and 5 tasks. Furthermore, we improve the CoT in both RB and question optimization perspectives to achieve state-of-the-art performance in BIGGSM. We hope that this framework paves the way for further research on understanding and enhancing LLMs' reasoning capabilities.where N i1 = ∂B * (x1,x2,...,xn|m) ∂xi . Then the original formula is expressed as:B(t 1 , t 2 , . . . , t n |m) ≈ 1 (n -1) n i=1 Ni1 ti-bi -k i (13)Given the minimal change in the derivative within the observable range, N i1 is treated as a constant N i in this task for simplicity. Experimental results show that, if sub-RBs are separated independently, k i is typically 0. Since t i cannot be directly quantified, we use basic form of B(t i |m) as its quantized substitute, thus simplifying the combination law as: To determine the constants, we first fit parameters to a model using a development dataset (or 20% of the test dataset if the development dataset is not available). This fitting process yields the corresponding constants. For a given task and prompt strategy, these constants remain fixed. Additionally, once the combination law constants are established, different reasoning boundaries are determined through a binary search on performance in a standard setting (3-shot CoT).B(t 1 , t 2 , . . . , t n |m) ≈ 1 (n -1)For instance, we use binary search to identify a reasoning boundary that ensures the accuracy of all problems below that boundary approaches 90%, achieving B Acc=90% . For one model, one task, and one prompt type, the reasoning boundary remains fixed. Zero-shot and few-shot settings share the same set of reasoning boundaries. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We have discussed the broader impacts of our work in Section 8. In addition, since our work is more like providing an empirical formula and has no additional social harmfulness, we do not discuss this part.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [NA] Justification: The paper does not use existing assets. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [Yes] Justification: As shown in Section J, We describe and analyze the details and ethical considerations of our crowdsourcing in detail.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [No] Justification: Since our region and institution are not required to provide IRB approval, we do not describe this section. We are convinced that our work complies with the NeurIPS Code of Ethics and the guidelines.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: From a practical standpoint, our mechanism framework exhibits universal adaptability, making it suitable for application in a wide range of scenarios. When confronted with a new problem context, the framework enables a systematic approach to problem-solving. A key feature of the framework is its reliance on the weighted harmonic mean, which imparts advantageous mathematical properties to its structure. Specifically, the framework operates effectively if the reasoning process can be segmented into relatively independent boundaries. This segmentation allows the framework to be fully leveraged in addressing diverse problems.Reasoning Boundary Application. In the case of a vertical domain problem based on CoT reasoning, the process can be divided into two key boundary levels: task planning and domainspecific reasoning. These can be modeled as follows:EQUATIONwhere: B p represents the task planning boundary, B v represents the vertical domain boundary, and k 1 is a constant reflecting the degree of boundary independence.Reasoning Boundary Definition & Segmentation. Neglecting any of these boundaries only results in an increase in k, but keeping the overall efficiency of the framework. If the reasoning boundary is well-defined and independent, the value of k approaches zero, showcasing the effectiveness of our mechanism framework.Further Reasoning Boundary Segmentation. Further refinement of the vertical domain boundary, B v , into B v1 and B v2 is straightforward. No additional complexity is introduced, as the following relationship holds:EQUATION)Thus, the overall boundary equation can be extended to:EQUATION)This formulation allows for flexible and systematic boundary division at multiple levels, enhancing the framework's practical utility across various problem domains.Challenging Reasoning Boundary Measurement. In addition, we propose an alternative method to measure the reasoning boundaries. This approach allows the model to provide direct answers without relying on CoT reasoning steps. By doing so, the model's reasoning process for a specific task depends solely on a single reasoning boundary, which can be represented as follows:EQUATION)For instance, in the MGSM task, assessing multilingual reasoning boundary is particularly challenging.To address this, we directly evaluate the model's performance using a direct prompting strategy without CoT outputs and use this performance to define the multilingual reasoning boundary, which in turn helps determine the corresponding normalization constant. Subsequently, we apply multilingual CoT reasoning to the MGSM task to calculate the combined boundary using the framework's combination law. This approach provides a more generalized solution and may be more adaptable to specific needs.To adequately assess the reasoning boundary of LLMs, it is essential to develop a dataset that encompasses a range of complexities and reasoning boundaries. To address these challenges, we propose a novel approach to constructing a mathematical reasoning dataset using manual synthesis and annotation which finally leads to the BIGGSM benchmark. Specifically, our proposed method involves the manual synthesis and annotation of a mathematical reasoning dataset. The construction process includes the following steps:Step 1: Domain Template Generation Initially, we employ a prompt-driven LLM (GPT-4) to generate complex scenarios necessitating multi-step calculations. This process also yields initial example templates. Specifically, the prompt given to the large model is as follows:Generate a scenario-related template involving multiple mathematical steps to solve a realworld problem. Ensure the scenario requires the application of different mathematical concepts. Please use "[VAR]" as a variable to mark the template of the question.Step 2: Natural Language Template Creation Recognizing that LLMs can produce errors and logical inconsistencies, we refine these initial templates to improve their accuracy and add mathematical calculations. To facilitate the generation of extended sequences, we decompose the templates into smaller, loopable segments that incrementally meet the multi-step reasoning demands.Step 3: Domain Template Augmentation To address the limited diversity in individual samples and provide a broader evaluation of LLMs' mathematical abilities, we use an LLM (GPT-4) to generate at least three alternative augmented templates for each original template and step. The generation prompt we use is as follows:Create three alternative versions of the following template that introduce different complexities or variables, ensuring each version demands an equivalent level of reasoning.Step 4: Numeric Filling Once all templates are prepared, we aim to test the upper-bound of the LLMs' computational reasoning boundary by introducing numerical values ranging from 1 to 1e5 in multiplication tasks. This step is designed to thoroughly assess the models' performance across a spectrum of numerical challenges. Step 5: Manual Annotation To ensure the quality and logical coherence of our synthetic samples, we manually review them to correct any errors introduced during the automated generation process. Finally, we hired three experts to mark whether the samples in the data set were correct. Only for those samples where more than two experts agreed did we retain the corresponding samples. The Cohen's kappa value marked by the experts was 0.97, which indicates the perfect agreement.Our dataset comprises 610 test samples, which is extensive when compared to the GSM8K dataset. It features a broader range of procedural steps, varying from 1 to 16 steps. Additionally, our dataset encompasses a wider spectrum of computational efforts, ranging from 6 to 3e5.This section outlines the process used to construct datasets for examining the existence of reasoning boundaries (RB) in basic arithmetic calculations. Initially, we identify the operations for investigation, namely addition, subtraction, multiplication, and division. We then determine the range of integer operands (x and y), starting from 1 to 1e10, subsequently extending to 1e20. A random number generator is employed to produce independent and unbiased pairs of x and y within the specified range. For each pair, we compute the expected correct outcome of the chosen operation using standard arithmetic procedures. In addition, in order to ensure that decimals do not affect the computational complexity, we restrict our analysis to integer operands and outcomes to control for complexity and randomly generate numerical values of x and y.The prompt configuration in our study involves inputting the structured data into a computational model to analyze the arithmetic accuracy. The following prompting is used for LLMs' input:Please calculate the formula given below:x op y= where op denotes the arithmetic operation (selected from addition, subtraction, multiplication, division). And x and y values are generated from Section D.1. The final experimental results are shown in Figure 13 .We propose that the natural language multi-hop CoT task comprises two sub-tasks: multi-hop planning and knowledge step reasoning for multi-hop question answering. To address the challenge of measuring knowledge difficulty, we utilize a NER model4 to identify the number of knowledge entities in each hop, thus marking the knowledge step reasoning RB in the single-step task. Formally, let B(h) represent the RB of multi-hop planning and B(e) denote the RB of knowledge step reasoning.The combined RB satisfies the following combination law:EQUATION)F Analysis for Complex-CoT and Least-to-Most within Reasoning Path Optimization Perspective Complex CoT Prompting can achieve better CoT within a specific RB by simplifying the calculation reasoning step. We believe that Complex CoT optimizes the performance of the model by allowing the model to reach its computational limit as much as possible in single-step reasoning. Therefore, the combined RB for Complex-CoT can be expressed as:B Complex (p, c) = lim B(c)→B Acc=100% (c) 1 N1 (B(c)-b1) + N2 (B ′ (p)-b2) (20)Assuming the premises of RB remain unchanged (B Complex (p, c) = B CoT (p, c)), it can obviously yield the solution B ′ (p) > B(p). Therefore, the model can accept more steps of reasoning boundary, that is, if the planning difficulty d p is less than reasoning capability B ′ (p), the accuracy is higher. In order to analyze this problem, we adopted a meta-analysis method. We count the performance of the work of Jin et al. [2024] , Fu et al. [2023] using Complex CoT. The relationship between the performance label and the number of steps is shown in Figure 6 (left). For most multi-step reasoning tasks, generally speaking, within a certain range, as the number of steps increases, the computational pressure of the model is relieved and the performance is improved, which is consistent with the theory and exploration of Feng et al. [2024] , Wang et al. [2023b] , Valmeekam et al. [2023] .However, We can also clearly recognize the flaws of Complex CoT. Once the difficulty of planning d p (that is, the number of planning steps) is greater than B ′ (p), it exceeds the capabilities of the model and the performance will decline. We can observe that for single-step calculation reasoning, as shown in Figure 6 (right), the performance of using Complex CoT will gradually decrease. The rest of mathematical reasoning will also decrease when the number of steps is greater than a certain threshold. This phenomenon can also be explained by our combination law. While the amount of calculation is reduced, the number of reasoning steps is also increasing. If the acceptable number of reasoning steps is exceeded, the reasoning boundary is exceeded, and the model performance will decline, which demonstrates that it is necessary to keep a balance between the number of reasoning steps and computational pressure (see Appendix G for detailed meta-analysis process).Limitation: Need to keep balance in the number of reasoning steps and calculation pressure.Least-to-Most Prompting can achieve better CoT within a specific RB by simplifying the planning reasoning paths. Least-to-most prompting structures problem-solving hierarchically, by breaking questions into smaller sub-questions and further solving them one-by-one. Accordingly, the Least-to-most RB can be divided into three sub-RBs, namely, the problem decomposition RB B(d), the problem planning RB B(p), and the single-step calculation RB B(c). Therefore, the combined RB for least-to-most can be expressed as:EQUATIONIdeally, if the problem decomposition ability of the model is excellent (B(d) → +∞), it can decompose the problem into sub-problems that can be solved in one step every time B(c) → 1, therefore the least-to-most RB can be expressed as:EQUATION)Assuming the premises of RB remain unchanged ( BLtM (d, p, c) = B CoT (p, c)), it can obviously yield the solution B ′ (c) > B(c). On the contrary, the model can accept larger difficulty d, which also shows that using least-to-most prompting can effectively increase the maximum of acceptable calculation RB under a given RB (as shown in Figure 7 ), thereby improving model performance. As shown in Table 1 , we find that LLM can be optimized by Least-to-most from vanilla CoT.However, the performance improvement of the model is not significant, which we attribute to the fact that the current model cannot push its performance to the ideal limit. As shown in Figure 15 (a) , the reasoning boundary of the model cannot make each reasoning step completely tend to 1, which also leads to the difference in reasoning performance in Figure 15 (b ). In the meantime, the model's ability to divide problems is also limited. What's more, as shown in Figure 15 (c), in around 90% of cases, the model will only divide less than 6 problems, which also limits the performance.Limitation: Although the pressure of local planning has been reduced, it has not actually effectively reduced the pressure of global planning, nor the pressure of optimization calculations.In order to discuss the advantages and limitations of Complex-CoT, we conduct two detailed metaanalyses through two distinct perspectives. The first one assesses the influence of the reasoning steps demonstrated in In-Context Learning (ICL) across various tasks, while the second evaluates the effects of employing a fixed number of reasoning steps in ICL on questions with different reasoning steps. These meta-analyses aim to compare the efficacy of these methods against prior studies systematically. Specifically, we conducted a systematic search for relevant studies addressing the same problem tackled by Jin et al. [2024] , Fu et al. [2023] , Shum et al. [2023] , Sun et al. [2023] , and Jiang et al. [2023b] . We ensured that retrieved studies were pertinent, focusing on studies that addressed the same problem and used similar evaluation metrics.From each selected study, including Jin et al. [2024] and Fu et al. [2023] , we evaluate the performance using Complex CoT. The relationship between performance and the number of Complex CoT's steps is shown in Figure 16 . For most multi-step reasoning tasks, as the number of steps increases within a certain range, the computational load decreases, and performance improves.However, as described in Appendix F, the flaws of Complex CoT are apparent. When the difficulty of planning (d p ), defined as the number of planning steps, exceeds B ′ (p), the model's capabilities are surpassed, leading to a performance decline. This is evident in single-step calculation reasoning, as shown in Figure 16 (c, e ), where performance using Complex CoT gradually decreases. Similarly, for other mathematical reasoning tasks, performance decreases when the number of steps exceeds a certain threshold. This phenomenon aligns with our combination law: while reducing the amount of calculation, the number of reasoning steps increases. Exceeding the acceptable number of reasoning steps surpasses the reasoning boundary, causing a decline in model performance. Therefore, maintaining a balance between the number of reasoning steps and computational pressure is crucial.Furthermore, to gain a nuanced understanding of the impact of Complex CoT when the number of steps exceeds the golden step number, we conduct further meta-analysis from Fu et al. [2023] , Shum et al. [2023] , Sun et al. [2023] , Jiang et al. [2023b] . Specifically, as illustrated in Figure 17 (a, b), our analysis reveals that for problems of low complexity and with smaller golden step numbers, Complex CoT tends to underperform compared to Vanilla CoT. Notably, it is only when the reasoning steps exceed two that Complex CoT outperforms Vanilla CoT. This suggests that Complex CoT effectively optimizes single-step computations and enhances model performance for complex [Minimum Reasoning Path Prompting] You need to perform multi-step reasoning, with each step carrying out as many basic operations as possible.[Acceptable Reasoning Prompting] Remember, you can only complete tasks that contain up to 5 basic operations per step, and multiplication operations must be less than 1.5e5. The upper limit of the multiplication operations decreases as the number of operations per step increases.Question: Leo's assignment was divided into three parts. He finished the first part of his assignment in 25 minutes. It took him twice as long to finish the second part. If he was able to finish his assignment in 2 hours, how many minutes did Leo finish the third part of the assignment? problems. However, it increases the cognitive load for simple problems, resulting in a performance decline.Interestingly, this phenomenon is also observed with the simpler Vanilla CoT, as shown in Figure 17 (c). The model achieves significant performance gains only when the number of reasoning steps aligns with the target output steps. If the complexity of the planned steps exceeds the necessary reasoning boundary, or if there is no effective optimization for reasoning boundary, the performance deteriorates.To address the two aforementioned limitations, we propose Minimum Acceptable Reasoning Paths (MARP). Firstly, to reduce the model's computational load, we introduce instructions that limit its single-step computing power, thereby optimizing its reasoning boundary. Secondly, to enhance the Model Acc. (↑) Input Token (↓) Output Token (↓)HotpotQA [Yang et al., 2018 model's acceptability, we increase the computation amount per step within this boundary and reduce the number of global planning steps, thus alleviating planning pressure.To control variables effectively, we make only the simplest modifications to the prompt to achieve the desired CoT optimization.Minimum Reasoning Path Prompting To alleviate the cognitive load associated with planning, it is essential to have the model respond to the question as succinctly as possible. This approach ensures that the focus remains on providing a short, clear and direct reasoning path. The following prompt is designed to achieve this objective:You need to perform multi-step reasoning, with each step carrying out as many basic operations as possible.Acceptable Reasoning Prompting To effectively utilize the model, it is crucial to define the upper-bound of reasoning boundary. This ensures that the complexity of the reasoning process is manageable and within acceptable bounds. The specific prompt to achieve this is as follows:Remember, you can only complete tasks that contain up to 5 basic operations per step, and multiplication operations must be less than 1.5e5. The upper limit of the multiplication operations decreases as the number of operations per step increases.This prompt is designed to set clear boundaries for the model's operations, thereby optimizing its performance and accuracy.Furthermore, it is necessary to enhance the demonstration within the corresponding in-context learning framework to meet the specific needs of our Model-Agnostic Reasoning Protocol (MARP). This involves refining the examples and instructions provided to ensure they align perfectly with the MARP requirements. Figures 18 and Figure 19 illustrate our MARP prompt, showcasing how to structure the demonstrations to facilitate effective learning and reasoning in natural language CoT and program-of-thought setting. By adhering to these guidelines, we can ensure that the model operates efficiently and produces reliable results.In summary, setting precise boundaries for reasoning boundary and optimizing in-context learning demonstrations are essential steps in enhancing the model's performance. By following the specified prompt and refining the MARP examples, we can achieve a high level of accuracy and efficiency in the model's reasoning processes.We employ 25 commonly used models to evaluate the extensibility of our framework to a broader range of models. The specific models are listed in Table 3 . For each model, we utilize the chat/instruct version whenever available to maximize their ability to follow instructions. Additionally, we deploy all models on the vLLM [Kwon et al., 2023] framework to ensure a fair comparison. Except for model OpenMath-series [Toshniwal et al., 2024] which does not conform to the vLLM format, all other models are deployed on vLLM for testing. All experiments on open-source models wereInspired by Qin et al. [2023] and Huang et al. [2023] , we propose that the multilingual mathematical CoT task comprises three sub-tasks: step planning, step calculation, and multi-modal expression. We evaluate the model's mathematical expression ability in different languages based on its zero-shot direct performance on MGSM, as reported by Qin et al. [2023] . For relevant parameter calculations, please see the "Challenging Reasoning Boundary Measurement" part of Appendix B. Formally, let step planning RB be denoted by B(p), step calculation RB by B(c), and multilingual expression RB by B(l). The combined RB satisfies the following law:EQUATION)As shown in Figure 10 , the performance distribution of RB (including B Acc=90% and B Acc=10% ) in the multilingual mathematical reasoning task aligns with the proposed combination law in Equation ( 23). Moreover, three distinct RBs are evident in Figure 10 .Data Access. Our data is adapted from GSM8K [Cobbe et al., 2021] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: We will release our code in the official version of the subsequent paper to provide reproduction and provide more help to the future community.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.See detailed analysis for these two strategies in Appendix F.The basic form B(ti|m) and the combined form B(+∞, . . . , +∞, ti, +∞, . . . , +∞|m) are not completely equivalent.https://huggingface.co/dslim/bert-base-NER 


Decision: The aim of this paper is to improve the ability of Large Language Models in solving complex reasoning tasks by addressing two main drawbacks in the Chain-Of-Thought (CoT) approach: the lack of quantification metrics and the absence of optimization guidance. The authors introduce the concept of Reasoning Granularity (RG) and a combination law for assessing the upper bound of CoT. They propose three categories of RGs to guide the optimization of CoT strategies and conduct extensive experiments on 4 different tasks and 25 models to validate this approach, highlighting its generality.

All reviewers agree that this study makes a significant contribution to the CoT approach in LLMs. They commend the well-organized paper, innovative and well-founded framework, and convincing empirical analysis. In summary, this is a strong piece of work.

To further enhance the quality of this study, I suggest including in the paper or the appendix some responses to reviewers’ questions. Notably, incorporating information on extending this approach to new incoming tasks, addressing concerns about the robustness of the combination law, and summarizing the discussion about assessing the difficulty level of an incoming task could be valuable additions.

Reviews: review: paper_summary: paper_summary: The paper introduces a novel reasoning granularity (RG) framework to quantify and optimize CoT capabilities in LLMs. The authors define RG to measure the upper bounds of CoT and establish a combination law for RG, enabling a practical quantitative approach. They categorize tasks into three categories based on accuracy and propose methods to optimize CoT for improvement. Extensive experiments across models and tasks demonstrate the framework's efficacy in explaining and optimizing CoT performance.
strength_weakness: strength_weakness: 1. While the framework is validated on 4 tasks, broader evaluations across more diverse tasks would strengthen the generalizability of the findings.

2. The evaluation requires the difficulty level of the task as input, which is not always available. The paper should discuss how to evaluate RG for a task without an explicit difficulty level.
questions: questions: 1. When evaluating RG on multi-hop question answering, the difficulty of sub-questions in each hop is measured by the number of entities, which may not necessarily be the case. Can you justify this choice?

2. Why does the categorization use 10% and 90% as the cut-off points? Is there statistical support for this categorization? Since most tasks fall in the difficult range of 10% to 90%, what insights does this framework offer for optimization within this range?

3. How would the RG framework perform with different types of reasoning tasks beyond those covered in the experiments?
limitations: limitations: Yes, discussed in paper.
paper_summary: paper_summary: This paper proposed a novel reasoning granularities (RG) methodological framework to quantitatively assess CoT capabilities and provide guidance on optimizing CoT performance. The experiement results show an upper bound of CoT, and the authors have proposed three catergories of RG to optimize CoT with combination laws focused on RG promotion and reasoning path optimization for CoT improvement.
strength_weakness: strength_weakness: No major weekness from my perspective
questions: questions: Check the writing and grammar. Some occassional typo or mis-used comma.
limitations: limitations: Yes.
paper_summary: paper_summary: The article introduced a novel framework for quantifying and optimizing the reasoning capabilities of large language models (LLMs). The concept of Reasoning Granularity (RG) is innovative and may have the potential to significantly impact the field of natural language processing with LLMs.
strength_weakness: strength_weakness: 1. Although the paper has demonstrated the effectiveness of the RG framework across several models and tasks, it could further strengthen its claims by discussing how these findings might generalize to other types of reasoning tasks or different domains beyond the ones tested.
2. Compared to GPT4, the multi-step reasoning capability of GPT3.5 used in this article might be insufficient. It would be better to add experiments based on GPT4 to prove that the improvement comes from the stimulation of model capabilities, rather than the introduction of a priori frameworks for specific tasks.
questions: questions: 1. Can you provide some comparative results based on GPT4 or other models with stronger reasoning capabilities as baselines to demonstrate the performance improvement of RG in path planning?
limitations: limitations: Yes.
paper_summary: paper_summary: The paper introduces a Reasoning Granularity (RG) framework that quantifies and optimizes Chain-of-Thought reasoning in large language models. Through extensive experiments, the authors validate the RG framework's effectiveness across various tasks and models, providing new insights into enhancing reasoning capabilities in LLM.
strength_weakness: strength_weakness: 1. **Lack of Theoretical Analysis**: While the paper provides an empirical framework and experimental validation, it may not delve deeply enough into the theoretical understanding of the concept of Reasoning Granularity (RG). A more rigorous theoretical foundation, although difficult, could strengthen the arguments and enhance the contribution of the paper.
2. **Limited Generalizability**: The RG framework, although validated across 25 models and 4 tasks, may not be fully generalizable to all types of large language models or reasoning tasks.
questions: questions: See the Weakness section.
limitations: limitations: Authors have adequately discussed limitations.
paper_summary: paper_summary: The paper presents a Reasoning Granularity (RG) framework designed to quantify and optimize the Chain-of-Thought (CoT) reasoning capabilities of large language models (LLMs). The framework introduces a new metric, RG, to measure the complexity of reasoning tasks that LLMs can handle. It also establishes a combination law to integrate multiple reasoning tasks and categorize RG into three distinct types. The study validates this framework through extensive experiments and demonstrates the effectiveness of various optimization strategies to enhance CoT performance.

Specifically, the authors: 
1. introduces the concept of RG to quantify the upper bound on task-specific reasoning complexity within a model.
Defines a combination law for RG, using the weighted harmonic mean to integrate multiple reasoning tasks.
2. proposes three categories of RG (Completely Feasible, Partially Feasible, Completely Infeasible) to guide the optimization of CoT performance.
3. introduces Minimum Acceptable Reasoning Paths (MARP) to optimize reasoning paths and reduce computational load.

To validate the effectiveness of their work, they:
1. validate the RG framework through extensive experiments on 25 models and 4 tasks, demonstrating its robustness and applicability.
2. Explains the effectiveness of 10 CoT strategies and provides optimization techniques like Tool Usage and Program-of-Thought (PoT).
3. They also establish a theoretical foundation for understanding the boundaries of CoT reasoning capabilities in LLMs. A combination law for RG is proposed to generalize the quantification in complex scenarios.

Overall, the paper advances both theoretical understanding and practical optimization of CoT reasoning in LLMs, providing a robust framework and concrete metrics to enhance model performance on complex reasoning tasks.
strength_weakness: strength_weakness: 1. Although the framework is validated on 25 models and 4 tasks, there may be concerns about how well these results generalize to other tasks or models not included in the study, as the solution for now are more task/RG-specific. 
2. The combination law for RG relies on certain assumptions that may not hold universally across all reasoning tasks or model architectures. Further empirical validation is needed to confirm these assumptions in diverse settings.
questions: questions: 1. Can you provide more detailed examples of tasks that fall into each of the three RG categories (Completely Feasible, Partially Feasible, and Completely Infeasible)? How do these categories affect the model’s optimization process?
2. How robust is the combination law of RG across different types of reasoning tasks? Are there specific scenarios where this law does not hold or requires adjustments?
limitations: limitations: 1. P2 in weaknesses. 
2. As far as I understand, the benchmark that is used in the evaluation, BIGGSM focuses more on math problems, like in MATH and GSM8K. This may raise concerns about the generality of the solution on other types of reasonings, for example in StrategyQA, or planning benchmarks. Improve the diversity of benchmarks could provide a more holistic view of the framework's impact.

Comments: title: Reply to Author
comment: Thank you for your thoughtful feedback and recognition. Your suggestion to evaluate the complexity of multi-hop questions is indeed a significant direction for future research.

1. **Concern about generalizability for difficulty calculation.**

Thank you for your concerns. In fact, our Combination Law is structured into two distinct parts: verification and application. 

**Verification Section:** Only the verification section necessitates annotated rationales, along with the corresponding number of steps and their associated difficulty levels. As long as there is a corresponding label, it can be applied to at least 5 tasks mentioned in our main paper and rebuttal.

Furthermore, we also provide another methods for better generalization. The model can be allowed to give answers directly without giving any CoT items. In this way, it can be ensured that the model's reasoning granularity on this task only depends on a certain reasoning granularity. It satisfies that:
$$
G=\frac{1}{\frac{1}{G_{1}}+k}
$$

For example, on the MGSM task, multilingual reasoning granularity is extremely difficult to measure. We directly use the performance of the direct prompting strategy on MGSM without any CoT output as the multi-language reasoning granularity to calculate the corresponding normalization constant. Further, we utilize multilingual CoT on MGSM to calculate combined combination law. This approach may be more general and fit your needs.

Additionally, as previously detailed in the rebuttal, in cases of approximate or incomplete segmentation, it is sufficient to consolidate these reasoning granularities into the constant $k$.

**Application Section:** In the application part, when using combination law to explain why Tool Usage and PoT work, from a theoretical perspective, these difficulties are not required for calculation. In addition, the MARP we proposed based on combination law can also dynamically adjust different reasoning granularities to ensure performance without difficulty calculation, and its generalization has also been verified in datasets with huge differences in multiple fields.

2. **Concern about difficulty calculation on HotpotQA.**

For HotpotQA complexity evaluation, we utilized the rationale manually annotated by HotpotQA, observing that few non-bridge entities appeared, providing an approximate measure of the single-step difficulty. For example:
- Q: Chikku Bhukku is based on a 2003 movie, which was directed by whom?
- A: Kwak Jae-yong
- R:
  - Step1: The movie is based on the [2003] movie ["The Classic"].
  - Step2: ["The Classic"] is a [2003] romance melodrama film directed by [Kwak Jae-yong].

In this instance, the single-step statement "The movie is based on the [2003] movie ["The Classic"]." is definitely less complex than "The movie is based on the [2003] movie." as it imposes a lower cognitive and knowledge load for LLMs.

All in all, we appreciate your valuable suggestions again and will incorporate additional discussions in the next version of our work.
Thanks for your responses. I have raised my score to 7.
Thank you for your response. However, my concern regarding using the number of named entities to represent the complexity of multi-hop questions remains unresolved. 

Firstly, the complexity of a multi-hop QA task should be generalized rather than influenced by biases introduced by specific construction methods used in a particular dataset. Secondly, even within HotpotQA, complexity is more closely related to the number of bridging entities rather than the count of all named entities in a question.

Additionally, this issue ties into Weakness 2, which was not addressed in the previous response.

Overall, the paper presents a method for qualitatively analyzing CoT reasoning, several technical decisions need further refinement to strengthen the work.
Thank you for your thorough review and thoughtful feedback on our work. We will carefully incorporate all the points of discussion mentioned above in future revisions.
Thank you for your constructive feedback. We recognize that our initial description may have been too theoretical. To clarify, let's consider the task of solving a multilingual mathematical reasoning problem. Depending on the segmentation method employed, we can encounter three scenarios:
1. **Insufficient Segmentation:** If the combined reasoning granularity (RG) is directly divided into multilingual RG and planning RG, while neglecting the mathematical calculation RG, the constant term k will not equal zero. Assuming the calculation difficulty remains unchanged, it can be treated as a constant, disregarding any additional complexity introduced by this factor.
2. **Sufficient Segmentation:** If the combined reasoning granularity is segmented directly into multilingual planning RG and mathematical calculation RG, the constant term k becomes zero.
3. **Further Segmentation:** Additionally, if we further divide the multilingual planning RG into multilingual RG and planning RG, this segmentation remains consistent with our combination law.
Thank you very much for your careful review and recognition of our work. We will address the concerns you have highlighted and incorporate them into the subsequent versions of our project.
Thank you for your thoughtful review and appreciation of our work.

I fully concur with your perspective. For complex reasoning tasks, entities alone may not adequately capture the complexity of the problem. However, in the context of the HotpotQA dataset, knowledge entities are fundamental to representing the intricacies of multi-hop knowledge-based reasoning. Both the answers and the bridging hops in this dataset rely heavily on entities. Therefore, there is no "open-analysis" problem like 'What are the economic impacts of the trade agreements signed by the United States in the 1990s?’ as you mentioned. For instance, multi-hop reasoning typically follows the paradigm:

entity$_1$ $\rightarrow$ entity$_2$ $\cdots$ $\rightarrow$ entity$_n$, where entity$_n$ represents the answer.

In this entity-centric reasoning process, a question like "Where was the capital of West Germany when the Berlin Wall fell, and who was the president of the United States?" is undoubtedly simpler than "Who was the president of the United States when the Berlin Wall fell?"

In addition, how to evaluate the complexity of the open-analysis problem you mentioned is indeed an issue worth exploring in Chain-of-thought evaluation, and we will conduct more exploration in the future.
Thanks for your responses. R1 seems to be quite theoretical. Maybe some examples on real reasoning tasks could help understand the claim.
Thanks for your response. However, for Q1, the number of named entities mentioned in a complex question does not necessarily represent the difficulty of the question. For example, the question 'Who was the President of the United States when the Berlin Wall fell, and which city was the capital of West Germany at that time?' involves multiple named entities like 'President of the United States,' 'Berlin Wall,' and 'West Germany,' but can be answered relatively easily. On the other hand, a question like 'What are the economic impacts of the trade agreements signed by the United States in the 1990s?' mentions fewer named entities but is more difficult.
Thanks for your detailed reply. They mostly clarify my concerns. I think this is a solid work if those are included in the revision. I'll raise my rate to 7.
Sounds great!