CreDes: Causal Reasoning Enhancement and Dual-End Searching for Solving Long-Range Reasoning Problems using LLMs

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 08:58:33

Abstract: Large language models (LLMs) have demonstrated limitations in handling combinatorial optimization problems involving long-range reasoning, partially due to causal hallucinations and huge search space. As for causal hallucinations, i.e., the inconsistency between reasoning and corresponding state transition, this paper introduces the Causal Relationship Enhancement (CRE) mechanism combining cause-effect interventions and the Average Treatment Effect (ATE) to guarantee the solid causal rightness between each step of reasoning and state transition. As for the long causal range and huge search space limiting the performances of existing models featuring single-direction search, a Dual-End Searching (DES) approach is proposed to seek solutions by simultaneously starting from both the initial and goal states on the causal probability tree. By integrating CRE and DES (CreDes), our model has realized simultaneous multi-step reasoning, circumventing the inefficiencies from cascading multiple one-step reasoning like the Chain-of-Thought (CoT). Experiments demonstrate that CreDes significantly outperforms existing State-Of-The-Art (SOTA) solutions in long-range reasoning tasks in terms of both accuracy and time efficiency.

Main paper: 
 introduction: 
 Reasoning aims to realize the causal transfer from the initial state to the goal state through several intermediate steps, which widely exists in the domains of Societal Simulation [1, 2, 3] , Economic Simulation [4, 5, 6] , Game Theory [7, 8, 9] and Gaming [10, 11, 12] , etc. LLMs like GPT-3 have shown competitive performances in many reasoning tasks [13, 14, 15] . However, their performances and efficiency are limited when dealing with complex combinatorial optimization problems that require multi-step long-range reasoning [16] .The first challenge is causal hallucinations, i.e., causality between one-step reasoning (OSR) and state transition in LLMs is not always guaranteed. Similar to pre-trained LLMs that are prone to produce hallucinations when processing certain factual information, causal hallucinations reflect the fact that LLMs lack rigor due to inherent randomness in accomplishing complex mathematical [17, 18, 19] , logical [20, 21] , or common-sense reasoning [22, 23, 24] , which is somehow entrenched in statistical inevitability and independent of the Transformer architecture or data quality [25] . For example, CoTbased finite-step reasoning methods [26, 27] suffer from causal hallucinations, which cannot effectively ensure the causality between OSR and state transition in LLMs, resulting in unreliable reasoning and relatively low success rates (especially for long-range reasoning problems with significant error accumulation effects). The reasonableness between OSR and state transition can be summarized as follows: There is a causal relationship between reasonable OSR and state transition. At the same time, there is only a correlation or no relationship between unreasonable OSR and state transition, which Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute.suggests that training with cross-entropy loss alone does not enable the model to have sufficient causal rigor. Inspired by this, we designed the CRE mechanism to make each step of reasoning correct and causally sound by including the causality measure between OSR and state transition as part of the training objective, thus more closely modeling the rigor, adaptability, and comprehensiveness of human reasoning [28] .The second challenge is that long-range reasoning problems have a huge search space. Although complex architectures such as CoT, Tree of Thought (ToT) [29] , and Program of Thought (PoT) [30] can effectively improve the reasoning accuracy of LLMs through external guidance, they are limited when handling long-range reasoning processes and task decomposition. A crucial reason is that long-range reasoning has a huge state space, i.e., each branch in the state transition process expands the search space approximately exponentially. Most of the existing LLM-based methods, e.g., MonteCarlo search [31] , are based on unidirectional reasoning, making them time-inefficient and easy to fall into local optima when dealing with reasoning problems with large search spaces. In this paper, a bi-directional Dual-End Searching method is developed, which first decomposes a long-range reasoning problem into a combination of short-range reasoning problems and then searches for the intersection of two causal probability trees starting from the initial and goal states, respectively.A structured and generalized reasoning framework, CreDes, is developed for long-range reasoning with LLMs in this paper, and the contributions can be summarized as follows:First, the CRE mechanism is introduced to improve the rigor of LLM-based long-range reasoning methods: Structural Causal Modeling (SCM) is exploited to enhance the causality between OSR and state transitions, involving performing causal interventions and optimizing the absolute value of ATE during training, which has effectively alleviated causal hallucinations in long-range reasoning of LLMs.Second, the DES method is developed to improve the search efficiency for long-range reasoning:After constructing causal probability trees starting from the initial states and ending at the goal states, long-range reasoning (e.g., 12 steps) is transformed into more manageable combinations of smaller segments (e.g., 2 or 4 steps) by minimizing the distances between leaves of the tree and employing end-matching techniques. By avoiding long-range sequential search from scratch, the DES method has greatly lowered the complexity when solving long-range reasoning problems.Third, simultaneous multi-step reasoning is realized to improve the time-efficiency of long-range reasoning: By integrating CRE and DES, CreDes can perform simultaneous multi-step reasoning within the model, i.e., avoiding the inefficiency of cascading single-step reasoning in frameworks such as CoT. While ensuring the accuracy of the reasoning process, CreDes can significantly reduce the time required for multi-step reasoning in LLMs.Fourth, adequate and rigorous testing of CreDes: CreDes has been extensively tested in the Blocksworld, GSM8K, and Hanoi Tower scenarios, respectively, and the experimental results show that CreDes outperforms existing SOTA regarding reasoning accuracy and time efficiency. 
related work: 
 Decision-Making Capabilities in LLMs: The core of intelligence partially lies in planning, which encompasses generating a sequence of actions aimed at accomplishing a predefined objective [32, 33] .Classical planning methods have found extensive application in robotics and embodied environments, where they are commonly employed to guide decision-making processes externally [34, 35] . Recent advancements, such as the Chain-of-Thought model [26, 36, 37] , have significantly bolstered the LLMs' capability to perform detailed reasoning [38, 39, 40] . This model breaks down intricate queries into a series of manageable steps, thereby enhancing the LLMs' decision-making ability.Subsequent initiatives like ReACT [41] have modified this approach to improve reasoning ability in decision contexts using a CoT-based framework. Additionally, Reflexion [42] provides a corrective mechanism that enables LLMs to recognize their errors during the decision-making process, reflect on these mistakes, and make accurate decisions in subsequent attempts. Further developments have led to the creation of tree-based decision-making frameworks that tailor LLM capabilities to specific scenarios. The Tree-of-Thought [29] utilizes Breadth First Search (BFS) and Depth First Search (DFS) algorithms to facilitate decision-making in activities such as the Game of 24, Creative Writing, and Mini Crosswords. Meanwhile, Reasoning via Planning (RAP) [43] employs the Monte Carlo Tree Search technique to optimize solutions across tasks like Blocksworld [44] , Math Reasoning [45] .N 1 N 2 N 3 N 1 N 2 N 3 T head T tail ℒ ℒ ℒ ℒDFSDT [46] proposed an efficient version of DFS for LLMs to make decisions, but it lacks the judgment ability to evaluate different decisions. JUDEC [47] utilizes an Elo rating system to enable LLMs to develop self-assessment capabilities, thereby enabling them to generate optimal solutions for a wide range of real-world tasks, independent of any task-specific expertise. Lastly, Graphof-Thought [48] represents the thoughts as nodes in a graph, combining thoughts non-sequentially.Encouraged by the studies above, we leveraged LLMs to solve long-range reasoning problems.Integrating Causal Analysis in LLMs for Multi-step Decision-Making: The causal analysis aims to discern and elucidate the causal relationships between actions, circumstances, or decisions. This method entails investigating the origins or causes leading to an event and the potential consequences that follow [49, 50, 51] . Although various causal models may produce identical observational distributions, they can yield distinct distributions when interventions are applied [52] . Therefore, using interventions allows for the distinction of possible causal frameworks that align with the observed data [53, 54] . Previous work suggests that, while CoT has been lauded for its potential to improve task performance, its application does not always lead to enhanced outcomes [36, 55] . Also, research has shown that the statistical pretraining of LLMs encourages models to achieve high empirical performance but not necessarily to reason [56, 57, 58, 59 ]. Inspired by this, we designed the CRE mechanism to control the causal hallucinations of LLMs to solve long-range reasoning problems.Solving Multi-step Problems with LLMs: Recent studies have shown that with substantial design, LLMs are capable of performing not only basic arithmetic tasks but also complex multi-step reasoning [60, 61] . For instance, increasing computational resources significantly enhances the accuracy of datasets like GSM8K [62] . Concurrently, Research [63] demonstrated that a 2B parameter LLM could achieve 89.9% accuracy in 5x5 multiplication tasks using curriculum learning with 50 million training instances. This evidence suggests that adequately scaled LLMs can process multiple reasoning steps effectively internally. While trees are frequently used to represent games (especially extensive-form games [64, 65] ) and sequential reasoning problems [66] , it was Shafer's groundbreaking work [67] that initially established a framework for understanding causality through the use of probability trees. Inspired by Shafer's approach, we recognized that LLMs tend to struggle with long-range reasoning problems involving multiple steps but excel in short-range reasoning tasks.By integrating causal probability trees, we can enhance search efficiency. This insight led to the development of DES.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification:Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 The pipeline of CreDes is illustrated in Fig. 1 . It comprises two main components: CRE and DES. In CRE, the inputs of LLMs for training are the initial state, goal state, and pathway (containing a series of OSRs), while for testing, the inputs are the initial and goal states only. The DES starts from the initial and goal states of the probability tree, expands them into two intermediate states, and uses the CRE-trained model to infer the pathway between them, ultimately producing the complete pathway.To further improve the capability of LLMs in solving combinatorial optimization problems that involve a finite number of discrete intermediate steps, we conducted experiments using the Blocksworld and Hanoi Tower datasets with 7B parameter models. The Blocksworld dataset includes 602 test cases categorized by the minimum number of required actions, ranging from 2 to 12 steps. For Hanoi Tower, cases are grouped based on the complexity related to the number of disks and poles, which directly influences the solution steps.For each category, our model is trained on 80 samples without common instructions. In the reasoning process, the following elements are included: initial state, OSR, state transition, next state, and goal state, as shown in Fig. 2 . During testing, the model was tested on new, categorically similar samples from different datasets, assessing its ability to transform the initial state to the goal state successfully. |ATE| ↑ |ATE| ↓ |ATE| ↓ |ATE| ↑ |ATE| ↓ |ATE| ↓ |ATE| ↑ |ATE| ↑ |ATE| ↓ |ATE| ↑ |ATE| ↑ |ATE| ↑ |ATE| ↓ |ATE| ↓ |ATE| ↓ |ATE| ↑ Figure 2: Schematic illustration of Causal Relationship Enhancement(CRE).Firstly, all the samples are classified into two categories: Correct and Incorrect. Within the Incorrect category, three scenarios exist, i.e., a correct OSR leading to an incorrect state transition, an incorrect OSR leading to an incorrect state transition, and an incorrect OSR resulting in a correct state transition.Given this, it is evident that we need to strengthen the causal connection between the OSR and the transition, and reduce the occurrence of samples where the OSR and the transition are non-causal. In CRE, we first use the ATE to estimate the causality between OSR and state transition quantitatively, and then embed the |ATE| into the loss function in the training process (the remaining is crossentropy), enhancing the causality of state transitions. As is shown in Fig. 2 , we leave the reasoning path selection to be controlled by the cross-entropy loss, while the suppression of hallucinations is handled by the |ATE| loss. Perplexity (PPL) is a metric used to evaluate the performance of a LLM, indicating how well the model predicts the next word in a sequence, and lower values signify better predictive accuracy. The estimation of ATE is detailed as the follows:Given binary variables X and Y indicating the correctness of OSR and next state (state transition), respectively, i.e., X, Y ∼ B(0, 1), and X = 1 (or Y = 1) means correctness. First, we calculate the cause-effect interventions between X and Y , then subsequently modify the distribution of Y by intervening in X. From a statistical correlation perspective, if X and Y are correlated, Y can be predicted using X. However, if there is no causal relationship between X and Y , intervening in X will not alter the distribution of Y . Hence, if X and Y are correlated but not causally linked, then manipulating or intervening in X would not lead to any changes in the distribution of Y . This distinction is crucial in statistical analysis and experimental design because it addresses the potential fallacy that correlation inherently means causation.EQUATIONIn ( 1) and ( 2), do(•) refers to Do-calculus [68] , which denotes an external intervention on the value of X without affecting the actual state of Y . Using interventions independent from other variables, we can obtain whether the treated variable X causes the target variable Y . Consequently, we can use ATE [69] to estimate the effect of the intervention, which compares the distributions of the target variable Y with and without the treatment. Y x is the potential outcome of Y under the intervention X = x. Then ATE is defined as follows:EQUATIONBased on (2), under the intervention, the proportion of positive and negative cases (hallucinations) in the model output samples remains roughly unchanged; the more robust the causal relationship between different OSRs and corresponding positive and negative cases, the lower the |ATE|. The reason is that cross-entropy basically ensures the majority of positive cases. At the same time, |ATE| reduces the occurrence of negative cases, making the distribution of positive and negative cases more stable. Consequently, we incorporate the ATE into the loss function, as is shown in ( 3) and ( 4), p 1|X and p 0|X denote the conditional probabilities of Y being 1 and 0, respectively, given the state of X.EQUATIONEQUATIONWe estimated the probabilities of correct and incorrect (hallucinations) samples in each path separately, and take |ATE| as part of the loss function for each case based on the sampling results between different paths, where |ATE| is smaller for the category with strong causal effects. This process allows the model to internalize the logical judgment between OSR and the next state during training, i.e., correct answers with strong causal effects and low |ATE|, and wrong answers with weak causal effects and high |ATE|. Therefore, with the loss function composed of cross-entropy and ATE, we can realize the synergistic optimization of path selection and hallucination elimination simultaneously.In this section, we improve the success rate of LLMs when solving long-range reasoning problems, like the 12-step scenarios in Blocksworld, by leveraging its higher success rates in simpler 2-step and 4-step scenarios. We construct two causal probability trees from the initial and goal states. Each node represents a state in the reasoning process, with arrows showing causal relationships. These trees outline possible reasoning outcomes within a limited number of intermediate steps. By matching the leaves of both trees, we identify several end-to-end permutation schemes to form a continuous and feasible path, as shown in the bottom of Fig. 1 .DES first calculates the ATE between tree unfolding and distance reduction, which in turn clarifies the causal relationship between tree unfolding and distance reduction and then infers a better unfolding direction and pruning process. At this point, the ATE calculation formula is:EQUATIONWhere A is the decrease in distance D of N i relative to N i-1 and B is the number of unfolded layers where the current leaf is located N i . We utilize the spatial positioning numbers of the blocks (disks) to calculate the distance D, and estimate the ATE of the reduction in distance, denoted as δD, relative to the number of layers N i of tree expansion. The distance is obtained by calculating the Euclidean distance for the current position of the block and the coordinates of the target position.In each layer of the tree expansion, we calculate the distance by comparing the current position with their target positions, ensuring that the reduction in distance and the direction of the tree's expansion have a strong causal effect. To avoid the expansion direction falling into local optimum, we conduct counterfactual assessments, hypothesizing alternative expansion routes that might have been takenduring the random expansion process, and incorporating the causal impacts of these hypothetical routes into consideration. Both these values are summed up to form the loss function, taking into account both the head T head and tail T tail trees.EQUATIONDuring the expansion process of the probability trees at both ends, we intervene by minimally altering the L, directing the expansion toward our desired outcome. Minimizing L realizes the pruning and unfolding direction judgment, prioritizing the direction with the lowest L as the unfolding direction.The whole process of DES is in Algorithm 1.Algorithm 1 DES (Taking the 12-step Blocksworld as an example) Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiment: 
 In this section, we validated the effectiveness of CreDes compared to baseline approaches.Blocksworld: There are n blocks initially placed randomly on a table [44] . The LLM's goal is to stack these blocks in a specified order. The LLM can perform four actions: pick up a block from the table, put down a block it is holding onto the table, unstack a block from another to hold it, and stack the block in its hand onto another block. The LLM can only manipulate one block at a time, and blocks with others on top are immovable.The GSM8K dataset [62] includes 1,319 diverse grade school math word problems curated by human problem writers. These tasks typically begin with a description and culminate in a final question requiring multi-step mathematical calculations contextual to the problem. To effectively tackle the final question, our approach involves decomposing it into a sequential series of smaller sub-questions, allowing for a structured solution process.Hanoi Tower: The Hanoi Tower problem [70] , a classic puzzle involving three pegs and a set of discs of varying sizes, serves as a key component of our experimental setup. The challenge requires moving the entire stack of discs from one peg to another, obeying the rules that only one disc can be moved at a time, and no disc may be placed on top of a smaller one. This task, structured around sequential and strategic disc placement, tests the model's ability to plan and execute a series of actions based on simple yet strict rules.Dataset: The datasets we used are the open source datasets Blocksworld [44] , GSM8K [62] ,AQUA [71] , QASC [72] , and our own production of Hanoi Tower. where the experiments for AQUA and QASC are in the Table 4 .The pre-trained models used in our study include: LLAMA-2-7B [73] , Phi-2-7B [74] , Mistral-7B [75] and Mixtral-8x7B [76] , Qwen1.5-7B[ [77] ], TAIDE-LX-7B1 , Mpt-7B[ [78] ],Baichuan2-7B[ [79] ],The model test results not mentioned in the main text will be supplemented in the Appendix.Train Parameter: In this paper, we primarily utilize the 7B models for training on a single NVIDIA A100 GPU and models are loaded in 4-bit.A technique that employs Monte Carlo Tree Search (MCTS) for exploration [43] . RAP transforms LLMs into both reasoning agents and world models, utilizing MCTS for strategic exploration and decision-making. This approach significantly enhances the LLM's ability to generate action plans and solve mathematical and logical problems, outperforming traditional methods and establishing new benchmarks in LLM's capabilities.CoT [26] : A technique having enhanced the reasoning capabilities of LLMs. By providing models with intermediate reasoning steps as examples, CoT demonstrates notable improvements across various complex reasoning tasks, including arithmetic, commonsense, and symbolic reasoning. CoT requires the model to generate a reasoning chain to improve the reasoning ability. We used all basemodels to carry out CoT in the experiment.RoT: A framework [80] to enhance the performance of tree-search-based prompting methods used in LLMs. This innovative approach leverages guidelines derived from past tree search experiences, allowing LLMs to avoid repeating errors and significantly improving their reasoning and planning capabilities across various tasks. We not only used the same basemodel as the original RoT, but also introduced other 7B models as a comparison.Blocksworld: We conducted ablation experiments on the Blocksworld dataset. Our methodology, detailed in Section 3, particularly focuses on scenarios with more than 6 steps. As is shown in Table 1 and Table 5 , for tasks up to 6 steps, results with our 7B models closely matched those with the benchmark's 70B models, suggesting robust inference capabilities even with reduced model size. For more complex tasks of 8 steps or more, DES improved its success rates by breaking down tasks into simpler segments, though it slightly lagged behind in performance compared to shorter tasks. This approach underlines the potential of our modified strategies in handling varying task complexities. By comparison, our CRE method not only outperforms benchmarks in terms of success rates on the 7B scale, but also achieves a higher success rate than the 70B+RAP method using the 7B model. For the arithmetic cases that use the full CreDes architecture, CreDes helps to improve the performance of the LLMs for long-range reasoning tasks.We further independently verified the capabilities of CRE based on the GSM8K dataset without introducing DES, to confirm that it helps to enhance the inference capabilities of large models. We found that our CRE is superior to the baseline methods RAP, RoT, and CoT, further demonstrating that completing multi-step reasoning in one go has more advantages than completing multiple single-step reasoning. See Table 2 . This example shows that CRE can not only help LLM solve highly structured problems, such as Blocksworld, but also has the ability to assist in solving some abstract mathematical problems.Hanoi Tower: Unlike the Blocksworld case, the longest reasoning steps for the Hanoi Tower have a fixed quantitative relationship with the number of rods and disks. Therefore, when training the model, we used combinations within 7 steps, i.e., 3 rods and 3 disks. For evaluation, we used problems within 15 steps, i.e., combinations of 3 rods and 4 disks, to test the reasoning ability. From this perspective, our reasoning process is based on a zero-shot setting. Due to the time complexity of the search-based method for long-range reasoning, we did not conduct experiments for too many reasoning steps, and its success rate can be recorded as '-.' As Table 3 shows, CreDes performed best among all the models. By comparing the Hanoi Tower scenario with the Blocksworld scenario, we find that the success rate under Hanoi Tower is lower than that of Blocksworld, and that the reasoning ability of the 7B+CRE group is slightly lower than that of the 70B+RAP group. We believe that this phenomenon occurs because Hanoi Tower has a stricter stacking order qualification relative to Blocksworld, and some of the intermediate steps may not hold at all, see Fig. 2 . From the results, the complexity of the Hanoi Tower problem is higher than that of Blocksworld. Time Efficiency: Using the CRE and DES architecture has significantly shortened the time to complete long-range reasoning tasks compared to benchmarks, as is shown in Fig. 3 . This is because CreDes can perform simultaneous multi-step reasoning, which is more efficient than other methods that generate answers multiple times and then cascade them together, which is more evident in longer-range reasoning. This study introduced the CreDes framework, which combines CRE and DES to improve LLMs' ability to handle long-range reasoning tasks. CRE ensures robust causal relationships between reasoning steps, and DES can lower the complexity of long-range reasoning by using a bidirectional search approach. Our experiments, particularly in the Blocksworld and Hanoi Tower scenarios, In scenarios with strict order of precedence, such as the Hanoi Tower, the accuracy is significantly lower compared to tasks like Blocksworld. The DES approach, while effective for moderate-length tasks, struggles with very long reasoning steps, leading to a decline in performance. Additionally, maintaining causal logic through CRE and DES introduces computational overhead, which may limit the framework's scalability and applicability in real-world scenarios with limited resources. Finally, our approach pays insufficient attention to the sequential ordering of steps, and the ATE can only determine whether the causal logic makes sense, rather than recognizing, for example, the assumption encountered in the Hanoi Tower problem that the larger disk must be placed under the smaller disk. 
conclusion: 
 By integrating CRE and DES, the CreDes framework has significantly advanced LLMs' capabilities in long-range reasoning tasks. This combined approach enhances the accuracy and efficiency of multi-step reasoning and maintains the problem-solving and reasoning abilities of pre-trained models across different tasks. Future work will focus on refining the framework to improve scalability and efficiency in various complex problem-solving scenarios. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification:Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The experiments in this paper did not involve error analysis and statistical significance. The data obtained from the experiments in this paper were averaged under independently repeated experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification:Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [No] Justification: Our paper deals only with a generalized framework for solving a class of mathematical problems using LLMs, independent of societal impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification:Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: To verify the success rate of our CRE method on other baseline tasks, we designed a control experiment to ensure that our approach does not impair the model's inherent problem-solving and reasoning abilities. Since DES is specifically designed for Blocksworld, a task with longer reasoning steps, the control experiments listed do not involve such lengthy reasoning steps; therefore, DES's performance is not tested in this section. The experimental results indicate that the CRE method can, to some extent, enhance the model's problem-solving capabilities on other baseline tasks without causing any reduction in performance. See Table 4 . We generated and produced the Hanoi Tower dataset in the paper. The production method is to randomly generate several states conforming to the placement rules of the Hanoi Tower based on a given number of rods and disks, e.g., three rods and three disks, and randomly select one of these states as the starting and target states for a single sample. For a single sample, the classical partition algorithm is used to derive the pathway, and according to the length of the pathway, the sample is categorized into different number of steps groups, e.g., 3-steps, 5-steps, 7-steps, and so on. An odd number is chosen for the allocation because the most complex solving step of Hanoi Tower in the case of three rods and n disks is 2 n -1 steps. We generated the dataset Hanoi Tower using exactly the same storage format and Prompt structure as Blocksworld and GSM8K. Justification:Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification:Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).http://taide.tw 


Decision: This paper introduces the CreDes framework, which enhances long-range reasoning in LLMs by combining causal relationship enhancement (Cre) and a bi-directional search method (Des). The reviewers recognized the potential of the method but raised concerns about scalability, generalization to less structured tasks, and clarity in the explanation of ATE usage. While the authors addressed most concerns through additional experiments and clarifications, some reviewers remain skeptical about the generalization of the method and the thoroughness of the theoretical analysis. The paper shows promise, but further improvements in presentation and more comprehensive validation are necessary. Overall, the recommendation leans towards rejection due to these unresolved issues.

Reviews: review: paper_summary: paper_summary: This paper introduces a new framework, CreDes, designed to enhance causal reasoning in large language models (LLMs) and solve complex, long-range reasoning problems. The framework integrates two main innovations: the Causal Relationship Enhancement (CRE) mechanism, which applies cause-effect interventions to maintain causal accuracy across reasoning steps, and the Dual-End Searching (DES) method, which approaches problem-solving by initiating searches from both the initial and goal states to efficiently navigate large search spaces. The efficacy of CreDes is demonstrated through rigorous testing on challenging datasets like Blocksworld and Hanoi Tower, where it outperforms existing state-of-the-art models in both accuracy and efficiency.
strength_weakness: strength_weakness: 1. Limited model sizes: The experiments are primarily conducted on 7B parameter models, which may not reflect performance on larger state-of-the-art LLMs.
2. Lack of error analysis: The paper doesn't provide a detailed analysis of the types of errors made by the model or how they differ from baseline methods.
3. Dataset validity and construction: More details is needed for the use of a custom-made Hanoi Tower dataset which potentially limiting the reproducibility and generalizability of the results.
4. Computational efficiency and scalability: As mentioned in the Limitation, the paper lacks a detailed discussion of the computational requirements and scalability of the CreDes framework.
5. Generalization to less structured tasks: The framework's effectiveness is primarily demonstrated on highly structured tasks but it's unclear about its applicability to more dynamic or open-ended reasoning scenarios.
6. Lack of statistical significance: The paper doesn't report error bars or statistical significance for its experimental results.
questions: questions: 1. Given that experiments were conducted on 7B parameter models (Section 4.2), how does the performance of CreDes scale with larger model sizes? Have you tested it on models larger than 7B parameters.
2. Regarding the Hanoi Tower dataset described in Appendix, can you provide more details on how it was validated? How does it compare to established benchmarks in testing causal reasoning capabilities?
3. Figure 3 shows an unusual lack of variation in reasoning speed as the number of steps increases for the CreDes (blue bar). Can you explain this phenomenon and discuss its implications?
4. The paper focuses on structured tasks like Blocksworld and Hanoi Tower (Section 4.1). How might CreDes be adapted or applied to less structured reasoning tasks that don't have clear initial and goal states?
5. It's mentioned that CreDes can perform simultaneous multi-step reasoning. Can you elaborate on how this works and provide examples?
limitations: limitations: Yes
paper_summary: paper_summary: This paper aims to improve LLMs in dealing with long-reason reasoning problems, especially the challenges of causal hallucination (inconsistency between one-step reasoning and corresponding state transition) and large search space. To tackle the first challenge, average causal effect of the one-step reasoning (treatment) on the state transition (outcome) is added to the loss function of the LLM; and for the second challenge, a dual-end (i.e. bi-directional) search approach is taken to improve efficiency. Experiments are conducted to demonstrate the effectiveness of the proposed method and its superiority over the compared existing methods.
strength_weakness: strength_weakness: 1. The soundness of the proposed CRE method (for dealing with the challenge of causal hallucination) is in doubt.

(a) It's not clear why the method aims to $\textbf{minimize} the absolute value of the average treatment effect (ATE) of the one-step reasoning on state transition. Assuming that the ATE can be accurately estimated, what we want here would be to maximize the ATE that can be achieved by the LLM, i.e. when the one-step reasoning is correct done will likely lead to a correct state transition.

(b) It's not clear how an unbiased estimation of the ATE can be obtained, and what assumptions are made in terms of ATE estimation.

(c) The definition or understanding of ATE is incorrect. In particular, formula (2) is wrong, and formula (5) is incorrect too. 

2. The presentation/technical quality requires improvement, including the presentation of related work. Please find below some examples:

(a) In Lines 42 to 44, it is said that the existing methods such as CoT are limited in task decomposition, but Lines 78-80 state that they can breakdown queries into manageable steps.

(b) Section 3.1 is titled as "Problem Definition", but it rather looks like a section on experiment setting.

(c) Lines 145-146 state that Fig. 1 shows "we leave the reasoning path selection to be controlled by the cross-entropy loss", but I cannot see this indicated in Fig. 2.

(d) Line 159: do(.) is an operator, specifically the do operator, rather do-calculus, although do-calculus uses this operator.

(e) Lines 159-160: the statement on the do(.) operator or do-calculus is incorrect, since an do operation on the treatment X would lead to the change of the outcome Y, especially if X is a cause of Y.
questions: questions: 1. How the ATE is estimated? Any assumptions, e.g. unconfoundedness, are made for the ATE estimation?
2. Line 166: what does "robust" mean here?
3. Why the model aims to minimize the absolute value of the ATE?
limitations: limitations: The authors have presented some discussions on the limitations of the proposed method. It would be better if the assumptions made could be presented more clearly and what the practical implications would be if the assumptions are violated.
paper_summary: paper_summary: The integration of Causal Relationship Enhancement (CRE) and Dual-End Searching (DES) mechanisms presents a novel solution to addressing causal hallucinations and large search spaces in long-range reasoning tasks. The CRE mechanism’s use of Structural Causal Modeling (SCM) and Average Treatment Effect (ATE) is  ensure causality between reasoning steps. Extensive testing on datasets such as Blocksworld, GSM8K, and Hanoi Tower demonstrates the effectiveness of the CreDes framework.
strength_weakness: strength_weakness: The method presented in this paper evaluates ATE on LLMs, but this approach's validity hinges on the assumption that LLMs can perfectly represent the real-world environment. The very reason we criticize LLMs for their reasoning issues is because their inferences are not accurate. Estimating ATE might only bring the prediction results closer to Y while maximizing the influence of the intervention factor on Y. However, it does not necessarily mean that the intervention factor is the true cause of Y. In other words, since there is no alignment with the causal relationships in real-world scenarios, the implementation of this method does not prove that the reasoning is causally sound.

The method lacks deeper thinking. The authors just apply the concept of ATE to the Chain-of-Thought (CoT) without thorough analysis. This oversight leads to a misalignment between the experimental results and the motivation of the paper. Suppose LLMs are not a good s simulations of the real world. In that case,  performing interventions on LLMs (whether they align with the real world or their identifiability) requires sound theoretical analysis and experimental validation. The current paper lacks a deep discussion on this matter.
questions: questions: 1. How does DES ensure that it does not fall into local optima during the search process?
2. What are the computational requirements and limitations of using CreDes in practical applications?
limitations: limitations: See weakness.
paper_summary: paper_summary: This paper introduces CreDes, a framework to improve the long-range reasoning capabilities of LLMs, consisting of two main components: Causal Relationship Enhancement (CRE) and Dual-End Searching (DES). CRE is developed to reduce causal hallucinations in LLMs by strengthening the causal relationships between reasoning steps and state transitions; it uses structural causal modeling and optimizes the Average Treatment Effect (ATE) during training. DES breaks down long-range reasoning tasks into shorter segments by simultaneously searching from both the initial and goal states on a causal probability tree. The authors evaluate CreDes on Blocksworld, GSM8K, and Hanoi Tower puzzles, showing improvements in both accuracy and efficiency compared to existing methods.
strength_weakness: strength_weakness: Major concerns: 

- The generalizability and scalability need better justification. The paper primarily tests the CreDes framework on Blocksworld, Hanoi Tower, and some mathematical reasoning tasks (GSM8K). These are relatively structured, rule-based problems that may not represent the full spectrum of reasoning challenges. In addition, the proposed method cannot be well scaled to long-range reasoning; for example, in Table 1, performance drops significantly for Blocksworld tasks beyond 8 steps, with success rates falling from 0.68 to 0.34 for 12-step problems using Llama-2-7B + CreDes. Table 3 shows even steeper declines for Hanoi Tower, with success rates dropping from 0.27 at 9 steps to just 0.07 at 13 steps for Llama-2-7B + CreDes. Notably, the authors explicitly acknowledge this limitation in Section 4.6, stating: "The DES approach, while effective for moderate-length tasks, struggles with very long reasoning steps, leading to a decline in performance."

- The presentation of this paper could be improved.

  -- In the problem definition, there is no explanation of the difference between training without common instructions and with common instructions.

  -- There is no detailed discussion of the differences between correlation and causation in Sec 3.2. I am confused about whether the correlation of two variables has anything to do with their distributions.

  -- While efficiency gains are mentioned, the added complexity of CRE and DES likely introduces some computational overhead, which could be further discussed.

  -- There is no analysis of the impact of the choices of hyperparameters on the methods, particularly in the CRE component.

- The proposed method lacks comparison to more recent state-of-the-art methods. The paper compares CreDes mainly to older baselines: Reasoning via Planning (RAP), Chain of Thought (CoT), and Reflexion of Thoughts (RoT). However, it doesn't evaluate against more recent advances in LLM reasoning, such as Tree of Thoughts (ToT) extensions in line 42, and the paper doesn't mention or compare to other recent works such as [a] and [b], which also address multi-step reasoning challenges. As a result, the technical contribution is not entirely clear.

[a] Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, & Xinyun Chen (2024). Large Language Models as Optimizers. In The Twelfth International Conference on Learning Representations.

[b] Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, & Zhiting Hu (2023). Reasoning with Language Model is Planning with World Model. In The 2023 Conference on Empirical Methods in Natural Language Processing.

Minor concerns: 

- Experiments are primarily conducted with 7B parameter models, leaving questions about scalability to larger models. How does the performance of CreDes scale with increasing model size (e.g., to 10B+ parameters)? The computational overhead may limit the framework’s scalability and applicability in real-world scenarios with limited resources.

- The approach achieves significantly lower accuracy in tasks with very strict ordering constraints, such as the Hanoi Tower problem.

- Since Blocksworld involves random steps, an analysis of the robustness of the performance may be needed.

 - More analysis/discussion on the sequential ordering of steps may be helpful. Notably, the ATE cannot recognize casual logic.

 - Some editorial issues, e.g., Line 110
questions: questions: Please see the weakness part.
limitations: limitations: The authors have discussed the limitations, and it is adequate to me. I do not see any potential negative societal impact.
paper_summary: paper_summary: This paper develops a structured and generalized reasoning framework, CreDes, for long-range reasoning in LLMs. In the framework, the Causal Relationship Enhancement (CRE) is used to guarantee the solid causal rightness between each step of reasoning and state transition, and the Dual-End Searching (DES) approach is proposed to seek solutions by simultaneously starting from both the initial and goal states on the causal probability tree, to improve the efficiency.
strength_weakness: strength_weakness: 1. The main concern is the understanding of ATE. This paper frequently uses ATE as part of the loss function and thinks the lower ATE can guarantee the solid causal rightness between each step of reasoning and state transition. However, ATE is used to measure the causal influence level between variables from the observational data, and causality does not mean rightness.
2. The DES section is not clear enough. It is suggested that more explanation be provided for the reason for the ATE as part of the loss. For example, if “B is the number of unfolded layers where the current leaf is located Ni”, what does E(A|do(B)) and E(A) mean in Formula (5)?
3. This paper needs to supplement the usage scenarios of methods, specifically in which scenario to use CreDes, in which scenario to use Cre alone, and whether Des is used separately.
questions: questions: CRE:
1. Row 136-142: Do you assume that only the transition from correct OSR to correct state is causal, and the other three incorrect scenarios are non-causal? This is not the case. Generally, the determination of causality is based on observational data rather than subjective perception.
2. Row 145: “enhancing the causality of state transitions” seems to contradict the subsequent mention of lower |ATE|. 
3. Row 146-147: Does the phrase “suppression of hallucinations is handled by the |ATE| loss” refer to reducing the causal relationship in the three wrong scenarios of incorrect OSR to state transitions? If so, is the causality in the correct category also weakened in the process?
4. Row 165-167: What is the basis for lower |ATE|? Normally, a low |ATE| does not signify a robust causal relationship but rather indicates a weak influence from X to Y.
5. Row 177-178: How to achieve “correct answers with strong causal effects|, and wrong answers with weak causal effects”?

DES:
1. Row 198: If the ATE is part of the loss function to be minimized, how could A and B have a strong causal effect?
2. Row 199: Please explain the detailed process of counterfactual assessments.
3. Row 204-205: Cannot understand why minimize ATE(A). Is there a tradeoff between tree unfolding and distance reduction? Can you explain it in an intuitive way?
4. Algorithm 1 step 5: What is the relationship between the “unfolded layers” of causal probability tree and the steps in “for every four steps do”? How to determine four steps rather than two steps?
5. Algorithm 1 step 6: Please explain the detailed process of “Determine intermediate steps and fill in details”.

Experiment:
1. Row 235-236: What is the training set? Please explain the training details.
2. Row 253-263: Is the 2-12 steps the shortest step for one problem in Blocksworld? How to determine the steps should be taken for one problem in Blocksworld?
limitations: limitations: None.

Comments: title: Statement on Reviewer MmUh's Relevant Opinions
Thanks for your detailed responses
To Reviewer BkzM and AC (Part 2)
To Reviewer BkzM and AC (Part 1)
Reply to the rebuttal
Error analysis provided to Reviewer nWAC
To ACs and Reviewers
comment: Dear All,

We extend our sincere gratitude for the opportunity to present our work and for the constructive feedback we have received from the reviewers. The review process has been invaluable in helping us refine and improve our submission. However, we would like to respectfully bring to your attention a few concerns regarding the feedback provided by Reviewer MmUh, particularly about their engagement during the rebuttal and discussion phases.

(a) The reviewer mentioned: "The authors just apply the concept of ATE to the Chain-of-Thought (CoT) without thorough analysis." 

This assessment may not fully reflect the distinctions between our approach and the CoT method. Our approach is significantly different, especially in the form of outputs, as outlined in our paper and highlighted in the error analysis provided to Reviewer nWAC. The logic behind our reasoning and the specific inputs to the LLM, including elements like prompts and instructions, differ markedly from those used in the CoT method.

(b)The reviewer mentioned: "But this approach's validity hinges on the assumption that LLMs can perfectly represent the real-world environment." and "In other words, since there is no alignment with the causal relationships in real-world scenarios, the implementation of this method does not prove that the reasoning is causally sound." 

This may need to be clarified. Our study focuses on widely recognized datasets like Blocksworld and GSM8K, as well as our custom Hanoi Tower dataset. Although examples such as Blocks World and Tower of Hanoi may seem simple compared to the real world, they are an abstract expression of the actual complex tasks in the real world. They can reflect the central core of long-range task decomposition, so they still have strong persuasiveness when validated on these datasets. For instance, besides the baselines we cited, other studies, such as those by Yu et al., Liu et al., and Zhang et al., have also used Blocksworld to validate task decomposition performance. These examples show that our experimental setup is practical and verifiable for evaluating reasoning capabilities. We welcome real-world datasets or validation scenarios supporting our method's applicability in practical contexts.

(c) This reviewer has not participated in the discussion. We and AC have repeatedly reminded the Reviewer MmUh but have not received a response. 

We have noticed that Reviewer MmUh has yet to engage in the discussion phase, which may suggest that the original review might not entirely reflect the current status of our work. We believe that active participation in the discussion is essential for a fair and comprehensive review process, and we hope this can be considered.

(d) Besides, considering that our paper introduces a straightforward and effective algorithm, presents comprehensive experiments, and raises no ethical concerns, it may not warrant a rejection. 

The definition of a reject is: 

>"3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility, and incompletely addressed ethical considerations." 

We hope our work's strengths will be considered in the final assessment.

We appreciate your understanding and consideration of these points. We are committed to improving our submission and ensuring that it meets the high standards expected by the community. Again, Thank you for your time and effort overseeing this review process.

Best Wishes,

Authors
Dear PC, SAC, AC, and Reviewers,

We express our deepest gratitude for the time and effort each of you has dedicated to reviewing and discussing our submission. The valuable feedback we have received has been instrumental in refining our work and strengthening its contributions to the field.
The following is a brief summary of the discussion process:

# Model Size and Scalability
Reviewer nWAC raised concerns about the limitations of the model sizes used in our experiments and the scalability of our approach. Reviewer yRix also highlighted the need for further demonstration of generalization and scalability, noting that the problems studied tend to be more structured, and they also questioned the performance scalability as the model parameters increase.

* Our Response:  
(a) During the rebuttal period, we supplemented our experiments with results from the LLaMA-2-13B model, constrained by our hardware capabilities. We also pointed out that the original paper provided experimental results from the Mixtral-8x7B model, which is larger than the 7B model. The results showed no significant difference between the 13B and 7B models, and this difference can be considered a random error due to different training instances. Comparing the performance of the 70B and 7B models under the baseline RAP method, the 70B model shows relative improvement. However, regarding inference speed, the 70B model is much slower than the 7B model and requires a certain level of quantization, which also leads to performance loss. The above experimental results will be included in the main body of the revised paper.\
(b) We addressed the concerns about problem structure and scalability by explaining that our ongoing work explores more unstructured, complex scenarios using the Minecraft simulator, aiming to advance embodied intelligence. We also noted that in similar concurrent and subsequent work, our results show advantages, confirming the effectiveness and verifiability of our experiments. Relevant references are provided in the rebuttal for Reviewer nWAC.
# Theoretical Analysis
Reviewer Y4qM and Reviewer BKzM raised several detailed questions about our methodology and theoretical analysis, such as why we use |ATE| as part of the loss function, the relationship between the strength of causal relationships and the size of ATE, and the unclear statements regarding our assumptions. Reviewer nWAC also pointed out similar issues in their follow-up questions.

* Our Response:
We are grateful for these questions, as they helped us fully recognize the shortcomings in our writing. We have provided a more detailed theoretical analysis:\
https://openreview.net/forum?id=azkuhJBZXi¬eId=cbDKQGUyra \
https://openreview.net/forum?id=azkuhJBZXi¬eId=RTQb3tqD03 \
In these responses, we further clarified the unclear and imprecise aspects of our paper, and we will provide a comprehensive explanation of our theoretical analysis in the revised version, correcting and refining any unclear parts. We want to sincerely thank Reviewer Y4qM, Reviewer BKzM, and Reviewer nWAC for their invaluable feedback.

# Error Analysis and Statistical Significance
Reviewer nWAC noted our initial submission's lack of error analysis and statistical significance.
* Our Response:
We apologize for the omission in the initial draft. We provided an example of error analysis during the rebuttal period, and in the appendix of the revised paper, we will include more examples. Additionally, we will add error bars to all experimental result data in the revised paper.

# Writing Style, Typos, and Presentation Quality
Reviewer nWAC, Reviewer yRix, Reviewer Y4qM, and Reviewer BKzM provided feedback on various issues related to writing style, typos, clarity of diagrams, and overall presentation quality.
* Our Response:
We are very grateful to the reviewers for their patience in reading the manuscript and providing specific suggestions. We have carefully recorded all the revision suggestions and will address each of them in the revised paper. We especially noted the requests from Reviewer BKzM and Reviewer nWAC for improvements in theoretical analysis, and we will make a concerted effort to incorporate these into the additional pages of the main text as fully as possible.

# Conclusion
After the Rebuttal and Discussion phases, we are pleased to see that we have received support from multiple reviewers:

Reviewer nWAC: This paper shows promise and could find its audience, but there's room for improvement in writing style and presentation. 

Reviewer Y4qM: I have more confidence on the soundness of the paper after reading your responses.

Reviewer yRix: I would like to keep my score and believe that this paper is lightly above the acceptance bar.

Reviewer BKzM: Thanks for the detailed responses, which have solved my major concerns. I suggest the authors continue to improve this submission.

Finally, we would like to thank all the reviewers and AC for supporting our paper. 

Best Wishes,

Authors
Dear Reviewer nWAC,

Thank you for your continued engagement and for your constructive feedback on our paper. We appreciate your recognition of the promise in our work and your thoughtful suggestions for improvement.

We will certainly take your recommendations to heart, especially regarding the need to refine the writing style and presentation. We also acknowledge the importance of including the detailed theoretical analysis and more comprehensive implementation details, particularly in light of the high efficiency and performance scores achieved on the datasets with CREDES performance. We will make these aspects a priority in our final draft to enhance the clarity and impact of our paper.

Your insights have been invaluable in guiding us toward a stronger submission, and we are committed to making the necessary revisions.

Thank you once again for your time and consideration.

Best regards,

Authors
Dear Reviewer BKzM,

Thank you very much for your thoughtful feedback and for taking the time to carefully review our responses. We are pleased to hear that our explanations have resolved your major concerns, and we sincerely appreciate your willingness to raise your rating accordingly.

We also value your insightful suggestions regarding the differentiation and elucidation of the concepts of ATE and variance for ITE, as well as the distinction between the significance and the value of ATE. We will certainly incorporate these points into our revised submission to further improve the clarity and impact of our work.

Once again, thank you for your constructive feedback and support. Your comments have been extremely helpful, and we are committed to making the necessary revisions to enhance our paper.

Best regards,

Authors
Thanks for the detailed responses, which have solved my major concerns. I will raise my rating accordingly.

Besides, understand the goal of achieving convergence in your results. In the submission, it may be necessary to differentiate and elucidate the concepts of ATE and variance for ITE. Additionally, it is important to distinguish between the significance of the ATE, as discussed in the provided references, and the value of the ATE itself. I suggest the authors continue to improve this submission.
Thank you for your detailed response. This paper shows promise and could find its audience, but there's room for improvement in writing style and presentation. The final draft should include the detailed theoretical analysis you've provided. I strongly recommend offering more comprehensive details on implementation, particularly in light of the high efficiency and performance scores achieved on those datasets with CREDES performance. I'm maintaining my current score.
Dear Reviewer BKzM,

We would like to express our sincere gratitude for your previous responses and engagement with our paper. We noticed that you raised some further questions after our initial rebuttal, and we have since provided additional responses to address your concerns.

We wanted to check in to see if our latest responses have resolved your queries, and to ask if there are any remaining questions or issues that you would like us to address. We apologize for reaching out so directly, but with the discussion period nearing its end, we will soon be unable to participate in further discussion.

Once again, thank you very much for your time and thoughtful feedback. We greatly appreciate your support and look forward to hearing from you.

Best regards,

Authors
Dear Reviewer nWAC,

We would like to express our sincere gratitude for your previous responses and engagement with our paper. We noticed that you raised some further questions after our initial rebuttal, and we have since provided additional responses to address your concerns.

We wanted to check in to see if our latest responses have resolved your queries, and to ask if there are any remaining questions or issues that you would like us to address. We apologize for reaching out so directly, but with the discussion period nearing its end, we will soon be unable to participate in further discussion.

Once again, thank you very much for your time and thoughtful feedback. We greatly appreciate your support and look forward to hearing from you.

Best regards,

Authors
Dear Reviewer Y4qM,

Thank you very much for your thorough review and for taking the time to carefully consider our responses. We are pleased to hear that our clarifications have increased your confidence in the soundness of our paper.

We sincerely appreciate your willingness to re-evaluate our work and for raising your score. Your feedback has been instrumental in helping us improve our research, and we value the opportunity to learn from your insights.

Thank you once again for your thoughtful review and consideration.

Best regards,

Authors
Thanks the authors for your detailed and helpful responses. I have more confidence on the soundness of the paper after reading your responses, and I have raised my score to 4.
Dear MmUh, 

Please acknowledge that you’ve reviewed the authors’ rebuttal and share any ongoing concerns. Do you feel that the authors’ explanations are still insufficient? 

Best,
AC
Dear Y4qM,

Please acknowledge that you have read the authors’ rebuttal and express remaining concerns. Are the authors’ explanation insufficient? if so in what aspect? Please also ensure score rating to reflect your latest opinion of the paper.
Dear Reviewer nWAC,

Thank you for your valuable feedback on our manuscript. The theoretical analysis you inquired about has been addressed in detail in our response to Reviewer BKzM's comments. We recommend that you refer to our reply to Reviewer BKzM for the relevant information.

https://openreview.net/forum?id=azkuhJBZXi&noteId=cbDKQGUyra

https://openreview.net/forum?id=azkuhJBZXi&noteId=RTQb3tqD03

We appreciate your support of our work, and please feel free to reach out if you have any further questions.

Best regards,

Authors
# Detailed Explanation
When $\sigma^2$  (variance) is small, the treatment effects for all individuals are very close to the mean $\mu $. This suggests that the treatment effect is highly consistent and stable across different individuals:

Causal Certainty: Due to the high consistency of individual effects, the treatment’s impact can be considered specific rather than incidental or person-dependent. This consistency is usually regarded as a hallmark of a strong causal relationship.

Robustness of Causal Effects: Even if the average effect (ATE) is small, this consistency indicates that the treatment has a similar impact on all individuals, making the causal relationship robust and strong.

Conversely, when  $\sigma^2$ is large, the treatment effects among different individuals may vary significantly, leading to a high level of inconsistency:

Causal Uncertainty: The treatment effect may manifest as a strong positive effect in some individuals and a negative effect in others. This high variability implies that the treatment's impact is uncertain, leading to a weaker causal relationship.

Inconsistency of Causal Effects: Even with a large average effect (ATE), the significant differences in individual responses make it difficult to assert that the treatment effect is consistent across all individuals. Consequently, such a causal relationship is generally considered inconsistent or weak.

$ 
\sigma^2 = \frac{1}{N} \sum_{i=1}^{N} (\tau_i - \text{ATE})^2
 $

# Practical Example and Application to Our Work
To better understand this extended perspective, let's consider a specific example:

Take the Blocksworld scenario as an example. At the beginning of training with a fixed OSR, various state transition results such as A, B, C, D, and E might indicate low consistency. By using methods like fine-tuning that rely on the model's cross-entropy loss, the model can be constrained to produce fewer state transition results, such as only A, B, and C. However, more is needed, as the ultimate goal of using ATE is to constrain the state transition results to only one outcome, specifically A.

Towards the end of a training process, the following two situations may occur:
Assume we plot A, B, C, D, and E on the X-axis and the frequency of the corresponding output samples on the Y-axis and observe the distribution. Our objective is to make the distribution tend toward one with low variability.

Scenario A: High Consistency, Low ATE: In this case, the variability of the distribution is small, meaning fewer types of situations occur.

Scenario B: Low Consistency, High ATE: In this case, the variability of the distribution is large, meaning more types of situations occur.

From here, we begin to consider the issue of consistency. These two scenarios have commonalities with the model training process. The combination of OSR and state transitions is not unique for a single sample under 2-class- and 3-class samples. Therefore, we aim to achieve a one-to-one correspondence between OSR and state transitions. Note that a one-to-one correspondence does not necessarily mean the combination of OSR and state transitions is correct. However, during the training process of a large model, the model's cross-entropy loss function also plays a role. Just as with methods like RAP and CoT that freeze the model, the model's cross-entropy loss function alone can somewhat control consistency. Additionally, because our model is not frozen, the output consistency of the model improves with each epoch during training. Therefore, our training process is an optimization process from Scenario B to Scenario A. ATE does not work in isolation but cooperates with the cross-entropy loss function.

The dimensions of cross-entropy loss and ATE are different. In the early stages of model training, the changes in cross-entropy loss (or PPL) are dramatic, far exceeding the changes in ATE. As a result, ATE plays a minor role at this stage. Only in the later stages of training when the cross-entropy loss decreases to a certain extent, and its rate of change approaches that of ATE, ATE begins to play a role, assisting the model in further optimization.

The above discussion focuses on a single-sample perspective. However, to evaluate the overall model training process across epochs, we assess the model by its accuracy in answering validation set questions, not by ATE.
# Conclusion
We believe that this reviewer's comments convey the same concerns that other reviewers may have, so we hope everyone can take a look.

In summary, the method presented in our paper involves fine-tuning during the training process, and the reviewer's perspective might have been overly broad.

Meanwhile, we have attached a relevant reference for your further understanding:

[1] Bao G, Zhang H, Yang L, et al. Llms with chain-of-thought are non-causal reasoners[J]. arXiv preprint arXiv:2402.16048, 2024.

Please review the 4 and 5 sections.

Thank you once again for your efforts and consideration.
Dear AC and Reviewer BKzM,

We sincerely appreciate the insightful feedback provided by the reviewer. It has helped us recognize potential misunderstandings that might exist among the reviewers.

We kindly request that the Area Chair share this QA exchange with all the reviewers to ensure these clarifications are understood broadly. Additionally, the 5000-word limit for the rebuttal is significantly constraining, and we respectfully request permission to exceed this limit to provide a more comprehensive response.

# Clarification on Our Perspective

Firstly, we urge the reviewer to revisit our rebuttal to the first point raised under "Weaknesses." We apologize for any lack of clarity in our initial response due to the word limit.

We must clarify that the reviewer's position is fundamentally different from the statement attributed to us: "The goal is to reduce the causal influence of OSR on state transitions, thereby making the results stable and unchanged under different inference paths." The reviewer's subsequent questions stem from this misunderstanding. We will now present a complete theoretical explanation, which we believe will address the three concerns that arose from this misinterpretation.

# Theoretical Background and Practical Considerations

In typical scenarios, the ATE (Average Treatment Effect) is expressed as follows:

$ \text{ATE} = \mathbb{E}[Y(1) - Y(0)] $

Isolating this formula in a purely metaphysical sense could lead to questions similar to the reviewer's. However, the context differs significantly when applied to large model inference. We perform numerous output trials to calibrate the model during the training process. From our experimental results, these output samples demonstrate a variety of possibilities, such as:

Type 1: Certain samples are challenging to answer correctly, regardless of training, resulting in near-random correct/incorrect states.

Type 2: There is a positive correlation between epoch count and correct answer frequency for some samples, significantly when aided by standard training techniques like RAP and CoT.

Type 3: Some samples can be answered correctly with minimal training, showing no correlation between epoch count and correct answer frequency.

As Blocksworld researchers, the current goal is to maximize the correct rate of Type 1 samples, effectively converting more Type 1 samples into Type 2.

# Explanation of Our Approach
It is crucial to note that we do not train for a specific problem in each training cycle but for all samples in the training set. Therefore, we must distinguish between aggregate and individual treatment effects. Suppose we define an individual-level treatment effect evaluation $ \tau_i $;  it can be written as:

$  \tau_i = Y_i(1) - Y_i(0)  $

$ 
\text{ATE} = \frac{1}{N} \sum_{i=1}^{N} \tau_i
 $

Among them:

-$ N $ is the number of individuals in the sample.

-$ \tau_i $ represents the processing effect of the $i $ th individual.

Given that large language models possess basic logical reasoning abilities (as discussed in our rebuttal), our objective is to enhance rather than reconstruct this capability. Our experimental data supports this (we acknowledge that we did not include this in the appendix but will include relevant visualizations in the revised submission). For repeated experiments on a single sample, the model's responses follow a normal distribution. 

$ 
\tau_i \sim N(\mu, \sigma^2)
$ 

Note that this involves a complete count of all possible answers, not just a binary correct/incorrect classification. The correct response occurs most frequently.

In this context, the mean of the normal distribution aligns with the expression for individual-level ATE, while the variance reflects the variability of individual effects. Based on this, we propose the following logical extension:

When individual effect consistency is high (low variance), the causal effect is more robust and consistent, resulting in a solid causal relationship even if the ATE is small.
When individual effect consistency is low (high variance), a large ATE does not necessarily indicate a robust causal relationship because the variability among individual responses may be significant.
Thanks for the detailed responses. The submission's statement suggests that the goal is to reduce the causal influence of OSR on state transitions, thereby making the results stable and unchanged under different inference paths. However, several issues require further clarification:

1. The Concept of Causality. The concept of "causality" is repeatedly mentioned in the paper, like "ensure the causality between OSR and state transition in LLMs," including the causality measure between OSR and state transition," enhancing the causality of state transitions," etc. Combined with the feedback in the rebuttal, does it mean "distribution of experimental results tends to be stable" is equivalent to "enhancing the causality"? If these two concepts are equivalent, please provide relevant references to support this equivalence.

2. The relationship between "Enhancing causality" and "Lower |ATE|". ATE is an estimand for causal effect estimation. "Low |ATE|" typically indicates a non-significant causal effect, what is the relationship between "lower |ATE|" and "enhancing causality"? The statement in the rebuttal, "we would like to clarify that the example indicated by the absolute drop in Fig. 2 ATE is causal" also raises this issue. If there is additional clarification regarding this, please illustrate with equations and references.

3. The Benefits of Stability of Results. Setting aside the causal effect estimand ATE, in the rebuttal, does "the results tend to be stable and unchanged under different inference paths" mean that the correctness or incorrectness of OSR has a minimal causal impact on state transition outcomes? What are the advantages of this approach? Is the stability of results inherently beneficial? If so, please provide references to support this.
Dear Reviewer yRix,

Thank you for taking the time to review our paper and for your thoughtful comments throughout the process. We greatly appreciate your recognition of our detailed responses and your continued support of our work.

We respect your decision to maintain your score and are grateful that you consider our paper to be above the acceptance bar. Your feedback has been invaluable in refining our research, and we are hopeful that the contributions we have made will positively impact the field.

Thank you once again for your efforts and consideration.

Best regards,

Authors
I appreciate the author's detailed response. 

I would like to keep my score and believe that this paper is lightly above the acceptance bar.
Dear Reviewer nWAC,

Thank you for your feedback and your willingness to reconsider your evaluation of our work. We appreciate your concerns regarding the generalization of our proposed method, particularly in relation to addressing long-range causal reasoning problems.

# 1. Regarding Experimental Comparison:
Many reasoning tasks or long-range sequence decomposition tasks in real life fundamentally follow the same paradigm as our research work. We understand that you may still have concerns about the real-world applicability of our approach, so we would like to provide a few scenarios to help clarify it.

For instance, in the scheduling of port container stacking or the arrangement of goods in warehouse areas, there are already mature algorithms in the logistics field. However, our work attempts to leverage Large Language Models (LLMs) to enhance reasoning in these contexts. The goal is to bridge the communication gap between human operators and algorithm engineers by using LLMs to facilitate clearer and more effective interactions. We hypothesize that LLMs can receive and understand human instructions, adjusting their actions accordingly, which would improve collaboration between humans and algorithms.

While we haven't yet validated our approach with real-world data, the essence of many real-world reasoning or long-range sorting tasks closely aligns with the experimental paradigm we employed in our paper.

To further illustrate this, consider the example of warehouse item arrangement. This task might involve organizing items based on criteria like size, weight, or frequency of access. Although it may initially seem like a complex, monolithic task, it can actually be decomposed into a series of smaller, more manageable sub-tasks. For instance, the first sub-task could involve categorizing items by size, followed by arranging them within sections based on weight, and finally, placing them in specific locations depending on access frequency. Each sub-task is interdependent, with the completion of one informing the next, thereby creating a continuous sequence of actions that leads to the overall goal.

We have observed that many related works do not establish a strong connection with real-world scenarios, and our experimental scope is similar to theirs. Specifically, other works have adopted similar test scenario and dataset, which strengthens our confidence that our experiments are robust enough to validate our ideas. For example, in addition to the baseline papers we cited, the following three papers also used the blocks world scenario to validate the performance of task decomposition. This demonstrates that the examples or experiments provided in our paper are highly effective for validating our reasoning capabilities and are indeed verifiable.

In fact, Blocksworld is recognized, and we have omitted three papers from our baseline. The following are relevant papers based on Blocksworld validation for effectiveness:

## In subsequent work:
[1] Yu F, Jiang L, Kang H, et al. Flow of Reasoning: Efficient Training of LLM Policy with Divergent Thinking.

The experimental results (based on LLaMA3 8B, which was released after the submission of our paper) are as follows:

 Model              | 2-step | 4-step | 6-step 
-|-|-|-
 Llama-3-8B    | 100.00   | 97.62   | 71.71  

[2] Liu Z, Hu H, Zhang S, et al. Reason for Future, Act for Now: A Principled Architecture for Autonomous LLM Agents.

The experimental results are presented in Fig.8 of their paper. Their baseline model, LLaMA-33B, shows performance similar to ours, but it struggles with 8-step, 10-step, and 12-step scenarios.

## Concurrent work with ours:

[1] Zhang S, Zheng S, Ke S, et al. How Can LLM Guide RL? A Value-Based Approach.

We noticed this work during the preparation of our paper. Due to their reliance on the OpenAI API for open-source code, we did not include it as one of our baseline comparison methods. However, they also compared our baseline method, RAP, as shown in Fig.5. Our Step-6 accuracy is higher compared to theirs.

Additionally, if you have any real-world datasets or validation scenarios, we would greatly appreciate it if you could share them with us. Access to such datasets or scenarios would allow us to further validate our work and ensure its practical applicability in real-world contexts.

# 2. Regarding Theoretical Analysis:
We would like to clarify which aspects of the theoretical analysis you would like us to elaborate on. We acknowledge that due to word limitations, the theoretical sections addressed to other reviewers were brief. If you have specific concerns, and if time permits, we will do our best to provide a complete explanation. In the revised version of the paper, we plan to enhance the theoretical analysis in the appendix, incorporating feedback from all reviewers, and we welcome any further suggestions you may have.



Thank you for your time and consideration.

Authors
Dear Reviewer Y4qM, 

We have submitted our rebuttal several days ago, and the discussion process is now more than halfway through. However, we have not yet received your response. We greatly value your insights and are eager to hear your further feedback on our paper. Your comments will be instrumental in helping us improve the manuscript.

Thank you for your time and consideration.

Sincerely,

Authors
Dear Reviewer MmUh, 

We have submitted our rebuttal several days ago, and the discussion process is now more than halfway through. However, we have not yet received your response. We greatly value your insights and are eager to hear your further feedback on our paper. Your comments will be instrumental in helping us improve the manuscript.

Thank you for your time and consideration.

Sincerely,

Authors
Dear Reviewer yRix, 

We have submitted our rebuttal several days ago, and the discussion process is now more than halfway through. However, we have not yet received your response. We greatly value your insights and are eager to hear your further feedback on our paper. Your comments will be instrumental in helping us improve the manuscript.

Thank you for your time and consideration.

Sincerely,
Authors
Dear Reviewer BKzM, 

We have submitted our rebuttal several days ago, and the discussion process is now more than halfway through. However, we have not yet received your response. We greatly value your insights and are eager to hear your further feedback on our paper. Your comments will be instrumental in helping us improve the manuscript.

Thank you for your time and consideration.

Sincerely,
Authors
Thank you for your detailed response; I am less concerned with the model size and error analysis and willing to increase my score a bit, but I am still unconvinced on the generalization of the proposed method on addressing the long-range causal reasoning problems without more experimental comparison and theoretical analysis as pointed out by other reviewers.
Dear Reviewers,

The authors have submitted the point-to-point response. 
Could you all read the rebuttal and see if it addresses the main issues raised by your review? 
Are there any unaddressed/unresolved issues?

AC
To Reviewer nWAC:

Due to the input limit of 6000 characters in the Rebuttal input box, we have no choice but to adopt this approach to improve our statement. This message may be released before the start of the discussion, and we apologize for any inconvenience caused. You should be able to see the complete statement after the debate begins.

# Error analysis provided to Reviewer nWAC:

Initial State:
```
The blue block is clear, the orange block is clear, the hand is empty, the blue block is on top of the yellow block, the orange block is on top of the red block, the red block is on the table and the yellow block is on the table. 
```
Goal State:
```
The orange block is on top of the yellow block.
```
Expected output:
```
<unstack the blue block from on top of the yellow block> 
<put down the blue block>
<pick up the orange block>
<stack the orange block on top of the yellow block>
```
Actual output:
### RAP:
A structure that unfolds in a tree shape:

First Layer:
```
<unstack the blue block from on top of the yellow block>
```
Second Layer:
```
<put down the blue block>  
or  <put down the yellow block> (hallucination)
```
Third Layer:
```
<unstack the orange block from on top of the red block> (hallucination)
or  <unstack the orange block from on top of the yellow block> (hallucination) 
or  <unstack the orange block from on top of the blue block> (hallucination)
```
Fourth Layer:
```
<stack the orange block on top of the yellow block> 
or  <stack the orange block on top of the blue block> (pruned)
```
### CoT:
The logic of CoT reasoning output is to solve complex problems by step-by-step reasoning and refining intermediate steps, ensuring the accuracy and reliability of the final answer.
First Input: Initial State

First Output:
```
<unstack the blue block from on top of the yellow block>
<put down the blue block>
```
Second Input: Initial State + First Output:

Second Output:
```
<pick up the blue block> (hallucination)
<stack the orange block on top of the blue block>
```
### CRE(Ours):
Model one-time output of the whole process:
```
<unstack the blue block>
<put down the blue block>
<pick up the orange block>
<stack the orange block>
```
It should be clarified that CRE's mistake lies in the possibility of incomplete answers as mentioned above.

***The above appendix will be included in our revised version of the paper.***
Dear AC and Reviewers,

We would like to express our sincere gratitude for the time and effort you have dedicated to the evaluation of our work. Your valuable insights and expertise are deeply appreciated.

In response to your review comments, we have prepared detailed answers to address the concerns and inquiries you have regarding our paper. Should there be any unresolved issues, or should you need further clarification or have additional questions, please do not hesitate to let us know. We stand ready to provide any information or clarification that may assist you in your review.

We have actually written a more informative and easy-to-understand Rebuttal, but since we need to comply with the 6,000 character limit on Rebuttal submissions, we have had to simplify the presentation, and we apologize for any confusion this may have caused you. We would be happy to receive further questions from you so that we can present it to you more fully.

Thank you once again for your invaluable time and consideration. We eagerly look forward to hearing from you, and we hope to have the opportunity to respond to any feedback or questions you may have.

With warm regards,

Authors