Decompose, Analyze and Rethink: Solving Intricate Problems with Human-like Reasoning Cycle
Authors: Shangzi Xue
Zhenya Huang
Jiayu Liu
Xin Lin
Yuting Ning
Binbin Jin
Xin Li
Qi Liu
Venue: NeurIPS 2024 oral (2024)
Keywords: Reasoning Tree
Large Language Models
Question Decomposition
Rationale Updating
Primary area: natural_language_processing

Abstract: In this paper, we introduce DeAR (_Decompose-Analyze-Rethink_), a framework that iteratively builds a reasoning tree to tackle intricate problems within a single large language model (LLM). Unlike approaches that extend or search for rationales, DeAR is featured by 1) adopting a tree-based question decomposition manner to plan the organization of rationales, which mimics the logical planning inherent
in human cognition; 2) globally updating the rationales at each reasoning step through natural language feedback. Specifically, the _Decompose_ stage decomposes the question into simpler sub-questions, storing them as new nodes; the _Analyze_ stage generates and self-checks rationales for sub-questions at each node evel; and the _Rethink_ stage updates parent-node rationales based on feedback from their child nodes. By generating and updating the reasoning process from a more global perspective, DeAR constructs more adaptive and accurate logical structures for complex problems, facilitating timely error correction compared to rationale-extension and search-based approaches such as Tree-of-Thoughts (ToT) and Graph-of-Thoughts (GoT). We conduct extensive experiments on three reasoning benchmarks, including ScienceQA, StrategyQA, and GSM8K, which cover a variety of reasoning tasks, demonstrating that our approach significantly reduces logical errors and enhances performance across various LLMs. Furthermore, we validate that DeAR is an efficient method that achieves a superior trade-off between accuracy and reasoning time compared to ToT and GoT.

Decompose, Analyze and Rethink: Solving Intricate Problems with Human-like Reasoning Cycle
Shangzi Xue!
Zhenya Huang Jiayu Liu' Xin lin' Yuting Ning' Binbin Jin Xin Li Qi Liu
I: State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China Institute Artificial Intelligence, Hefei Comprehensive National Science Center (xueshangzi, jy251 198, ,lixx, mingyt, bb0725) ustc. edu. cn; (huangzhy, leexin, (eeeii,iiliuug)eeee..eee fuuitg edu, cn
Abstract
In this paper, we introduce DeAR Decompose-Analysee Rethinkk, a framework that iteratively builds a reasoning tree to tackle intricate problems within single large language model (LLM). Unlike approaches that extend or search for rationales, DeAR is featured by 1) adopting a tree- rree-based question decomposition manner to plan the organization of rationales, which mimics the logical planning inherent in human cognition; 2) globally updating the rationales at each reasoning step through natural language feedback. Specifically, the Decompose stage decom- poses the question into simpler sub-questions, storing them as new nodes; the Analyze stage generates and self-checks rationales for sub-questions at each node level; and the Rethink stage updates parent-node rationales based feedback from their child nodes. By generating and updating the reasoning process from more global perspective, DeAR constructs structures for complex problems, facilitating timely error correction compared to rationale-extension and search-based approaches such as Tree-of- Tree-of-Thouglee Thoughts (ToT) and Graph-of-TThoughts (GoTT). We conduct extensive experiments on three reason- ing benchmarks, including ScienceQA, StrategyQA, and GSM8K, which cover variety of reasoning tasks, demonstrating that our approach significantly reduces logical errors and enhances performance across various LLMs. Furthermore, we validate that DeAR is an efficient method that achieves a superior trade-off between accuracy and reasoning time compared ToT and GoT.
adaptive and accurate logical
Introduction
Learning to perform intricate reasoning, including commonsense reasoning (23). knowledge reason- ing [28), and mathematical reasoning [8], is a crucial step towards achieving general artificial intelli-
gence [49 20 25 26 21 24). The tasks always present significant challenge as they require many human-like intricate problem-solving abilities, such as abstract thinking and logical inference, which
could consolidate many decision-maker applications in real-world scenarios 38
36
Recent advances have witnessed remarkable performances of scaled-up up large language models (LLMs) in various reasoning tasks, including GPT (5), LLaMA [40], and ChatGLM 9
They
could enable several state-of-the-art prompting approaches like Chain-of- Thain-of-Thought (CoT) 45 Tree-
of-Thoughts (ToT) (49), Graph-of-Thoughh (GoT) (3, etc., enhancing reasoning capabilities. They not only improve problem-solving performance but also reveal their intrinsic reasoning steps
*Corespondiigg Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Question: J Janet's ducks lay 16 eggs per day, She eats three for breakfast every momiig and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?
Answer:
Question
Question
Site makes
f $2 pir
Jamee's ducks lay I6 egus per day. she cars the number of eggs day multiplied
The money sach day is equal to
Jamet's ducks loy 16 eggs per day. She solls 92 fresh duck ogg.
thmee for breakrastt every morning, she has 18-32 17 eggs, (x)
qt: W
selling price of one
Q2: How many eggs dors annt have per day?
egn?
Each egg is sold For
She has 16-1544) eggs per day.
She seills for per frnssh ogn, so the total namber she corns every day is 17
43: Hew
q4: How many breakfast every day?
@5: How many
She makes 17 $34.
Shee mikess 16 532.
does the ducks lay daih? ?
does anet bake
muffins every day?
-2 $34. (x)
The ducks lay 16 eggs per day.
She cats 3 Lggs for Irveakiast.
She uses eggs bake muuffns.
Answer: 34 (x)
(a) The simulation of ToT Reasoning
(b) The simulation of DeAR Reasoning
Figure Comparison between Tree-of- Theuuhh- (ToT) Reasoning and our DeAR (Decompose-
Reasoning on a reasoning -ased problem. (a) The simulation of Tree-of- Th-ouughsss (ToT) (branch = 3). (b) The simulation of DeAR Cecoompos--Analyee Analyze-Rethink) Rethink) Reasoning.
(i.e., rationales) [47] through linear, tree-based, or graph-based structures. For example, in Figure ! (a). given a math problem "Janet's ducks ... in dollars ... market?". ToT maintains tree of thoughts with intermediate nodes to generate the rationales step by step. Specifically, through several operations including exploration, termination, and traceback on the nodes, ToT ultimately identifies the complete reasoning path, highlighting two-step rationales (green nodes) leading to the answer. perform the reasoning process explicitly, such a rationale-extension and search-based reasoning paradigm is still far from numan-like intelligence and limits problem-solving abilities to some extent: On one hand, this tree-like structure is rigid and sometimes illogical. The ToT approaches often require setting a fixed number of thought branches ("3" branches in Figure [ (a)) each time it expands, which can result in either missing information or redundancy. Its reasoning process essentially extends previous rationales at each step, but falls short of the logical planning inherent in human thinking [33 42]. On the other hand, ToT generates rationale paths sequentially, and errors along the path, such incorrectly calculating "she has 16-3+4=17 eggs", cannot be promptly corrected. This allows mistakes to propagate to subsequent steps, ultimately leading to incorrect final outcome (e.g., "34").
However, although ToT and its variants 27 35
To address these challenges, we propose a novel reasoning paradigm DeAR (Decompose-Analyze Rethink), which enhances LLMs' capacity for complex problem-solving by emulating human reasoning (Figure This approach inspired by several theories in cognitive science 43 30 Specifically, reasoning simplification theory (33 suggests that when confronted with an intricate question, humans tend to break it down into simpler ones, which help in organizing thoughts and solving problems more logically. Referring back to Figure (b), can break down the logic by first solving two sub-questions (q1 and 92). Upon examining 92. we find it can be further divided into three additional sub-questions (g3, q4, and 95). By sequentially resolving these sub rub-questions and using their results as feedback to update answers for previously generated -ubb-uussions (41 and 92), ultimately arrive at the final answer ("18").
To implement such a human-like problem-solving process, we introduce a Decompose-Anulyze Rethink cycle. This involves gradually constructing a reasoning tree guided by sub nub-questions, following a top-to-bottom reasoning process as illustrated in Figure (b). The process begins with the Decompose stage (black arrows in Figure (b)). where prompt- method breaks down the question into simpler sub-questions at subsequent nodes. Then, the Analyze stage (green box at each node) takes charge of problem-solvingg at the node level. The stage also introduces a self-check module to ensure the quality of the generated rationales, thus refines the reasoning process. Last, in the Rethink stage (indicated by green arrows), the result at the current node is evaluated to determine if the reasoning in parent nodes requires further updates, providing a global perspective. After multiple cycles, the answer can be summarized from the root node.
Compared to ToTs 49 27 35
and GoT R, our approach presents the following highlights. First, unlike ToT/GoT methods which directly generate rationales as branches from the original question,

---

DeAR breaks it into sub-quese question tree nodes to guide the generation. Second, our tree structure more flexible and adaptable, as each node is generated and updated autonomously by the large language model based on the problem's 8 logic, without relying on predefined settings. Third, DeAR enables timely correction of rationales, ultimately ensuring the correctness of the root node's answer.
We conduct extensive experiments on three complex reasoning benchmarks including ScienceQA (28), Strategy [2]. and GSM8K 3 Experimental results show that our approach enhances the reason- ing performance with different backbones such as GPT-3.5 [1. LLaMA2 40),, and ChatGLM3 (9. Compared state-of-the-art art methods such as Tree-of-Thougee
(ToT) and Traph-of-Thoughhe (GoT), DeAR demonstrates a significant improvement in reasoning accuracy across all backbone LLMs, validating its generalizability and scalability. Additionally, by measuring the relationship between reasoning accuracy and reasoning time across different datasets, DeAR exhibits greater efficiency, further underscoring its advantages in practical applications.
Related Work
2.1
Prompt-based Approaches in LLM Reasoning
There has been a growing interest in LLM reasoning research, with various prompting schemes applied in areas such commonsense 2
mathematical [8] and knowledge reasoning [29], etc.
Early methods appends examples on top of the input question (few-shot prompting (5 or performs
in-context learning (ICL) (37),, ) or includes examples at all (zero-shot prompting) 44
Recent research has sought to enhance the capabilities of large language models (LLMs) by intro- ducing intermediate reasoning steps into the prompting process, epitomized by methods such as the Chain-of-Though (CoT) (45). By prompting LLMs to solve problems step by step, the CoT method demonstrates outstanding performance in multi-step reasoning tasks. Self-consistency [41] is a significant improvement upon COT, where multiple CoT paths are initially generated, and the best one is selected the final result, thereby improving the reliability of the outputs. In parallel, other prompting methods design scarch- based schemes for LLMs, such as Tree-of-Thought (ToT) [9 and Graph-of- Thhouhhs- Thoughts (GoTT) 3 which innovate by structuring the reasoning process into or graph structures. These structures are created to take advantage of the many reasoning paths that LLMs can generate, greatly expanding the range and depth of exploration for any given question. More recently, Reasoning via Planning (RAP) [16] repurposes the LLM as both a world model and a reasoning agent to conduct reasoning. These methods expand the reasoning space of LLMs, which can fully leverage the diverse thinking paths generated by LLMs.
2.2 Question Decomposition
Question decomposition, which decomposes complex questions into multiple sub-ones, has been shown to largely improve models' reasoning ability. Early works 4 decompose questions with hand-crafted rules and lexicon-syntactic features. These works heavily rely on human efforts, which are hard to extend to general domains and tasks. Recently, researchers utilize neural network models For example, Min al. [32 focused on directly training model to produce sub-questions using question spans; BREAK A6 followed an alternative paradigm of collecting full question decomposition meaning representations (QDMR) annotations. However, primary challenge lies in the scarcity of annotations for training a decomposition model [32].
to decompose questions 39 [8 52
More recently, in the era of LLMs, there lot of work exploring LLMs for question decomposi- tion [50 19 17 10 7 22 51 For example, ToT [49] prompts the LLM to decompose the rationales by searching intermediate steps. Least-to-most prompting (56) leverages few examples teach LLMs to decompose each problem into series of simpler sub-problems. These prompting-bases based question decomposition methods serve an important step in reasoning and planning with LLMs.
3 Problem Formulation and Preliminaries
3.1 Problem Definition
In this paper, we focus on the intricate reasoning task, The input of the task the question Q (e.g.. "Janet's ducks ... market?" in Figure D] The output is rationale R (21, .., Tk) with & word
3

---

Decompose
Analyze
Rethink
She malkes
She makes
Shee mikkss
9*14427 per day.
Oo Swg77 per dkyy
gP2es18 per day.
of: W hat price egu?
qf: How eggs have per day?
ef: W hat is the selling price ose ege?
: Blo
a W hat
q How many does Janet have day?
eggs dues per day?
sellng egg?
Ff: Fach egg lood
Fg: egg
Stw sggs day
rj Each suld for
per eggg
Sell cholis
Shi 3P 2-21188
18
nestion: Janet's ducks lay 16 eggs per day, eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' markest daily for s2 per fresh duck egg, How much in dollars does she make every day at the farmens' markeet?
41: What the selling petoo one ags? Back Sars2
aी Howe manty saga dow
Janet have per day?
Decompose
Rethink
Son has (H-
sago por day
Analyze
cass does
et: raany
dem, Jane
aqu
ol Hov
hone per day?
uggs dees Junet
base per day?
Reasoning Tree 7
booiifes
Figure 2: A demonstration of the DeAR (Decompoxe-Anulyze-eelll cycle.
tokens ("She makes 9 x $2 = $18 per day."), and the answer A ("11") derived from R. Given the input question Q, we aim design reasoning framework with LLM backbone pe generate the rationale R and answer A as outputs.
3.2 Reasoning Tree
Motivated by the reasoning simplification theory 33 propose a novel reasoning structure for LLMs, named Reasoning Tree T, as shown Figure lbb. Overall, this Reasoning Tree decomposes and resolves sub-questions using a top-down approach, while concurrently updating existing solutions through a bottom-up process. Formally, the Reasoning Tree T can be defined as T = (N, E) where r, s) N contains question g
N is the set of tree nodes and E is the edge set. Each node as a sub-question of the target Q (c.g., 42 "How many eggs does Janet have per day?"), a rationale r to ("She has 16-(3+4) directed edge e = (Np, n2)) E E means that the upper-level sub-question 4p in the parent node np is decomposed into lower-level one q. in the child node ne (e.g., the parent 42 "How many eggs ... have per day" is decomposed into three children "How many eggs lay", q4 "How many eggs breakfast". and q5 "How many eggs ... muffins").
eggs per day."), and
evaluating the logical coherence of r. Each
Our Reasoning Tree is progressively constructed and updated. The target question Q the root node is decomposed into sub-questions step by step, from sub-questions in the higher levels the ones in the lower levels (i.c., the black directed edges in Figure [I. For example, Q is first decomposed into Qi and 92. then d2 is further decomposed into q3. Qa and Ø5. Furthermore, humans could also rethink the rationales generated earlier (in the higher nodes) based on the ones generated later (in the lower nodes). For example, the rationales for 94 ("She eats 3 eggs for breakfast") could be used to update rationales for 92 ("She has 16-(3+4) = 9 eggs per day") through the dashed lines green.
3.3 Framework Overview
To construct the aforementioned Reasoning Tree T, which imitates human-liae reasoning, we propose a novel DeAR (Decompose-AAualye- Rethink) cycle as the core of our framework, as illustrated in Figure 2] The cycle is composed of three stages: Decompose, Analyze and Rethink. Specifically, in the Decompose stage, one upper-level question decomposed into several lower-level ones. In the Analyze stage, the framework solves the newly generated sub-questions by generating and self- checking rationales. the Rethink stage, the newly generated rationales are used to update existing ones in the parent nodes. The three stages work in cycle to build the reasoning tree T.

---

DeAR (Dcooppse-Audly--Relhine Cycle
In this section, we will demonstrate how the reasoning tree T is constructed with the Decompose- mualyze-Rething cycle, as demonstrated in Figure 2
Initially, the target question is set the question go in the root node nuy The framework selects an existing edge node n, = (gt, Ts, St) (t is the level of the node) from T (e.g., ng with Q "Janet's ducks market?") to start the cycle. First, in the Decompose stage (4.1),
prompt LLMs to decompose the question gr in the node into sub-questions 4t possible, and store them in nodes M++1 at level 1 (e.g., qi "What is ... one egg?". and of "How many ... per day?"). 'hen, in the Analyze stage (4.2), we conduct reasoning and answers the newly generated questions 4++1 by generating rationales r1+1 for them (ri "Each egg is sold for S2" for q. and rf "She has 16 eggs per day" for q checking their correctness and evaluating the coherence scores st+1 (Eq. ( Next, in the Rethink stage 4.3). we use the newly generated r++11 to update rationales in existing upper-level nodes ri I t) (e.g., use ri and ri to update ro into r()). After that, the framework selects another edge node and returns the Decompose stage (e.g.. decompose of into 92 o and q3). The cycle continues until the LLMs determine that no further decomposition is possible, thereby forming the reasoning tree for Q.
As Q is the question go for the root node no, after the tree-construction process, we consider the rationale ro in the root node as the overall solution for Q and extract the answer A from ro- The whole procedure is described in Algorithm|] [ In the following sections, we will technically describe the three stages in the cycle and make detailed analyses.
4.1 Decompose Stage
Algorithm I Decompose--uutlze-RRtiin
Input: Question Q
According to the Analogical Reasoning the- ory 2]. when humans conduct reasoning, they often analogize the logical processes of new questions to those of similar questions. There- fore, to make the decomposition logic of sub- questions qe each level more closely re- semble that of humans, we first use human- annotated question decomposition examples (Appendix(A.) (A.]) as demonstration pool P. Then we calculate the cosine similarity of the repre- sentations between Q and each Q in P and select top-K nearest neighbors in the vector space. After that, we concatenate each Q with its numan-annotated sub-questions subgs' (subgi subg2, .... subg',) ) to form K question- decomposition examples (Appendix(A. A.
Parameters: LLM pe, natural language prompts
(c1 co), threshold 61 for Decompose, threshold
62 for Rethink
Output: Rationale R, Answer A Create an empty node queue N Enqueue (9 Q, ro None, 80 1) into N while N is not empty and current level < max deptt do
Dequeue current node nu(g,, st) from N if ne end node Bendd then continue
else #; > E1 then
// Stage I: Decompose
(r++d) Decompose(po: h1, tha, q:)
Stage Analyze
r4t Solvelps, h2, 4(+1) 3
F4 Self SheckCChe ha, Lo r441)
lhq (Q subgs') (i 1,2...., K).
(1)
8/41 Score(po, ha, Z.41 p24)) (5)
Set m/ (16-1: P 82 2.41
These examples are regarded as "logic heuristics" that inspire the model to decompose questions in a manner closely aligned with human reasoning.
Enqueue ni/u, into N Stage Rethink
if 8/41 > 62 then
La + Extract(po, hs, L, a
After obtaining lhg. we utilize them to decom- pose the sub-question ge level & into multi- ple sub-questions at level + 1. Specifically, given question qt- if its coherence score St (Eq. (5)) is higher than a threshold é1, We ask the LLM whether it needs be further decom- posed. If requires decomposition, we then prompt the L.M autonomously break down into several sub-questions {ui+1.j 1,.... J). It is worth noting that in our decomposition ap- proach, we do not pre-specify the number of sub-questions; instead, we allow LLMs to adap-
r' + Update(po. ha, ne nelg r)s), (
ne 8) ne(g,
else end if end while
Enqucue Rend into N
<- ro Extract answer A from R return R, A
'Sec 5.1.2 for max depth and branch settings.
5

---

tively determine it based on the logic of each question. However, the number of sub-questions is capped at predefined maximum branch limit ensure computational efficiency and manageabil- ity 5.1. This enhances adaptability and more closely aligns with human logical characteristics when compared to existing methods like ToT [49] and GoT 3]. etc. To facilitate this process, we design a heuristic-enhanced prompt that consists of a prompt head h and "logic heuristics" Thg. The prompt head describes the question decomposition task in natural language. This process is formulated in Eq. 2 Additionally, we validate the effectiveness of using logic heuristics, and provide detailed explanations and templates in Appendix A.I
1 L,,.., J} f Decompose(po- h1, lhq. qt).
After decomposition, each 4/41 is added as new node n141 at level t + 1, with a directed edge from n, ni++ (denoted as e = (ng, 1)) )). If the LLM determines that qe does not require further decomposition, we create a leaf node aand as a child of n,
4.2 Analyze Stage
In Analyze stage, we reason the answers for all the sub-questions {u/-1> } at level /+ 1. To be specific, first prompt the LLM generate the essential rationale r041 for each sub-question 7441
7 + Solve(po, l22 d/-1)-
(3)
Here, h2 denotes the prompt head, which is natural language sentence that asks the model to generate detailed solutions (see Appendix A.22..
After obtaining the rationales for the sub- sub-questions, we evaluate and correct them, as large language models (LLMs) often tend to hallucinate during problem- boolem--olvingg [54]. Using generated rationales without verification can propagate errors, leading to incorrect outcomes. To address this issue, we develop self-check method that promptly identifies and corrects these errors while providing coherence score (Eq. (5)) for each node.
Specifically, we first instruct the LLM to perform a self-check on the rationale r/+1 generated for the sub-question 4'41 (see Appendix A.2 for the prompt head h2) identify any potential errors. If the LLM detects errors in the original rationale ri+1 it modifies the rationale to fft1; otherwise, the rationale output unchanged. Take the case in Figure 2 as an example, we expect the LLM to identify the error "Each egg is sold for $3" in ri. and correct it to "Each egg is sold for $2". This process is denoted as:
P+1 Seel_Seeeehhlee Chhekk(pp. h3, d411 1(+1).
Then, we prompt the LLM to evaluate the logical coherence between the refined rationale P/+1 and the question o by generating coherence score s442 (see Appendix A.2 for prompt head hy):
L +- Score(ps, h4s dit 12
(5)
The score s/__ can also be obtained through voting or classification methods. Here, we specifically
investigate the effectiveness of directly prompting LLMs to generate numerical values as scores.
At the end of the Analyze stage, we fill the obtained rationales and scores into nodes n (j 2 1):
(a'4 Pl P: $t
where 8/41 can support the current or subsequent cycles in Rethink 4.3 and Decompose 4.1p
4.3 Rethink Stage
in cognitive science, humans constantly update and reflect on their previous reasoning results based on the current information. This allows us to correct past mistakes and ultimately achieve consistent and stable answer. For example in Figure person might initially answer question Q ("Janet's 's ducks ... How much ... market?") with the rationale ro "She makes 9 3 = $27 per day". affer considering responses to sub-questions of ("What
According to self relf-reflection theories 13,6 6

---

is the selling price of one egg?") and oी ("How many eggs does Janet have per day?"), he/she realizes an error in ro- The correct calculation, using the values "2" for the price per egg and "9" for the daily number of eggs, should be "2 x 9 $18".
Nevertheless, existing methods like ToT 48 search reasoning paths based solely on preceding steps, lacking the ability to retrospectively update carlier content based on the influence of later steps. To address this, we introduce Rethink stage that mirrors the human reflective process.
Specifically, during the rethinking process, humans first identify which existing reasoning steps may require revision. We aim to automate this by using LLMs to detect logical connections between ancestral and newly generated nodes, updating ancestral nodes based on insights from the rationales of new nodes. In our proposed "Reasoning Tree", we essentially use information from lower-level nodes to "rethink" higher-level nodes, closely mirroring the human cognitive simplification process in problem-solving (33]
To achieve this, after obtaining node n²41 in Analyze Stage, we first check its coherence score 8'41 (Eq. (.. If 41 exceeds the threshold E2, we then examine the correlation between 4'41 and all sub-questions above level t, specifically, {qi, S t}. Next, we extract a subset of most related nodes Lg from I, _ (m, S i] (the specific nodes be extracted are determined by the LLM)):
Ls + Extract(pa hs, L, 1. LA S L.
(7)
where hi is a prompt head (Appendix A.3) Next, we use the rationale e of sub-question 4.4 to update the rationale of each extracted node ne in Lb:
r' + Update(po; he, nelg, r, 8), (
(8)
Finally, we replace r with the updated rationale y':
Te (g. r', c no (g,
(9)
5 Experiments
In this section, we demonstrate the generality and effectiveness of DeAR by applying wide range of tasks, including knowledge reasoning, logical reasoning and mathematical reasoning. The results across these tasks validate DeAR's adaptability and highlight its capability to effectively tackle a diverse range of challenging reasoning tasks.
5.1
Experimental Setup
5.1.1 Datasets and Baselines
We employ the ScienceQA (28) dataset for the knowledge reasoning task. And we use Strate- gyQA [12] for logical reasoning that requires multiple reasoning steps. We also verify the mathemati-
cal reasoning ability of our framework by applying it to GSM8K dataset [ The details of these datasets available in Appendix B.I.I
In our main results, we compare DeAR with multiple prompt-based methods including Few-shot prompting (5), Chain-of-Thoure
(CoT) prompting (45), and state-of-ther the-art Tree-of-Thoughe (GoT) 3 prompting. Besides, we also list extra comparison results with another two state-of-the-art prompt-based methods Least-to-moo most Prompting [56) and SelfCheck 31 (see Appendix B. B.1.2 for all baseline details).
(ToT) 49 and
5.1.2 Implementation Details
We conduct experiments with three LLM backbones GPT-3.5 (1. LLaMA2-7B 40 and ChatGLM3-
6B [9. For GPT-3.5, we use the OpenAl API to invoke the -gpr-3.5-turbo- 106" model. For LLaMA2-
7B and ChatGLM3-6B,, load the checkpoints from huggingface P and use the models directly
bhttss/tuuginnffaccco/HUUUU http://f......eaoooeoeeee Twttee/ececccTTUDDD/ichannn- bhtpss//huggingfffcccoomm co/meta laamaLLama-22-7
7

---

Table 1: Overall results of our DeAR Framework three intricate reasoning datasets. (+: p 0.05).
ScienceQA
StrategyQA
GSMBK
GPT-3.5
LLaMA2
ChatGLM3
GPT-3.5
LLaMA2
CharGLM3
GPT-3.5
LLaMA2
CharGLM3
Fow-shor
73,97
66.35
42.46
67.71
61.21
54.41
74.26
72.25
51.02
CiT
75.17
46.35
69.26
63.86
57.18
79.55
74.04
53.85
ToT
49.58
71.89
59.21
83.42
55.88
GoT
82.34
68. 86
49.26
72.02
66.61
59.88
84.77
75.95
56.01
DeAR
83,68*
70.57*
51.08*
73,36
68.33*
61.02
86.82*
78.01
58.54
to most
47.45
70.55
58.36
81.25
54.21
SeltChock
75.81
49.23
68.87
66.35
61.22
79.88
75.28
56.72
without fine-tuning as the backbone For each dataset, we randomly sample 10% of its training set as a validation set to select different combinations of thresholds E and é2. The combination that achieves the best performance on the validation set is then used for inference on the test set. We observe that the threshold combinations obtained through this method also yield optimal inference results the test set. In Section 5.6 we visualize the inference accuracy on the test sets across different datasets based on GPT-3.3, using diffenrent threshold combinations. The implementation and prompting templates (i.c., natural language prompts h1 ~ 09 for Decompose, Analyze and Rethink) are shown in Appendix A To ensure computational efficiency, we set the maximum depth to 4 and the maximum number of branches to 3 during the construction of the reasoning tree in DeAR. This prevents the tree from becoming excessively deep and avoids redundancy in sub-questions. For baselines, the settings used in the experiments are consistent with those described in the original papers. For a concise description of baselines, please refer to Appendix B.1.2
5.2 Experimental Results
We conduct experiments verify the effecttivness of our framework DeAR, and report the results in Table 1 We use the accuracy (ACC) as metric for all three datasets. We statistically test the improvement over baselines with paired t-test, and find the improvement to be significant with < 0.05 (marked with "*"). We get the following observations. First, DeAR performs better than all baselines, which indicates improvements over ToT highlight the advantage of Decompose stage which adaptively decomposes questions based on their characteristics rather than extending a fixed number of thought branches. Third, DeAR performs better than GoT which lacks rationale updating. This reflects the superiority of the Rethink stage to identify correlations between reasoning steps and update previous rationales. Besides, the accuracy increase on GSM8K is greater than ScienceQA and StrategyQA. That is probably because problems in GSM8K require longer rationales to be solved (Table 2). Furthermore, DeAR outperforms the Least-to-mon most (56] and SelfCheck [31 methods across all datasets. The Least- to-most method sequentially solves sub-problems derived from the decomposition without updating content that has already been generated; SelfCheck updates rationales but it does not decompose the original question. In contrast, DeAR not only generates rationales based on decomposed sub- questions but also updates existing rationales in each cycle. This further underscores the necessity of the Decompose and Rethink phase in DeAR for enhancing the reasoning capabilities of L.LMs.
more effective in enhancing LLMs' reasoning ability. Second, the
We have also validated that DeAR enhances stronger LLMs (e.g., GPT-4) on complex reasoning tasks (e.g., MATH), as shown in Appendix. Appendix B.3 includes an ablation study on the self-check method in the Analyze stage, its removal does not structurally impact the other stages.
5.3 Analyses of the Reasoning Tree
Table Characteristics of T different datasets.
For each question Q, DeAR constructs a reason- ing tree T represent the reasoning process, as shown in Figure (b). The structure of T provides insights into the complexity of Q. analyze the nature of questions across datasets, we examine reasoning trees from three datasets using three metrics: "Avg Branch," "Avg Depth," and "Avg Length of R." "Avg Branch" indicates the average branching factor of Tf "Avg Depth" reflects the average depth of T', and "Avg Length
ScienceQA
Strategy
GSM8K
Avg Branch
1.58
2.43
2.06
Avg Depth
3.62
1.96
2.55
Avg Length of R
66.34
61.55
85.27
*Our code is available at: https:: www.tauu..oom/maaaaaaaee www.hhwww..coo/aaaaaneeeee//aaae
8

---

Table 3: ROSCOE evaluation results ratio- nales generated by Tree-of-Thoughe (ToT), Graph-of-Thoughe (GoT) and DeAR on dif- ferent datasets. SC Source-Consistency: RA =Reasoning Aligmment.
66
Go
51
46
ScienceQA
StrategyQA
GSM8K
24
25
24
SC
RA
RA
SC
RA
ScienceQA
Strategy
GSMSK
ToT
0.44
0.31
0.47
0.33
0.56
0.41
Figure 3: The distributions of annotators' se- lections. More annotators considered DeAR's rationales to be more logical.
GoT
0.42
0.35
0.44
0.38
0.53
0.45
DeAR
0.48
0.42
0.52
0.43
0.58
0.50
of R represents the length of rationale R derived from the root node nig upon tree completion, e.g..
R = "Se makes 9*2=$18 per day" in Figure
Using GPT-3.5 as the backbone, results in Table 2 reveal the following: ScienceQA questions have the highest "Avg Depth" and lowest "Avg Branch," indicating fewer sub-questions per Decompose stage but more rounds required. Strategy questions have the lowest "Avg Branch" but the highest "Avg Depth," suggesting fewer Decompose rounds but more sub-questions per round. For GSM8K, the root node ng has longer rationales R, suggesting that these questions require more extensive explanations than those the other datasets.
5.4 Logical Coherence the Generated Rationales
We assess the logical coherence of rationales generated by DeAR using both automatic and human evaluation methods. For automatic metrics, we apply the Alignment" (RA) from the ROSCOE evaluation suite [14]. SC measures logical entailment between question and rationale, while RA evaluates alignment with ground truth. As shown Table DeAR outperforms ToT and GoT on all datasets. For human evaluation, 100 questions were sampled from each dataset, with annotators selecting most logical rationale among those generated by ToT, GoT, and our method (details in Appendix B.4). Results in Figure 3 confirm that DeAR (using GPT-3.5) produces rationales with superior logical coherence compared to ToT and GoT.
(SC) and Reasoning
Table Comparisons between different portions of "Random Update" and DeAR.
5.5 Effectiveness of Rethink
Random Update
ScienceQA
Stralegy
GSM8K
In Rethink stage, our DeAR employs the same backbone LLMs to determine which nodes' ra- tionales need to be updated. To validate its ef- fectiveness, based on GPT-3.5, we compare our method with "Random Update" method which randomly selects nodes to update different proportions. The results in Table 3 demonstrate that, compared to "Random Update", our method performs better in terms of accuracy. Additionally, unlike approaches that require 100% update of all generated rationales, DeAR's targeted updates allow model to autonomously select nodes that need refinement, thus minimizing unnecessary inference.
0%
82.77
72.84
85.09
20%
81.77
72.21
83.96
40%
82.59
73.03
84.35
60%
82.06
72.29
85.07
80%
81.49
72.04
86.01
100%
81.16
71.79
85.32
DeAR
83.68
73.36
86.82
SciemasQA
Strotegy&Q
GSMBK
5.6 Combinations of Thresholds
=00
In this subsection, visualize the impact of different combinations of threshold values 6j and 62 on the in- ference accuracy of DeAR (with GPT- 3.5 backbone) across the test sets of all three datasets. and e2 are set for the Decompose stage (Section 4.T)
m
00
01
05
Figure 4: Combinations of threshold values (61, 62) and corresponding ACCs on test sets (GPT-3.5 backbone). respectively, with their value combinations selected based on
and Rethink stage (Section
4.3

---

SciencoQA
StratogyQA
GSM8K
0.735
0.835
b=t.58
b-2 23d-119
0.86
1-2, 6-
0,750
0,830
be
0,725
0.85
ACC
ACC
b=2d=3
ACC
0.720
0.84
beiddw
1-2d-22
0.715
b=3d=3
0.83
0.815
beildw4
0.710
0.810
b=2
6-2
be2d+3
0.82
0.705
sec/question
see/question
see question
GoT
Figure 5: Efficiency comparison between DeAR and variants of ToT/GoT.
performance on the validation set (Section 5.1.2 We observe from Figure thhtt, that, DeAR achieves the highest accuracy when setting é1 = 0.4 and 62 0.6 for ScienceQA and StrategyQA. For GSM8K, the highest accuracy is obtained with 61 0.4 and 62 = 0.4. The threshold combinations that optimize DeAR's 's performance on the test set are consistent with those obtained from the validation set (c.g., Val: E = 0.4: E2 = 0.6 for ScienceQA), demonstrating the validity of the value selection method. Additionally, the smaller optimal E2 value for GSM8K suggests that tackling GSM8K prob- lems requires a more frequent or active rethinking process compared to ScienceQA and Strategy This difference highlights the varying nature of reasoning demands across different tasks, where the threshold tuning helps adapt DeAR's ' reasoning process accordingly.
5.7 Efficiency
Compared to the rationale extension in ToT and GoT, DeAR incorporates question decomposition and rationale updating. Thus, will the efficiency of reasoning be affected? To investigate this, we use ChatGLM3-68 as the backbone model and measure the average inference time per question (scconds/question) and accuracy (ACC) for each method. The results are in the form of scattered points as shown in Figure 5 We set the fixed branch numbers and depths for these variants of ToT and GoT (e.g., b=3, d=4), and compare them with DeAR. In ToT/GoT, we set "b" and "d" (integers) as close to DeAR's average values as possible to ensure fairness. We can observe that points closer to the upper-left corner, and farther away vertically from the diagonal, represent methods that achieve better trade-off between reasoning accuracy and time. The points corresponding to DeAR clearly exhibit this characteristic, hence we can conclude that has higher efficiency. Moreover, in Appendix B.5 to further validate this conclusion, we measured the average number of API calls made by DeAR, ToT, and GoT per question in the ScienceQA dataset using GPT-3.5, as well as their reasoning accuracy. DeAR consistently requires fewer API calls average to solve question, while simultancously achieving higher accuracy.
Conclusion
In this paper, we introduced DeAR Decompose-Analyzz-Reeiine a framework designed to mimic human reasoning patterns in tackling intricate problems by constructing a reasoning tree in a top- down, iterative manner.This approach is coupled with a Sertemppoe--nnayy--RRReile the rationale at each node is generated, evaluated, and refined through feedback loops. Specifically, the Decompose stage applies logic heuristics to decompose the original question, the Analyze stage produces and self-checks rationales, and the Rethink stage integrates these insights by updating parent nodes based on child-node feedback. Extensive experiments demonstrate that DeAR not only improves reasoning performance across different large language models (LLMs) but also surpasses current state-of-the-an art methods like Trre-of-Thoughhe (ToT) and Traph-of-Thoughhe (GoT) in logical coherence and accuracy. DeAR's 's rationale update mechanism enhances logical consistency by iteratively refining previously generated rationales, achieving more accurate and interpretable results. Additionally, compared to ToT and GoT, DeAR strikes better balance between reasoning accuracy and inference time, further improving efficiency. Case studies also demonstrate that DeAR produces more interpretable reasoning process (Appendix B.6).
cycle, in which
10

---

Acknowledgments and Disclosure of Funding
This research was partially supported by grants from the National Natural Science Foundation of China (No.62106244, 62337001), and the Key Technologies R&D Program of Anhui Province (No.202423k09022003))
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad. Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] Paul Bartha. Analogy and analogical reasoning. 2013.
[3] Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. arXiv preprint arXiv:2308. 09687, 2023.
[4] BK Boguraev. Fact-based question decomposition in deepqa. IBM Journal of Research and Development, (6(3.4):13-1, 2012.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[6] Tyler Burge. Cognition Through Understanding: Delf-Knowledge, Interlocution, Reasoning, Reflection: Philosophical Essays, Volume 3, volume 3. Oxford University Press, USA, 2013.
[7] Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211. 2022.
[8] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14166,, 2021.
[9] Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 320-335, 2022.
[10] Dheeru Dua, Shivanshu Gupta, Sameer Singh, and Matt Gardner. Successive prompting for decomposing complex questions. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1251-1265, 2022.
[11] Shane Frederick. Cognitive reflection and decision making. Journal of Economic perspectives, 19(4):25-42 42, 2005.
[12] Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did Aristotle Use Laptop? Question Answering Benchmark with Implicit Reasoning Strategies. Transactions of the Association for Computational Linguistics (TACL), 2021.
[13] Alex Gillespie. Becoming other: From social interaction to self-reffection. IAP, 2006.
[14] Olga Golovneva, Moya Peng Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. Roscoe: A suite of metries for scoring step-by-step reasoning. In The Eleventh International Conference on Learning Representations, 2022.
[15] Zheng Gong and Ying Sun. Graph reasoning enhanced language models for text-to-sql. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2447-2451, 2024.

---

[16] Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhit- ing Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305. 14992, 2023.
[17] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners:
Extracting actionable knowledge for embodied agents. In International Conference Machine Learning, pages 9118-9147. PMLR, 2022.
[18] Mohit lyyer, Wen-tau Yih, and Ming- expressed in decomposed question sequences. arXiv preprini arXiv: wexiv::611.01220 1.01242, 2016.
Ming-Wei Wei Chang. Answering complicated question intents
[19] Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. In The Eleventh International Conference on Learning Representations, 2023.
[20] Xin Lin, Zhenya Huang. Hongke Zhao, Enhong Chen, Qi Liu, Defu Lian, Xin Li, and Hao Wang. Learning relation-enhanced hierarchical solver for math word problems. IEEE Transactions Neural Networks and Learning Systems, 2023.
[21] Xin Lin, Zhenya Huang, Hongke Zhao, Enhong Chen, Qi Liu, Hao Wang, and Shijin Wang. Hms: A hierarchical solver with dependency-enhancee understanding for math word problem. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 4232-4240, 2021.
[22] Zhan Ling, Yunhao Fang, Xuanlin Li, Zhiao Huang, Mingu Lee, Roland Memisevic, and Hao Su. Deductive verification of chain-of-thought reasoning. Advances in Neural Information Processing Systems, 36, 2024.
[23] Jiacheng Liu, Alisa Liu, Ximing Lu,, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi, and Hannanch Hajishirzi. Generated knowledge prompting for commonsense reasoning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3154-3169, 2022.
[24] Jiay Liu, Zhenya Huang, Xin Lin, Qi Liu, Jianhui Ma, and Enhong Chen. A cognitive solver with autonomously knowledge learning for reasoning mathematical answers. In 2022 IEEE International Conference on Data Mining (ICDM), pages 269-278. IEEE, 2022.
[25] Jayy Liu, Zhenya Huang, Zhiyuan Ma, Qi Liu, Enhong Chen, Tianhuang Su, and Haifeng Liu. Guiding mathematical reasoning via mastering commonsense formula knowledge. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1477-1488, 2023.
[26] Jiay Liu, Zhenya Huang, Chengxiang Zhai, and Qi Liu. Learning by applying: A general frame- work for mathematical reasoning via enhancing explicit knowledge learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 4497-4506, 2023.
[27] Jieyi Long. Large language model guided tree-of tee-of-thouugt. arXiv preprint arXiv:2305.08291, 2023.
[28] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
[29] Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. Language models of code are few-shot commonsense learners. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 1384-1403, 2022.
[30] Kenneth Ian Manktelow. Reasoning and thinking. Psychology press, 1999.
[31] Ning Miao, Yee Whye Teh, and Tom Rainforth. Selfcheck: Using llms to zero-shot check their own step-by-step reasoning. In The Twelfth International Conference on Learning Representa- tions, 2023,
12

---

[32] Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi. Multi-hop reading comprehension through question decomposition and rescoring. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6097-6109, 2019.
[33] Allen Newell, Herbert Alexander Simon, et al. Human problem solving, volume 104, Prentice- hall Englewood Cliffs, NJ, 1972.
[34] Hongbin Pei. Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-gen: Geo- metric graph convolutional networks. In International Conference on Learning Representations, 2020.
[35] Leonardo Ranaldi and Fabio Massimo Zanzotto. Empowering multi-step reasoning across languages via tree-of-thoughts. arXiv preprint arXiv:2311.08097 2023.
[36] Revant Gangi Reddy, Xilin Rui, Manling Li, Xudong Lin, Haoyang Wen, Jaemin Cho, Lifu Huang, Mohit Bansal, Avirup Sil, Shih-Fu Chang. et al. Mumuqa: Multimedia multi-hop news question answering via cross- moos- knowledge extraction and grounding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11200-1 1208, 2022.
[37] Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. arXiv preprint arXiv:2112.0080 0863333, 2021.
[38] Zhengxiang Shi, Qiang Zhang, and Aldo Lipani. Stepgame: A new benchmark for robust multi- hop spatial reasoning in texts. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 1 11321-118 11399, 2022.
[39] Alon Talmor and Jonathan Berant. The web as knowledge-base for answering complex questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long Papers). pages 641-651, 2018
[40] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:230.009288 2023.
[41] Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistnncy improves chain of thought reasoning in language models. arXiv preprint ar arXiv:2203.1111 2022.
[42] Yingxu Wang and Vincent Chiew. On the cognitive process of human problem solving. Cognitive systems research, 11(1):81-92 2010.
[43] Peter Cathcart Wason and Philip Nicholas Johnson-Laird. Psychology of reasoning: Structure and content, volume 86. Harvard University Press, 1972.
[44] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. arXiv preprint arxiv:2109.01652, 2021.
[45] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of -f-thuught prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-2483 2022.
[46] Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. Break it down: A question understanding benchmark. Transactions of the Association for Computational Linguistics, 8: 183--99 2020.
[47] Shangzi Xue, Zhenya Huang, Xin Lin, Jiayu Liu, Longhu Qin, Tianhuang Su, Haifeng Liu, and Qi Liu. Enhancing the completeness of rationales for multi-step question answering. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 2753-2763, 2763, 2024.
[48] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas I. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv: 10601, 2023.
13

---

[49] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths. Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in Neural Information Processing Systems, 36, 2024.
[50] Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, and Yongbin Li. Large language models are versatile decomposers: Decomposing evidence and questions for table-based rea- soning. In Proceedings of the 46th International ACM SIGIR Conference Research and Development in Information Retrieval, SIGIR '23, page 174-184, New York, NY, USA, 2023. Association for Computing Machinery.
[51] Eric Zelikman, Qian Huang, Gabriel Poesia, Noah Goodman, and Nick Haber. Parsel: Algo- rithmic reasoning with language models by composing decompositions. Advances in Neural Information Processing Systems, 36:31 1466-115222 2023.
[52] Haoyu Zhang, Jingjing Cai, Jianjun Xu, and Ji Wang. Complex question decomposition for semantic parsing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4477-4486, 2019.
[53] Jinghan Zhang, Xiting Wang, Weijieying Ren, Lu Jiang, Dongjie Wang, and Kunpeng Liu. Ratt: Athought structure for coherent and correct Ilmreasoning, arXiv preprint arXiv:2406.02746. 2024.
[54] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's song in the ai ocean: A survey on hallucination in large language models. arXiv preprint 2rXiv:2309.01219 2023.
[55] Hongke Zhao, Songming Zheng, Likang Wu, Bowen Yu, and Jing Wang. Lane: Logic alignment of non-tuning large language models and online recommendation systems for explainable reason generation. arXiv preprint arXiv:2407.0288. 2024.
[56) Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe V Le, et al. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference Learning Representations, 2022.
14

---

A More Details and Prompt Templates of DeAR
A.1 Prompts for Decompose Stage
Table 5: An example of Heuristic-enhanced Prompt ej in the Decomposition Stage.
Example A.1: Prompts for Decompose Stage
Prompt Head h; Your task is decompose the given question Q into sub-questions. You should based on the specific logic of the question to determine the number sub-questions and output them sequentially. If you consider the question to be sufficiently simple and no further decomposition is needed, then output "End." will provide you with three questions similar 9. along with their decomposed sub-questions as examples. You can learn from these examples on how to decompose such questions, and then apply what you't 've learned to decompose Q.
Logic Heuristics thQ:
Example question 1: Will Queen Elizabeth be buried in the Pantheon?
Decomposition: (I): The Panthéon reserved as a mausoleum for citizens of which country? (2): Is Queen Elizabeth from (1)?
Example question 2: Was Elizabeth II the Queen during the Persian Gulf War?
Decomposition: (1): When did Elizabeth II become the Queen? (2): When was the Persian Gulf War? (3): Was Elizabeth II alive in (2)? (4): Is (2) after (1)?
(5): Are the answers to (3) and (4) both yes?
Example question 3: Does Elizabeth II reign over the Balearic Islands?
Decomposition: (1): What all the areas Queen Elizabeth II rules over? (2): What country owns the Balearic Islands? (3): Is (2) included in (1)?
The given question Q: : Does the actress who played Elizabeth II speak fluent Arabic?
Please note that: If Q can be decomposed, you should output multiple sub-questions as shown in the above Logic Heuristies. Otherwise please output "End".
The decomposed sub-questions for Q is:
In the Decompose stage (4. we design heuristic-enhanced prompt to facilitate the question decomposition process. The prompt consists of prompt head hı and "logic heuristics" Uhg. The prompt head 1 describes the question decomposition task in natural language and "logic heuristics" thq are K (K = 3 in this paper) demonstrations of how similar questions are decomposed.
For the we specifically outline the process for obtaining these. We begin by constructing the question decomposition demonstration pool P. For GSM8K and StrategyQA QA, we directly utilize the existing question decomposition annotations from the training sets P. the case of ScienceQA, we generate question decomposition data from portion of its training set using annotations produced by GPT-4, which are subsequently verified through manual checks. Next we employ a BERT encoder F&g to transform target question Q and questions Qi 1, 2.. from decomposition pool P into vector representations 2; and x, respectively. Then we calculate the cosine similarity of the representations between Q and Q and select top-/K nearest neighbors in the vector space:
2T | 1, 1,2,..,,)
La = argTon opk
(10)
where Ta is an index set of the top-K similar questions of Q in the demonstration pool. Fi- nally, We concatenate each Qi E Ta) with its human-annotated sub-questions subgs' (subat, suba,, -.., subgg, to form K question-decomposition examples.
lho = (Q subqs (i = 1,2.
(11)
In this paper, we use the TracenneTransoomerrrr as Eg to transform questions into embeddings, and set K 3.
hhtps//www.sbbet.eed/
15

---

(iblelyyssplays the prompt used in the Decomposition stage for a particular question in StrategyQA dataset.
We also conducted experiments demonstrating that incorporating logic heuristics effectively enhances the overall performance of the Decompose Stage. As shown in the table on the ScienceQA dataset, DeAR w/o logic heuristics indicates the removal of logic heuristics from the prompts used in the Decompose Stage. This adjustment results in lower ACC compared DeAR, highlighting the necessity of constructing heuristic-enhanced prompts for improved performance.
Table 6: Performance comparison of DeAR with and without logic heuristics on the ScienceQA dataset.
GPT-3.5
LLaMA2-7B
ChatGLM3-6B
DeAR w/o logic heuristics DeAR
83.06
69.85 70.57
50.17
83.68
51.08
A.2 Prompts for Analyze Stage
we use the Solve method to prompt the LLM to generate rationales for them. The prompt used for this purpose h2. For use ha to correct the errors in the generated rationales. For Score, we use h4 to prompt the LLM to score the logical coherence rationales. Examples Table demonstatttt h2, ha and h4.
For the generated sub-questions, in the Analyze stage 4.2
we
Table 7: Demonstrations of h2, ha and h4 in the Analyze Stage.
Example A.2: Prompts for Analyze Stage
Solve prompt:
h2: Answer the following question and provide a detailed reasoning process.
Question: How many eggs does Janet have per day?
Your reasoning process:
Self Check prompt:
hg: There might be some errors in the rationale for the following question. If you believe there errors, please correct them and provide the accurate reasoning process. Otherwise, output the original reasoning process.
Question: How many eggs does Janet have per day?
Rationale: Janet has 16 eggs per day.
Your output:
Score prompt:
hy: Please rate the overall correctness and logic of the following rationale on a scale from 1 to 10, where I is the lowest score and 10 is the highest score. Divide the chosen integer by and output it as the final score.
Rationale: Janet has 16 eggs per day.
Your score:
A.3 Prompts for Rethink Stage
In the Rethink Stage we first extract previous sub-questions that are relevant to the newly generated one by using prompt <s, then we use the newly generated rationale to update rationales of these previous sub-questions by using cg. Examples shown in Table g
16

---

Table 8: Demonstrations of ht and hi in the Rethink Stage.
Example A.3: Prompts for Rethink Stage
Extract prompt: h5: Please extract questions from the following list that might the answer of q to update their rationales.
Question list:
1. What is the selling price of one egg?
2. How many eggs does Janet have per day?
q: How many eggs does the ducks lay daily?
The answer of q: The ducks lay 16 eggs per day.
Your extracted questions:
U pdate prompt:
hg: Please update the answer to question b based on the answer to question a.
Question a: How many eggs does Janet eat for breakfast every day?
The answer to question She eats 3 eggs for breakfast. Question b: How many eggs does Janet have per day? The answer question b: She has 16 eggs per day.
The updated answer to question b is:
B Appendix for Experiments
B.1
Datasets and Baselines
B.1.
Datasets
Here, introduce the three datasets used in our experiments in detail. For each dataset, it has publicly released training/validationnness set partitions. Following established practices in previous works, adopt the same partitions fairly compare performance.
ScienceQA [28]
a benchmark for science question answering. which requires machines to reason diverse range of science topics. It is collected from elementary and high school science curricula, and contains 21,208 multiple-choice science questions. Most questions are annotated with grounded lectures (83.9%) and detailed explanations (90.5%). The lecture and explanation provide general external knowledge and specific reasons, respectively, for arriving at the correct answer.
Strategy 12 is a guestion-answering benchmark focusing on open-domain questions where the required reasoning steps are implicit in the question and should be inferred using a strategy. StrategyQA includes 2,780 examples, each consisting of strategy question, its decomposition, and evidence paragraphs. To guide and evaluate the question answering process, each example in Strategy QA was annotated with a decomposition into reasoning steps for answering it, and Wikipedia paragraphs that provide evidence for the answer to each step.
GSMSK [8] is dataset of 8.5K high quality linguistically diverse grade school math word problems created by human problem writers. The dataset is segmented into 7.5K training problems and 1K test problems. These problems take between 2 and 8 steps to solve, and solutions primarily involve performing sequence of elementary calculations using basic arithmetic operations to reach the final answer. bright middle school student should be able to solve every problem. It can be used for multi-step mathematical reasoning.
B.1.2 Baselines
In this subsection, we introduce the baselines used in our experiments, including Few-shot prompting, Chain-of-Thoughts (CoT) prompting, Tree-of-Thought (ToT) and Craph-of-Thoughhe (GoT).
17

---

Few-shot prompting [5]. Few-shot prompting is a paradigm where the language model is provided with a limited number of examples for a specific task, allowing it to generalize and generate the desired output when presented with new instances of the task. This approach leverages a small amount of task specifie information to guide the language model's behavior and enable it to perform effectively on novel examples.
Chain-of-thoughts (CoT) [45] CoT prompts the language model to generate intermediate explanations during the reasoning process preceding the final answer. This deliberate emphasis providing step-by-step rationale enhances the model's capacity to produce more accurate and contextually grounded results. The inclusion reasoning explanations contributes to more robust and insightful generation of answers by guiding the language model through a thoughtful and systematic thinking process.
Trre-of-Thoughhe (ToT) [49]. ToT extends the capabilities of language models by enabling deliberate decision makiing through the exploration of multiple reasoning paths. It incor- porates various search algorithms, allowing the model to traverse diverse routes during the decision-making process. This approach enhances the model's ability consider alternative perspectives and reasoning strategies, contributing to more nuanced and informed outputs.
Trrph-of-Thhuughee (GoT) [.. GoT is an innovative framework that builds upon the advancements introduced by ToT, pushing the boundaries of prompting capabilities in Large Language Models (LLMs). Unlike ToT, GoT represents the information generated by LLM an arbitrary graph, introducing a more flexible and comprehensive structure. Furthermore, GoT incorporates an expanded set of thought transformation operations, allowing for a richer and more diverse modeling of the underlying thought processes within the language model.
Least-to-most [56]. The term least-to-most prompting is borrowed from educational psychology. where used to denote the technique of using a progressive sequence of prompts to help a student to learn a new skill. The key idea in Least-to-mon strategy is to break down a complex problem into a series of simpler subproblems and then solve them in sequence. Solving each subproblem is facilitated by the answers to previously solved subproblems.
SelfCheck [3T]. SelfCheck is zero-shot step-by-step checker for self-identifying errors in LLM reasoning chains. SelsCheck uses the LLM to individually check the conditional correctness of each step in the chain based on the preceding steps, in a manner similar to human going back to check their working. The results of these individual checks are then integrated to form an overall correctness estimation for the whole reasoning chain.
Table 9: Performance comparison of more baseline methods on MATH dataset.
Methods (with GPT-4 backbone)
ACCs on MATH datasets
CoT
56.99
CoT+SC (sample solutions each time)
57.24
ToT
57.18
ToT-variant
57.02
GoT
58.78
DeAR
62.25
B.2 Comparison with More Strong Baselines
In this subsection, we present comparison results with additional strong basclines, including one variant of ToT ("ToT-variant") [27] and CoT with self-consistency ("CoT SC") [4T] We conduct experiments on the more challenging MATH dataset, using GPT-4 as the backbone model, to further demonstrate the effectiveness of our DeAR approach. The MATH dataset is specifically designed to assess the mathematical reasoning and problem-solving abilities of AI models. It consists of 12,500 complex competition-level problems across diverse topics such as algebra, geometry, calculus, number theory, and combinatorics. For our experiments, we use the "gpt-4-01225previin of the GPT-4 model. The results are summarized in Table 9 From the results, we can see that our DeAR method achieves significant improvements over the two newly added baseline methods,
version
18

---

demonstrating the effectiveness of our approach in further enhancing the reasoning capabilities of GPT-4 on more complex reasoning tasks, Moreover, we observe that, based on GPT-4, GoT performs better than both ToT and CoTT methods. Although CoT has the lowest ACC, when combined with self-consistency sampling, its performance surpasses that of ToT.
B.3 Ablation Study of the Self-Check Method in the Analyze Stage
Table 10: Performance comparison of DeAR with and without self-check on the ScienceQA dataset.
GPT-3.5
LLaMA2-7B
ChatGLM3-6B
DeAR w/o self-check DeAR
82.76
69.44
50.35
83.68
70.57
51.08
The construction of the reasoning tree relies on the indispensable interplay of DeAR's three stages: Decompose, Analyze, and Rethink. An ablation study omitting any of these stages would disrupt the entire process. For instance, removing the Decompose stage would prevent both the Analyze and Rethink stages from functioning, as they depend on analyzing and updating the sub-questions generated during decomposition. Similarly, eliminating the Analyze stage would hinder the generation of rationales for each node, rendering the Rethink stage infeasible. Without the Rethink stage. the first two stages would lose their purpose, reducing the entire framework zero-shot approach, where the problem is directly solved at the root node without any further reasoning.
The only reasonable point for ablation is the self-check method within the Analyze stage (Section 4.2 as removing it does not structurally affect Decompose stage and Rethink stage. Therefore, we have included an ablation study the self-check method using the ScienceQA dataset, shown in Table TO The results demonstrate that across various LLM backbones, DeAR consistently outperforms its counterpart without the self-check, confirming the method's effectiveness.
B.4 Human Evaluation
Janet's ducks lay 16 eggs per day. She cats three for brcakfast every moming and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' markett?
Questione
Options
(A) Determine total daily egg production of Jamet's ducks, which is 16. Subtract the eggs consumed for breakfast (3) and those used for baking muffins (4). Multiply the remaining eggs by the selling price of $2 per egg to find the daily income at the farmers' market. The answer $18.
(B) Janet's ducks lay 16 eggs per day, She uses 3 4 <<3+4-77077 eggs per day for breakfast and muffins. She sells 16
7 7:9-7990 eggs at the farmers' market. She carns for each egg sold, so she makes $2
=<<9*2=188a> =8 per day. A nswwen 18.
(C) Let's start by calculating the
number eggs e Janet's ducks lay per day: Janet's ducks lay eggs per day. Next, let's calculate how many eggs she uses every moming for breakfast: She eats eggs for breakfast every moming.
Now, let's deternine how many eggs she uses for baking muffins: She uses eggs bake muffins for her friends
every day. Now, we'll find out the number of eggs that are left to sell the farmers' market: Total eggs laid- (Eggs
ealen for breakfast + Eggs used for baking) Remaining eggs to sell Remaining eggs to sell = 16 +4) Remaining eggs to sell 16 7 Remaining eggs sell 9. Now, we know she sells each egg for $2: Total money made Number of remaining eggs* Price per egg. Total money made =9* $2. Total money made $18. A nswer 18.
Plase sclett the one that has the best logical coherance
Figure 6: The interface of our Human Evaluation method.
we analyze the logical coherence of rationales generated by our framework. We randomly sampled 100 questions from each of the three datasets, and for each question, we used ToT, GoT, and DeAR generate three distinct rationales. Each question, along with the rationales from ToT, GoT, and DeAR, was presented to 10 well-educated annotators (all with at least bachelor's degree). To ensure impartiality, we did not disclose the model that generated each rationale, and the order of the rationales was randomized. The annotators were asked to select the rationale they found most logical, following a majority-vote approach: each question was annotated by all 10 annotators, with the rationale receiving the highest vote count chosen as the final result. Figure 6 shows the template used for annotation. The annotation achieved a Kappa score of 0.70, indicating good agreement among annotators.
In Section 5.4
19

---

Table 11: Comparison of Avg API calls and ACC between DeAR, ToT, and GoT on ScienceQA.
DeAR
ToT (b=2, d=4)
GoT (b=2, d=4)
Avg API calls
9.82
11.35
13.74
ACC
0.837
0.826
0.831
B.5 More Discussion about Efficiency
In Section 5.7 we compared the reasoning time and accuracy DeAR with ToT and GoT across different datasets based on ChatGLL3-66, concluding that DeAR achieves a better trade-off between time and aceuracy. To further validate this conclusion, we measure the average number API calls made by DeAR, ToT, and GoT for each question in ScienceQA based on GPT-3.5, along with their reasoning ACCs, as shown in Table TT It's clear that DeAR makes fewer API calls on average, which implies less time under the same conditions, while simultancously achieving higher ACCs.
B.6 Case Study
The film whose director is younger, Zakhm or Telefono Rosso, is produced by which country?
Question:
These films likely represent different cultural and cinematic contexts, reflecting the diversity of the global film industry.
Step I: "Zakhm" is an Indian film, and was released in 1998, On the other hand, "Telefono" |
"Zakhm,' directed by Mahesh Bhatt, is Bollywood film
Rosso" is an Italian film, and it was released in
known for its poignant
1986 (x)
narrative and emotional depth.
Step 2: Considering the release years,
"Zakhm"" was released earlier 1986, making the film whose directoris younger (x)
The chronological order release indicates that "Telefono Rosso" predates 'Zakhm' by 12 years.
'T Thi chronological difference in release years provides insights into the temporal context of these two films
Step "Zakhm" is an Indian film. Therefore, the answer is India (x)
Figure 7: Case of ToT (GPT-3.5)'s reasoning process.
We conduct case study to demonstrate that our framework's reasoning process aligns more with human logic, effectively reduces logical errors, and is more interpretable. We present one typical intricate reasoning case from StrategyQA with GPT-3.5 backbone in Figure and Figure As shown in Figure 7 ToT generates fixed number of thoughts at each level (3 in this case), and it stops generating at depth of It searches for an optimal path ("Step 1", "Step 2" and "Step 3")) from the tree structure as the reasoning process and discards other thoughts. However, in this example, the logical relationships between the generated thoughts are not clear (e.g.. "Question" and "Stepl") and there are logical errors in intermediate step "Step 1 leading to errors in the subsequent step ("Step 2" and "Step 3*).
In contrast, DeAR establishes a clear logical structure through Decompose stage (black directed arrows). For example, to answer the comparison question "#2": Which of these two directors has smaller age?",, DeAR decomposes it into more -naiind sub-questions "#3": What is the age of Zakhm's 's director?" and "#4": What is the age of Telefono Rosso's director?". Then, with Analyze stage, we obtain the rationales (texts in the green dashed box) for each sub-question, which are then utilized in the Rethink stage to update the existing upper-level rationales (green dashed arrows). For example, the rationale "Nanni Moretti directed Telefono Rosso." for sub-question "#6" is used to update the rationales of "#2", "#1" "#2","#1" and the original "Question". The three-stage cycle iterates until no questions can be further decomposed and we get the tree structure as shown in Figure Finally,
20

---

Answer: Italy (v)
Questtion: The film whose director is younger, Zakhm
or Telefono Rosso, is produced by which country?
Mariish Bhatt diracta 7atam, no yoars aid nato: Kann Monra
directed Talefono Rosso, he is years old now. Nanni Moretti has a Telefono Rosso
directed Telefonn produced by Ital,, so the answer is Italy.
smalles
FBE Which film does the smaller age
correspond to in berms of the director?
Nanmi Murethi has smaller age 71, and be dimettdd Telefonn Rosso
#R Which of these bwo directors
smaller age?
has smaller age
#E W hat is the age of Zakhm's director?
#1 W hat is the age of Telefono Rosso 's director?
Mabesh Bhat was bom now
yours
Nanni Monetl was born in 1953, be is 71 years nnn.
#fi W ho is the director of the film Zakhm?
A W ho is the director of the film Telefono Rosso?
Maheeh Bhatt was boo in 1948, he is 76 years old"
Nanm Monetti dimeted Telefono Resso.
Figure 8: Case of DeAR (GPT3.5)'s reasoning process.
can extract the answer "Italy" (correct in this case) from the rationale of the original question the root node. Our reasoning structure effectively avoids the errors generated by ToT, while being more logical and interpretable.
We further demonstrate the error correction process in the reasoning during the Rethink stage for previously generated nodes, as illustrated in Figure The left part shows the state of the reasoning tree when solving sub-question "#2", At this point, the original question and the answers to "#1" and "#2" are incorrect (marked in red). The right part shows the state after decomposing "#2" into sub- questions "#3" and "#4", solving them, and updating the reasoning tree. The purple text represents the rationales obtained from solving the newly decomposed sub-questions "#3" and "#4", which are subsequently used to update the rationale for "#2", correcting the wrong answer "Mahesh Bhatt has smaller age 70" to "Nanni Moretti has a smaller age 71". This correction also impacts earlier nodes, replacing the original wrong answer with the correct one (with the correct parts shown in blue). Without using the newly obtained rationale update the previous nodes in DeAR, the above errors would not have been corrected. Therefore, the Rethink stage is crucial for DeAR to achieve accurate results in reasoning.
Quandions direkeris younger
Questions
direttr younger or
groduced which country?
Rosso, pompaaee
Mateoaa Zakha, years non:
Matess
be
2on; has
airected Teletono Russo,
discctd Telefono Rosso he years old now.
producee
smalllee age 73 and be dineted Tedetions Ruseu Telecono Roeso
India
ipreduoel by Iaaly
ensoee fuly
ØE W Watch film dons the staalie age comeepond to
ABL W Wich film does the smalie age comrspood to inl borns of the dinettr?
turns the dinctor?
Maheah Slor: las smaller age 70, and be directed Zakim.
Noria Mocelll lis smaille 71, and direc Leletooo Horgo
6G Which of thow ben dreetors los smalll ogo?
AL Which of theee twn dine ton has a small ot ago! Niceeta
Makeas Blue has smaller
At What is the ogent Zakams dincaor Matzel Uizil was 1913, 35
What's the age of Telefioo Hospo dimetoc) Tomnt More Wis bori TH6X Tail piars old now
Figure 9: Case of DeAR (GPT3.5)'s emrrr correction process in the Rethink stage.
21

---

C
Broader Impacts
This work endeavors advance the field of natural language processing through the introduction of novel DeAR (Decompose-Analyz) (nalyze-Rethink) reasoning framework, leveraging LLMs for enhanced reasoning capabilities. The potential broader impact of our research lies in implications for natural language understanding and reasoning systems. By dynamically generating and updating rationales, our framework contributes to the development of more effective and interpretable language models.
The societal consequences of our work include the potential improvement in the interpretability and reliability of machine-generated reasoning, which can have positive implications across various domains, such as education, decision support systems, and natural language processing applications. However, is essential approach these advancements with a critical lens, considering the ethical implications and societal impact of widespread deployment.
D Limitations and Future Work
While the DeAR framework can significantly enhance reasoning capabilities for large language models, several limitations merit attention. First, shown in Figure although DeAR achieves a better trade-off between time and accuracy, the reasoning time for complex problems remains relatively long. This is due to the significant overhead associated with the iterative cycles of the Decompose, Analyze, and Rethink stages. Therefore, there still room for improvement enhancing reasoning efficiency for practical applicatiioss in the future. Second, in the Decompose Stage, while logic heuristics contribute to overall performance improvement (as illustrated in Table constructing these heuristics requires additional annotation. For the datasets used in this paper, most of the training data already includes annotated question decompositions, saving considerable time in preparation. However, extending DeAR to other datasets may necessitate the development of more efficient methods to reduce or eliminate the need for the annotation. Third, while DeAR outperforms existing methods like ToT and GoT in flexibility and error reduction, its real-world applicability requires further validation across broader range of datasets to fully assess the framework's 's versatility and robustness, well as explore the potential of more LLMs for complex reasoning. In the future, may consider exploring methods to enhance LLMs' reasoning capabilities on more complex tasks, such as those in STEM fields and programming. Recent studies, including OpenAl's ol model. utilize large-scale reinforcement learning algorithms to teach the model how to think productively through its chain of thought. Based on these advancements, a promising approach would to investigate how to integrate similar methods with the model's 's self-thinking capabilities to enhance adaptive learning and facilitate continuous improvement in reasoning.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification: We made claims about contributions and scope in the Abstract and the fifth paragraph of
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Due to space limit, present the Limitations section in Appendix D
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We present implementation details and prompting exemplars in Section 5.1.2 5.12 and Appendix [A] We also include our code and describe the setups needed to reproduce the experimental results at httph::t///ttt http://ggtttt.... com/ i Xue/DeAR
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
24

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Our dataset and code is available at httpp:///gtth/: httpg/ttuuu..cmmSSaaaiiiiitu
DeAR
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The implementation details are described Section 5.1.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We analyze the significance of our experimental results in Section 5.2 with paired (-test. This paper also evaluates the quality of DeAR's R's rationales by human annotators. In order ensure the consistency of annotators, calculate the Kappa score and the result is 0.70, which ensures the credibility of our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We present time of execution information. We conduct efficiency experiments about the inference time in Section 5.7 and Appendix B.5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps:/neurips.co http:///wwppp..cccppbb....ttttaa
Answer: [Yes]
Justification: This paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We present a "Broader Impact" section in Appendix C to discuss the potential positive/negative impacts of our work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
26

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The assets including data and bascline models used in this paper are properly cited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We include the full template given the human annotators Appendix B.4
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: In this paper, we invite human annotators only to assess LLMs' outputs mi.e..maee choices). The annotators themselves are not the subjects of the evaluation and are not being tested. Besides, as shown our annotation template in Appendix B.4, this evaluation process does not collect personal information or privacy of the annotators, and the annotators are fully aware of the purpose of the evaluation and have consented to its use.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
28

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "TPkxe1XuO2", "reviewer": "Reviewer_dFYv", "summary": "The paper presents a novel reasoning framework DeAR (Decompose-Analyze-Rethink), which aims to advance the capabilities of large language models (LLMs) in handling complex reasoning tasks. DeAR introduces a Decompose-Analyze-Rethink cycle that involves breaking down intricate problems into simpler sub-questions, analyzing these to form rationales, and revisiting prior answers to refine the reasoning process. Different from the rigid structures of existing methods like Tree-of-Thoughts (ToT) and Graph-of-Thoughts (GoT), this approach allows each branch to be independently generated without preset configurations, thereby enhancing logical coherence. Extensive experimentation on several benchmarks are conducted to demonstrate the effectiveness of the framework.", "questions": "1. Can the authors present more case studies where the DeAR framework has been or could be effectively applied, and how the problem-solving process might benefit from the enhanced reasoning capabilities?\n\n2. Does the method work with stronger LLMs? The paper should present the framework performance with GPT4 as its backbone to validate its effectiveness. If the computation cost is too high, the authors can consider running experiments on subsets of the original datasets.", "limitations": "The authors have discussed the limitations of the proposed DeAR framework, including potential computational overhead, variability in reasoning quality, and the need for broader real-world testing.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The DeAR framework introduces a novel reasoning cycle that mimics human cognitive reasoning, offering a fresh perspective on how LLMs can tackle complex problems.\n\n2.By decomposing problems into sub-questions and rethinking rationales, DeAR ensures greater logical consistency compared to traditional methods like ToT and GoT. \n\n3.Experimental results show that DeAR achieves significant improvements over state-of-the-art methods, particularly in reducing logical errors and enhancing the reasoning process with different LLMs.\n\n4.By constructing a reasoning tree through a three-stage framework, DeAR provides a clear and interpretable reasoning process, which aids in understanding the decision-making of LLMs.", "weaknesses": "1. While the framework is designed to enhance reasoning accuracy and flexibility, the iterative nature of the cycle may lead to increased computational demands, particularly when dealing with highly complex problems. The authors can further discuss this point.\n\n2. The paper should include experimental comparisons between the DeAR framework and more strong baselines, such as other variants of ToT and CoT+SC approach, an enhanced Chain-of-Thoughts approach that incorporates self-consistency checks [1][2]. \n[1] Long J. Large language model guided tree-of-thought[J]. arXiv preprint arXiv:2305.08291, 2023.\n[2] Mo S, Xin M. Tree of uncertain thoughts reasoning for large language models[C]//ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024: 12742-12746.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your affirmation of the motivation of our paper, the significance of our experimental results and the novelty of DeAR.\n\n**Q1**: The iterative nature of the cycle may lead to increased computational demands.\n\n**A1**: Thank you for your comments. In section 5.7, we compare the efficiency of our framework with different variants of ToT/GoT with ChatGLM3-6B backbone. As shown in Figure 5, compared to state-of-the-art ToT/GoT methods, the point corresponding to our method achieve better ACC with less time.\n\n**Q2**: Include comparisons with more strong baselines (one variant of ToT**[1]** and CoT+SC**[2]**).\n\n**A2**: Thank you for your comments.  We conducted further comparisons with one variant of ToT**[1]** and CoT+SC**[2]** based on GPT-4 backbone, on a more challenging dataset MATH, to address your inquiry. The results are presented in the table below.\n\n| Methods (with GPT-4 backbone) | ACCs on MATH | \n|  ----  | ---- |\n|CoT     | 56.99  |\n|CoT+SC **[1]** (sample 5 solutions each time)  | 57.24  |\n|ToT     | 57.18   | \n|ToT-variant **[2]** | 57.02  |\n|GoT     | 58.78  |      \n|DeAR    | **62.25**  | \n\n**Q3**: More case studies.\n\n**A3**: Here, we use the case in Figure 9, Appendix C.4, to further explain how the reasoning process benefit from DeAR’s decomposition, analyze and rethink. First, to answer the comparison question “#2”: Which of these two directors has a smaller age?”, our framework decomposes it into sub-questions “#3”: What is the age of Zakhm’s director?” and “#4”: What is the age of Telefono Rosso’s director?”, which are easier for LLMs to solve. Second, in the Analyze stage, DeAR obtains the answers of sub-question #3 and #4, and also corrects the wrong answer “Mahesh Bhatt was born in 1954, he is 70 years old now” to the right one “Mahesh Bhatt was born in 1948, he is 76 years old now”. After that, the corrected answer to #3 is used to update the answer of #2, and corrects #2’s answer “Mahesh Bhatt” to “Nanni Moretti”. Through the above process, DeAR is able to help correct wrong reasoning steps and avoid error propagation, which is crucial in enhancing model’s reasoning ability. We will include more detailed cases in the revised version.\n\n**Q4**: Does the method work with stronger LLMs? \n\n**A4**: Thank you for your insightful comments and valuable feedback. In response to your interest, we have conducted further experiments using the GPT-4 backbone to robustly illustrate the effectiveness of our DeAR framework. As indicated in the response to \"W2,\" we present these results to demonstrate the superiority of our approach. On the MATH dataset, a comprehensive benchmark that challenges models with a variety of mathematical reasoning tasks, DeAR has demonstrated superior performance compared to different SOTA methods, including CoT, CoT-SC, ToT, a variant of ToT, and GoT. \n\n**[1]** Self-consistency improves chain of thought reasoning in language models.\n\n**[2]** Large language model guided tree-of-thought.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hkWetrRu7c", "reviewer": "Reviewer_vfU8", "summary": "The paper presents **DeAR**, a new reasoning framework for large language models to perform intricate reasoning tasks. Inspired by human cognition, it decomposes problems into sub-questions within a Reasoning Tree, refining solutions through iterative Decompose-Analyze-Rethink cycles. Compared to existing state-of-the-art approaches like **ToT** and **GoT**, **DeAR** offers more flexibility and continuous rationale refinement, leading to reduced logical errors and improved performance across various reasoning benchmarks.", "questions": "- How large is the (human-annotated question decomposition) demonstration pool for each of the datasets?\n- I notice that the authors employ a cosine similarity-based strategy to pick appropriate demonstrations when constructing prompts at the ***Decompose*** stage, is the same strategy used to test the performance of the baseline prompting methods?\n- How to assign the proper values for $\\epsilon\\_1$ and $\\epsilon\\_2$ if there is no validation set available?", "limitations": "Yes, the authors have covered several limitations in Appendix 4.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- A novel reasoning framework implemented with a **Decompose-Analyze-Rethink** (**DeAR**) cycle has been proposed to enhance the capabilities of LLMs in solving intricate problems.\n- The proposed framework is capable of generating rationales with better logical consistency while achieving better accuracy in less time per question.\n- Extensive experiments on three complex reasoning benchmarks demonstrate the superiority of **DeAR** over state-of-the-art approaches (e.g., **ToT**, **GoT**), showcasing its ability to improve performance for intricate reasoning with different LLMs.", "weaknesses": "- Lack of ablation studies to analyze the contribution of each individual step, i.e., ***Decompose***, ***Self-Check*** and ***Rethink***.\n- A small number of participants in human evaluations leads to statistically unreliable conclusions. I notice that different prompting methods elicit the LLM to produce responses of different lengths, which is also a confounding factor that can affect the choice, as humans prefer more concise responses.\n- The values for threshold hyperparameters, i.e., $\\epsilon\\_1$ and $\\epsilon\\_2$ should be carefully set.\n- Typo. In Line 322, *Table 3* should be *Tabel 4*.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive comments on the novelty and efficiency of our DeAR and the affirmation of its superior performance over SOTA methods.\n\n**Q1**: Lack of ablation studies to analyze the contribution of each individual step.\n\n**A1**: Thanks for your insightful comments. The construction process of our proposed reasoning tree is such that the three stages—decompose, analyze, and rethink—are indispensable. If we conduct a ablation study that omits one of these stages, for example, removing the decompose stage, then both the analyze stage and the rethink stage would be unable to proceed, as the latter two stages must analyze and update the sub-problems generated from the decomposition. Similarly, eliminating the analyze stage would result in the inability to obtain the rationales for each node, thereby preventing the rethink stage from taking place. Removing the rethink stage would also render the first two stages pointless; the entire framework would then devolve into using a zero-shot approach to directly solve the problem at the root node and obtain the result.\n\nHere, the only part where an ablation study can be reasonably conducted is the self-check method within the analyze stage, as removing self-check will not structurally affect the other two stages. Therefore, we have added an ablation study on the self-check method using the ScienceQA dataset, as shown in the table below. It can be observed that, based on different LLM backbones, DeAR consistently performs better than DeAR without self-check, which also proves the effectiveness of the self-check method. We will include this experiment in the updated version.\n\n|     |     | ScienceQA|      |   \n|  ----  | ---- | ---- |  ----  | \n|     | GPT3.5|LLaMA2-7B|ChatGLM3-6B|\n|**DeAR w/o self-check**| 82.76 | 69.44  | 50.35 | \n|**DeAR**| 83.68  | 70.57  | 51.08  | \n\n**Q2**: Human evaluations.\n\n**A2**: Thank you for your comments. We have adopted some approaches to help minimize the bias of annotators towards rationales of varying lengths. For example, we provided each annotator with detailed annotation instructions, allowing them to select the most logical response from the answers given by different models, as shown in Figure 6, Appendix C.2. At the same time, we performed multiple random samplings from the dataset, each time with a different set of annotators, to further prevent the unreliability of results due to the subjective factors of individual annotators. We have five annotators, which is a similar number to that used in other studies employing human evaluation methods, such as in [1]. We will include more details about the sampling and annotation process in the updated version.\n\n[1] Guiding Mathematical Reasoning via Mastering Commonsense Formula Knowledge\n\n\n**Q3**: The values for threshold hyperparameters.\n\n**A3**: Thank you for your insightful comments. We set the thresholds by conducting the threshold combination experiment in Section 5.6. We selected the threshold combination that yields the highest reasoning accuracy for our configuration.\n\n**Q4**: Typo about table’s name: Table 3 should be Tabel 4.\n\n**A4**: Thank you for the reminder, we will correct it in the updated version.\n\n**Q5**: How large is the demonstration pool?\n\n**A5**: For the ScienceQA dataset, we randomly selected some questions from each topic in the training set and annotated 500 examples as a demonstration pool. For GSM8K and StrategyQA, since their training sets already have annotations for problem decomposition, we directly chose 500 items from them as the demonstration pool. We will include this in the updated version.\n\n**Q6**: I notice that the authors employ a cosine similarity-based strategy to pick appropriate demonstrations when constructing prompts at the Decompose stage, is the same strategy used to test the performance of the baseline prompting methods?\n\n**A6**: Thank you for your question.  Selecting demonstrations for problem decomposition prompts will only be effective for methods that include a problem decomposition step. Among the baselines in this experiment, only the least-to-most method includes a problem decomposition step. Therefore, for least-to-most, we use the same demonstration pool and cosine similarity selection method as DeAR. As for other baselines, such as CoT, ToT, GoT, since they do not include a problem decomposition step, naturally we do not select decomposition demonstrations for prompting.\n\n**Q7**: How to assign the proper values if there is no validation set available?\n\n**A7**: Thank you for your question. A portion of the data from the training set can be selected as a validation set to verify the effects of different threshold combinations, and the optimal combination can be chosen for testing on the test set.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MiheSrGBBw", "reviewer": "Reviewer_WGVX", "summary": "This paper proposes a recursive method for LLMs to solve complex reasoning tasks. The approach formulates problem-solving as a hierarchical tree structure, where each problem is broken down into a tree of sub-problems. Each sub-problem is then analyzed and updated. This method has been evaluated on datasets such as ScienceQA, StrategyQA, and GSM8K, demonstrating improved accuracy on LLMs like Llama-2, GPT-3.5, and ChatGLM3.", "questions": "Q1. Is the method effective for more complex tasks? In the context of math reasoning, the community might be more interested in results on MATH or MathQA, as opposed to GSM8K, which is relatively simple for models like GPT-3.5.\n\nQ2. The paper provides an efficiency analysis based on ChatGLM3. Could the authors provide a more detailed analysis based on GPT-3.5? For example, could they present a comparison of the number of API calls and the number of input tokens compared to ToT and GoT? This is beneficial to verify the method's efficiency on API-based LLMs.", "limitations": "The authors addressed the limitations in Appendix D: 1. The self-check method may add more computational complexity. 2. The autonomy in generating branches might result in inconsistency in the reasoning quality. 3. A broader range of datasets should be considered to validate its real-world applicability.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "S1. The concept of the proposed framework is sound, and the cycle algorithm is shown with clear examples. The main idea, which mimics human reasoning, is easy to understand and is presented in a straightforward way.\n\nS2. The performance improvements over SOTA methods like ToT and GoT are significant. The experiments on different LLMs (GPT3.5, Llama2, ChatGLM3) also demonstrate the method’s versatility.\n\nS3. The structure of the framework is more flexible and reasonable compared to CoT, ToT and GoT. The method can generate the reasoning path based on the specific logic of the problems and timely correct errors.", "weaknesses": "W1. The effectiveness of the “self-check” method in “Analyze Stage” may need further validation. The paper (Jie Huang et al, \"Large Language Models Cannot Self-Correct Reasoning Yet\") show that LLMs cannot correct themselves. \n\nW2. Could the authors provide a more detailed explanation of each step of the algorithm's execution, including the input and output results, in the case study? For instance, in the example in Figure 9, only the final reasoning process of each node is shown. It would be better if the authors could explain how the contents of these nodes are updated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your acknowledgement of on our study motivation, model design, experimental results, and presentation. Your suggestions are insightful for us.\n\n**Q1**:  The effectiveness of the “self-check” method in “Analyze Stage” may need further validation. \n\n**A1**: Thank you for your insightful comments. Here, we add an ablation study focusing on the self-check mechanism, utilizing the ScienceQA dataset for our analysis, as illustrated in the table below. The results demonstrate that across three LLM backbones, the DeAR model outperforms its counterpart without the self-check method, thereby validating the self-check method's efficacy. Although other methods that employ LLMs for self-correction may not be sufficiently effective, our experiments have demonstrated that incorporating a self-check method during the Analyze stage is very necessary. We intend to incorporate these findings into the updated version of our work.\n\n|     |     | ScienceQA|      |   \n|  ----  | ---- | ---- |  ----  | \n|     | GPT3.5|LLaMA2-7B|ChatGLM3-6B|\n|**DeAR w/o self-check**| 82.76 | 69.44  | 50.35 | \n|**DeAR**| 83.68  | 70.57  | 51.08  | \n\n**Q2**: More detailed explanation of case studies.\n\n**A2**: Yes. We'll break down how DeAR enhances the reasoning process with a real example from Figure 9 in Appendix C.4. Imagine we need to solve a comparison question, \"#2\": \"Who is younger between these two directors?\" In the **Decompose** stage, DeAR breaks \"#2\" into simpler sub-questions, \"#3\" and \"#4\", asking for the ages of the directors of \"Zakhm\" and \"Telefono Rosso,\" which are more manageable for Large Language Models (LLMs) to figure out.\nOnce we have the answers to these sub-questions, DeAR moves on to the **Analyze** stage. Here, it not only gets the specifics but also spots and fixes a mistake: it corrects the age of Mahesh Bhatt from 70, born in 1954, to the accurate 76 years old, born in 1948. With the correct information in hand, in **Rethink stage**, DeAR then revisits the original question and makes the necessary update, correcting the initial guess of \"Mahesh Bhatt\" to the right answer, \"Nanni Moretti.\"\nThis step-by-step approach allows DeAR to catch and correct any faulty reasoning along the way, stopping errors from spreading. In the next version of our work, we'll add more such detailed examples to paint a clearer picture.\n\n**Q3**: Is the method effective for more complex tasks? \n\n**A3**: Thank you for your comments. We conduct further experiments based on GPT-4, particularly on the more challenging MATH dataset, to address your inquiry. The results are presented in the table below. For more complicated questions in MATH, DeAR also performs better.\n\n| Methods (with GPT-4 backbone) | ACCs on MATH | \n|  ----  | ---- |\n|CoT     | 56.99  |\n|CoT+SC **[1]** (sample 5 solutions each time)  | 57.24  |\n|ToT     | 57.18   | \n|ToT-variant **[2]** | 57.02  |\n|GoT     | 58.78  |      \n|DeAR    | **62.25**  | \n\n**[1]** Self-consistency improves chain of thought reasoning in language models.\n\n**[2]** Large language model guided tree-of-thought. \n\n**Q4**: Could the authors provide a more detailed efficiency analysis based on GPT-3.5? \n\n**A4**: Yes. On the ScienceQA dataset, using GPT3.5 as backbone, to ensure a fair comparison, we compare DeAR (with parameters b=1.58 and d=3.62) with ToT, which has the closest values for branch b and depth d (b=3, d=4). We've looked at the average number of API calls for each question, and ACCs on test set, as shown in the table below. It's clear that our method makes fewer API calls on average, which means less time under the same conditions, and achieves higher ACCs at the same time. We'll add more detailed experiments about the average input tokens in the updated version.\n\n|  | DeAR|  ToT (b=2, d=4) | GoT (b=2, d=4) |\n|  ----  | ---- | ---- | ---- |\n|**Avg API calls**  | **9.82** | 11.35 | 13.74\n|**ACC**  | **0.837** | 0.826 | 0.831", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uKTpLBYk7K", "reviewer": "Reviewer_Zsa1", "summary": "The paper proposes DeAR-prompting (decompose, analyze, rethink) as a new prompting paradigm. The framework basically consists of decomposing the original question into subquestions, answering the subquestions and analyzing the answers and potentially rethinking answers to earlier questions based on the new answers to correct mistakes. The approach is evaluated experimentally and shows significant improvement over ToT and GoT prompting on three benchmarks.", "questions": "I was surprised by the first question example in B1. Was there a rationale for adding so much information in the decomposition example that goes beyond the original question? It's hard to evaluate how often this really happens experimentally, but did you look into some examples to see how far the subquestions go beyond the original question?", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I find the idea very intuitive, well motivated and mostly well presented. The experiments indicate that DeAR improves the state-of-the-art and this seems intuitively plausible.", "weaknesses": "From my perspective, there are some weaknesses, but most of them I wouldn't consider as serious issues, but rather as starting points for future work.\n- one problem is decomposing the question. This step can probably itself be improved by different prompting strategies. In B1, I was indeed surprised by the first decomposition example. It seems to me that there is nothing that suggests that the Pantheon is a Mausoleum or that it is reserved for citicens of a particular country. It seems to me that the example is much more than just a decomposition as it involves a lot of Background Knowledge that may or may not be available. The second and third example seem much more natural.\n- another problem is evaluating the answers. It seems very naive to ask the LLM for a score. While some People claim that LLMs can give meaningful quantitative evaluations, there is also a lot of evidence to the contrary. Given the nature of LLMs, it seems to me that there is a good chance that the LLM will just return a score that frequently occured in \"similar contexts\" during training and is not particularly meaningful. The idea of applying voting methods for this step sounds more convincing to me.\n- the paper currently argues that particular stages are important because  they do not exist in other frameworks (e.g., \"the improvements over ToT highlight the advantage of Decompose stage\"). It would be more convincing to do an ablation study.\n- there are quite a few typos and grammatical problems in the paper. It would be good to apply a spell checker. Just two examples:\n ** line 114: an novel -> a novel\n ** line 257: Graph-of-Thoughtss -> Graph-of-Thoughts?\n- finally, a minor philosophical point that will not affect my evaluation: personally, I am not a fan of the whole ANN-vs-human discussion. A neuron in an ANN is just a numerical parameter, a human neuron is a biological cell, which can itself be seen as a primitive life form with its own metabolism. Perhaps something intelligent will evolve from ANNs, but comparing them to biological NNs seems rather far-fetched to me. I appreciate that the paper does not really go into that direction, but does the whole human cognitive reasoning discussion really add anything to the paper? I agree that the proposed approach is more natural than other prompting approaches, but does it really resemble what humans do? Decomposition is certainly a part of what humans do, but do we really have to go back and revise our previous answers because we hallucinated a random answer at some point? This does not really seem to be a reasoning problem in general, but an artifact of the probabilistic-generative nature of LLMs. Of course, it's important to deal with this in LLMs, but do we really need to sell this as human-like?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your affirmation of the motivation of our idea, and the implementation of our approach DeAR. As for your concerns:\n\n**Q1**: Question decomposition examples.\n\n**A1**: Thanks for your insightful comments. In our approach, the logic heuristics provided in the problem decomposition prompt vary dynamically depending on the question. The logic heuristics presented in Table 5 are specific to a particular case of a question. For a given question, we select problem decomposition examples from the demonstration pool based on the calculated cosine similarity. Different examples may be selected as part of the prompt for different questions (see Appendix B.1 for a detailed description). This method can effectively adjust the prompt according to the differences in questions, allowing for better decomposition tailored to the characteristics of each problem compared to a fixed prompt.\n\n**Q2**: Using LLMs for answer scoring.\n\n**A2**: Thanks for your valuable comments. In section 4.2, line 217, we mentioned that scoring for the answer could also be achieved using other voting or classification methods. In the ToT method, a comparison was made between \"value\" and \"vote\" scoring approaches, demonstrating that both are effective for verifying the accuracy of answers[1]. In our paper, for simplicity, we explored the method of directly generating scores using the backbone LLMs , and we will supplement the results obtained using voting in the future.\n\n[1] Tree of thoughts: Deliberate problem solving with large language models.\n\n**Q3**: An ablation study about Decompose Stage.\n\n**A3**: Thanks for your insightful comments. Given that each stage in our method is essential for constructing the reasoning tree, it is challenging to perform an ablation study by simply removing one stage. If we were to conduct an ablation study that eliminates the decompose stage, then the subsequent analyze and rethink stages would be hindered, because these stages rely on analyzing and updating the sub-questions that result from the decomposition process. A possible way to validate the effectiveness of the decompose stage is to replace its prompt with prompts from other methods, such as those used for problem decomposition in the Least-to-Most approach. In the table below, we have included supplementary comparative experiments on ScienceQA that demonstrate the superiority of our designed decompose stage. As shown in the table, the performance will decline after we replace our prompt with Least-to-Most decomposition prompt, indicating the effectiveness of our method. We will consider designing additional experiments to further validate the effectiveness of different stages.\n\n|  | DeAR+GPT3.5| DeAR+GPT3.5 (Least-to-Most decomposition prompt)|\n|  ----  | ---- | ---- |\n|Accs on ScienceQA |83.68 | 81.33 |\n\n**Q4**: Typos and grammatical problems.\n\n**A4**: Thanks for your suggestions, we will correct these errors in the revised version.\n\n**Q5**: Human cognitive reasoning discussion.\n\n**A5**: Thank you for your insightful comments and for raising a philosophical consideration regarding the comparison between ANNs and human cognition. Regarding the human cognitive reasoning discussion in our paper, we included it with the intention of drawing parallels to human problem-solving strategies, like the decompose, analyze and rethink stages, which can provide intuitive understanding and potentially guide the development of more natural and effective AI systems. \n\n**Q6**: The first question example in B1.\n\n**A6**: Thank you for your comments. For each question, we employ cosine similarity to select the most semantically similar questions from the demonstration pool to construct decomposition examples. For instance, regarding the original question \"Does the actress who played Elizabeth II speak fluent Arabic?\", the questions chosen from the demonstration pool are \"Will Queen Elizabeth be buried in the Pantheon?\", \"Was Elizabeth II the Queen during the Persian Gulf War?\", and \"Does Elizabeth II reign over the Balearic Islands?\". These selected questions and their decomposition examples might contain additional information. However, compared to direct prompting methods (such as using the least-to-most decomposition prompt for prompting), this method is more effective, as we have also demonstrated in our response table for Q3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9LW7tGXNoA", "reviewer": "Reviewer_ANC9", "summary": "The paper introduces a reasoning framework called Decompose-Analyze-Rethink (DeAR) for enhancing the reasoning capabilities of large language models (LLMs). DeAR mimics human cognitive reasoning by decomposing complex problems into simpler sub-problems using a Reasoning Tree structure, analyzing these sub-problems independently, and rethinking the answers in light of new insights from sub-problem solutions. This iterative cycle allows for dynamic adjustments and error corrections in the reasoning process. The proposed framework shows promising results across multiple reasoning datasets.", "questions": "Please address the concerns outlined in Weaknesses.", "limitations": "No negative societal impact", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1) The paper makes solid progress on improving how LLMs solve complex problems, an important area in AI research.\n2) The paper is well-organzied and the proposed method is well-explained. The entire idea is reasonable and well-aligned with human thinking. \n3) The experiments are sound. The proposed method is thoroughly tested on different types of complex problems and the performance improvement over SOTA methods (e.g., CoT, ToT and GoT) is significant.", "weaknesses": "1) The process for obtaining decomposition demonstrations in the logic heuristics lacks a detailed explanation.\n2) As mentioned in Section 2.2, there are also other works that explore problem decomposition in LLMs. Lacking further discussion or evaluations on how their work differs from existing paradigms of problem decomposition may limit the technical contribution of the paper.\n3) The experimental design could be enhanced by providing a more detailed analysis. For example, the effectiveness of the “self-check” mechanism is not well evaluated. The authors may consider showing the error rates of the generated rationales and demonstrate how the “self-check” stage contributes to reducing these errors.\n4) The paper could benefit from improvements in presentation. For example: a) Table 4 is incorrectly referenced within the text, b) a typo in Algorithm 1 (stgt;).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your affirmation of the contribution of our paper. For your concerns:\n\n**Q1**: The process for obtaining decomposition demonstrations in the logic heuristics lacks a detailed explanation.\n\n**A1**: Thank you for your comments. We have provided a detailed explanation for the obtaining of question decomposition demonstrations and examples of prompts for the Decomposition Stage in **Appendix B.1**. We use a BERT encoder to transform target question Q and human-annotated question decomposition demonstrations into vector representations. Then we use cosine similarity to select the top K (K=3 in our setting) similar demonstrations as logic heuristics.\n\n**Q2**:  Lacking further discussion or evaluations on question decomposition.\n\n**A2**: Thank you for your insightful comments. Other methods that use question decomposition to solve complex problems typically decompose the original question into sub-questions through a simple prompting method and then solve them step by step, such as the least-to-most approach. In contrast, the DeAR framework we propose not only uses logic heuristics during the **Decompose** stage to enhance the logic of the question decomposition but also provides more refined planning and updating of the problem-solving process during the **Analyze** and **Rethink** stages. This further ensures the reliability of the problem-solving process, preventing the spread of errors. Additionally, the reasoning tree generated by DeAR makes the reasoning process more interpretable.\n\n**Q3**:  The effectiveness of the “self-check” mechanism is not well evaluated.\n\n**A3**: Thank you for your insightful comments. Here, we verify the effectiveness of the self-check method by comparing the predict accuracy of DeAR and DeAR(w/o self-check) on the ScienceQA dataset. DeAR(w/o self-check) refers to the version where the self-check part is removed during the Analyze stage, while the rest of the implementation process is identical. The experimental results are shown in the following table, from which it can be seen that under different backbones, DeAR has higher predict accuracy, demonstrating the necessity of the self-check method for correcting errors. \n\n|     |     | ScienceQA|      |   \n|  ----  | ---- | ---- |  ----  | \n|     | GPT3.5|LLaMA2-7B|ChatGLM3-6B|\n|**DeAR w/o self-check**| 82.76 | 69.44  | 50.35 | \n|**DeAR**| 83.68  | 70.57  | 51.08  | \n\n**Q4**: The paper could benefit from improvements in presentation.\n\n**A4**: Thank you very much for your suggestions. We will carefully review and correct any writing errors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers’ efforts in reviewing our paper. We would like to thank all of them for providing constructive and valuable feedback, which we will leverage to improve this work. We are encouraged by the positive comments from reviewers, including: 
- **Motivation**: “offering a fresh perspective on how LLMs can tackle complex problems.” (Reviewer dFYv), “A novel reasoning framework” (Reviewer vfU8), “I find the idea very intuitive, well motivated and mostly well presented” (Reviewer Zsa1)

- **Method**: “novel” (Reviewer dFYv, Reviewer vfU8), “DeAR ensures greater logical consistency compared to traditional methods” (Reviewer dFYv), “The concept of the proposed framework is sound” (Reviewer WGVX), “The entire idea is reasonable and well-aligned with human thinking.” (ANC9), “The structure of the framework is more flexible and reasonable compared to CoT, ToT and GoT” (Reviewer WGVX)

- **Experimental Results**: “ DeAR achieves significant improvements” (Reviewer dFYv), “ the superiority of DeAR over state-of-the-art approaches” (Reviewer  vfU8), “The experiments also demonstrate the method’s versatility” (Reviewer WGVX), “DeAR improves the state-of-the-art and this seems intuitively plausible” (Reviewer Zsa1), “The experiments are sounds” (Reviewer ANC9).

We will specify the detailed responses to all reviewers as follows.

Meta-review: In order to solve intricate reasoning problems, this study introduces the paradigm of  Decompose-Analyze-Rethink (DeAr) cycles for Large Language Models. The key idea is to iteratively build a reasoning tree in a top-down way by breaking the question into sub-question nodes, analyzing them to form rationales, and revisiting answers in parent nodes to refine the reasoning process. This iterative decomposition approach is evaluated experimentally and shows significant improvement over ToT and GoT on several benchmarks.

All reviewers concur that this study makes a significant contribution to solving complex reasoning problems with LLMs. The paper is well-organized, the framework is well-explained, and the comparative experiments are conclusive. In summary, this is a strong piece of work.

While the paper is in good shape, additional explanations about the cycle components could be helpful (Reviewers #ANC9 and #WGVX). It would also be beneficial to incorporate some of the answers to reviewers’ questions into the paper (or the Appendix). Notably, the additional experiments with stronger baselines (Reviewer #dFYv) or more complex tasks (Reviewer #WGVX) would clearly demonstrate the effectiveness of DeAR. In addition, insights from ablation studies (Reviewers #vfU8 and #Zsa1) and the importance of the self-check method (Reviewers #WGVX and #ANC9) could be incorporated.
Decision: Accept (oral)