SEER: Facilitating Structured Reasoning and Explanation via Reinforcement Learning (2024)

Abstract: Elucidating the reasoning process with structured explanations from question to answer is crucial, as it significantly enhances the interpretability, traceability, and trustworthiness of question-answering (QA) systems. However, structured explanations demand models to perform intricately structured reasoning, which poses great challenges. Most existing methods focus on single-step reasoning through supervised learning, ignoring logical dependencies between steps. Moreover, existing reinforcement learning (RL) based methods overlook the structured relationships, underutilizing the potential of RL in structured reasoning. In this paper, we propose SEER, a novel method that maximizes a structure-based return to facilitate structured reasoning and explanation. Our proposed structure-based return precisely describes the hierarchical and branching structure inherent in structured reasoning, effectively capturing the intricate relationships between different reasoning steps. In addition, we introduce a fine-grained reward function to meticulously delineate diverse reasoning steps. Extensive experiments show that SEER significantly outperforms state-of-theart methods, achieving an absolute improvement of 6.9% over RL-based methods on EntailmentBank, a 4.4% average improvement on STREET benchmark, and exhibiting outstanding efficiency and cross-dataset generalization performance. Our code is available at https://github.com/Chen-GX/SEER.

heading: 1 Introduction
text: Navigating machines to understand and articulate the thought process from posing a question to arriving at an answer has been a long-term pursuit in the AI community (McCarthy, 1959; Yu et al., 2023). Current QA explainable systems adeptly furnish brief supporting evidence (Rajani et al., 2019; DeYoung et al., 2020). However, they often fail to clarify the reasoning process from prior knowledge
to the derived answer. By elucidating the reasoning process of answers generation from the language models, we can greatly improve interpretability, trustworthiness, and debuggability (Dalvi et al., 2021; Ribeiro et al., 2023). As illustrated in Figure 1, when generating answers for the question "Which natural material is best for making a table?", the reasoning process with structured explanations, such as entailment trees (Dalvi et al., 2021) or reasoning graphs (Ribeiro et al., 2023), explains why "sturdy wood" is the best answer.
Deriving such complex structured explanations poses a great challenge. Previous methods (Dalvi et al., 2021; Tafjord et al., 2021) consider structured explanations as linearized sequences and generate the entire reasoning process in one go. However, these methods lack controllability and may hallucinate unreliable reasoning steps. To ad-
5901
dress these concerns, recent studies (Hong et al., 2022; Neves Ribeiro et al., 2022; Yang et al., 2022) decompose structured explanations and focus on single-step reasoning via supervised learning. Nevertheless, this kind of approach may not always yield optimal results as they fail to consider the interdependencies between different steps. FAME (Hong et al., 2023) attempts to compensate for these shortcomings by leveraging Monte-Carlo planning (Kocsis and Szepesvári, 2006), which significantly increases the running time and inadvertently explores numerous ineffective steps (as shown in Table 1). Furthermore, FAME still concentrates on isolated single-step reasoning, which lacks support for structured reasoning. As a general framework for solving sequential decisionmaking problems, reinforcement learning (RL) is employed in RLET (Liu et al., 2022) to enhance multi-step reasoning. However, RLET defines the return (a.k.a. cumulative reward) using the standard chain structure, thus lacking the ability to represent the tree (Dalvi et al., 2021) or graph (Ribeiro et al., 2023) logical structures inherent in structured reasoning. As a result, the potential of RL for structured reasoning is not fully exploited.
To address the above issues, we propose SEER, a novel method that facilitates Structured rEasoning and Explanation via Reinforcement learning. In structured reasoning, we observe that the logical dependencies between different steps no longer follow a chained trajectory but instead adhere to the inherent tree or graph structure. Therefore, we propose the structure-based return to precisely describe a tree or graph logical structure, effectively capturing the complex interdependencies between different steps. Additionally, we refine the reward function to meticulously delineate diverse reasoning steps, specifically targeting redundant ones that do not contribute to the final structured explanations. Through experiments in Sec. 5.4, we find that redundant steps represent the exploration in the environment, and appropriate penalization contributes to improved reasoning performance.
Our contributions are summarized as follows: •We propose SEER, a novel RL-based method that facilitates structured reasoning and explanation. To our knowledge, SEER is the first general framework that accommodates scenarios of chained, treebased, and graph-based structured reasoning. •We propose the structure-based return to address the intricate interdependencies among different reasoning steps, effectively stimulating the potential of RL in structured reasoning. •We conduct extensive experiments to demonstrate the superiority of SEER over state-of-the-art methods. Our method facilitates the effectiveness and efficiency of structured reasoning and exhibits outstanding cross-dataset generalization performance.

heading: 2 Related Work

heading: 2.1 Explanation for Question Answering
text: Extensive research has delved into various forms of interpretability in QA systems (Thayaparan et al., 2020; Wiegreffe and Marasovic, 2021; Lamm et al., 2021; Chen et al., 2023). Different from the freeform texts susceptible to hallucinations (Rajani et al., 2019; Wei et al., 2022) or the rationales that only provide supporting evidence (DeYoung et al., 2020; Valentino et al., 2021), the structured explanations, such as the entailment trees (Dalvi et al., 2021) and reasoning graphs (Ribeiro et al., 2023), offer a novel way to generate explanations. These structured methods utilize tree or graph formats to clearly outline what information is used and how it is combined to reach the answer. Despite the remarkable interpretability, the intricately structured reasoning also poses significant challenges (Yu et al., 2023; Xu et al., 2023).

heading: 2.2 Natural Language Reasoning
text: Natural language reasoning, a process that integrates multiple knowledge to derive new conclusions, has attracted significant attention (Saha et al., 2020; Tafjord et al., 2021; Sanyal et al., 2022; Chen et al., 2024). Among these, the entailment trees and reasoning graphs, which involve structured reasoning and reasoning path generation tasks, present considerable challenges (Yu et al., 2023). Dalvi et al. (2021) attempt to transform structured reasoning into a linearized sequence to fit generative models, which may generate hallucinations and invalid reasoning. To alleviate this issue, recent studies (Neves Ribeiro et al., 2022; Hong et al., 2022; Neves Ribeiro et al., 2022; Hong et al., 2023) per-
form premises selection and reasoning in a step-bystep manner. Nevertheless, these methods decompose structured reasoning and solely leverage isolated single-step supervision to train models. This kind of approach neglects the interdependencies between different steps, which may not always yield optimal results. Therefore, in light of the advancements of RL in various reasoning tasks (Poesia et al., 2021; Le et al., 2022), RLET (Liu et al., 2022) attempts to incorporate RL into the entailment trees. However, it has to enumerate all potential actions, which is unacceptable for practical scenarios. Furthermore, RLET still defines returns in chained trajectories to facilitate multi-step reasoning, which is not suitable for tree/graph-based structured reasoning. In contrast, our SEER showcases superior adaptability to chained, tree-based, and graph-based structured reasoning via the structurebased return, which significantly enhances both the reasoning performance and efficiency.

heading: 3 Method

heading: 3.1 Task Formulation
text: As illustrated in Figure 1, the input of the task comprises a set of facts X = {x1, x2, . . . , xn} and a hypothesis h. The output of the task is the reasoning steps in a structured form, such as an entailment tree T or a reasoning graph1. The entailment tree T consists of tree-structured reasoning, whose leaf nodes are selected from the relevant facts (x∗) and intermediate nodes represent the derived intermediate conclusions (i∗). We represent the annotated ground-truth entailment tree as Tgold, with its leaf nodes signifying Xgold.

heading: 3.2 Overview
text: We model the structured reasoning as a reinforcement learning (RL) task, the goal of which is to learn the optimal reasoning policy. Figure 2 illustrates the overall framework of SEER, which mainly includes trajectory rollout and policy optimization. For trajectory rollout, we generate trajectories based on the current policy, and each trajectory is produced iteratively until the stopping criteria are satisfied (Appendix C.1). For policy optimization, we assign rewards to the collected
1Although the reasoning graph (Ribeiro et al., 2023) is a more general structure, to be consistent with the majority of previous work, we use the entailment tree (Dalvi et al., 2021) as an example to formalize the task and illustrate our method. Our proposed method is also applicable to the task described in the form of a reasoning graph.
trajectories and update both the policy and critic using the structure-based return. Algorithm 1 (Appendix A) outlines our proposed method for further reference.

heading: 3.3 Fine-grained Component of SEER
text: State At reasoning step t, we define the state st = {h, Pt, Ct} as a combination of the hypothesis h, existing reasoning steps Pt and candidate sentences Ct. Pt contains the reasoning steps so far, and Ct is the set of sentences that can be selected as premises. Each sentence in Ct is either unused facts or intermediate conclusions It generated by previous steps, i.e., Ct = {X ∪ It\Ut}, where Ut is the set of used sentences. For the initial state, s1 = {h, P1 = ∅, C1 = X}.
Action Given the state st, we consider two types of actions at ∈ A(st): (1) "Reason: <premises>": the entailment module is invoked to generate a new intermediate conclusion it based on the given <premises>. Here, <premises> are selected from Ct. Then, the state is updated as follows: Pt+1 = Pt ∪ {<premises> → it}, Ut+1 = Ut ∪ {<premises>}, and It+1 = It ∪ {it}. (2) "End": This action signifies the end of the reasoning process and returns the trajectory τ .
Policy The action type "Reason: <premises>" induces a large action space, since premises can be any combination of sentences from the candidate set C. To enumerate the probabilities of all potential actions and then sample an action to execute, previous studies (Liu et al., 2022; Hong et al., 2022) limit combinations to pairwise premises, such that the action space is reduced to ( n 2 ) , where n is the size of the set C. However, such a simplification incurs some potential drawbacks. First, as the number of candidate sentences increases, the number of potential actions grows exponentially. This renders them impractical for complex reasoning tasks with limited computational resources. Second, by restricting combinations to pairs only, the interdependencies among multiple premises are ignored, which may limit the effectiveness and richness of the derived conclusions.
To address this issue, we adopt a generative model to represent the policy π, which can directly sample from the action space A(st). Using the generative model essentially expands the action space where the combinations of premises can be arbitrary. This enables the policy to extensively
explore better actions during RL training, not limited to paired premises. Further, to speed up RL training, we first generate the top-k actions using policy π:
a1t , a 2 t , ..., a k t ∼ π(a|st), a ∈ A(st), (1)
where the input is a linearized state st (i.e., the concatenation of h, Pt, and Ct). Then, we proceed with re-normalization to form an appropriate probability distribution over the top-k actions, and sample from it to select the action at to be performed in the current reasoning step, that is,
π′(ait|st) = π(ait|st)∑k j=1 π(a j t |st) , i = 1, ..., k, (2)
at ∼ π′(a|st), a ∈ {a1t , a2t , ..., akt }. (3) Entailment Module If the action at is "Reason: <premises>", we invoke the entailment module to derive the intermediate conclusion to obtain the next state. The entailment module is also a generative model with its input being <premises>. Following Hong et al. (2022); Liu et al. (2022), we finetune the entailment model in a supervised manner and freeze the parameters during the reinforcement learning process, as shown in Figure 2.
Reward To evaluate the correctness of the entailment tree, Dalvi et al. (2021) proposed an alignment algorithm based on Jaccard similarity to align each intermediate node of the predicted tree Tpred with Tgold. However, different from the fully supervised learning methods, we observe that during the RL process, the policy explores different actions to
identify the optimal reasoning process, inevitably attempting some redundant steps that do not contribute to reaching the final hypothesis. Existing RL-based work (Liu et al., 2022) simply treats redundant steps with the same penalty as erroneous steps. This simplification may negatively affect the learning process which discourages necessary exploration in the action space. Furthermore, it lacks detailed feedback to guide the policy toward optimal policy, as it fails to differentiate between innocuous actions (redundant steps) and incorrect actions (erroneous steps).
To this end, we propose a fine-grained reward function that assigns different reward values for correct steps, erroneous steps, and redundant steps, as shown in Equation 4. For a trajectory τ , we assume that the last intermediate conclusion is our predicted hypothesis since the policy deems it should End here. Then, we backtrack to construct the predicted entailment tree Tpred (see Appendix C.6 for more details). Note that there might be some steps not participating in Tpred, which are regarded as redundant steps. Then, as illustrated in Figure 5, we consider steps that perfectly match via the alignment algorithm (Dalvi et al., 2021) as correct steps and regard others as erroneous steps.
rt =    1, if perfectly match, −0.5, if it /∈ Tpred, −1, otherwise.
(4)
Critic To enhance training stability, we introduce the critic to estimate the state-value function V (st). The input of V (st) is a linearized state, and its
output is a scalar representing the return (i.e., cumulative reward) when starting from state st. In the simplest case, the return is the chained sum of the rewards. Accordingly, one-step temporal difference (TD) (Sutton, 1988) is often used to estimate V (st), which is updated by the TD-target:
Gt = rt + γV (st+1), (5)
where γ is the discount factor. However, in structured reasoning, reasoning steps typically adhere to inherent tree (Dalvi et al., 2021) or graph (Ribeiro et al., 2023) structures, with the chained structure being merely a special case. Thus, Equation 5 just describes the chained multi-step reasoning, which may not effectively capture the intricate logical dependencies between steps in structured reasoning.
Therefore, we propose the structure-based return, where the TD-target is expressed in a more general formulation:
Ĝt = rt + γ 1 |P(st)| ∑
sj∈P(st) V (sj), (6)
where P(st) represents the parent node of state st in the entailment tree Tpred or reasoning graph. When st /∈ Tpred, P(st) = st+1. It can be seen that our structure-based return (Equation 6) adapts to structured reasoning involving chained, tree-based and graph-based structured scenarios. Especially, entailment tree is a special case of the reasoning graph, in which each state typically has only one parent node, and thus Equation 6 degenerates into Ĝt = rt + γV (P(st)). Furthermore, as shown in Figure 6 (Appendix E), for equivalent trajectories s1 → s2 → s3 → s4 and s2 → s1 → s3 → s4, previous method (Liu et al., 2022) would assign different returns for state s1 and s2, even though they represent the same tree in the end. Conversely, our method, by precisely delineating the intricate interdependencies between reasoning steps, consistently allocates the same return to any equivalent trajectories, thereby enhancing both stability and effectiveness.

heading: 3.4 Optimization
text: Our objective is to enhance the structured reasoning capabilities of the policy through RL. To alleviate issues of training instability and sample inefficiency in RL (Zhou et al., 2023; Roit et al., 2023), we employ the proximal policy optimization (PPO) algorithm (Schulman et al., 2017) to train the policy
π (parameterized by θ), as follows: Lπ = Et[min ( π′θ(at|st) π′θold(at|st) Ât, clip ( π′θ(at|st) π′θold(at|st) ,
1− ϵ, 1 + ϵ ))
Ât + βE(π′θ)], (7)
where π′ represents the probabilities normalized by Equation 2, θ and θold are parameters of the new and old policies, ϵ is a hyperparameter defining the clipping range, β is the entropy exploration coefficient, and E is the entropy bonus, which encourages sufficient exploration:
E(π′θ) = Eat∼πθ [− log π′θ(at|st)]. (8) Futhermore, Ât is the estimate of the advantage function for state st, defined as follows:
Ât = Ĝt − V (st). (9) To accurately evaluate return and guide the policy towards better updates, we train the critic by minimizing the difference between its prediction and the TD-target:
LV = Et [ (V (st)− Ĝt)2 ] . (10)
Supervised Warm-up Incorporating the supervised warm-up strategy before RL offers a relatively stable initial policy, which facilitates faster adaptation to the environment, particularly for complex reasoning tasks (Ramamurthy et al., 2023; Wu et al., 2023). Therefore, we convert the structured reasoning into single-step supervised data to warm up the policy as follows:
Lwarmup = − ∑
i
log p(yi|st, y<i). (11)
where y is the golden action at st.

heading: 4 Experiments

heading: 4.1 Datasets
text: Tree-structured reasoning We conduct experiments on EntailmentBank (Dalvi et al., 2021), the first dataset that supports structured explanation with entailment trees. Following (Hong et al., 2023), we also conduct experiments on EntailmentBankQA (Tafjord et al., 2022), whose objective is to reach the answer based on the entailment tree.
Graph-structured reasoning We conduct experiments on the STREET benchmark (Ribeiro et al., 2023) to assess the performance of graphstructured reasoning. Please refer to Appendix B for more details about the dataset statistics.

heading: 4.2 Baselines
text: For EntailmentBank, we compare with single-pass methods, such as EntailmentWriter (Dalvi et al., 2021), and step-by-step methods including METGEN (Hong et al., 2022), IRGR (Neves Ribeiro et al., 2022), RLET (Liu et al., 2022), NLProofs (Yang et al., 2022) and FAME (Hong et al., 2023). For EntailmentBankQA, we compare with Selection-Inference (SI) (Creswell and Shanahan, 2022) and FAME (Hong et al., 2023). For the STREET benchmark, we compare with the method proposed in (Ribeiro et al., 2023). Furthermore, we conduct comparisons with GPT-4 (OpenAI, 2023) equipped with Chain-of-Thought (CoT) (Wei et al., 2022), Tree of Thought (ToT) (Yao et al., 2023a) and ReAct (Yao et al., 2023b).

heading: 4.3 Implementation Details
text: For a fair comparison2, the policy is built with a T5-large model (Raffel et al., 2020), while the critic is the encoder of T5-large combined with a MLP (tanh as the activation function). For a supervised warm-up, we set a learning rate of 1e-5, a batch size of 16, and train the model for 20 epochs. For RL training, we set learning rate 2e-6 for both policy and critic, discounter factor γ as 0.95, batch size as 3, buffer size as 12, buffer training epochs NK as 2, ϵ as 0.2, and β as 1e-4. More implementation details can be found in Appendix C.

heading: 4.4 Evaluation Metrics
text: For EntailmentBank, we evaluate Tpred with the following dimensions: Leaves, Steps, Intermediates, and Overall AllCorrect. For STREET benchmark, we evaluate the reasoning graphs with two dimensions: Answer Accuracy and Reasoning Graph Accuracy. Note that Overall AllCorrect and Reasoning Graph Accuracy are extremely strict metrics, where any deviations will result in a score of 0. More metrics details can be found in Appendix D.

heading: 5 Result Analysis

heading: 5.1 Structured Reasoning
text: EntailmentBank As shown in Table 2, our SEER outperforms all baseline methods on the most strict metric, "Overall AllCorrect", across all three tasks.
2Previous studies have consistently utilized T5-large as the base model. Despite the existence of more advanced generative models (Du et al., 2022; Touvron et al., 2023), using T5-large enables us to maintain a fair comparison.
Specifically, our method achieves an absolute improvement of 1.7%/1.4%/1.0% in Task 1/2/3 compared to the strongest baseline. The steps dimension, i.e., premises selection, is the core of EntailmentBank3, contributing to enhancing the accuracy of both leaves and intermediates dimensions, thereby improving the overall AllCorrect metric. (1) Compared to SOTA supervised methods, such as NLProofs and FAME, our method exhibits significant advantages in the steps dimension. This demonstrates that focusing solely on isolated single-step reasoning through supervised learning may yield suboptimal solutions in intricate structured reasoning tasks, even though employing advanced planning algorithms, such as Monte-Carlo planning in FAME. (2) Compared to the SOTA RLbased method, our method outperforms RLET by 5.8%/9.0%/6.0% in Task 1/2/3. Our method employs a generative model as the policy to circumvent the issue of enumerating actions, facilitating the policy’s understanding of structured reasoning tasks (generating potential actions by itself). Moreover, our proposed structure-based return more effectively captures the tree-structured logical dependencies between steps and can assign stable returns for equivalent trajectories, which significantly improves reasoning abilities. Subsequent ablation studies will further demonstrate this. (3) Compared to GPT-4 with CoT, ToT, and ReAct, our method achieves an absolute improvement of 1.9% in Task 3. Although GPT-4 exhibits outstanding reasoning capabilities surpassing many other baselines, its performance relies on a vast number of parameters. Details about the prompts of GPT-4 can be found in Appendix F.
EntailmentBankQA Following Creswell and Shanahan (2022), we introduce the halter module to generate answers based on Tpred and substitute hypothesis with question and option during the reasoning process. As illustrated in Table 3, our method surpasses FAME by an absolute margin of 1.2%/7.4% in Task 1/2. While both FAME and SI are supervised methods, FAME significantly outperforms SI by enhancing the model’s reasoning and exploration capabilities through MonteCarlo planning. However, our method enhances the structured reasoning capabilities of the policy rather than focusing solely on single-step reasoning, which can significantly improve the quality of the entailment tree to aid in answering, especially
3A comprehensive error analysis is detailed in Appendix G.
in complex reasoning environments.
STREET As shown in Table 4, compared to GPT-4, our method has achieved absolute improvements of 4.8%/3.4%/4.1%/5.2% across various datasets, although the Reasoning Graph Accuracy is a very strict metric (Ribeiro et al., 2023). While GPT-4 excels at answering questions (far surpassing other methods), its parameter is thousands of times greater than other methods. Moreover, during the reasoning process, GPT-4 is prone to hallucinations (Rawte et al., 2023), resulting in poor performance in structured reasoning, particularly evident in the "Reasoning Graph Accuracy" metric. Since SCONE contains sufficient data as well

heading: Method SCONE GSM8K AQUA-RAT AR-LSAT Answer Accuracy

heading: Reasoning Graph Accuracy
text: as similar QA and reasoning patterns, we observe that the STREET method would outperform GPT4 on SCONE. However, by obtaining high-quality reasoning graphs, our method achieves absolute improvements of 2.8%/11.0%/8.9%/5.5% compared to the STREET method, significantly improving answer accuracy and trustworthiness. In reasoning graphs, a state may have multiple parent nodes. Our structure-based return (Equation 6) still precisely describes the cumulative reward for each
state, thereby facilitating reasoning performance in graph-structured reasoning.

heading: 5.2 Cross-dataset Performance
text: To evaluate the generalization performance, we conduct cross-dataset experiments on eQASC and eOBQA4 (Jhamtani and Clark, 2020). We apply the policy of Task 2 for selection without training on eQASC or eOBQA. As illustrated in Table 5, our method exhibits significant superiority in crossdataset generalization. Compared to supervised methods, our SEER, following the inherent structural nature of entailment trees, can better capture the logical dependencies between reasoning steps, which can effectively promote the generalization ability of the policy. The experimental results further validate the effectiveness of our method.

heading: 5.3 Ablation Studies
text: To evaluate the contribution of each component, we conduct extensive ablation studies. As shown in Table 6, we investigate three different variations of SEER in Task 3 of EntailmentBank: (1) w/o redundant neglects redundant steps by assigning a reward of -1. (2) w/o structure-based return removes the structure-based return and calculates it using the chained sum of rewards (Equation 5). (3) w/o RL removes the RL phase, relying solely on supervised warm-up. We discover that overlooking redundant steps may potentially inhibit the exploration of policy, leading to a performance decline. In addition, the results shown in Table 6 also
4More details about the setting of eQASC and eOBQA can be found in Appendix B.
demonstrate that removing the structure-based return severely affects the performance. It not only adequately addresses the equivalent trajectory problems, but also elegantly captures the logical relationships inherent in entailment trees, which is crucial for structured reasoning. Furthermore, it can be seen that removing the RL phase reduces performance by 3.8% of Overall Allcorrect, which is a significant impact for this strict metric. This indicates that relying solely on supervised learning may overlook the logical relationships in structured reasoning, thereby falling into suboptimal solutions.

heading: 5.4 Parameter Sensitivity Analysis
text: As illustrated in Figure 3, we further investigate the impact of rredundant and β on the performance in Task 3. We observe that compared to treating redundant and erroneous steps equally (rredundant = −1), not penalizing (rredundant = 0) may have more detrimental effects, which allows for unrestricted exploration. Moreover, a suitable β (the coefficient of entropy bonus) is crucial for performance enhancement, as it encourages the policy to break away from the "stereotypes" of supervised warm-up.

heading: 6 Conclusions
text: We propose SEER, a novel approach that facilitates structured reasoning and explanation via RL. To our knowledge, SEER is the first general framework capable of enhancing chained, tree-based, and graph-based structured reasoning. Our structurebased return precisely delineates the hierarchical and branching structure inherent in structured reasoning, effectively facilitating reasoning ability. Furthermore, SEER employs a generative model to represent the policy and refines the reward function, ingeniously circumventing the limitations of existing works. Comprehensive experimental results demonstrate that SEER significantly outperforms state-of-the-art methods and exhibits outstanding cross-dataset generalization performance.

heading: Limitations
text: Although our method has achieved excellent performance in structured reasoning and explanation, there remains one issue that deserves further exploration for future work: how to perform structured reasoning in the context of multimodal data. This includes combining content from images, tables, or audio data, a form of multimodal structured reasoning that is increasingly prevalent and demanding in real-world scenarios. In future work, we plan to extend our SEER to accommodate multimodal scenarios.

heading: Ethics Statement
text: This work focuses primarily on structured reasoning and explanation problems, and its contributions are entirely methodological. Therefore, this work does not have direct negative social impacts. For the experiments, we have open-sourced the code and utilized openly available datasets commonly used in previous research, without any sensitive information to our knowledge. The authors of this work adhere to the ACL ethical guidelines, and the application of this work does not present any apparent issues that may lead to ethical risks.

heading: Acknowledgements
text: This work is supported by the National Key R&D Program of China (NO.2022ZD0160102). Chao Yang is supported by the Shanghai Post-doctoral Excellent Program (Grant No. 2022234).

heading: A Algorithm Details
text: Algorithm 1: The training process of SEER Input: Structured reasoning dataset D;
Training epochs Nwarmup, N and NK ; batch size bwarmup and bmini;
Output: The optimal parameter of policy /* (1) Supervised Warm-up phase */
1 Initialise policy parameters πθ 2 Convert D into single-step data Dstep 3 for epoch = 1 to Nwarmup do 4 for i = 1 to |Dstep|/bwarmup do 5 sample minibatch from Dstep 6 update parameters πθ by Eq. 11
/* (2) RL phase */ 7 Initialize the critic parameters V 8 for epoch = 1 to N do 9 Initialise training buffer B ← ∅
// Filling the replay buffer 10 while B not full do 11 sample {h,X, Tgold} from D 12 collect trajectory τ via πθ 13 assign a reward for each step in τ 14 fill buffer B with {st, at, rt} from τ
// Performing k-epoch updates per buffer
15 for epochk = 1 to NK do 16 sample {(st, at, rt)}bmini from B 17 compute E(π′θ) and Ât by Eqs. 8, 9 18 update policy πθ by Eq. 7 19 update critic V by Eq. 10
Algorithm 1 describes the overall training process of our proposed SEER in detail, which primarily consists of two phases: supervised warm-up and reinforcement learning (RL). In the supervised warm-up phase, the structured reasoning is first decomposed into single-step reasoning data (Line 2). Then, we employs supervised learning to guide the policy πθ to quickly adapt to the complex reasoning environments (Lines 3-6). This is particularly beneficial when the number of parameters in the policy is relatively small (Akyurek et al., 2023; Liu et al., 2023). In the RL phase, we initially populate the replay buffer B through the policy πθ (Lines 9-13). Then, we update the parameters of the policy and the critic using the buffer data. To improve sample efficiency, NK updates are performed for each replay buffer (Lines 14-18).
For the inference process, we only need to use the policy (without the critic) for structured reasoning. Specifically, as illustrated in the trajectory rollout of Figure 2, we update the state by the policy and the entailment module. Then, we end the reasoning process until the stopping criteria are satisfied. Finally, we backtrack to construct the entire structured explanation, taking the last intermediate conclusion as the hypothesis for entailment tree (or the answer for the STREET benchmark).

heading: B Datasets Details
text: Datasets of Structured Reasoning Table 7 describes the statistics of datasets in detail. In the answer types, “K-Way MC” stands for multiple choice answer with K options.
EntailmentBank (Dalvi et al., 2021) comprises 1,840 expert-annotated entailment trees with an average of 7.6 nodes spanning across 3.2 entailment steps. The facts are derived from the WorldTree V2 corpus (Xie et al., 2020). Based on different facts X , there are three progressively more challenging tasks: Task1 (no-distractor), Task2 (distractor) and Task3 (full-corpus). For GPT-4, we employ all the data in Task 3 from EntailmentBank to evaluate its performance. EntailmentBank was originally designed for post-hoc tree reconstruction tasks instead of QA, Tafjord et al. (2022) converted it into EntailmentBankQA where the task is to choose the correct answer given multiple choice options rather than deriving hypothesis h.
To construct the STREET benchmark, Ribeiro et al. (2023) standardized many QA datasets, such as ARC (Clark et al., 2018), SCONE (Long et al., 2016), GSM8K (Cobbe et al., 2021), AQUARAT (Ling et al., 2017) and AR-LSAT (Zhong et al., 2021), in the graph-structured explanation format, where the tasks are converted into answering the question based on the predicted reasoning graphs. Please note that ARC in STREET is congruent with Task 1 of EntailmentBankQA (Ribeiro et al., 2023), hence, we do not repeat the experiment for this task in Table 4. Due to the high cost of GPT-4, we randomly sample 50 instances from each dataset in the STREET benchmark to evaluate GPT-4’s performance.
Datasets of Cross-dataset Experiments To evaluate the generalization performance of our method, following Hong et al. (2022), we conduct crossdataset experiments on eQASC and eOBQA (Jhamtani and Clark, 2020), which collect one-step entail-
ment trees for questions from QASC (Khot et al., 2020) and OpenBookQA (Mihaylov et al., 2018), respectively. The goal of this task is to select valid one-step trees from the candidate set and evaluate the results with P@1 and NDCG metrics (Hong et al., 2022). Questions with no valid tree are filtered. The candidate sets for eQASC and eOBQA are composed of 10 and 3 sentences respectively.
C Implementation Details

heading: C.1 Stopping criteria
text: For a fair comparison, we use the T5-large model to represent the policy. However, we observe that T5-large tends to perform "Reason" actions more frequently, which is caused by the smaller number of model parameters and the issue of having only a few "End" instances. Moreover, unlike GPT-4, T5large is less able to recognize when a hypothesis has been inferred and when to stop. Therefore, we attach two extra stopping criteria in addition to the "End" action: (1) The semantic similarity between the intermediate conclusion and the hypothesis exceeds a predefined threshold, i.e., BLEURT(i∗, h) > 1. (2) Exceeding the maximum number of reasoning steps (set to 20 in this paper).

heading: C.2 Alignment algorithm
text: Following (Dalvi et al., 2021), we evaluate the intermediate steps based on Jaccard similarity. Specifically, the intermediate nodes i∗ in Tpred are aligned with the intermediate nodes in Tgold that have the maximum Jaccard similarity. If the Jaccard similarity between the intermediate node in Tpred and all intermediate nodes in Tgold is zero, it is aligned with "NULL". Note that only the intermediate node that is perfectly matched with a node in Tgold, i.e., the Jaccard similarity is 1, is considered as a correct step. Figure 5 provides a detailed illustration of this process. The alignment process is similar in the reasoning graphs (Ribeiro et al., 2023).

heading: C.3 Retriever for Task 3
text: In Task 3 of EntailmentBank, first, it is necessary to retrieve relevant sentences from the corpus (Dalvi et al., 2021). The research focus of this paper is to enhance the structured reasoning ability of the policy. Therefore, we directly adopt the retriever and its associated parameters proposed in previous work (Hong et al., 2023), which is based on the all-mpnet-base-v2 model (Reimers and Gurevych, 2019). For a fair comparison, we retrieve the top 25 most relevant sentences as X for Task 3.

heading: C.4 Entailment Module
text: The entailment module is also based on the T5large model, taking premises as input and generating intermediate conclusions. Our primary focus is to enhance the structured reasoning ability of the policy through RL, therefore, we directly employ the entailment module that has already been trained in previous work (Hong et al., 2023), which also aids in a fair comparison.

heading: C.5 Halter Module
text: In EntailmentBankQA, we employ the Halter module (Creswell and Shanahan, 2022) to answer questions based on the predicted entailment trees. In this paper, the Halter module is built upon the T5large model. The module is trained with a learning rate of 1e-5 and a batch size of 16.

heading: C.6 Entailment Tree Construction
text: To evaluate the correctness of each reasoning step, we have to reconstruct the trajectory into an entailment tree Tpred and compare it with Tgold. Figure 5 illustrates this reconstruction process. We consider the last intermediate conclusion as the hypothesis and then construct the predicted entailment tree based on the reasoning relationship of each step. The reconstruction process is similar in the reasoning graphs (Ribeiro et al., 2023).

heading: C.7 Running time
text: In our experimental setting, the average training time per entailment tree in SEER is 6.98 seconds, and the average inference time per entailment tree in SEER is 3.91 seconds. As reported in their papers, the inference time per entailment tree in RLET (Liu et al., 2022) and FAME (Hong et al., 2023) are 9.34 seconds and 30.77 seconds, respectively. FAME leverages Monte-Carlo planning, necessitating the exploration of numerous nodes to enhance the reasoning capability of the policy, thus requiring considerable computational time. Our proposed SEER significantly surpasses FAME in terms of both efficiency and effectiveness.

heading: C.8 Experiment Environments
text: All experiments were conducted on Ubuntu 22.04 equipped with NVIDIA A100 GPUs. Our code mainly depends on python 3.105 and PyTorch 2.0.16. The pre-trained language models are derived from HuggingFace Transformers7.

heading: C.9 Details of Reasoning Graphs
text: For the reasoning graphs in the STREET Benchmark, the implementation details are slightly different from the entailment trees. In the reasoning graphs, reasoning steps may possess multiple parent nodes, and a fact (x∗) or intermediate conclusion (i∗) may be utilized multiple times (Ribeiro et al., 2023). Therefore, in the reasoning graph, we refrain from incorporating previously used premises into Ut, instead continually expanding the candidate sentence set Ct through newly derived intermediate conclusions. In other words, the state in the reasoning graphs is updated according to the following rules: Pt+1 = Pt ∪ {<premises>→ it}, Ct+1 = {X ∪ It+1}, and It+1 = It ∪ {it}.

heading: C.10 Other Implementation Details
text: For GPT-4, we set the temperature to 0.7. For Tree of Thought, we set b = 5 candidates at each step, and then vote to select the optimal action. Details regarding the prompts of CoT, ToT, and ReAct can be found in Appendix F. For all baselines, we obtain the optimal results based on experimental results or hyperparameter settings derived from the original papers. For our method, we initialize the critic with the encoder of the warm-up policy to
5https://www.python.org/ 6https://pytorch.org/ 7https://huggingface.co/
expedite the convergence of the critic and facilitate policy updates. The hidden layer dimension of the MLP in the critic is set to 512.

heading: D Metrics Details
text: For EntailmentBank, we follow (Dalvi et al., 2021) and evaluate the entailment tree Tpred using three dimensions: • Leaves: To evaluate the leaf nodes of Tpred, we compute F1 by comparing Xpred with Xgold. • Steps: To evaluate the structural correctness of each step, we compare all steps between Tpred and Tgold and then compute F1. A predicted step is considered structurally correct if its premises (e.g., x∗, i∗) exactly match the gold premises. • Intermediates: To evaluate the intermediate conclusions, we compare the aligned intermediate conclusions and then compute F1. A predicted intermediate conclusion is deemed correct if the BLEURT score (Sellam et al., 2020) exceeds 0.28.
For each dimension, the AllCorrect score is 1 if F1 is 1, otherwise 0. Given the above scores, we employ the Overall AllCorrect metric to comprehensively evaluate Tpred, which takes a value of 1 if and only if all leaves, steps, and intermediates are correct. Note that this is an extremely strict metric, where any deviations in Tpred will result in a score of 0.
For the STREET benchmark, we follow (Ribeiro et al., 2023) and adopt two metrics, namely, the answer to the question and the quality of the reasoning graphs, to evaluate different methods. • Answer Accuracy: This metric measures the ability to correctly answer questions. The answer accuracy serves as an upper bound for other metrics, as any reasoning graph generated with incorrect answers is also labeled as incorrect. • Reasoning Graph Accuracy: This metric compares the predicted reasoning graph and the golden reasoning graph from the aspects of the graph structure and the content of intermediate conclusions. Please note that this is a stringent metric, with minor deviations from the golden reasoning graph resulting in the prediction being incorrect.
E Illustrations and Case Study of SEER
Given a hypothesis h and initial facts X , we first obtain the trajectory through the reasoning process, as shown in Figure 4. The state update follows the Markov decision process (Bellman, 1957), meaning the current state only depends on the previous
state. Figure 4 is an erroneous reasoning example to better illustrate the following steps. Then, we convert the trajectory τ into an entailment tree Tpred and align it with Tgold to assign reward for each intermediate conclusion (as presented in Figure 5). Furthermore, Figure 6 elucidates the issue of equivalent trajectories, and previous work can not accurately describe the logical relationship between different states in entailment trees.

heading: F Prompts for GPT-4
text: Figures 7 and 8 show the Chain-of-Thought (CoT) (Wei et al., 2022) and ReAct (Yao et al., 2023b) prompts for GPT-4, and figures 9 and 10 show the prompts of thought generator and state evaluator in Tree of Thought (ToT) (Yao et al., 2023a), respectively. We provide a detailed introduction to the task definition and guide the model to respond in the required format. We randomly selected three examples for in-context learning. For a fair comparison, we use the same examples for CoT and ReAct, attributing similar thoughts to them. ReAct divides the dialogue into two rounds, "Thought" and "Action", to query GPT-4. For ToT, following (Yao et al., 2023a), we generate candidate actions using a thought generator and subsequently select and execute the optimal action through a state evaluator. Due to its exceptional reasoning capabilities and self-evaluation strategy, ToT achieves superior results compared to CoT and ReAct, as shown in Table 2. However, ToT requires higher costs in comparison to CoT and ReAct.

heading: G Error Analysis
text: We conduct a comprehensive error analysis on Task2 and Task3 of EntailmentBank.

heading: G.1 Error Analysis of Task2
text: We randomly sample 50 entailment trees where SEER made incorrect reasoning. We find the following four types of errors.
(1) Reasoning Step Error (62%). This is the main source of errors and predominantly depends on whether the policy can select the correct premise. We observe that a small portion of the errors (accounting for 12.9% of this error type) use all the gold leaves, but have errors in the combination order. Compared to other reasoning step errors, the policy identified the correct premise. For example, the gold steps are "x24 & x5 → i1;
i1 & x23 → h" and the error steps predicted by SEER are "x23 & x5 → i1; i1 & x24 → i2".
(2) Early Termination Error (18%). We observe that the reasoning process may terminate prematurely and the existing entailment steps are all correct. On one hand, T5-large outputs “End” prematurely, unlike GPT-4 which can accurately judge when to stop. On the other hand, the entailment module might erroneously infer a hypothesis, leading to premature termination.
(3) Intermediate Conclusion Error (10%). This error type is different from the above error (where the entailment module prematurely infers a hypothesis). Intermediate conclusion error denotes errors triggered by incorrect entailment in the intermediate conclusions, despite having correct leaves and steps. For a fair comparison, we used the entailment module that has already been trained in previous work (Hong et al., 2023). It is noted that the reasoning part, which is the focus of our paper, is completely correct in this type of error, and this type of error can be mitigated by training a better entailment module.
(4) Imperfect Evaluation (10%). We discover that some trees deemed as invalid are valid in fact, indicating that current automated metrics underestimate the validity of the trees. The most common reason is that there are multiple valid ways to construct an entailment tree. For example, consider the structure of a gold tree: "x1 & x2 & x3 → h" may be predicted as: "x1 & x2 → i1; i1 & x3 → i2".

heading: G.2 Error Analysis of Task3
text: Task 3 requires retrieving an initial set of facts X from the corpus. Therefore, in addition to the errors in Task 2 described above, we found that Task 3 has its own unique set of errors.
(1) Missing Gold Leaves Error (58%). Missing gold leaves error refers to the case where the gold leaves are not included in the facts X retrieved from the corpus. This case will inevitably lead to an error in the predicted entailment tree, regardless of how powerful the policy is. The bottleneck of this error lies in the retrieval model. For a fair comparison, we directly use the retrieval model provided in previous work (Hong et al., 2023).
(2) Reasoning Errors (42%). The four error types described in G.1 account for 42% in Task3.
We also discovered that the reasoning graph contains similar error types as found in entailment trees, as they both belong to structured reasoning.
Question: Melinda learned that days in some seasons have more daylight hours than in other seasons. Which season receives the most hours of sunlight in the Northern Hemisphere? Answer: summer Hypothesis ℎ: northern hemisphere will have the most sunlight in summer
�𝐺𝐺1 = 𝑟𝑟1 + 𝛾𝛾𝑉𝑉(𝑠𝑠4) �𝐺𝐺2 = 𝑟𝑟2 + 𝛾𝛾𝑉𝑉(𝑠𝑠4)
trajectory 𝜏𝜏
𝑖𝑖4 𝑖𝑖1
𝑖𝑖2 𝑖𝑖2
𝑥𝑥10
𝑥𝑥13 𝑖𝑖1
𝑥𝑥4
𝑥𝑥25
𝑠𝑠1 𝑠𝑠2 𝑠𝑠3 𝑠𝑠4
𝑖𝑖3 𝑥𝑥3
𝑥𝑥17
𝑟𝑟1 = 1 𝑟𝑟2 = −1 𝑟𝑟3 = −0.5 𝑟𝑟4 = −1
trajectory 𝜏𝜏′
𝑖𝑖4 𝑖𝑖1
𝑖𝑖2 𝑖𝑖2
𝑥𝑥10 𝑥𝑥13 𝑖𝑖1 𝑥𝑥4 𝑥𝑥25
𝑠𝑠2 𝑠𝑠1 𝑠𝑠3 𝑠𝑠4
𝑖𝑖3 𝑥𝑥3
𝑥𝑥17
𝑟𝑟1 = 1𝑟𝑟2 = −1 𝑟𝑟3 = −0.5 𝑟𝑟4 = −1
𝑥𝑥4 𝑥𝑥25
𝑖𝑖1
𝑥𝑥10 𝑥𝑥13
𝑖𝑖2
𝑖𝑖4
𝐺𝐺2 = 𝑟𝑟2 + 𝛾𝛾𝑉𝑉(𝑠𝑠3)
𝐺𝐺2′ = 𝑟𝑟2 + 𝛾𝛾𝑉𝑉(𝑠𝑠1)
𝐺𝐺1 = 𝑟𝑟1 + 𝛾𝛾𝑉𝑉(𝑠𝑠2)
𝐺𝐺1′ = 𝑟𝑟1 + 𝛾𝛾𝑉𝑉(𝑠𝑠3)
Structure-based return
previous return
Figure 6: An illustration of the equivalent trajectory and the definition of return. As the reasoning steps of s1, s2, and s3 are mutually independent, the execution order among these steps can be arbitrary. Thus, τ and τ ′ are equivalent trajectories because they can be converted to the same entailment tree (Dalvi et al., 2021). As shown in blue box, previous work (Liu et al., 2022) defines the return (a.k.a cumulative reward) in a chained trajectory and would assign different returns to s1 and s2 in these equivalent trajectories. In contrast, as shown in red box, our structure-based return is defined based on the tree or graph structure inherent in structured reasoning, which is the same source of rewards. Our structure-based return will consistently allocate stable returns to equivalent trajectories, thereby promoting training stability and convergence. Furthermore, maintaining consistency between the sources of rewards and returns can significantly enhance the effectiveness of the policy.

heading: Now reason about the following:

heading: Here are some examples:

heading: Now reason about the following:

heading: Here are some examples:

heading: Here are some examples:

heading: 1. reason: sentX & intX -> new conclusion

heading: The Prompt of Thought Generator:

heading: Vote: The best choice is 4

heading: END

heading: Candidate action:

heading: Be careful to follow the output format. Here are some examples:

heading: The Prompt of State Evaluator:

heading: Future Work
text: Although our method has achieved excellent performance in structured reasoning and explanation, there remains one issue that deserves further exploration for future work: how to perform structured reasoning in the context of multimodal data. This includes combining content from images, tables, or audio data, a form of multimodal structured reasoning that is increasingly prevalent and demanding in real-world scenarios. In future work, we plan to extend our SEER to accommodate multimodal scenarios