SlowGuess/NAACL_2022_OCR huggingface dataset:
Mokanarangan Thayaparan, Marco Valentino, and André Freitas. 2020. A survey on explainability in machine reading comprehension. CoRR, abs/2010.00389.
Marco Valentino, Mokanarangan Thayaparan, and André Freitas. 2021. Unification-based reconstruction of multi-hop explanations for science questions. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, pages 200-211. Association for Computational Linguistics.
Sarah Wiegreffe and Ana Marasovic. 2021. Teach me to explain: A review of datasets for explainable natural language processing. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1).
Zhengnan Xie, Sebastian Thiem, Jaycie Martin, Elizabeth Wainwright, Steven Marmorstein, and Peter A. Jansen. 2020. Worldtree V2: A corpus of science-domain structured explanations and inference patterns supporting multi-hop inference. In Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, pages 5456-5473. European Language Resources Association.
Vikas Yadav, Steven Bethard, and Mihai Surdeanu. 2019. Quick and (not so) dirty: Unsupervised selection of justification sentences for multi-hop question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 2578-2589. Association for Computational Linguistics.
Vikas Yadav, Steven Bethard, and Mihai Surdeanu. 2020. Unsupervised alignment-based iterative evidence retrieval for multi-hop question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 4514-4525. Association for Computational Linguistics.
Qinyuan Ye, Xiao Huang, Elizabeth Boschee, and Xi-ang Ren. 2020. Teaching machine comprehension with compositional explanations. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 1599-1615. Association for Computational Linguistics.
Hongming Zhang, Xinran Zhao, and Yangqiu Song. 2020. Winowhy: A deep diagnosis of essential commonsense knowledge for answering winograd schema challenge. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 5736-5745. Association for Computational Linguistics.
A Entailment Modules Training Details
A.1 Synthetic Data
We follow the ParaPattern (Bostrom et al., 2021) to collect synthetic training data for the entailment modules. Since they only consider the substitution and contraposition deductions, we extend the method to conjunction and if-then deductions by designing the specific syntactic templates and construction rules. Table 9 shows the used syntactic patterns. We use Spacy<sup>3</sup> to match sentences from Wikipedia (version "20200501.en"). In total, we collect about 24k, 443k, and 97k sentences for substitution, conjunction, and if-then modules, respectively. We follow Bostrom et al. (2021) to train the modules on the synthetic data with a learning rate of 3e-5 for 1 epoch.
A.2 Reasoning Type Annotations of EntailmentBank
The original steps in the EntailmentBank are not annotated with reasoning types. We manually annotated the reasoning types of 400 steps in the training split (Train-manual) and 275 steps in the development split (Dev-manual). To label the remaining steps in the training split, we train a classifier with the Train-manual steps. We use the Roberta-large (Liu et al., 2019) as our classifier. It achieves an accuracy rate of  $88\%$  on the Dev-manual steps. We use the classifier to predict the reasoning types of the remaining 2-premise steps and take the predicted types as the pseudo labels (Train-pseudo). Table 8 shows the statistics of the reasoning type annotations.
Split	Sub.	Conj.	If-then	All
Train-manual	211	105	84	400
Train-pseudo	2,441	812	535	3,788
Dev-manual	153	71	51	275
Table 8: Statistics of the step reasoning type annotations.
B Controller Training Details
Training Data. We decompose the gold entailment trees into several intermediate states for training. For example, the tree in Figure 1(c) can be decomposed into the following positive states:  $R_0:H \Leftarrow \{s_1,s_2,s_3,s_4,s_5\}$ ,  $R_1:H \Leftarrow \{s_1,s_3,s_4,i_1\}$ , and  $R_2:i_1 \Leftarrow \{s_1,s_2,s_3,s_5\}$ . The state  $R_0$  has two distractors  $s_1$  and  $s_3$ , one positive deductive
step  $s_2 + s_5 \to i_1$ , and one positive abductive step  $H - s_4 \to i_1$ . We add disturbances to the trees to make positive and negative states. For the state  $R_1$ , the fact  $i_1$  is the conclusion of gold step  $s_2 + s_5 \to i_1$ . We use a deductive module to predict a conclusion  $\hat{i}_1$  given  $s_2$  and  $s_5$ . If the predicted  $\hat{i}_1$  is correct, we replace  $i_1$  with  $\hat{i}_1$  to make new positive states  $R_1^+ : H \Leftarrow \{s_1, s_3, s_4, \hat{i}_1\}$ . The  $R_1^+$  can be used to perform further reasoning. Otherwise, we replace  $i_1$  with  $\hat{i}_1$  to make negative states  $R_1^-$ . The  $R_1^-$  contains an incorrect conclusion  $\hat{i}_1$  and thus should not be used for further reasoning. The reasoning controller should be trained to learn to distinguish between  $R_1^+$  and  $R_1^-$  and give the  $R_1^+$  a higher state score than  $R_1^-$ . To judge whether the generated  $\hat{i}_1$  is correct, we follow the evaluation metrics (Dalvi et al., 2021) to use BLEURT. The predicted  $\hat{i}_1$  is considered correct if the BLEURT score between  $\hat{i}_1$  and the gold  $i_1$  is larger than 0.28.
C Reasoning Algorithm and Hyperparameter Analysis
Algorithm 1 shows the whole reasoning process. The hyperparameters are selected with the development split, as shown in Figure 6. We select a beam size of 10, a max reasoning depth of 5, a distractor threshold of 0.001, and a step sampling rate of  $10\%$ . We only consider the steps whose sentences have word overlap. When constructing the entailment tree, we use the BLEURT scores to align the target of a state to the most similar fact. Note that when making a new reasoning state with the step  $p$  and the novel intermediate fact  $i$ , if the step  $p$  is a backward abductive step, we replace the original target hypothesis with  $i$  and treat the  $i$  as the target hypothesis which the new state aims to prove (as shown in Figure 2). We run our method three times and report the average performance.
D Heuristic Reasoning Algorithm without the Controller
To investigate the effect of the reasoning controller for entailment tree generation, we design a heuristic generation algorithm that does not use the reasoning controller. Since the cost of traversing the entire search space is unaffordable, we adopt the beam search. In each reasoning state, we try all possible steps with entailment modules and make new candidate reasoning states. To select the correct states, we use the BLEURT scores as the heuristic information to guide the search process. Specifically,
Algorithm 1 Reasoning Algorithm
Input: Hypothesis  $H$ , fact sentences  $S$ , controller, deductive modules  $M_{ded}$ , abductive modules  $M_{abd}$
Parameter: Beam size  $K$ , max reasoning depth  $D$ , distractor threshold  $\theta$ , step sampling rate  $\tau$
1: // Construct initial reasoning state
2: Remove  $s_i$  with fact score less than  $\theta$  in  $S$
3:  $R_{init} \gets (H \Leftarrow \text{the filtered sentences } S')$
4:  $\mathcal{R}_{\text{beam}} \gets \{R_{\text{init}}\}$ ,  $\mathcal{R} \gets \mathcal{R}_{\text{beam}}$
5: // Reasoning with beam search
6: while the depth does not reach  $D$  do
7:  $\mathcal{R}_{beam}^{\prime}\gets \{\}$
8: for  $R\in \mathcal{R}_{beam}$  do
9: // Select promising steps
10: for  $p \in$  steps of  $R$  with top  $\tau \%$  step score do
11: // Single-step entailment reasoning
12: for  $m\in M_{\text{ded}}$  or  $m\in M_{\text{abd}}$  do
13: execute step  $p$  with module  $m$  and obtain a novel intermediate fact  $i$
14: construct a new state  $R_{new}$  with the step  $p$  and the fact  $i$
15:  $R_{beam}^{\prime}\gets R_{beam}^{\prime}\cup \{R_{new}\}$
16: end for
17: end for
18: end for
19: // Verify and select states
20:  $\mathcal{R}_{beam} \gets K$  states with the highest state scores from  $R_{beam}'$
21:  $\mathcal{R} \gets \mathcal{R} \cup \mathcal{R}_{\text{beam}}$
22: end while
23: // Construct the entailment tree
24: for  $R\in \mathcal{R}$  do
25: Align the target of  $R$  to the most similar fact sentence of  $R$  to make a tree  $T$
26: end for
27: Select the tree  $\hat{T}$  with highest score
28: Return The entailment tree  $\hat{T}$
given a candidate state  $R:H \Leftarrow S$ , we estimate the similarity between a fact  $s_i \in S$  and the target  $H$  by
$$
G _ {\text {f a c t}} ^ {\prime} \left(s _ {i}\right) = \operatorname {B L E U R T} \left(H, s _ {i}\right), \tag {8}
$$
and then score a candidate state by
$$
G _ {\text {s t a t e}} ^ {\prime} (R) = \frac {1}{n} \sum_ {s _ {i} \in S} G _ {\text {f a c t}} ^ {\prime} \left(s _ {i}\right). \tag {9}
$$
The top-  $K$  candidate states with the highest state scores are selected to perform further reasoning,
where  $K$  is the beam size. We use the same beam size as the algorithm with the controller uses.
E Experiment Details on eQASC and eOBQA
For each question+answer pair, the eQASC/eOBQA provides the corresponding hypothesis  $H$ , about 10/4 facts as  $S$ , and a candidate set of steps. Each candidate step is a 2-premise single step from two facts to  $H$  (e.g.,  $s_1 + s_2 \rightarrow H$ ) and can be viewed as a one-step entailment tree with three nodes. The target is to select the correct trees/steps from the candidate set. There might be more than one correct tree in the candidate set. We conduct experiments on the questions with at least one correct entailment tree (677 eQASC questions and 79 eOBQA questions). Since the given  $S$  contains distractors, we adopt the Task2 models trained on EntailmentBank (without further fine-tuning on eQASC and eOBQA) to perform cross-dataset experiments.
For our method, we follow our Task2 reasoning algorithm to select from the candidate trees/steps. Specifically, we first filter out the facts in  $S$  with low fact scores using a threshold (selected using the development split). Then we predict the step scores for the candidate steps and select the step with the highest score. For the EntailmentWriter, we feed the  $S$  and  $H$  to the EntailmentWriter and score each candidate step with the  $\frac{1}{\mathrm{PPL}}$ , where PPL is the perplexity of the sequence segment representing the step (e.g., sent1 & sent2 for  $s_1 + s_2$  in the official EntailmentWriter implementation).
We follow the official evaluation metrics of eQASC and eOBQA. The P@1 (Precision@1) measures the fraction of cases where the selected tree (topmost ranked) is correct. It is equivalent to the Overall AllCorrect score between the top-1 predicted one-step tree and the best-matching gold tree. The NDCG (Normalized Discounted Cumulative Gain) metric measures how well ranked the candidate trees are when ordered by the predicted scores. It reflects the model's ability to distinguish the validity of trees and rank the correct trees ahead of the incorrect ones.
F Main Experimental Environments
We deploy all models on a server with 500GB of memory and one 40G A100 GPU. Specifically, the configuration environment of the server is ubuntu 21.04 and our code mainly depends on
Type	Dependency Patterns	Example
Substitution	1. nsubj:NNS$0 amod:' such' prep:IN'as' pobj:$1 ROOT:VBP$2
2. nsubj:NNS$0 prep:IN'like' pobj:$1 ROOT:VBP$2
3. nsubj:NNS$0 prep:VBG'include' pobj:$1 ROOT:VBP$2	Original Sentence:
Slime molds like Physarum polycephalum are useful for studying cytoplasmic streaming.
Premises:
Physarum polycephalum are a slime mold.
Slime molds are useful for studying cytoplasmic streaming.
Paraphrased:
The polycephalum is a slime mold.
The slimy molds are useful for studying streaming.
Conclusion:
Physarum polycephalum are useful for studying cytoplasmic streaming.
Conjunction	1. nsubj:NN|NNS$0 cc:' and' conj: NN|NNS $1 ROOT:VBP$2
2. acomp:JJ$0 cc:' and' conj:JJ$1 ROOT:VB|VBP|VBZ$2
3. ROOT:VB|VBP|VBZ$0 attr:$1 nsubj:WDT relcl:VB|VBP|VBZ$2
4. ROOT:VB|VBP|VBZ$0 attr:$1 acl:$2 nsubj:NN|NNS$3	Original Sentence:
Aman is an Indian anti-war movie directed by Mohan Kumar.
Premises:
Aman is an Indian anti-war movie.
Aman is directed by Mohan Kumar.
Paraphrased:
Aman is a Indian film that is anti-war.
Mohan Kumar was the director of Aman.
Conclusion:
Aman is an Indian anti-war movie directed by Mohan Kumar.
If-then	1. mark:IN'if' advcl:$0 nsubj:$1 ROOT:$2 nsubj:$3	Original Sentence:
If the rebels occupy territory, they make a gain.
Premises:
If the rebels occupy territory they make a gain.
The rebels occupy territory
Paraphrased:
The rebels are able to make a gain if they hold on to territory.
The territory was occupied by the rebels.
Conclusion:
The rebels make a gain.
Table 9: The syntactic patterns used on data scraping and the training examples for deductive entailment modules. Pattern nodes are donated as dep: POS‘lemma’ \( \\( i \) ,where dep contains the dependency relations of the matching token,POS contains the part-of-speech tags of the matching token,'lemma' contains the lemmatized form of the matching token,and \( \$ i \) indicates that a matching token and its subtree will be used as a match variable for rule-based rewriting. | means "or".
Figure 6: Hyperparameter analysis on the Task2 development spilt.
python 3.8.10 and PyTorch 1.7.1. We use the pre-trained language models from HuggingFace Transformers<sup>4</sup>. We use the Adafactor optimizer (Shazeer and Stern, 2018) implemented by HuggingFace Transformers.
G Discussion on the Automatic Evaluation
As discussed by Dalvi et al. (2021), the automatic entailment tree evaluation metrics might misjudge in some cases (e.g., tree structure variation) and still need to be improved. In fact, how to quantitatively evaluate a predicted tree remains a challenging problem. In the existing metric, the first step is the tree alignment algorithm (Dalvi et al.,
Figure 7: An example case illustrating the potential inaccuracy of the automatic evaluation metrics. In the predicted tree, the fact  $s_3$  is a distractor and the step  $s_1 + s_3 \rightarrow i_1$  is not a valid entailment. Following the official evaluation code, the nodes  $i_1, H$  in the predicted tree are aligned to the  $H, H$  in the gold tree, respectively (the dotted line). By comparing the aligned intermediate nodes ( $i_1$  vs.  $H, H$  vs.  $H$ ), the predicted tree achieves a Step F1 score of 0.0 and an Intermediate F1 score of 1.0. The Intermediate F1 score being 1.0 should have indicated that the predicted tree has perfect intermediate conclusions. However, the  $i_1$  is not entailed by the  $s_1$  and  $s_3$ .
2021). The nodes in the predicted tree  $T_{pred}$  are aligned to the nodes in the gold tree  $T_{gold}$  for further comparison. Each non-leaf node  $i_{pred}$  of  $T_{pred}$  is aligned to the first non-leaf node  $i_{gold}$  where the Jaccard similarity of their respective leaf sentences is maximum. For any  $i_{pred}$  with zero Jaccard similarity to all gold nodes, it is aligned to a dummy gold node with a blank conclusion. In the official implementation, (1) each  $i_{gold}$  may correspond to more than one  $i_{pred}$ , while there is no penalty for duplication when calculating Intermediate F1; (2) the root node (the given hypothesis sentence which is identical in  $T_{pred}$  and  $T_{gold}$ ) is trivially viewed as a normal intermediate node (the novel generated intermediate sentence). Because of these two reasons, the Intermediate F1 might achieve a high score (indicating the  $T_{pred}$  can draw correct intermediate conclusions from the premises), even when the Step F1/AllCorrect is relatively low (indicating the  $T_{pred}$  does not select the correct premises for the intermediate nodes). For example, the EntailmentWriter (T5-11B) for Task3 achieves an Intermediate F1 of 36.4% while the Step F1/AllCorrect is only 7.8%/2.9% (Dalvi et al., 2021). Figure 7 shows a specific case.
To alleviate the inaccuracy caused by the above reasons, we mainly use the more strict metrics (i.e., Leaves/Steps/Intermediates/Overall AllCorrect) for comparison. Furthermore, we adopt manual evaluation on the full trees and individual steps to make a more accurate comparison (Sec. 6.2).
H Case Study
We show some entailment trees generated by our METGEN-separated on the Task2 questions in Figure 8, 9, 10, 11. METGEN can generate a valid entailment tree which may have a different structure with the gold one (Figure 8). METGEN can handle medium-complexity questions, generate valid entailment trees and provide the reasoning types of steps (Figure 9 and 10). The questions which require more complex reasoning (e.g., the gold tree in Figure 11 requires 11 leaf facts and 8 entailment steps) remain challenging. Although the full tree generated by our method for such complex question can be not entirely correct, the intermediate conclusions (e.g.,  $i_1$ ,  $i_2$  in Figure 11) are still reliable.
Question Q: Which phase of the Moon occurs after a waxing gibbous?
Answer A: full moon
Hypothesis H: a full moon is the moon phase that occurs after a waxing gibbous
Facts S:
s1: state of matter is a property of matter and includes ordered values of solid / liquid / gas
s2: the moon is earth's moon
s3: usually means most of the time
s4: phase means state
s5: occur is similar to appear
s6: the moon orbits the earth
s7: to be found in means to be contained in
s8: revolving around something means orbiting that something
s9: the moon orbiting the earth occurs once per month
s10: a complete revolution / orbit of the moon around the earth takes 1 / one month
s11: amount is a property of something and includes ordered values of none / least / little / some / half / much / many / most / all
s12: a waxing gibbous is a kind of phase of the moon
s13: warm / becoming warm means heat is added
s14: a phase change is when matter / a substance changes from one state of matter into another state of matter
s15: a full moon occurs after a waxing gibbous moon
s16: the moon reflects sunlight towards the earth
s17: generally means usually
s18: to happen means to occur
s19: type of moon / kind of moon means moon phase
s20: a full moon is a kind of phase of the moon
s21: visible means able to be seen
s22: motion / movement means moving / to move
s23: the moon completes a lunar cycle over a period of 29 days
s24: the moon rising occurs once per day
s25: lunar phase is synonymous with moon phase
Predicted Tree
Figure 8: Case 1. The predicted entailment tree consists of two 2-premise steps, while the gold tree consists of one 3-premise step. Under the automatic evaluation metric, the predicted tree would be rated as invalid (Overall AllCorrect = 0), since the predicted steps do not match the gold step. However, the predicted tree should be valid because each step in the tree is a valid entailment (i.e., the 3-premise step can be decomposed into two valid 2-premise steps). It would be rated as valid under manual evaluation.
Question Q: According to the Periodic Table of the Elements, which set of elements has similar properties?