SlowGuess/NAACL_2021_OCR huggingface dataset:
Table 4 shows the quality of the generated proofs. Only the samples that can be proved (either by "proof" or "inv-proof") are compared with our method's proofs. The number of samples subject to this comparison is indicated by "Cnt". The results in the table demonstrate that EVR obtains high-quality proofs most of the time, regardless of the proof depth.
Table 3: Answer accuracy of EVR variants trained on DU1 (top) and DU5 (bottom), and evaluated on all data depths (DU5).
D	0	1	2	3	4	5	all
Cnt	1934	1934	1934	1934	1934	1934	11604
DU1
EVR1	96.4	91.9	86.3	75.7	73.1	64.5	81.3
EVR2	96.1	91.2	86.7	77.9	76.8	67.4	82.7
EVR3	96.4	93.0	89.8	83.2	84.1	75.1	86.9
DU5
EVR1	96.0	90.6	86.6	76.8	74.6	66.0	81.8
Table 4: Justification quality of our approach. "Cnt" indicates the number of questions eligible for proof comparison.
Subset	B1	B2	E1	E2	E3	E4	all
Cnt	40	40	162	180	624	4224	5270
DU1
RT	100.0	100.0	88.9	80.0	93.9	97.5	96.2
EVR1	72.5	72.5	100.0	100.0	98.1	98.4	98.1
EVR2	77.5	72.5	100.0	100.0	98.1	93.6	94.3
EVR3	87.5	77.5	100.0	100.0	99.0	95.7	96.2
DU5
RT	97.5	100.0	96.9	98.3	91.8	76.7	80.2
PR	95.0	95.0	100.0	100.0	89.7	84.8	86.5
EVR1	70.0	75.0	64.8	60.0	63.1	61.1	61.6
EVR1c1	75.0	72.5	100.0	75.0	98.1	93.8	93.6
Table 5: Evaluation of EVR answers, under the zero-shot learning scenario on the birds-electricity dataset. There are 6 subsets in the dataset, denoted from B1 to E4.
Subset Cnt	B1 28	B2 28	E1 72	E2 90	E3 312	E4 1206	all 1736
DU1
EVR1	28.6	32.1	100.0	100.0	73.1	90.5	86.3
EVR2	35.7	46.4	100.0	100.0	73.1	87.6	84.6
EVR3	50.0	67.9	100.0	100.0	75.0	91.0	88.0
DU5
EVR1	28.6	28.6	58.3	40.0	45.2	68.4	61.1
EVR1c1	28.6	35.7	100.0	50.0	70.8	81.6	77.2
Table 6: Evaluation of EVR-generated proofs, under the zero-shot learning scenario on the birds-electricity dataset.
4.3 Zero-Shot Results on the Birds-Electricity Dataset
Table 5 shows the performance of EVR when tested under a zero-shot learning scenario on the birds-electricity dataset. The results show that EVR yields good generalization ability and outperforms the other two baseline methods in general. Notably, EVR1 trained on DU1 considerably outperforms the baseline methods trained on DU5. A surprising result is that RT trained on DU1 yields a better results than that trained on DU5. The RT creators explained that it is because some extremely rare cases in the training data are not well learned by their DU5 model.
Surprisingly, EVR1 trained on DU5 yields a low performance in the evaluation (e.g., 61.6 on all datasets). We inspected several outputs of the reasoning steps generated by EVR1, and observed
D	0	1	2	3	4	5	all
Cnt	6299	4434	2915	2396	2134	2003	20192
10k
PR	—	—	—	—	—	—	87.1
EVR1	99.9	99.1	96.8	90.4	84.3	82.9	94.8
30k
PR	—	—	—	—	—	—	97.8
EVR1	99.6	98.5	96.8	93.4	89.6	88.5	96.0
70k
PR	100.0	99.0	98.8	99.1	98.8	99.3	99.3
EVR1	99.7	98.8	97.8	95.5	91.7	90.7	97.0
that the major reason for this failure is because pattern 4 is not successfully learned due to the bias in DU5 data. In DU5 training data, there are at least 7 facts for each question (i.e., at least 2 fact buffers when the fact buffer size is set to 5). In this case, the target output for pattern 4 would be "I want to judge whether fact buffer 1 ... OR I want to judge whether fact buffer 2 ..." . In contrast, in the birds-electricity dataset, some questions have only 3 supporting facts, so there is only 1 fact buffer. In this case, T5 should generate "I want to judge whether fact buffer 1 can prove [query]." without further disjunctions. However, since such examples never appear in the DU5 training data, the T5 still generates instructions to loop over multiple fact buffers, which causes the reasoning program to fail (due to the attempt to access non-existent fact buffers).
To verify the above observation, we trained pattern 6 and 10 using DU5, and all other patterns on DU1. We report this model's performance (indicated by  $\mathrm{EVR}^{c1}$ ) in Table 5 and 6. These results indicate that our intuition was correct, as  $\mathrm{EVR}^{c1}$  yields good generalization performance on the birds-electricity dataset.
Table 6 shows EVR generates high-quality proofs overall, but the proofs on B1 and B2 are poor. The reason is that the B1 and B2 datasets contain some examples that have proofs unsupported by our model (e.g., to directly prove a positive statement is False by contradiction).
4.4 Results Using Less Training Data
Tables 7 and 8 show that EVR1 yields stable performance and proofs when trained with considerably less data (10k and 30k examples). In the lowest data configuration (10k), EVR outperforms PR considerably, even when PR is trained on DU5. This
Table 7: Evaluation of the predicted answers of EVR1, when trained on less data.  ${70}\mathrm{k}$  is the number of original training examples. PR is trained on DU5 whereas EVR1 is trained on DU1.
D	0	1	2	3	4	5	all
Cnt	1934	1934	1934	1934	1934	1934	11604
10k	96.4	92.1	86.0	73.6	70.4	62.5	80.2
30k	96.4	91.5	84.9	75.0	72.6	64.8	80.9
70k	96.4	91.9	86.3	75.7	73.1	64.5	81.3
Table 8: Evaluation of the generated proofs of EVR1, when trained on less data.
experiment supports our claim that EVR suffers less from the compositionality generalization problem by recursively decomposing complex problems and reasoning over simple ones.
Neural Symbolic Methods: One branch of neural-symbolic reasoning methods is to design different components in the network, but keeping the whole network differentiable. Typical works include the Differential Neural Computer (DNC) (Graves et al., 2016), End-to-end Memory Networks (Sukhbaatar et al., 2015), Dynamic Memory Networks (DMN) (Kumar et al., 2016) and Compositional Attention Networks (MAC) (Hudson and Manning, 2018). Another direction are the neural modular networks, where what components to use are determined dynamically for each question (Gupta et al., 2019; Jiang and Bansal, 2019). However, it is hard to prove the components are actually fulfilling the designed functionality after training due to the distributed nature of the intermediate representations. In contrast, we explicitly evaluate the performance of each component of EVR after training, achieving better faithfulness (Subramanian et al., 2020).
Formal Theorem Prover: Neural components have been used to augment formal theorem proving in several ways. Polu and Sutskever (2020) apply a Seq2Seq neural network for mathematical theorem proving by training the neural network to generate the proof at each step. Some works seek to use distributed representations to augment the rule-based backward chaining (Weber et al., 2019; Dong et al., 2018). However, these works still highly rely on the formal representations and they do not generate the natural language subgoals at each step.
Problem Solver and Cognitive Architectures: Our work is also largely inspired by cognitive architectures such as ACT-R (Anderson et al., 1997) and SOAR (Laird, 2012), which originate from Newell's GPS. These cognitive architectures employ symbolic systems to simulate the human general cognitive processes, but have not been used on
complex reasoning problems in NLP.
Internal Monologue: Internal monologue is the subjective experience of language without overt articulation, and it plays important roles in cognition (Alderson-Day and Fernyhough, 2015). The role of internal monologue in problem solving/reasoning is mixed. Studies have been shown that internal monologue might not be crucial to visual reasoning (Phillips, 1999), whereas in verbal reasoning tasks, some subjects indeed rely more on the internal monologue than the visual imagery (Bacon et al., 2003). However, there is still not a wide consensus on the form/grammar of the internal monologue.
Question Decomposition: Our work is also different from several existing works about question decomposition, where the strategy of decomposition is largely reflected by the question itself (Min et al., 2019; Wolfson et al., 2020). In contrast, the expressions of our questions are already simple, and don't reflect the decomposition strategies.
6.1 Discussion of the Current Method
Does EVR solve the problems raised in Section 2.4? We believe our neural GPS at least partially solves the issues mentioned in Section 2.4. First, EVR decomposes a hard problem into several simple ones, thus resembling the human thinking process more. In addition, this modular strategy also enables EVR to suffer less from the compositionality generalization problem (as shown in Section 4). Second, during each step of reasoning, all the sub-goals and the derived statements are expressed in natural language, thus making the reasoning proofs interpretable.
Could this task be addressed with a pure rule-based approach? Due to the language variations introduced (by the authors) to this dataset, a pure rule-based method is probably not easily applicable. For example, during the generation of pattern 10 training data, we use the provided meta data (formal language, no variation, not available at test time) to help us produce the necessary supervision. We found it otherwise hard to compose the rules directly from the natural language representation.
Can the operator proposer be replaced by a rule-based one? Due to the synthetic natural of the dataset, it is possible to replace the operator proposer in Section 3.3 with some rule-based algorithm (e.g., patterns 1, 3, 5, 7, 9, 11 can be pro
cessed with pure rules). However, this is not the goal of our work. In this paper, we study how well a Seq2Seq model can learn under more realistic conditions, i.e., in real-world scenarios, the agent might only have access to input-output training pairs (rather than rules).
6.2 Challenges of Applying EVR on Real World Problems
While EVR achieved the goals set in this work, several important questions remain for future work:
Limited language variation: Although the data used here contains some variations in language, it is still considerably simpler than real-world natural language. Thus, it remains unanswered whether the Seq2Seq components can achieve a robust performance under actual natural language.
Can other complex problems be reduced to simple ones? It is unclear whether most of the real-world multi-hop reasoning problems can be reduced to a (not too large) set of basic cognitive processes that can be learned.
Non-recursive problems and memory management: Due to the recursive nature of the problem we solve in this paper, the memory access/modification can be simplified. However, it is unknown whether recursive and context-free patterns are the only way in which human think. If not, the access and modification of working memory will become a challenging problem.
Acquisition of training data: Finally, the acquisition of high-quality training data is not always easy in real world. It is possible that low-quality training data introduce dangerous cascading errors.
In this paper we propose the Explainable multihop Verbal Reasoner (EVR) to solve a synthetic question answering problem that requires multihop reasoning (Clark et al., 2020). EVR answers a question by reducing a complex one to several simple ones, and guiding all reasoning steps with natural language for better interpretability. Evaluation of EVR shows it achieves high accuracy, suffer less from the compositionality generalization problem, and generalizes well when training data are not abundant.
Ben Alderson-Day and Charles Fernyhough. 2015. Inner speech: development, cognitive functions, phenomenology, and neurobiology. *Psychological bulletin*, 141(5):931.
John R Anderson, Michael Matessa, and Christian Lebiere. 1997. Act-r: A theory of higher level cognition and its relation to visual attention. Human-Computer Interaction, 12(4):439-462.
Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong. 2019. Learning to retrieve reasoning paths over wikipedia graph for question answering. In International Conference on Learning Representations.
Alison Bacon, Simon Handley, and Stephen Newstead. 2003. Individual differences in strategies for syllogistic reasoning. *Thinking & reasoning*, 9(2):133-168.
Pratyay Banerjee, Kuntal Kumar Pal, Arindam Mitra, and Chitta Baral. 2020. Careful selection of knowledge to solve open book question answering. In 57th Annual Meeting of the Association for Computational Linguistics, ACL 2019, pages 6120-6129. Association for Computational Linguistics (ACL).
Peter Clark, Oyvind Tafjord, and Kyle Richardson. 2020. Transformers as soft reasoners over language. arXiv preprint arXiv:2002.05867.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Honghua Dong, Jiayuan Mao, Tian Lin, Chong Wang, Lihong Li, and Denny Zhou. 2018. Neural logic machines. In International Conference on Learning Representations.
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. 2016. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626):471-476.
Nitish Gupta, Kevin Lin, Dan Roth, Sameer Singh, and Matt Gardner. 2019. Neural module networks for reasoning over text. In International Conference on Learning Representations.
Michael Hahn. 2020. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics, 8:156-171.
Drew A Hudson and Christopher D Manning. 2018. Compositional attention networks for machine reasoning. In International Conference on Learning Representations.
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. 2020. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67:757-795.
Yichen Jiang and Mohit Bansal. 2019. Self-assembling modular networks for interpretable multi-hop reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4464-4474.
Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher. 2016. Ask me anything: Dynamic memory networks for natural language processing. In International conference on machine learning, pages 1378-1387.
John E Laird. 2012. The Soar cognitive architecture. MIT press.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2019. Multi-hop reading comprehension through question decomposition and rescoring. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6097-6109.
Allen Newell. 1966. On the analysis of human problem solving protocols.
Allen Newell. 1994. Unified theories of cognition. Harvard University Press.
Allen Newell, John C Shaw, and Herbert A Simon. 1959. Report on a general problem solving program. In IFIP congress, volume 256, page 64. Pittsburgh, PA.
Louise H Phillips. 1999. The role of memory in the tower of london task. Memory, 7(2):209-231.
Stanislas Polu and Ilya Sutskever. 2020. Generative language modeling for automated theorem proving. arXiv preprint arXiv:2009.03393.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67.
Swarnadeep Saha, Sayan Ghosh, Shashank Srivastava, and Mohit Bansal. 2020. Prover: Proof generation for interpretable reasoning over rules. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 122-136.
Sanjay Subramanian, Ben Bogin, Nitish Gupta, Tomer Wolfson, Sameer Singh, Jonathan Berant, and Matt Gardner. 2020. Obtaining faithful interpretations from compositional neural networks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5594-5608.
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-to-end memory networks. In Advances in neural information processing systems, pages 2440-2448.
Leon Weber, Pasquale Minervini, Jannes Münchmeyer, Ulf Leser, and Tim Roktäschel. 2019. Nlprolog: Reasoning with weak unification for question answering in natural language. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6151-6161.
Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. 2020. Break it down: A question understanding benchmark. Transactions of the Association for Computational Linguistics, 8:183-198.
Vikas Yadav, Steven Bethard, and Mihai Surdeanu. 2019. Quick and (not so) dirty: Unsupervised selection of justification sentences for multi-hop question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2578-2589.
Source code: https://github.com/clulab/releases/tree/master/naacl2021-evr
Computing infrastructure: Experiments are done with Nvidia RTX2070 and Nvidia V100.
Average run time: The running time of EVR1: it takes about 4 to 6 hours to train the three T5 models on RTX2070. On V100 this time is reduced by about  $40\%$ . Evaluation costs a lot of time. EVR1's evaluation on about 20000 test data (depth from 0 to 5) takes about 84 hours. Time using V100 is reduced by about  $40\%$ . This is because for EVR1, every question requires running T5 for many times, and the time grows with depth.
Number of parameters: T5 small:60 million; RT (RoBERTa-large): 355 million; PR: uses RoBERTa (355 million).
Validation performance: Not provided.
Evaluation Measure: Provided in the result section.
Bounds for hyper-parameter Only 3 hyperparameters. The learning rate is set to 1e-4, according to recommendations of other tutorials of using T5. The fact buffer size and rule buffer size are manually tuned.
Number of training and evaluation runs: Only run for one time, because evaluation consumes a lot of time. But random seed is set to 0 to ensure reproducibility.
Hyperparameter Configuration: Provided in paper. Fact buffer size  $= 20$  and rule buffer size  $= 10$ . This is manually tuned.
Statistics of results: Not applicable because models are evaluated for only once.
Number of training samples: Provided.
Data processing: discussed in paper and appendix. For the detailed process please look at our code: https://github.com/clulab/releases/tree/master/naacl2021-evr
Train/Dev/Test spits: provided.
Downloadable link: https://github.com/clulab/releases/tree/master/naacl2021-evr
Name of language: English
Data collection process: Discussed in paper and appendix.
A.2 Full Table of All Data Pattern Generations
Table 9 shows the generation strategies of all the 12 patterns of training data. Note that the pattern 12's generated goal (i.e., "i want to prove [statement]") has exactly the same format as the pattern 1's input goal (i.e., "i want to prove [statement]").
Ptn Generation Specifications
Table 9: The generation strategy for all the 12 patterns of training data. In this verbal reasoning problem, any question with any depth reasoning can be solved by solving these 12 patterns of small problems.
A.3 Details of the Implementation of the Executor
As discussed in section 3.3, the executor has the three functions: (1) it first parses the instructions generated by the operator proposer; (2) it calls the corresponding components according to the parsed instruction; (3) it updates the working memory.
Instruction Parsing: In this paper, the executor needs to determine whether the instruction is "GENERATE_SUBGOALS", or "GET(FACT BUFFER_[i]) THEN RUN(EPISODIC BUFFER, FACT BUFFER_[i])" or "GET(RULE BUFFER_[i]) THEN RUN(EPISODIC BUFFER, RULE BUFFER_[i])". So we simply use "if-elif-else" to judge which instruction is generated.
Handling the Parsing Instruction: The executor calls the corresponding module according to the parsed instruction.
If the instruction is "GENERATE_SUBGOALS", then the executor runs T5 with this instruction (the input-output examples are pattern 2, 4, 8 in Figure 4).
If the instruction is "GET(FACT BUFFER_[i]) THEN RUN(EPISODIC BUFFER, FACT BUFFER_[i])", the executor first gets the text of the facts of FACT BUFFER_[i], then it runs T5 using the text of the current episodic buffer and the text of the retrieved fact buffer (an input-output example is pattern 6 in Figure 4).
If the instruction is "GET(RULE_BUFFER_[i]) THEN RUN(EPISODIC_BUFFER, RULE_BUFFER_[i])", the executor first gets the text of the rules of RULE_BUFFER_[i], then it runs T5 using the text of the current episodic buffer and the text of the retrieved rule buffer (an input-output example is pattern 10 in Figure 4).
Updating Working Memory: After the executor calls the correct modules according to the parsed instruction, new textual information will be generated. If the operator is "GENERATE_SUBGOALS", the generated textual information is a new subgoal, and this new subgoal will be used to replace the old goal in the working memory. If the operator is "GET(RULE_BUFFERER_[i]) THEN RUN(EPISODIC_BUFFERER, RULEbufferer_[i])" and there are matched rules after running this operator, a textual subgoal will also be generated and be used to replace the old subgoal in the working memory.
A.4 Elaborated Description of Pattern 10 Generation
During the generation of pattern 10, we use the formal representations of the facts and rules provided by the data to generate the input-output pairs. Note that this formal representation is only used in the generation of training data and not used at testing time.
All the facts have the formal representation "subject-verb-object". This "S-V-O" combination is called a triple. And all the rules have the formal representation "triple1 (and optionally triple 2) -> triple". For example, the statement "Bob is green" has the formal representation "(\" bob\" \\" is\" \\" green\" \\"+\")" and "All green things are rough" has the formal representation (((\\" something\" \\" is\" \\" green\" \\"+\")) -> (\\" something\" \\" is\" \\" rough\" \\"+\"))". The "+" at the end means the polarity, i.e., whether there is negation in the triple.
Note that there are some variations when this formal representation is converted to natural language. For example, “((\\" something\" \\" is\" \\" green\" \\"+\")) -> (\\" something\" \\" is\" \\" rough\" \\"+\")” can be translated to “all green things are rough” or “if someone is green then it is rough” (or other expressions).
We use the formal representations of facts and rules to determine the formal representation to be generated, and then translate the generated formal representation to natural language. Assume a rule "tripleA and tripleB -> tripleC", we call tripleA and tripleB "preconditions", call tripleC "Effect". Table 10 shows how to generate the preconditions when a rule's effect matches a query. Assume the query has the form " $E_q - V_q - O_q$ " (query subject entity, query verb, query object). Similarly, " $E_p - V_p - O_p$ " means a precondition triple and " $E_e - V_e - O_e$ " means an effect triple. "S" means "something" or "someone". Note that in order for a rule to be matched with a query,  $V_q$  should be the same as  $V_e$  and  $O_q$  should be the same as  $O_e$ .