SlowGuess/NAACL_2021_OCR huggingface dataset:
4. Composition (Where was 44th President born?): For such questions(only present in HotpotQA), we need to first find an intermediate entity  $e_1$  that would be the answer to a sub-question in  $qc$  (e.g. Who is the 44th President?). This intermediate entity is used by the second sub-question to get the final answer. Given the two gold paras  $d_1$  and  $d_2$ , where  $d_2$  contains the answer, we use the mention of  $d_2$ 's title in  $d_1$  as the intermediate entity. While we could use the entire complex question vocabulary to create hints, we can reduce some noise by removing terms that appear exclusively in the other document. So the final hints are:
$$
\begin{array}{l} p _ {1} = d _ {1}; a _ {1} = e _ {1}; z _ {1} = \zeta (q c, d _ {1}, d _ {2}) \\ p _ {2} = d _ {2}; a _ {2} = a; z _ {2} = \zeta (q c, d _ {2}, d _ {1}) + e _ {1} \\ \end{array}
$$
where  $\zeta (q,d_1,d_2)$  indicates the terms in  $\Phi (q)$  that appear in  $d_{2}$  but not in  $d_{1}$ .
5. Conjunction (Who acted as  $X$  and directed  $Y$ )? These class of questions do not have any intermediate entity but have two sub-questions with the same answer e.g. "Who is a politician and an actor?" If the answer appears in both supporting paragraphs, we assume that it is a conjunction question. The hints for such questions are:
$$
\begin{array}{l} p _ {1} = d _ {1}; a _ {1} = a; z _ {1} = \zeta (q c, d _ {1}, d _ {2}) \\ p _ {2} = d _ {2}; a _ {2} = a; z _ {2} = \zeta (q c, d _ {2}, d _ {1}) \\ \end{array}
$$
While decomposition datasets such as BREAK could be used to obtain more direct supervision for these hints, we focus here on the broader feasibility of distant supervision. We observe that our current approach generates hints for  $89\%$  of the questions and can find decompositions that lead to the gold answer for  $50\%$  of them. So while the hints cannot
If not found, we ignore such questions.
<sup>9</sup>We use the same for comparison questions in HotpotQA.
be used directly to produce decompositions, the next-question generator is able to generalize from these examples to generate decompositions for all questions with  $81\%$  of them leading to the gold answer. App. D provides more details and example of hints for each question class.
As described earlier, given these input hints and our sub-task question models, we can generate the sub-question for each step and the appropriate sub-model (based on the model that produced this question). We use nucleus sampling to sample 5 questions for each reasoning step. To improve the training data quality, we also filter out potentially noisy decompositions. $^{10}$  We train a BART-Large model, our next-question generator, on this training data to produce the next question given the complex question and previous question-answer pairs.
We use best-first search (Dijkstra et al., 1959) to find the best decomposition chain and use the answer produced at the end of the chain as our predicted answer. We sample  $n_0$  sub-questions from the next-question generator using nucleus sampling. Each question is then answered by the appropriate QA sub-model (defined by the prefix in the question). This partial chain is again passed to the next-question generator to generate the next  $n_1$  sub-questions, and so on.[11] A chain is considered complete when the next-question generator outputs the end-of-chain marker [EOQ].
We define a scoring function that scores each partial chain  $u$  based on the new words introduced in the sub-questions compared to the input question. For a complete chain, we additionally add the score from a RoBERTa model trained on randomly sampled chains (chains that lead to the correct answer are labeled as positive). Concretely, we use the negative class score from this classifier,  $\delta(u)$ , to compute the final chain score as  $\theta(u) + \lambda \delta(u)$ , i.e., lower is better.
To evaluate our modular approach, we use two datasets, DROP and HotpotQA, that contain ques
tions answerable using a SQuAD model and a math calculator. We identify 14.4K training questions in DROP that are within the scope of our system, $^{14}$  which forms  $18.7\%$  of the dataset. $^{15}$  We similarly select 2973 Dev questions (from 9536), and split them into 601 Dev and 2371 Test questions.
We evaluate our system on the entire HotpotQA dataset. Since the test set is blind, we split the Dev set (7405 qns.) into 1481 Dev and 5924 Test questions. For training, we only use  $17\%$  of the training dataset containing 15661 questions categorized as "hard" by HotpotQA authors.[16]
5.1 Explanation and Interpretability
A key aspect distinguishing MODULARQA is that it can explain its reasoning in a human-interpretable fashion, in the form of simpler sub-questions it creates via decomposition. Table 1 illustrates six sample reasoning explanations; the question context and sub-models are omitted for brevity. We see that MODULARQA is able to take oddly phrased questions to create clean sub-questions (example 4), handle yes/no questions (example 6), recognize the unit of comparison (example 1), and map the phrase "smaller" to the appropriate direction of comparison without any manual rules (example 2).
Analyzing such explanations for 40 Dev questions (20 from each dataset), we found that among the 28 questions MODULARQA answered correctly, it produced a valid reasoning chain in as many as  $93\%$  of the cases, attesting to its strong ability to provide understandable explanations.
To further assess the human readability of MODULARQA's explanations, we compared them with those produced by DecompRC (Min et al., 2019b), the only decomposition-based system for the considered datasets. We identified 155 questions that are within the scope of MODULARQA and for which both systems produce a decomposition.[17] We then asked crowdworkers on Amazon Mechanical Turk to annotate them along three dimensions: (1) given the two explanations, which system's answer do they trust more; (2) which system's explanation do they understand better; and (3) which system's explanation do they generally prefer.
$^{10}$ if an intermediate answer is unused or vocabulary of question chain is too different from the input question. See Appendix A.3 for more details.
11To enable early exploration, we use exponential decay on the number of generated questions:  $n_i = 10 / 2^i$
$^{12}\theta(u) = \#$  new words/#words in input question
13For more details, refer to App. A.4.
14See App. D for how this subset is automatically identified.
15Previous modular systems (Gupta et al., 2020) have targeted even smaller subsets to develop modular approaches.
16Increasing the training set didn't affect performance.
17DecompRC failed to produce chains on 6x more questions than our system. See App. C for details on how these questions were selected and how they were normalized.
DROP F1	HotpotQA F1
All	Diff	Comp	Cmpl	All	Br	Comp
Interpretable Cross-Dataset Models (§5.2)
MODULARQA	87.9	85.2	81.0	96.6	61.8	64.9	49.2
WordOverlap	80.5	82.5	58.3	95.8	57.5	61.7	40.5
Greedy	60.2	52.2	52.9	76.3	42.4	44.8	33.0
Limited Versatility (§5.3)
NMN-D†	79.1*	-	-	-	-	-	-
SNMN†	-	-	-	-	63.1	63.7	60.1
DecompRC†	-	-	-	-	70.3	72.1	63.4
Limited Interpretability (§5.4)
NumNet+V2	91.6	86.5	94.5	95.5	-	-	-
Quark	-	-	-	-	75.5	78.1	64.9
Table 2: F1 scores on the DROP and HotpotQA questions and the individual classes: Difference(Diff), Comparison(Comp), Complementation(Cmpl) and Bridge(Br). TOP: Comparison to variations of MODULARQA that work across datasets. MIDDLE: Comparison to targeted interpretable systems. BOTTOM: Comparison to targeted blackbox systems. MODULARQA is competitive with previous approaches on DROP and mainly lags behind systems on HotpotQA that are able to exploit artifacts.
Trust	Understand	Prefer
DecompRC	50 (33%)	34 (22%)	49 (32%)
MODULARQA	105 (67%)	121 (78%)	106 (68%)
Table 3: Human evaluation of the explanation quality. Across all dimensions, crowdsourced workers preferred the explanations of MODULARQA over DecompRC.
Table 3 summarizes the aggregate statistic of the majority labels, with 5 annotations per question. Crowdworkers understood MODULARQA's natural language explanations better in  $78\%$  of the cases, trusted more that it pointed to the correct answer, and generally preferred its explanations.
5.2 Interpretable Cross-Dataset Models
With MODULARQA being the first interpretable model for DROP and HotpotQA, there were no comparable existing cross-dataset systems. We instead consider two baselines obtained by modifying MODULARQA: (1) only the word-overlap based scoring function  $\theta(u)$  for chains (no RoBERTa classifier); and (2) greedy inference, i.e., use the most likely question at each step (no search).
As shown in Table 2 (top rows), MODULARQA outperforms the purely word-overlap based approach by 7pts F1 on DROP and 4pts on HotpotQA. A simple coverage-based decomposition is thus not as effective, although HotpotQA suffers less because of decompositions being explicit in it.[18] Performance drops much more heavily (18pts on DROP and 19pts on HotpotQA) when we do not employ search at all. This is primarily because
the optimal sub-question can often be unanswerable by the intended sub-model while an alternate decomposition may lead to the right answer.
5.3 Comparison to Dataset-Specific Models
To assess the price MODULARQA pays for being versatile, we compare it to three interpretable systems that target a particular dataset. Two are Neural Module Networks, with modules designed specifically for a subset of DROP (referred to as NMND) (Gupta et al., 2020) and for HotpotQA (referred to as SNMN) (Jiang and Bansal, 2019). The third is DecompRC, whose split-based decomposition, human annotations, as well as answer composition algorithm was specifically designed for HotpotQA.
As seen in Table 2 (middle rows), MODULARQA actually substantially outperforms the DROP model NMN-D while being able to produce textual explanations (rather than attention visualization). On the HotpotQA dataset, MODULARQA is comparable to S-NMN but underperforms compared to DecompRC. Note that DecompRC can choose to answer some questions using single-hop reasoning and potentially exploit many artifacts in this dataset (Min et al., 2019a; Trivedi et al., 2020).
5.4 Comparison to Black-Box Models
To assess the price MODULARQA pays for being interpretable, we compare it to two state-of-the-art black-box systems that not only lack interpretability but are also targeted towards specific datasets: NumNet+V2 (Ran et al., 2019) for DROP
<sup>19</sup>Since NMN-D focuses on a different subset, we report its score on the shared subset, on which MODULARQA achieves an F1 score of 92.5 (not shown in the table)
<sup>18</sup>Recall that our word-overlap based score penalizes missed question words and words introduced during decomposition.
and Quark (Groeneveld et al., 2020) for HotpotQA. Since we use the SQuAD QA system in our model, we first fine-tune the LM in both of these systems on the SQuAD dataset, and then train them on the same datasets as MODULARQA.
As seen in Table 2 (bottom rows), we are competitive with the state-of-the-art model on DROP but underperform compared to the Quark system. Note that Quark relies on supporting fact annotation and trains a single end-to-end QA model, thereby being more likely to exploit dataset artifacts.
Upon analyzing MODULARQA's errors (defined as questions with F1 score under 0.5) on HotpotQA, we found  $65\%$  of the errors arise from intermediate questions having multiple or yes/no answers. These are not handled by modules in our current implementation, suggesting a path for improvement.
We also analyzed the errors on the DROP dev set and identified question decomposition (53.3%) and QA models (33.33%) as the main sources of error.[20] Within question decomposition, the key cause of error is higher RoBERTa score for an incorrect decomposition (50% of errors). Both the SQuAD and Math QA models were responsible for errors, with the latter erring only due to out-of-scope formats (e.g., date ranges 1693-99). Appendix E provides more details.
5.5 Additional Benefits of TMNs
The last set of experiments support two distinct benefits (besides interpretability) of our approach even against state-of-the-art black-box models.
Higher Robustness. We evaluate on the DROP contrast set (Gardner et al., 2020), a suite of test-only examples created for assessing robustness via minimally perturbed examples. On the 239 (out of 947) questions that are within our scope using the same logic as before, we find that MODULARQA outperforms NumNet+V2 by  $7\% -10\%$ :
Contrast Test	EM	F1
MODULARQA	55.7	63.3
NumNet+V2	45.2	56.2
Learning with Less Data. We next evaluate the sample efficiency of MODULARQA by considering training sets of 3 different sizes:  $100\%$ ,  $60\%$ , and  $20\%$  (14448, 8782, and 2596 questions, resp.) of the training questions selected for DROP.[21] As
shown below, the gap (in F1 score) between MODULARQA and the state-of-the-art model steadily shrinks, and MODULARQA even outperforms it when both are trained on  $20\%$  of the data.
Portion of Train set	100%	60%	20%
MODULARQA	87.8	89.3	87.0
NumNet+V2	91.6	88.3	85.4
We introduced Text Modular Networks, which provide a general-purpose framework that casts complex tasks as textual interaction between existing, simpler QA modules. Based on this conceptual framework, we built MODULARQA, an instantiation of TMNs that can perform multi-hop and discrete numeric reasoning. Empirically, MODULARQA is on-par with other modular approaches (which are dataset-specific) and outperforms a state-of-the-art model in a limited data setting and on expert-generated perturbations. Importantly, MODULARQA provides easy-to-interpret explanations of its reasoning. It is the first system that decomposes DROP questions into textual sub-questions and can be applied to both DROP and HotpotQA.
Extending this model to more question classes such as counting ("How many touchdowns were scored by X?") and Boolean conjunction ("Are both X and Y musicians?") are interesting avenues for future work. To handle the former class, the first challenge is building models that can return a list of answers—a relatively unexplored task until recently (Hu et al., 2019b; Segal et al., 2020). For Boolean questions, the challenge is identifying good sub-questions as there is a large space of questions such as "Did musicians work for X?" that may have the expected yes/no answer but are not part of the true decomposition. Semantic parsing faces similar issues when questions have a large number of possible logical forms (Dasigi et al., 2019). Finally, end-to-end training of the next-question generator and QA models via REINFORCE (Williams, 1992) can further improve the score and allow for faster greedy inference.
We thank the Aristo team at AI2 for helpful input, Beaker team for their support with experiments, Dirk Groeneveld for providing the output of the Quark system for evaluation, and Jonathan Berant, Matt Gardner, and Hanna Hajishirzi for invaluable feedback on initial drafts of this paper.
20Remaining errors are due to dataset and scope issues. 21For simplicity, we train MODULARQA on the DROP questions only here. To obtain sufficient examples, we increase the number of questions sampled for each decomposition step. See App. A.6 for more details.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Learning to compose neural networks for question answering. In NAACL-HLT.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic Parsing on Freebase from Question-Answer Pairs. In EMNLP.
Gino Brunner, Y. Liu, Damián Pascual, Oliver Richter, and Roger Wattenhofer. 2020. On identifiability in transformers. In ICLR.
Pradeep Dasigi, Matt Gardner, Shikhar Murty, Luke Zettlemoyer, and Eduard Hovy. 2019. Iterative search for weakly supervised semantic parsing. In NAACL.
Edsger W Dijkstra et al. 1959. A note on two problems in connexion with graphs. Numerische mathematik, 1(1):269-271.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In *NAACL*.
Yuwei Fang, Siqi Sun, Zhe Gan, Rohit Pillai, Shuo-hang Wang, and Jingjing Liu. 2020. Hierarchical graph network for multi-hop question answering. In EMNLP.
Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Quan Zhang, and Ben Zhou. 2020. Evaluating NLP models via contrast sets. In Findings of EMNLP.
Dirk Groeneveld, Tushar Khot, Ashish Sabharwal, et al. 2020. A simple yet strong pipeline for hotpotqa. In EMNLP.
Nitish Gupta, Kevin Lin, Dan Roth, Sameer Singh, and Matt Gardner. 2020. Neural module networks for reasoning over text. In ICLR.
Sanda Harabagiu and Andrew Hickl. 2006. Methods for using textual entailment in open-domain question answering. In ACL.
Chris Hokamp and Qun Liu. 2017. Lexically Constrained Decoding for Sequence Generation using Grid Beam Search. In ACL.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In ICLR.
J Edward Hu, Huda Khayrallah, Ryan Culkin, Patrick Xia, Tongfei Chen, Matt Post, and Benjamin Van Durme. 2019a. Improved Lexically Constrained Decoding for Translation and Monolingual Rewriting. In NAACL.
Minghao Hu, Yuxing Peng, Zhen Huang, and Dong sheng Li. 2019b. A multi-type multi-span network for reading comprehension that requires discrete reasoning. In EMNLP.
Alon Jacovi and Y. Goldberg. 2020. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In ACL.
Yichen Jiang and Mohit Bansal. 2019. Self-assembling modular networks for interpretable multi-hop reasoning. In EMNLP-IJCNLP.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In *NAACL*.
Tushar Khot, Peter Clark, Michal Guerquin, Paul Edward Jansen, and Ashish Sabharwal. 2020. QASC: A dataset for question answering via sentence composition. In AAAI.
Louis Kirsch, Julius Kunze, and David Barber. 2018. Modular networks: Learning to decompose neural computation. In NeurIPS.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In ACL.
Percy Liang, Michael I Jordan, and Dan Klein. 2013. Learning dependency-based compositional semantics. Computational Linguistics, 39(2):389-446.
John McCarthy. 1988. Epistemological challenges for connectionism. Behavioral and Brain Sciences, 11(1):44-44.
Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019a. Compositional questions do not necessitate multi-hop reasoning. In ACL.
Sewon Min, Victor Zhong, Luke S. Zettlemoyer, and Hannaneh Hajishirzi. 2019b. Multi-hop reading comprehension through question decomposition and rescoring. In ACL.
Dan I. Moldovan, Sanda M. Harabagiu, Marius Pasca, Rada Mihalcea, Roxana Girju, Richard Goodrum, and Vasile Rus. 2000. The structure and performance of an open-domain question answering system. In ACL.
Arvind Neelakantan, Quoc V Le, Martin Abadi, Andrew McCallum, and Dario Amodei. 2017. Learning a natural language interface with neural programmer. In ICLR.
Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela. 2020. Unsupervised question decomposition for question answering. In EMNLP.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP.
Qiu Ran, Yankai Lin, Peng Li, Jie Zhou, and Zhiyuan Liu. 2019. Numnet: Machine reading comprehension with numerical reasoning. In EMNLP-IJCNLP.
Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. 2018. Routing networks: Adaptive selection of non-linear functions for multi-task learning. In ICLR.
Subhro Roy and Dan Roth. 2018. Mapping to declarative knowledge for word problem solving. TACL, 6:159-172.
Elad Segal, Avia Efrat, Mor Shoham, Amir Globerson, and Jonathan Berant. 2020. A simple and effective model for answering multi-span questions. In EMNLP.
Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi, Oren Etzioni, and Clint Malcolm. 2015. Solving geometry problems: Combining text and diagram interpretation. In EMNLP.
Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In ACL.
Paul Smolensky. 1988. On the proper treatment of connectionism. Behavioral and brain sciences, 11(1):1-23.
Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. In *NAACL*.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2020. Is multihop qa in dire condition? measuring and reducing disconnected reasoning. In EMNLP.
Ming Tu, Kevin Huang, Guangtao Wang, Jui-Ting Huang, Xiaodong He, and Bufang Zhou. 2020. Select, answer and explain: Interpretable multi-hop reading comprehension over multiple documents. In AAAI.
Sarah Wiegrefe and Yuval Pinter. 2019. Attention is not not explanation. In EMNLP/IJCNLP.
Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256.
Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. 2020. Break it down: A question understanding benchmark. TACL, 8:183-198.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP.
A Model Settings
Each BART-Large model (406M parameters) is trained with the same set of hyper-parameters - batch size of 64, learning rate of 5e-6, triangular learning rate scheduler with a warmup of 500 steps, and training over 5 epochs. Each RoBERTa model is trained with the same set of hyper-parameters but a smaller batch size of 16. We selected these parameters based on early experiments and did not perform any hyper-parameter tuning thereafter. All the baseline models are trained with their default hyper-parameters provided by the authors. All the experiments are performed on single-GPU machines (either V100 GPUs or RTX 8000s).
We always used nucleus sampling to sample sequences from the BART models. To sample the sub-question using the SQuAD sub-question generator, we sampled 5 questions for each step with  $p = 0.95$  and max question length of 40. To sample the question decompositions during inference, we additionally set  $k = 10$  to reduce the noise in these questions.
A.1 Training SQuAD Question Generator
We use the SQuAD 2.0 answerable questions to generate the training data for our SQuAD question generator. We use the nouns, verbs, nouns, adjectives and adverbs (pos tags=[NOUN, VERB, NUM, PROPN, ADJ, RB]) from the question to define the vocabulary hints (after filtering stop words). To simulate the noisy vocabulary, we also add distractor terms with similar pos tags from other questions from the same paragraph. We sample  $j \in [2, \dots, 7]$  distractor terms for each question and add them to the vocabulary hints.
A.2 Generating sub-questions
For every step in the reasoning process, we generate 5 questions using nucleus sampling. We select the questions that the corresponding sub-model is able to answer correctly. For each sub-question, we generate 5 questions in the next step (and so on). At the end, we select all the successful question chains (i.e each sub-question was answered by the sub-model to produce the expected answer at each step).
A.3 Selecting Question Decompositions
It is possible that some of these sub-questions, while valid answerable questions, introduce other words mentioned in the paragraph. However, these