SlowGuess/NAACL_2021_OCR huggingface dataset:
may not be valid decompositions of the original question. E.g., for the complex question: "When was the 44th US President born?", the sub-question may state "Who was the 44th President from Hawaii?" While this a valid question with the expected intermediate answer, it introduces irrelevant words that would not be possible for the next-question generator to learn.
To filter out such potentially noisy decompositions, we compute three statistics based on nonstopword overlap. We compute the proportion of new words introduced in a decomposition  $u = \{., q_i, a_i, \dots, a_n\}$  that were not in the input question or any of the previous answers, that is:
$$
\theta (u) = \frac {\left| \bigcup_ {i} \{w \in q _ {i} \mid w \notin q c \text {a n d} \forall j <   i w \notin a _ {j} \} \right|}{\left| \{w \in q c \} \right|}
$$
We also compute the number of words from the input question not covered by the decomposition:
$$
\mu (u) = \frac {\left| \{w \in q c \mid \forall i w \notin q _ {i} \} \right|}{\left| \{w \in q c \} \right|}
$$
Lastly, we compute the number of answers  $\nu$  that were not used in any subsequent question, i.e., the sub-question associated with this answer is irrelevant:
$$
\begin{array}{c} \nu (u) = \big | \{a _ {i} \mid \neg (\exists w \in a _ {i} \text {s . t .} w \in q _ {j} \text {w h e r e} j > i \\ \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \text {o r} w \in a _ {n}) \} \big | \end{array}
$$
We only select the decompositions where  $\theta < 0.3, \mu < 0.3, \theta + \mu < 0.4$ , and  $\nu = 0$ . To prevent a single question from dominating the training data, we select up to 50 decompositions for any input question. These hyper-parameters were selected early in the development and gave reasonable results. Minor variations did not have a substantial impact and hence were not tuned on the target set.
A.4 Inference Parameters
We sample  $n_i$  questions in the  $i^{th}$  question decomposition step. To ensure sufficient exploration of the search space, we initially sample a larger number of questions but scale them down every step for efficiency. Due to the pipeline nature of our system, it is difficult for our model to recover from any missed question early in the search. We set the number of sampled questions as  $n_i = N * r^i$  where  $N = 15$  and  $r = \frac{1}{2}$ . When we use greedy inference, we just sample one most-likely question using beam search with width=4. For the QA
E	G	R	DROP	HotpotQA
All EM | F1	Diff F1	Comp F1	Cmpl F1	All EM | F1	Bridge EM | F1	Comp. EM | F1
MODULARQA	✓	✓	✓	86.6 | 87.9	85.2	81.0	96.6	48.5 | 61.8	50.7 | 64.9	39.8 | 49.2
BART	×	✓	×	26.7 | 28.0	7.7	77.5	12.8	38.0 | 49.0	40.4 | 52.9	28.0 | 33.7
NMN-D†	✓	×	×	71.0 | 79.1*	-	-	-	-	-	-
NumNet+V2	×	×	×	90.6 | 91.6	86.5	94.5	95.5	-	-	-
SNMN†	✓	×	×	-	-	-	-	50.0 | 63.1	48.8 | 63.8	54.8 | 60.1
Quark	×	×	×	-	-	-	-	61.7 | 75.5	62.5 | 78.1	58.3 | 64.9
Table 4: Expanded version of the quantitative part of Table 2, reporting both F1 and EM scores in each case. The first three columns, as before, denote qualitative capabilities of each model: whether it can Explain its reasoning, Generalize well to multiple datasets, or Re-use existing QA models.
Figure 5: A sample inference chain scored by MODULARQA for a negation DROP question. For each  $n_0$  question generated in the first step, we will explore  $n_1$  questions in the second step (and so on). We use our scoring function  $w$  to select the optimal inference chain+answer (and prune incomplete low-scoring chains).
models, we always select the most likely answer. When there are multiple input paragraphs (e.g., HotpotQA), we run the QA model against each paragraph independently and select the most likely answer based on the probability.
To score each generated question, we again rely on the same word-overlap statistic used to filter decompositions. We only use the  $\theta$  metric that captures the number of new words introduced in a question chain. The other two metrics are non-motonic i.e they could go down depending on future questions and answers in the chain. At the end, we use a chain scorer (described next) to score each decomposition chain. While we use the  $\theta$  metric to guide the search, we primarily rely on the chain score  $\delta$  to select the right answer. As a result, the final score for a chain  $u$  is a weighted combination of these two metrics with higher emphasis on  $\delta$
$$
\operatorname {s c o r e} (u) = \theta (u) + \lambda \delta (u)
$$
where  $\lambda = 10$  (was set initially during development and not fine-tuned).  $\delta$  can only be computed for a complete decomposition and is set to zero for the intermediate steps. Note that higher this score, the worse the chain i.e. we need to find the chain with the lowest score. This scoring function is monotonically increasing as any continuation of a chain will have the same or higher score. We can thus ignore any partial chains with higher scores once
we find a complete chain with the lowest score.
A.5 Chain Scorer
To train the scorer, we first collect positive and negative chains by running inference with just the  $\theta$  metric. For efficiency, we set the inference parameter  $N = 5$  here. For every complete chain, we compute the F1 score of the final answer with the gold answer. If the F1 score exceeds a threshold (0.2 in our case), we assume this chain to be a positive example. We collect such positive and negative chain examples from the training set and then train a RoBERTa model to classify these chains. We use the RoBERTa model's predicted probability for the negative class as the score  $\delta$ .
A.6 Less Data Training
Since we sample the training data for our next-question generator, we can generate more training data by sampling more questions. When training on  $20\%$  of the training data, i.e. only 2600 questions, we sample 15 questions at each step when we are generating the sub-questions(App. A.2). Similarly we increase  $\mathrm{N} = 10$  to generate more chains for the chain scorer(App. A.5).
Table 4 expands upon the quantitative results in Table 2 and reports both F1 and EM (exact match)
Operation	QA Model, A_C	Question Generator, G_C
diff(X, Y, [Z])	Return absolute difference between X and Y. If Z ∈ {days, months, years}, find the difference in Z.	Generate questions with all possible date/number pairs as X, Y. If Z ∈ {days, months, years} is mentioned in the question, add Z
not(X)	Return 100 - X	Generate questions for every number ≤ 100 as X.
if_then(X[<>≠] Y, Z, W)	If X is [<>≠] Y, return Z else return W	Generate questions with all possible date/number pairs as X and Y. Use pair of entities in the question as Z and W.
Table 5: Set of operations handled by the symbolic calculator model  $\mathbb{A}_{\mathcal{C}}$  and the corresponding approach to generate such questions in  $\mathbb{G}_{\mathcal{C}}$ .
scores in each setting considered.
To identify a subset of questions that might be within the scope of our system, we used the human-authored BREAK decompositions (Wolfson et al., 2020) and filtered out questions that require Boolean operations or list operations. We identified the former by the presence of patterns such as "Which is true" and latter by the presence of plural return terms in the sub-questions.
On this resulting subset of 253 questions, MODULARQA is comparable to the DecompRC system with only a 2pt F1 gap. Out of the 253 in-scope dev questions, DecompRC did not produce a chain at all (i.e., relied on single-hop reasoning) for 79 questions, whereas MODULARQA did produce a chain. In contrast, MODULARQA failed on only 12 questions on which DecompRC succeeded in producing a chain (both systems failed on 3 questions).
We used crowdworkers to annotate the generated explanations $^{22}$  on 155 questions where both systems produced a chain. The annotation covered three dimensions: (1) given the explanation, which system's answer do they trust; (2) which system's explanation do they understand better; and (3) overall, which system's explanation do they prefer (subjective).
D Hints for Complex QA Tasks
To apply Text Modular Networks to any complex QA dataset, we need to be able to extract the hints needed by the sub-task question model. As mentioned earlier, these need not have full coverage or have  $100\%$  precision.
D.1 HotpotQA
The questions  $qc$  in HotpotQA have two supporting gold documents:  $d_{1}$  and  $d_{2}$ . Additionally they are also partitioned into two classes: Bridge and Comparison questions.
D.1.1 Bridge Questions
There are two forms of bridge questions in HotpotQA:
Composition questions: These questions need to first find an intermediate entity  $e_1$  that is referred by a sub-question in HotpotQA. This intermediate entity points to the final answer through the second sub-question. Generally this intermediate entity is the title entity of the document containing the answer. Say  $d_2$  is the document containing the answer and  $d_1$  is the other document. If we are able to find a span that matches the title of  $d_2$  in  $d_1$  and the answer only appears in  $d_2$ , we assume it to be a composition question. We set  $e_1$  to the span that matches the title of  $d_2$  in  $d_1$ .
For the question vocabulary, we could use the terms from the entire question for both steps. Also the second sub-question will use the answer of the first sub-question, so we add it to the vocabulary too. However, we can reduce some noise by removing the terms that are exclusively appear in the other document. The final hints for this question are:
where  $\zeta (q,d_1,d_2)$  indicates the terms in  $q$  that appear in  $d_{2}$  but not in  $d_{1}$
Conjunction questions: These class of questions do not have any intermediate entity but have two sub-questions with the same answer e.g. "Who is a politician and an actor?". If the answer appears in both supporting paragraphs, we assume that it is a conjunction question. The hints for such
22We normalized explanations from both the systems for a fair comparison, e.g., lower-cased our explanations, used the system's answers for both, and converted symbolic terms in both explanations to natural language.
Complex Q	Input Hints z	Output Sub-Questions
qc: How many years did it take the services sector to rebound after the 2002 decrease?	a1: 2002, v1: Φ(qc), p1: p	q1: When did the services sector take a decrease?; t1: S
a2: 2003, v2: Φ(qc), p2: p	q2: When did the services sector rebound?; t2: S
a3: 1, v3: ["diff", "2002", "2003"], p3: p	q3: diff(2002, 2003); t3: C
qc: Which ancestral group is smaller: Irish or Italian?	a1: 12.2, v1: Φ(qc), p1: p	q1: How many of the group were Irish?; t1: S
a2: 6.1, v2: Φ(qc), p2: p	q2: How many Italian were there in the group?; t2: S
a3: 1, v3: ["if_then", "12.2", "6.1"], p3: p	q3: if then (12.2 < 6.1, Irish, Italian); t3: C
qc: How many percent of the national population does not live in Bangkok?	a1: 12.6, v1: Φ(qc), p1: p	q1: What percent of the national population lives in Bangkok?; t1: S
a2: 87.4, v2:[not", "12.6"], p2: p	q1: not (12.6); t2: C
qc: Little Big Girl was a Simpsons episode directed by the animator and artist of what nationality?	a1: Raymond S Persi, v1: ζ(qc, d1, d2), p1: d1	q1: Who directed "Little Big Girl"?; t1: S
a2: American, v2: ζ(qc, d2, d1)+ a1, p2: d2	q1: What nationality was Raymond S?; t2: S
Table 6: Sample hints and the resulting decomposition for DROP and HotpotQA examples. The function  $\Phi$  selects non-stopword words and  $\zeta(q, d_1, d_2)$  selects the words from  $\Phi(q)$  that don't exclusively appear in  $d_2$ .
questions are simple:
$$
p _ {1} = d _ {1}; a _ {1} = a; z _ {1} = \zeta (q c, d _ {1}, d _ {2})
$$
$$
p _ {2} = d _ {2}; a _ {2} = a; z _ {2} = \zeta (q c, d _ {2}, d _ {1})
$$
D.2 Comparison Questions
These questions compare certain attribute between two entities/events mentioned in the question. E.g., "Who is younger: X or Y?". We identify the two entities  $e_1$  and  $e_2$  in such questions and find dates/numbers that are mentioned in documents. For every  $n_1$ ,  $n_2$  number/date mentioned in the document  $d_1$  and  $d_2$  respectively, we create the following hints:
$$
p _ {1} = d _ {1}; a _ {1} = n _ {1}; z _ {1} = \zeta (q c, d _ {1}, d _ {2})
$$
$$
p _ {2} = d _ {2}; a _ {2} = n _ {2}; z _ {2} = \zeta (q c, d _ {2}, d _ {1})
$$
$$
p _ {3} = \phi ; a _ {3} = a; z _ {3} = [ \text {i f \_ t h e n}, n _ {1}, n _ {2}, e _ {1}, e _ {2} ]
$$
The final set of hints would be used by the calculator generator to create the questions: if_then  $(n_{1} > n_{2},e_{1},e_{2})$  and if_then  $(n_{1} < n_{2},e_{1},e_{2})$ .
D.3 DROP
For the questions in DROP, we first identify the class of question that it may belong to and then generate the appropriate hints. Note that one question can belong to multiple classes and we would generate multiple sets of hints in such cases. The questions  $qc$  in DROP have only one associated context  $p$ .
D.3.1 Difference Questions
We identify these questions based on the presence of term indicating a measurement: "how many" and terms indicating difference such as "shorter", "more", "days between", etc. We remove questions that match patterns indicating counting or minimum/maximum such as "shortest", "how many touchdown", etc. Table 7 shows the regexes that must match and ones that must not match for a question to be categorized as a difference question.
Finally we check for two dates or numbers in the context such that their difference (in all units) can lead to the final answer. If these conditions are satisfied, for every pair  $n_1, n_2$  where the difference (in units u) can lead to the final answer, we generate the hints:
$$
p _ {2} = p; a _ {2} = n _ {2}; v _ {2} = \Phi (q c)
$$
$$
p _ {3} = \phi ; a _ {3} = a; v _ {3} = [ \mathrm {d i f f}, n _ {1}, n _ {2}, u ]
$$
D.3.2 Comparison questions
We identify these questions based on the presence of the pattern: “ques:  $e_1$  or  $e_2$ ”(specifically we match “ $([\hat{\mathbf{\Gamma}},] + )[:,]$  (.  $\star$  ) or (.  $\star$  )\?”). We handle them in exactly the same way as HotpotQA. Since DROP contexts can have more dates and numbers, we select numbers and dates that are close to the entity mentioned (Gupta et al.,
Must match
".*how many (dayslmonthslyears).*", ".*how many.*(dayslmonthslyears).* between .*", ".*how many.* shorter.+than .*", ".*how many.* shorter.+compar.*", ".*how many.* longer.+than .*", ".*how many.* longer.+compar.*", ".*how many.* less.+than .*", ".*how many.* less.+compar.*", ".*how many.* more.+than .*", ".*how many.* more.+compar.*", "*difference.*"
Should not match
".*minimum.*", ".*maximum.*" ".*longest.*", ".*shortest.*", ".*highest.*", ".*lowest.*", ".*first.*", ".*last.*", ".*second.*", ".*third.*", ".*fourth.*", ".*how many touchdown.*", ".*how many field goal.*", ".*how many point.*", ".*more touchdown.*", ".*more field goal.*", ".*more point.*"
Table 7: Regexes used to identify difference questions and filter out false positives.
Type	Sub-Type	Description	#Errors
Decomposition	Low Score	RoBERTa chain scorer model returned a lower score for the correct decomposition	5
Incomplete	The decomposition missed a key part of the complex question (e.g. "When did Killgrew marry?" instead of "When did Killgrew marry Catherine?")	3
Sampling	A correct and better scoring decomposition exists but was not generated during the search	3
Long Question	A sub-question exceeded the max token length of 40 set during question generation	2
Missed Decomp	Valid decomposition was not generated	2
Noisy Q	Questions in the generated decomposition were ill-formed	1
QA	Incorrect Ans	SQuAD QA model produced an incorrect answer	3
No Answer	SQuAD QA model produced no answer i	2
Partial Answer	SQuAD QA model produced a partial answer span	1
MathQA format mismatch	Math QA model was unable to handle input format (e.g. if_then (1683-99 > 1591-92, . . .)	4
Out-of-scope	Question can not be handled by our sub-models	2
Dataset	Question makes assumptions not stated in text	2
Table 8: Break down of errors in 30 DROP questions incorrectly answered (F1 < 0.5) by MODULARQA
$$
p _ {1} = p; a _ {1} = n _ {1}; v _ {1} = \Phi (q u e s) + e _ {1}
$$
$$
p _ {2} = p; a _ {2} = n _ {2}; v _ {2} = \Phi (q u e s) + e _ {2}
$$
$$
p _ {3} = \phi ; a _ {3} = a; v _ {3} = [ \text {i f \_ t h e n}, n _ {1}, n _ {2}, e _ {1}, e _ {2} ]
$$
D.3.3 Complementation questions
We identify these questions purely based on the presence of \* not \*.
in the question(specifically we match
"^(.*percent.*)(\Wnot\W|n't\W) (.*)\$"
and a number in the context  $n_1$  such that the  $a = 100 - n_1$ . The hints are pretty straightforward too:
$$
p _ {2} = \phi ; a _ {2} = a; v _ {2} = [ \mathrm {n o t}, n _ {1} ]
$$
E Drop Error Analysis
See Table 8 for the different error types and their counts. Since the search does not explore all possible decompositions, it is possible that there are other decompositions not considered in this analysis. For example, we marked a question to have an error due to "sampling" if running inference again
found a higher scoring, valid decomposition that led to the correct answer. However, it is possible that an exhaustive search would find an invalid decomposition with an even lower score. Similarly the error cases due to "Incomplete" decomposition or "No Valid" decomposition could also be due to sampling issues.