Chain-of-Questions Training with Latent Answers for Robust Multistep Question Answering
Authors: Wang Zhu
Jesse Thomason
Robin Jia
Venue: EMNLP 2023 Main (2023)
Keywords: multistep reasoning
question answering
latent variable learning
TL;DR: We propose Chain-of-Questions, a framework that generates sub-questions and sub-answers one at a time, trained by latent answer learning.

Abstract: We propose Chain-of-Questions, a framework that trains a model to robustly answer multistep questions by generating and answering sub-questions. We obtain supervision for sub-questions from human-annotated question decomposition meaning representation (QDMR),
but QDMR does not include annotated answers to sub-questions. To overcome this technical challenge, we treat sub-answers as latent variables and infer them with a novel dynamic mixture of Hard-EM and MAPO. Chain-of-Questions is effective and robust, greatly outperforming strong neuro-symbolic methods by 9.0 F1 on a DROP contrast set and GPT-3.5 by 24.3 F1 on a HotpotQA adversarial set.

Chain-of-Questione Training with Latent Answers for Robust Multistep Question Answering
Wang Zhu
Jesse Thomason Robin Jia
University of Southern California, Los Angeles, CA, USA (wangzhu, jessetho, robinjiaJ@usc.edu
Abstract
Single-step QA Approach
Which side suffered more casualties, the Federales the Americans?
Question:
We propose Seain-of-Quuetions framework that trains model to robustly answer multi- step questions by generating and answering sub-questions. We obtain supervision for sub- questions from annootttd question de- composition meaning representation (QDMR), but QDMR does include annotated answers to sub-questions. To overcome this technical challenge, we treat sub-answers as latent vari- ables and infer them with novel dynamic mixture of Hard-EM and MAPO. Chain-of- Questions effective and robust, greatly out- performing strong neuro-symbolic methods by 9.0 FI on a DROP contrast set and GPT-3.55 by 24.3 Fl on HOTPOTQA adversarial set.
Context:
12 troops were killed
taken prisoner; 45 Federales were casualties...
T5
Answer: Americans
Chain-of ooeeorttioueussssseeeeee Approach
Which side suffered more casualties, the Federales or the Americans?
Question:
Context:
12 troops were killed
taken prisoner; 45 Federales were casualties...
T5
Sub- Qee-tuuustiinnn retuun casualties of the Federales? Sub-Answer 1: 45
Sub-Question 2: return casualties the Americans? Sub-Answer
T5
1 Introduction
T5
Final Question: return which side was more Federales (45), Americans (12) Final Answer: Federales
Multistep question answering (QA) poses a rea- soning challenge that current Mtte-offtheear QA models have not fully addressed. Strong fine- tuned QA models like UnifiedQA (Khashabi et al., 2020a) can achieve impressive results on various QA tasks through multitask training, but exhibit subpar performance on multistep reasoning. More- over, because some multistep reasoning bench- marks contain annotation artifacts or reasoning shortcuts (Jiang and Bansal, 2019), dedicated mod- els trained these benchmarks often have much lower Fl performance contrast sets (Gardner al., 2020) and adversarial sets (Schlegel al., 2021), indicating their lack of robustness.
Figure I: Single-step QA vs. Chain-of caterooof-uussooon....m We show that single-model with sub-question generation and answering works better than single-step QA on questions that require multistep reasoning.
of multistep QA (Fu et al., 2021; Perez et al., 2020), and thus cannot be easily extended to other multi- step QA settings. Prompting methods (Chen et al., 2020; Dua et al., 2022) have shown promise in gen- erating multistep solutions to questions, but they require very large language models (LMs) as well as careful prompt engineering, and still lag behind fine-tuned methods (OpenAI, 2023).
Prior research has attempted to tackle this chal- lenge with various question decomposition strate- gies to explicitly incorporate reasoning chains into the question answering process. However, as show in experiments, existing methods (An- dor et al., 2019; Chen et al., 2020) that perform explicit reasoning steps still suffer from robustness issues. Moreover, multi-step reasoning methods are often engineered for a specific domain or type
To develop robust multistep QA system, we propose novel framework, Chain-of-Questioon training with latent answers. Our framework trains model to generate sub-questions and their cor- responding sub-answers one time, as shown in Fig. 1, then aggregates those sub-answers to an- swer the original question. To define an appropriate

---

set of sub-questions, we use question decomposi- tion meaning representation (QDMR), an existing dataset with human-annotated sub-questions for questions from multiple multistep QA benchmarks. While QDMR is helpful, it only contains anno- tated sub-questions, not sub-answers, which makes training a QA system to generate sub-answers tech- nically challenging. We view the sub-answers in the intermediate steps as latent variables, and apply Hard-EM (Neal and Hinton, 1998) to op- timize these latent variables during training. To further improve performance, we use a memory buffer to store trajectories with high FI score, in- spired by Memory-Augment
datasets. Gardner et al. (2020) created contrast set for DROP (DROP-CS) by modifying test in- stances in ways that often change the correct an- swer. HOTPOTQA-ADY. (Jiang and Bansal, 2019) adds adversarial paragraphs that do not change the correct answer but fool models that rely too heavily on reasoning shortcuts. We experiment on DROP, HOTPOTQA, and their robustness evaluation sets.
Training with Question Decomposition. Wolf- son et al. (2020) introduce QDMR, human- annotated question decomposition format and dataset. Since QDMR's for each question were annotated without looking at evidence passages. the QDMR dataset does not include sub-answers to any sub-questions within each QDMR. Subsequent work has used QDMR to help train QA models. TeaBReaC (Trivedi et al., 2022b) uses the QDMR decomposition graph to generate large multistep QA dataset with synthetic contexts for pretraining, We show that TeaBReaC has complementary bene- fits with Chain-of-Queste Questions, which adds explicit multistep reasoning inference time. Guo et al. (2022b) train a model to generate QDMR's s and provide them as context to a single-step QA model. Unlike this model, our model explicitly attempts to generate sub-answers of QDMR sub-questions. We compare with a single-step baseline similar to Guo et al. (2022b), and show that learning to generate sub-answers improves performance.
Policy Optimiza- tion (MAPO; Liang et al., 2018), 8), previously used for semantic parsing. Because starting with MAPO alone does not converge well, we design dy- namic loss function that combines the Hard-EM and MAPO objectives for fast improvement at the beginning and better final convergence.
We conduct experiments on DROP (Dua et al., 2019), HOTPOTQA (Yang et al., 2018), and their contrast and adversarial sets to evaluate the perfor- mance of our proposed Chain-of-Questions frame- work. On the contrast set of DROP, Chain- of-Questions outperforms neuro-symbolic base- lines by 9.0 on Fl score, and outperforms Chain- of of-Thotgh on GPT-3.5 by 16.8 despite using a much smaller model (Tr-Large, 770M parameters). On the adversarial set of HOTPOTQA, Chain-of- Questions outperforms Longformer by 5.5 on Fl score, and outperforms Chain-of-Thought Thought on GPT- 3.5 by 24.3. Our experimental results demonstrate that -aaustof-Qusssions successfully leverages ex- isting QDMR annotations train an effective and robust multistep QA model.
Other work on multistep QA, such as Decom- pRC (Min et al., 2019b), ONUS (Perez et al., 2020) and RERC (Fu et al., 2021), generate a decompo- sition with one model and the answers for the sub- questions with another model, although none of these use QDMR. These methods require a single- step QA model trained with other QA data, whereas our approach does not. Moreover, they rely on entity matching to decompose questions; such ap- proaches do not naturally extend tasks requiring forms of multistep reasoning that are not entity- centric, such as numerical reasoning.
2 Background and Related Work
We introduce the multistep QA benchmarks we use, well as other methods using question decompo- sition during training and prompting.
Multistep QA Benchmarks. We focus on two popular multistep QA benchmarks-DROW and HOTPOTQA. DROP (Dua et al., 2019) focuses on questions that require discrete and symbolic reasoning. Most of its questions require mul- tiple steps of retrieval and numerical execution. HOTPOTQA (Yang et al., 2018) contains 2-hop questions over 10 paragraphs. Other work has constructed contrast and adversarial sets to eval- uate the robustness of models trained on these
Neuro-symbolic methods such as BERT- Calculator (Andor et al., 2019) and NeRd (Chen et al., 2020) generate functional programs for mul- tistep numerical reasoning. However, they require the model generate accurate programs in single run, without observing the results of intermedi- ate stages of computation. This process can make them more susceptible to learning simple reasoning shorteuts, compared with Chain-of Chain-of-Questions

---

Input
Output
Training
Question: Which side suffered more casualties, Federales or Americans? Context: 12 US troops... ..
reuuateee casualties of Federales [QDMRANS5]45)
Beam search
Replay buffer
Run #1:
@ From the top-k samples, add those with F1 no less than 0.8 to the buffer
Question: Federales
Append
Run #2: Context: 12 US troops... ...
[QDMR] return casualties of Americans
[QDMR] [QDMR-ANS] (45)
[QDMR-AN5[12)
@ Select out out-of-ig of-memory examples in top-5, sample most in in-memory examples from the buffer, compute the MAPO loss
Question: Which suffered casualties, the Federales Americans?
Append
troops...
Run #3: [QDMR] return
(QDMR) widch more of [45), (12)
[45]
BND-QDMR] (QDMR-ANS](eederale)
① Sample top-k candidates, select the best to compute the Hard-EM loss
[QDMR] return casualties of the Americans [QDMR-ANS]
Chain-of- -otbb--o-Qussionss framework and its training process with Hard-EM and MAPO. In the left panel, the blue box contains the input question and context, the pink box shows the intermediate sub-questions and sub-answers, the orange box the final sub-question and final answer. Words in braces { are sampled as latent variables, and the other words learned with supervision. In the right panel, during training, we first select the best of top-k candidates to compute the Hard-EM loss, then we combine these candidates with samples from the buffer compute the MAPO loss. Finally, add high-reward candidates into the buffer.
Figure 2:
Prompting with Question Decomposition. Chain-of-Though prompting (Wei et al., 2022) inserts explicit reasoning chains into prompts to help language models answer compositional and multistep questions, especially ones involving mathematical reasoning. Subsequent work such as successive (Dua al., 2022; Zhou et al., 2023), it- erative (Zelikman al., 2022), modularized (Khot et al., 2022) or tool-based prompting (Yao et al., 2023) filter or refine the reasoning process. However, LLMS are computationally expensive and require careful prompt engineering. We show that smaller, more efficient LMs outperform LLMs when trained output reasoning chains.
QDMR Parser. We assume access to a QDMR parser model 9(: Φ). parameterized by o, that takes in question and generates a corresponding QDMR, proposed by Wolfson et al. (2020). The parser is trained with the original QDMR annota- tion, and can predict QDMR for any new question.
3.2 Multistep Question Answering
A QA model takes in a question and context pas- sage and outputs predicted answer à. We train model training dataset of question-context answer triples (q, c, a). Our multistep QA model answers a question q by generating and answering QDMR sub-questions one at a time.
3 Problem Formulation
Training Data. We assume that all QA train- ing examples have a QDMR sub corresponding to question q. However, the Human-annoturl QDMR dataset only has gold QDMR's annotated for a small fraction of QA examples. We con- sider two settings: (1) only use examples with gold QDMR data for QA training (DQDMR): (2) we use the QDMR parser to generate silver QDMR for the rest of the QA training data (DQDMR+). The sub-answers sub [a .a ...., (ib' to each sub-question are included in the QDMR, QDMR annotators only looked at the question without supporting passages. We use auu [aauh suhh .-.., اريية to denote the model predicted
We define the notations related to the question de- composition annotation QDMR, as well as the mul- tistep QA training and testing setups.
3.1 QDMR
Formally, QDMR for a question is a corre- sponding list of natural language sub-questions quu = ln sub .... qn (dn' where answering each sub-question would lead to answering q (Wolf- son et al., 2020). The number of sub-questions n = 1qد" varies for different questions q; requires multistep reasoning, n is usually > 1.

---

sub-answers. At inference time the model is given only the question q and context e as input, and must generate both sub-questions and sub-answers.
intermediate sub-answers as latent variables and use Hard-EM and RL optimize them.
Hard-EM. variant of the EM algorithm, Hard- EM (Neal and Hinton, 1998) assigns the most likely values to all latent variables and maximizes the expected log-likelihood of the label based on these values. Hard-EM helps to filter spurious ways derive the correct answer. Min al. (2019a) use Hard-EM for weakly supervised training for multi- mention QA tasks.
4 Proposed Methods
We discuss how to use the QDMR annotation and the QDMR parser, as well how train the QA model with Hard-EM and reinforcement learning in this section. We combine question decompo- sition and latent variable learning to construct a generalizable multistep reasoning framework.
Since it is computationally infeasible to enumer- ate all possible sets of sub-an answers to find the best set, we approximately compute the best asub with beam search (see Appendix A. for details), In particular, we pick the sequence of sub-answers asub where anth and a @im-1 has the highest likelihood to predict
4.1 Chain-of-Questinn Framework
Our model ((;0) predicts each QDMR sub- question and its corresponding sub-answer one at time. During training, we feed the question and context into our model of Fig. 2, and first ask it predict the first sub- question, 91th along with its corresponding sub- answer, aiub. To separate the sub-question and the uub-answer, we append a special [QDMR] token to the sub-question and another special [QDMR-ANS] token to the sub-answer in the model input and out- put. In the second iteration, we append the gold qlub and predicted ànك after the question and con- text, thereby allowing the model to predict the sec- ond sub-question and its sub- sub-answer based on pre- vious steps. This process is repeated until all sub- questions and sub-answers have been predicted. In the final iteration, the model outputs a [END-QDMR] tag indicating the end of iterations. We use the final sub-answer àou as the answer to the input question.
in the blue box
iin-1 arg max pa(a c, C, 9thh, ;imm-1)-
FIn-1
Following Hard-EM, we train the model to maxi- mize the probability of both quub and aaub, which is equivalent to minimizing negative log likelihood:
FIEMM (0, q, c, q sub asuby
= log pa (q asub
log pa (agub
sub sub
q. c, 9i:j Big
==1
+ (SL 4, a.
(2)
Reinforcement Learning. In another perspec- tive, we view each sub-answer as an action and the whole sub-answer set as a trajectory. By do- ing so, we can reinforcement learning methods such as Memory-Augmentted Policy Optimization (MAPO: Liang al., 2018), originally designed for semantic parsing, to optimize the latent sub- answers. MAPO reduces the variance of policy gradient estimates with a memory buffer that stores high-reward trajectories (in our case, sequences of predicted sub-answers).
4.2 Learning with Latent Answers
We illustrate how the model f learns to predict the sub-questions and sub-answers. The sub-quese questions are provided in the annotation, ply supervised learning to optimize the likelihood of ground-truti sub-questions given the question, the context and any sub-answers as in Eq. (1).
can simply ap-
(0, q, c, ahh)
We adapt MAPO to multistep QA as follows. While the original MAPO algorithm samples many independent trajectories using the model f. we in- stead use the trajectories from the beam, which reduces sampling time and yields better quality trajectories. During training, we maintain a replay buffer B of high-quality sequences of predicted sub- answers. For each example (g, c, a), we choose the
- log po b
q, e, -19-1 1 a15-1), (I)
sub asub
suh where Tig denotes the set {q." |= and likewise for ai.y Notice that the model is trained generate the gold next sub-question regardless of whether its previous predicted sub-answers are correct (Bengio et al., 2015). Because the ground truth sub- answers are not provided the training time, we regard the
See details of the original MAPO in Appendix A.2

---

Testing Data (FI)
Model
Method
Training Data
DROP
DROP-CS
DERT-Calculan
DROP
81.7
55.8
BERT
NeRd
DROP
81.8
59.5
Chain-of-
Thaou-of Thought (4-shot)
59.7
51.7
GPT-3.5
Standard Prompting (0-shot)
40.4
34.1
DROP
51.6
44.8
Single-step
DROP DoDMR-
45.0
T5-B
DROP w/ DoNME
46.6
44.3
DROP w/ DODMR-
74.4
63.8
Chain-of-Questions
MAPO
73.7
62.9
Regex
59.1
56.4
DROP
73.9
53.7
Single-step run
DROP w/ DODMR.
75.2
55.3
TS-L
DROP w/ DODMR
65.8
55.4
DROP w/ DOUMR+
84.4
67.9
Chain-of-Questions
w/o MAPO
83,5
67.8
w/o Regex
78.1
64.4
Single-step run Chain-of-Qucstions
DROP
81.4
60.1
TB-T5-L
DROP w/ DODMR-
85.6
68.2
Table 1: F1 scores on DROP dev set and DROP-CS. Blue bold is the best model, purple bold is the second best. Chain-of- oft-o-ffouuuussssseeeeee outperforms other baselines on both the in in-distribution dev set and the contrast set. Integrating -iti-o--Quuttions with TeaBReaC Pretraining further improves the performance. *We report reproduced results of GPT-3.5, which is slightly lower than their official report (64.1 on DROP with few-shot prompting).
top 5 trajectories from the beam that are not in the replay buffer as out-of-memory trajectories. Next, we sample at most in-memory trajectories from the replay buffer. We thus have total of m different sub-answer trajectories (5 I m < 10), denoted as "a for each (q, e) example. Fi- nally, we use the Fl score of the final predicted sub-answer àguh as the reward R(āsub) of the tra- jectory, The MAPO training objective uses both sets of trajectories derive unbiased stratified sampling estimator of policy gradient objective:
tially no sampled trajectories receive high reward. Hard-EM provides useful training signal the start of training, but MAPO can help training converge to a better final model once some successful tra- jectories are added to the buffer. Thus, we apply a mixture weight À to dynamically balance Eq. (2)
and (3). The overall Chain-of ciiit-fff-uusttonn (CoQ) loss for a given example (4, c, a) is defined as:
tceQ A6MAPO + (1 A)6H-EM.
(4)
where À is the proportion of examples with at least one trajectory in the replay buffer. We rely on Hard-EM at the beginning of training, but switch to MAPO the model finds successful trajectories. Note that Est. is used in both ÊAPO and EILEM.
(APO (0, suh {á.
sub
N=1)=
Σ
R(a,th
sub
log logPo(( (a.th q. mhth)
m
w EB
1 TB m
R(a,th) log (a,"b q, q, qDu)
Experiments
a g
m
We present our experimental setup and show CoQ outperforms other multistep fine-tuning or prompt- ing baselines over multiple benchmarks.
(f, q, q, a.
(3)
m
where TB is the ratio of the number of trajecto- ries in the buffer the total number of sampled trajectories. As step in Fig. after computing the objectives, we update the replay buffer with high-reward examples from the beam.
5.1 Experimental Details
Datasets. For in-distribution evaluation, DROP and HOTPOTQA. DROP contains 77,400 training examples and 9,536 validation examples.
HOTPOTQA contains 72,928 training examples
4.3 Chain-of-Questions Training Algorithm As mentioned in Agarwal et al. (2019), MAPO works poorly at the beginning of training, as ini-
and 5,901 validation examples.? DODMR contains
"We use the two- paragraph HOtPOTQA version released by Fisch et al. (2019) for efficient training.

---

the question decomposition of 7.705 training exam- ples from DROP and 6,233 training examples from HO We use a T5-base (Raffel al., 2020) QDMR parser to create DODMR+, which has ques- tion decompositions of all training examples. We do not use QDMR examples from other datasets during training. e.g., we only use the DROP ex- amples in PDDMR for training DROP. For ro- bustness evaluation, set DROP-CS (Gardner et al., 2020) containing 947 examples, and the adversarial set HOTPOTQA- ADV (Jiang and Bansal, 2019) containing 3,627 examples.
with the given dataset, and the other is fine-tuning with QDMR-augmented contexts. For the latter, we concatenate all sub-questions from the question decomposition to (q. c) as the input to the model and fine-tune it to perform single-step QA (i.e., di- rectly generate the answer). At inference time, we first use the QDMR parser y to generate the sub- questions for each question, and input them along with (g. c) to the model. This baseline similar to (Guo al., 2022b), which uses the same QDMR parser and the same model. The only difference is they train the QDMR parser with Mard-EM.
evaluate the contrast
Models. For DROP, we apply CoQ with T5- Base (T5-B), T5-Large (T5-L), and TeaBReaC T5- Large (TB-T5-L) with a batch size of 16. For HoT- POTQA, the context length of examples in the 4- paragraph HOTPOTQA-ADY exceeds 512, which is the token number limit of T5. We apply CoQ with LongT5- Base (LongT5-Br B; Guo et al., 2022a) with batch size of 8.
5.2 Task-Specific Modifications
To match the in in-distribution performance of state- of-thc-art systems, we make lask-specific modifica- tions for DROP and HoTPOTQA.
Modifications for DROP. Smaller language models such T5-B and T5-L with numer- ical operations. Since DROP focuses numerical reasoning, analogous to how BERT-Calculate help the model to do arithmetic, we add a regular ex- pression matching module that can handle basic numerical operations. We note that only the last sub-question of DROP example may require nu- merical operation. Hence, we add regular expres- sion matching and the "[REGEX]" tag in the last step.
We compare Chain-of- ofeeosstiouuussss to several fine-tuning methods and prompting meth- ods. For fine-tuning methods, we compare with: BERT-Caleulator (Andor et al., 2019): A uni- fied model that uses BERT to predict programs that execute to answers on DROP.
Baselines.
NeRd (Chen et al., 2020): A neuro-symbolic model that extends BERTTCaleelator from one- step operators to two-step compositions.
For each example, we take the last sub-question generated by the model f. parse program based on the keyword matching and ex- ecute it. If the parsing and execution process are both successful, we put the "[REGEX]" token in front of the numerical execution result and input them together into the answer generation process. Else, we keep the answer generation process un- changed as in Fig. 2.
Longformer (Beltagy et al., 2020): A trans- former encoder that combines local and global attention to process long documents.
functional
For prompting methods, compare with the fol- lowing methods using GPT-3.5:
Chain-of- Chain-of-Thought (Wei al., 2022): We insert few-shot in-context examples with explicit rea- soning chains to solve multistep problems. Due to the limited context window of Transformer models, we use 4-shot prompting for DROP and 2-shot prompting for HOTPOTQA. Our prompt examples consist of QDMR's 's paired with manu- ally written sub-am sub-answers.
Modification for HOTPOTQA. Predicting the supporting facts is an auxiliary task HotPOTQA used in many models (Groeneveld et al., 2020; Belt- agy et al., 2020). Following the same input format as Longformer, we add the supporting fact (SF) prediction and the span prediction (SP) tasks in the encoder as auxiliary tasks. We use two-layer feed- forward network for supporting fact prediction, and one-layer classification head for span prediction. We perform these two tasks at each run of model f and add the two cross-entrops losses to tcoQ-
Standard Prompting (Brown et al., 2020): We prompt GPT-3.5 to generate the answer without providing any in-context examples.
We engineered prompts to make these baselines as competitive as possible, detailed in Appendix C.
Single-step Baselines. We compare to two single-step baselines. One is standard fine-tuning

---

Testing Data (FI)
Model
Method
Training Data
HOTPOTQA
HOYP
Longformer
Longformer
HOTPOTQA
85.6
77.7
Chain-of- Shain-of-Thought (2-shot)
66.8
58.9
GPT-3.5
Standard Prompting (0-shol)
69.7
57.1
HOtPOTQA
77.5
Single-step run
HOTPOTQA A w/ DDDMRR
78.2
LongTT-B
HOtPOTQA w/ DODMR
74.7
72.8
HOTPOTQA w/ DODMR.
85.1
83.2
w/o MAPO w/o SP
83.0
82.0
78.1
76.5
Table 2:
scores on HOTPOTQA dev set and HOTPOTQA-ADY, Blue bold the best model, purple bold is
the second best. Chain-of- -oiinttf-Quusionns matches the performance of Longformer on the in-distribution dev set, and outperforms all baselines on the adversarial set.
5.3 Results
MAPO works better on HOTPOTQA than DROP. Using MAPO in addition Hard-EM, as opposed to Hard-EM alone, leads larger gains in Fi on HOTPOTQA (+(1.2-2.51%) than on DROP (+[0.1-0.9]%). As the sub-answers should be spans in the context for both DROP and HOTPOTQA, our hypothesis is that the span prediction task provides an inductive bias for actions to focus on spans in the context, which makes the model more likely to gen- erate correct answers. The context of 2- 2-paragraph HOTPOTQA is shorter than DROP, which makes the action space smaller.
We show results for DROP and DROP-CS in Ta- ble 1, and results for HotPoTQA and HOTPOTQA- ADV in Table 2. The results indicate the effective- ness and robustness of Chain-of-Questionn
Chain-of-Questions outperforms other baselines. Table 1, CoQ w/ DODMR+ on DROP outper- forms all baselines on F1 by 2.6% in-distribution and 7.8% on the contrast set. Moreover, the T5- L. version is 3.5% better than recently released GPT-4 Fl score the DROP (OpenAI, 2023). The model can be further improved by initializ- ing with TeaBReaC Pretraining. Similarly, in Ta- ble 2, CoQ w/ DODMR+ on HOtPOTQA on-par in-distribution with Longformer and 5.5% higher on the adversarial set. On both datasets, CoQ has a smaller performance gap between the in- distribution dev set and the robustness evaluation set compared with the baselines, indicating its strong robustness.? 3
Task-specific modifications are necessary. Both the regular expression module in DROP and the auxiliary tasks (SF & SP) in HOTPOTQA improves Fl by more than 5% on the in-distribution dev sets and 3% on the robustness evaluation sets in Table and 2. This shows that task-specific modifications are an important part of Chain-of Chain-of-Questions.
Chaun-of-Questions can generalize to bench- marks with no QDMR annotation. To test if our framework is still effective when annotated QDMR's s are not available for the training dataset, conduct additional experiments where omit the QDMR annotations for one dataset, then run CoQ on that dataset using only QDMR generated by QDMR parser trained on other datasets.
Chain-of-Thoughe prompting weaker than fine-tuning methods on multistep QA. On both DROP and HOTPOTQA, prompting methods have lower FI than fine-tuning methods, which indicates the difficulty of prompting large language models to do multistep QA. Moreover, Chain-of-Thought is 2.9% worse than zero-shot Standard Prompting HOtPOTQA A F1 score. We find it difficult to design prompts for GPT-3.5 that lead clean and concise answers under the Chain-of ThoThht setup. HOTPOTQA-ADY benefits from question decom- position, which suggests Standard Prompting may take reasoning shortcuts on HOTPOTQA examples.
Recall that in our main DROP experiments, DODMR+ includes both human-annotated gold QDMR and silver QDMR generated by QDMR parser trained with the QDMR annotation of train QDMR parser
DROP. Instcad, with only the QDMR annotation of COMPLEXWE BQUESTIONS (Talmor and Berant, 2018) and HoT- POTQA. Then, we use that QDMR parser to gen- crate bronze QDMR augmentation for the whole DROP dataset, and use these to run CoQ. this
"The one exception is the prompting baselines on DROP. which are 30% lower on DROP and 20% lower on DROP-CS than CoQ.

---

Train Data
DROP
DROP-CS
sub-questions from DROP (Ist) and HOTPOTQA (3rd) examples are correct decompositions. In the DROP-CS (2nd) example, the generated QDMR is also a valid decomposition, although answering the first two generated sub-questions require additional reasoning steps. Note that the QDMR is anno- tated by only looking at the questions (Wolfson et al., 2020), and the QDMR sub-question genera- tion in Chain-of- learning. The QDMR annotation may cause the sub-question generation focus only on the ques- tion, and thus generate sub-questions that require multiple reasoning steps from the passage, which are hard for the model to answer.
DROP
w/ DOOMR
46.6
44.3
w/ bronze QDMR
69.1
59.8
w/ DOOMR+
74.4
63.8
Train Data
HOTPOTQA
HOTPOTQA-ADV
HOTPOTQA
w/ DOOMR
74.7
72.8
w/ bronze QDMR
84,9
81.5
w/ DODMR+
85.5
83.2
is trained by supervised
Table 3: F1 scores of Chain-of of-uueston models on DROP dev set, DROP-CS, HOTPOTQA dev set and HOTPOTQA-ADV set, trained with different QDMR data. By assumiing DROP and HoTPOTQA has no QDMR annotation, CoQ models trained on bronze QDMR drops 4-5% Fl on DROP and DROP-CS, 1-2% Fl on HOTPOTQA and HOTPOTQA- are still much better than the gold-only QDMR training. as well as the single-run baselines.
The final answer of the question is generated in the first sub-answer in the HOTPOTQA (3rd) ex- ample, which is not correct answer to the first sub-question. The model may use reasoning short- cuts to generate the final answer as sub-answers, which is beneficial to generate the final answer at the last step, but not answering the sub-question.
while they
way, we consider what would happen if no anno- tated QDMR was available for DROP. Similarly, for HOTPOTQA, we train the QDMR parser with the QDMR annotation of COMPLEXWEBQUES TIONS and DROP, and generate bronze QDMR for the TOOPOOQAA dataset.
How do the generated sub-questions help the QA model? We hypothesize generated sub- questions helps the model return the right answer for the right reason, and we try to detect this by looking at whether it can correctly identify sup- porting facts in HOTPOTQA-ADV. For each ques- tion, the model predicts two supporting facts in the context using the encoder output embedding. We observe that the accuracy of supporting fact prediction improves as we incorporate question de- composition in the input.
Compared to training on DODMR+, CoQ train- ing on bronze QDMR results in decreases of 4-5% Fl on DROP and DROP-CS and 1-2% Fl on HOTPOTQA and HOTPOTQA-ADV, Nonetheless, these Fl scores are still much better than the single- run baselines on the contrast and adversarial sets. These experiments show that CoQ can be effective even on benchmarks without QDMR annotations. Training on bronze QDMR is also much more effec- tive than training only on gold DODMR, shown in Table 3. This suggests that having large amount of training data greatly helps CoQ, even if that data is lower quality; future work could explore using more unannotated data to further improve perfor- mance.
For example, given a question "What govern- ment position was held by the woman who por- trayed Corliss Archer in the film Kiss and Tell?". the ground truth supporting facts are two sentences: "Kiss and Tell is.. Shirley Temple as Corliss Archer." and "As an adult, she ... served as Chief of Pro- tocol of the United States.", from two different paragraphs.
5.4 Qualitative Analysis on QDMR
In the first run, the input to the encoder is only the question and context; the model predicts the first supporting fact correctly, while it takes sen- tence "As an adult, she ... served as Chief of treaty of the United States." from an adversarial para- graph, which refers to a different person, as the sccond supporting fact. However, in the second run, when the input to the encoder contains the first sub-question and sub-answer ("[QDMRJ return
We conduct qualitative analysis on QDMR to check its quality and effectiveness.
How good are the generated sub-questions and sub-answers? We list three multistep QA exam- ples in Table 4 to show quality of generated QDMR sub-questions and sub-answers." The generated
WWe choose the examples based on following the criteria (1) the example must be multistep QA (2) we manually check around 3 examples per dataset satisfying (1) and select one
based on how well the context and the relevant information can he visualized.

---

Question
Context
Ground truth QDMR
Generated QDMR & Answeers
How many years aller
After the fall of Pegu in December 1599 but the king died during the campaign on March
1. return year that Pegu fell: return that the king did die: return difference 42 and
1. return the year when Pegu fell: 1599;
Pegu fell did the king die? (DROP)
2. return the year when the king died: 1606;
3. return difforence of 1606 and 1599; 7.
Which cities did they before Manchester and Derby? (DROP-
The Jucobile ammy... .. enteblue England November ... Two days later they reached Carlide they reached Preston followed by Manchester
1. return the date they went to Manchester:
1. return vities they gu before Manchester::
Preston:
26 November,
return the they went Derby:
2. return they go before Derby: Manchester;
CS)
3. return the cities they went hefore 1
November
return the cities they went before 42;
3. return
in both Preston and Manchester: Preston.
entebluc Derby on 4 December...
5. return the cities in both #5 and #4,,
Which airline partners with Gogo Inllight In- ternet and opened in 1984? (IoTPPTQA)
Gogo Inflight 17 airlines partner with Gogo to provide in-llight Witi including.... Virgin Atlantic, trading name
1. return airline that partners with Gogo Inllight Internet;
1. return which airline partners with Gogo Inllight Internet: Vingin Atlantic:
2. return #1 that opened in 1984,
2. return Virgin Atlanric open in 1984: Virgin Atlantic.
established 1984
Table 4: Qualitative analysis of in the context, blue hold to mark correct final answers, and red bold to mark wrong final answers. We find (1) most generated sub-questions are correct; (2) a generated sub-question by CoQ models can contain multiple reasoning steps, (3) the model may use reasoning shorteuts generate the final answer as sub-answers for early sub-questions.
reasoning chains. We use black bold to mark relevant information
the woman who portrayed Corliss Archer in the film Kiss and Tell [QDMR-ANS] Shirley Temple"), the model selects the second sentence from the paragraph about Shirley Temple, thus pre- dicting both supporting facts correctly.
et al.. 2019) requires complex multistep reasoning between a background context paragraph and sit- uation context paragraph. We could either train models on these datasets if we can eliminate our reliance on QDMR data, or test whether models trained with CoQ can transfer well to these other datasets.
For all the 2-hop questions in HOTPOTQA- ADV, the supporting fact prediction accuracy is 75.6% during the first run, and rises to 83.2% dur- ing the second run. This accuracy increase shows the effectiiveness of generating sub-qucstions step- by-step. which helps the model gradually filter out close but irrelevant context.
Limitations
Due to GPU resource constraints, we were unable to scale up our method to larger models such as T5- 3B. However, the smaller models we were able to experiment with already show good performance. Similar, for computational efficiency, we did not try other more advanced on-policy reinforcement algorithms, but we find that MAPO already yields good improvement on F1.
6 Discussion
We present Chain-of-Questione Qufetteesonss (CoQ), robust sub-question generation and answering framework that shows strong performance on DROP and Hor- POTQA. CoQ uses a combination of Hard-EM and MAPO for training, effectively optimizing the la- tent variables associated with sub-answers of inter- mediate questions.
Chain-of -fiiossiinueeusss still requires task-specific modifications for different multistep QA benchmarks we did not find out a good way to build universal model that is highly effective on all datasets. UnifiedQA (Khashabi et al., 2020b) constructs unified model for multiple QA benchmarks, but their largest model (T5-1 IB) still has poor performance on DROP, again suggesting the importance of dataset-specific modifications.
We envision multiple directions for future work. CoQ requires supervision from QDMR; other fam- ilies of RL methods we did not explore may be used to reduce our reliance on this supervision, and instead allow the model to learn appropriate decom- positions from scratch. On the other hand, we could also explore using different question decomposi- tions, such as ones generated by LLMs like GPT- 3.5. Either approach could help us extend CoQ to other multistep reasoning datasets with no QDMR annotation. Similar to DROP, FINQA (Chen et al.. 2021) consists of numerical reasoning questions over financial data. In a similar format as HOT- POTQA, MUSIQUE (Trivedi et al., 2022a) contains 3-hop and hop retrieval questions. ROPES (Lin
Finally, the best version of Chain-of- -f--o-tinnn requires QDMR annotation during training, which is only available for some datasets. In order to be in- dependent of the QDMR annotation, we show some generalization of CoQ by assuming no QDMR an- notation on DROP and training on bronze ques- tion decompositions generated by QDMR parser. CoQ could also be tested on other datasets without QDMR annotation to evaluate its transfcrability. by having the QDMR parser for bronze annota-

---

tions. On the other hand, the method can be further explored work with question decompositions generated by LLMs.
William Yang Wang. 2021. FinQA: A dataset of nu- merical reasoning over financial data. In Proceedings of the 2021 Conference Empirical Methods in Nat- ural Language Processing, pages 3697-3711, Online and Punta Cana, Dominican Republic. Association for Computational Linguisties.
Acknowledgements
We thank Qinyuan Ye. Ameya Godbole, Kristina Toutanova, Jonathan Berant and the members of USC NLP Group for their valuable feedback. This work was supported in part by an Open Phi- lanthropy research grant and a Google Research Scholar Award.
Dheeru Dua, Shivanshu Gupta, Sameer Singh, and Matt Gardner. 2022. Successive prompting for decom- posing complex questions. In Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pages 1251-1265, Abu Dhabi, United Arah Emirates. Association for Com- putational Linguistics.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019, DROP: A reading comprehension benchmark requir- ing discrete reasoning over paragraphs. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies. Volume 1 (Long and Short Papers), pages 2368-2378, Min- neapolis, Minnesota. Association for Computational Linguistics.
References
Rishabh Agarwal, Chen Liang, Dale Schuurmans, and Mohammad Norouzi. 2019. Learning to generalize from sparse and underspecified rewards. In Interna- tional Conference on Machine Learning.
Daniel Andor, Luheng He, Kenton Lee, and Emily Pitler. 2019. Giving BERT a calculator: Finding opera- tions and arguments with reading comprehension. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- pages 5947-
Adam Fisch, Alon Talmor, Robin Jia. Minjoon Sco, Eunsol Choi, and Danqi Chen. 2019. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering. pages 1-13, Hong Kong. China. Association for Computa- tional I Linguistics.
guage Processing 5952, Hong Kong, China. Association for Computa- tional Linguisties.
Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv:2004.05150
Ruiliu Fu, Han Wang. Xuejun Zhang, Jun Zhou, and Yonghong Yan. 2021. Decomposing complex ques- tions makes multi-hop QA easier and more inter- pretable. In Findings of the Association for Compu- national Linguistics: EMNLP 2021, pages 169-180. Punta Cana, Dominican Republic. Association for Computational Linguistics.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. Advances neural information processing systems, 28.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei, 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems.
Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi. Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hamnanch Hajishirzi, Gabricl Ilharco, Daniel Khashabi. Kevin Lin, Jiangming Liu, Nel- son F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020. Evaluating models' local decision boundaries via contrast sets. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1307-1323, Online. Association for Computational Linguisties.
Xinyun Chen, Chen Liang, Adams Wei Yu, Denny Zhou, Dawn Xiaodong Song, and Quoe V. Lc. 2020, Neural symbolic reader: Scalable integration of distributed and symbolic representations for reading compre- hension. In International Conference on Learning Representations.
Dirk Groeneveld, Tushar Khot, Mausam, and Ashish Sabharwal. 2020. A simple yet strong pipeline for HotpotQA. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 8839 8845, Online. Association for Computational Linguistics.
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and
Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei

---

Yang. 2022a. LongTS: Efficient text-to-text trans- former for long sequences. In Findings of the Associ- ation for Computational Linguistics: NAACI. 2022, pages 724-736, Seattle, United States. Association for Computational Linguistics.
the Association for Computational Linguistics, pages 6097-6109, Florence, Italy. Association for Compu- tational Linguistics.
Radford M. Neal and Geoffrey E. Hinton. 1998. A view of the em algorithm that justifies incremental, sparse, and other variants. In Learning in Graphical Models.
Xiao-Yo Yu Guo, Yuan-Fang Li, and Gholamreza Haffari. 2022b. Complex reading comprehension through question decomposition. In Proceedings of the The 20th Annual Workshop of the Australasian Language Technology Association, pages 31-40, Adelaide, Aus- tralia, Australasian Language Technology Associa- tion.
OpenAL. 2023. Gpt-4 technical report. abs/2303.08774.
ArXiv,
Ethan Perez, Patrick Lewis. Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela. 2020. Unsupervised question decomposition for question answering. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8864-8880, Online. Association for Computational Linguistics.
Yichen Jiang and Mohit Bansal, 2019. Avoiding reason- ing shortcuts: Adversarial evaluation, training, and model development for multi-hop QA. In Proceed- ings of the 57th Annual Meeting the Association for Computational Linguistics, pages 2726-2733 Flo- rence, Italy. Association for Computational Linguis- tics.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Han- naneh Hajishirzi. 2020a. UNIFIEDQA: Crossing format boundaries with a single QA system. In Find- ings the Association for Computational Linguistics: EMNLP 2020.
Viktor Schlegel, Goran Nenadic, and Riza Batista- Navarro. 2021. Semantics altering modifications for evaluating comprehension in machine reading. In Proceedings of the AAAI Conference on Artificial Intelligence.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Han- naneh Hajishirzi. 2020b. UNIFIEDQA: Crossing format boundaries with a single QA system. In Find- ings of the Association for Computational Linguistics: EMNLP 2020, pages 1896- 1907, Online. Association for Computational Linguistics.
Alon Talmor and Jonathan Berant. 2018. The web as knowledge-base for answering complex questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume I (Long Papers).
Tushar Khot, Harsh Trivedi, Matthew Finlayson. Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sab- harwal, 2022. Decomposed prompting: A modular approach for solving complex tasks. In International Conference on Learning Representations.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022a. MuSiQue: Multi- hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics.
Chen Liang, Mohammad Norouzi, Jonathan Berant, Quoe V Le. and Ni Lao. 2018. Memory augmented policy optimization for program synthesis and se- mantic parsing. In Advances in Neural Information Processing Systems.
Harsh Trivedi. Niranjan Balasubramanian. Tushar Khot, and Ashish Sabharwal. 2022b. Teaching broad rea- soning skills for multi-step QA by generating hard contexts. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6541 1-6566, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gard- ner. 2019. Reasoning over paragraph effects situ- ations. In Proceedings of the 2nd Workshop on Ma- chine Reading for Question Answering, pages 58-62, Hong Kong. China. Association for Computational Linguistics.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F, Xia, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. In Con- ference on Neural Information Processing Systems.
Sewon Min, Danqi Chen, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019a. A discrete hard em ap- proach for weakly supervised question answering. In Conference on Empirical Methods in Natural Lan- guage Processing.
Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gard- ner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant, 2020. Break it down: A question understand- ing benchmark. Transactions of the Association for Computational Linguistics.
Sewon Min, Victor Zhong, Luke Zettlemoyer, and Han- nanch Hajishirzi. 2019b. Multi-hop reading compre- hension through question decomposition and rescor- ing. In Proceedings of the 57th Annual Meeting of

---

Zhilin Yang, Peng Qi. Saizheng Zhang, Yoshua Bengio. William Cohen, Ruslan Salakhutdinov, and Christo- pher D. Manning. 2018. HotpoiQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2369-2380, Brussels, Belgium. Association for Com- putational Linguistics.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafram, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Good- man. 2022. STar: Bootstrapping reasoning with rea- soning. In Advances in Neurul Information Process- ing Systems.
Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoe Lc, and Ed Huai hsin Chi. 2023. Least-to-most prompting enables complex soning large language models. In International Conference on Learning Representations.

---

A Chain-of-Questions Algorithm Details We list omitted details of Hard-EM and MAPO in the Chain-of-Quee -fiiss---Quesionss algorithm.
Notice that sampling new trajectory is expensive in our scenario, especially with a large model. So instead of use samples from a policy model, we instead use the trajectories from the beam, which reduces sampling time and yields better quality trajectories.
A.1 Sub-Answer Sampling Strategy
Given qutb and a we compute the likelihood to predict the sequence of qtth and asub by:
On the other hand, MAPO stores trajectories with rewards greater than 0 in B. However, wrong answers with some corrects words could also get score greater than with our reward function. Hence, we select R( > 0.8 and store these trajectories in the replay buffer 13.
po Pa(quu asub q.
II Po (auu
q. c, Ti.z. 11g
sub asub 1)
72
II
po (g)
aub
g, c, Ti
1. agg 1)
(5)
B Implementation Details
We list the implementation details and hyperparam- eter choice as follows.
We use beam search to sample the sub-answers and keep a beam of size k. In each iteration, we expand each beam with b different answers and select the top-k likelihood answers construct new beam following Eq. (5). We choose 25 and b = in our experiments. Thus, we have 5 different candidates for aqub in the first run, 25 different candidates for (ā-ub, a55D)) after the second run, and in general 25 candidates for 1 for the j-th run for j }>> 2.
B.1
Datasets
Jiang and Bansal (2019) constructed HOTPOTQA- ADV by generating up to 8 adversarial paragraphs for given HOTPOTQA example. Because we trained on the 2-paragraph HOTPOTQA, we gen- crate a 4-aaragraph version of HOTPOrQA-ADV to reduce length generalization. We take the ad- versarial set from Jiang and Bansal (2019), filter the examples with at least 2 adversarial paragraphs, and select their intersection with the validation set of 2- paragraph HOTPOTQA. We randomly choose the adversarial paragraphs, and randomly order them with the supporting paragraphs.
The challenge is to ensure the sampling will pro- vide some correct sub-answers. However, notice that roughly 10% of annotated QDMR's across DROP and HOTPOTQA are single-step decomposi- tion (i.e., n = 1). Our hypothesis is the model may to learn to do single-step QA from these examples, which will also help it learn to produce meaning- ful sub-answers to sub-questions that come from multistep QDMR's.
B.2 DROP Regular Expression
During the execution time, we detect the output [END-QDMR] token for the last sub-question.
A.2 Difference from Original MAPO
We search for specific keywords in the last sub- question and match them with numerical opera- tions, e.g., we match "higher" to max and "less" to min. The keyword matching perfectly matches the ground truth QDMR annotation numerical oper- ations. We use 7 operators in total: max, min, sum, di ff, mul, div, or.
Given policy model (1; f), and replay buffer B, the original MAPO objective is:
MAPO = r' aud NH (a R(asub)
+(1 - TB) (B Hom (a R( R(asub)
where R(a"uh) denotes the reward of the trajectory, and rg is the ratio of the number of trajectories in the buffer the total number of sampled trajecto- ries, used derive an unbiased stratified sampling estimator of the gradient. nTi (āstb) and He (asuh) are the normalized probability distribution inside and outside the buffer, defined as:
We take the last sub-question generated by the model f. parse it to a functional program based on the keyword matching and execute it. If the pars- ing and execution process are both successful, we put the "[REGEX]" token in front of the numerical execution result and input them together into the answer generation process. For example, if the last sub-question generated is "[QDMR] return the largest of 4 and 3 [END-QDMR]", we will in- put "[QDMR] return the largest of 4 and
Tg. (a-uh) = me(āauh) ((rau))/// [[asuh < 13)
To (asub) = Tuba //1 rB) 1 sub & B

---

Model
DROP
DROP-CS
The final answer must be short and concise. Don't t repeat the question in the final answer. An\n"
TB-T5-I.
85.6+0.5
68. 68.2+00.8
Model
HotPOTQA
HOTPOTQA-ADV
LongT5-B
85.1
83.0+0.55
The last two sentences help the model to gener- ate clean and concise answers. Without these two sentences in the prompt, the zero-shot FI score will drop by more than 10% in DROP and HOtPOTQA benchmarks. Without the thinking step-by-step sen- tence, the Fi performance will drop more (-8%) on DROP but Icss (-2%) on HOTPOTQA, indicating HOTPOTQA A has more validation questions contain- ing reasoning shortcuts.
Table 5: The Fl and the standard deviation on the evaluation benchmarks of the best CoQ models. The results are based on three different runs.
3 [END-QDMR] [REGEX] 4" into the decoder for answer generation, in both training and inference time.
However, with these constraints and ex- plicit format provided in the prompt, the GPT-3.5 model still hard to generate the precise answer the format we want. For example, apart from 'The answer is' GPT-3.5 generates different strings front of the answer, such as 'The final answer is', 'Final answer is', 'Answer:'. During the answer parsing, we consider all different variations of the pre-answer string to parse for the answer.
B.3 Hyperparameters
In training DROP, we train a total of 50k iterations with batch size of for T5-B and a batch size of 4 for T5-L. One of DROP training with T5-B on one NVIDIA V100 GPU takes 40 hours. One run of DROP training with T5-L on one NVIDIA A A100 GPU takes 60 hours.
In training HOTPOTQA, we train total of 30k iterations with batch size of 16 for LongT5-B. One run of HOTPOTQA training with LongT5-B NVIDIA V100 GPU takes 30 hours.
C.2 Few-shot Prompting
The few-shot prompt we used to evaluate the model
In all training. we use the AdamW optimizer with a weight decay of 0.01. The learning rate is set as 1e We evaluate the model every 500 steps and set an early-stopping criterion on the validation Fl score, with max patience of 3.
is
[EXAMPLE] [REASONING] [ANSWER] [EXAMPLE] [REASONING] [ANSWER] [EXAMPLE]" where
"[INSTRUCTION]
[INSTRUCTION] = "We provide 4 examples for answering the question given the context, by thinking step-by-step. Please answer the last question in the same format. \n\n"
B.4 Standard Deviation
We report the mean Fl score and the standard de- viation over three runs of the best CoQ models in Table 5, which are the TB-T5-L model for DROP and the LongT5-B model for HOTPOTQA.
[EXAMPLE] = "question: context:
An". [REASONING] "question decomposi tion: sub; af ...; In a An"
C Prompt Engineering
We show the effort we made on prompt enginecring to get the best possible performance on DROP and HOTPOTQA using GPT-3.5.
[ANSWER] = "The answer is \n\n" We tried different variations for each tag as follows and evaluated on 100 random examples choose the best prompt over their combinations.
C.1
Zero-shot Prompting
The zero-shot prompt we used to evaluate the model is
[INSTRUCTION] = "Please answer the question after reading the context. You should start with the thinking and reasoning steps, such as retrieving relevant content from the context and solving the question step-by-step. We provide 4 examples for answering the question given the context, by thinking step-by-step. You should give the final
"Please answer the question after reading the context. You should start with the thinking and reasoning steps, such as retrieving relevant content from the context and solving the question step-by-step. You should give the final answer in the format of 'The answer is

---

answer in the format of ' 'The answer is The final answer must be short and concise. Don't repeat the question the final answer An\n"
[REASONING] = "Let's s think step-
qi o ...; u an An".
[ANSWER] "Answer: An\n"
The results 100 random validation examples of DROP and HOTPOTQA both show that shorter and more concise instruction with a more formal reasoning format achieves better score.
In terms the cost, we did not optimize for the best few-shot examples for validation data. Instead, we selected 5 random set of few-shot examples and chose the best set over them by F1 score over 100 validation examples. For each few-shot exam- ples set, wrote the sub-answers for the QDMR annotations and inserted these sub-answers into the QDMR question if they are referenced. The an- swer parsing mechanism is the same as in zero-shot prompting.
D Random Qualitative Examples
We numpy. random. choice select 2 random examples from DROP and 2 random examples from HotpotQA. Unlike in Table 4, we omit the manually written ground truth QDMR for the dev set and use the ground truth answer instead.

---

Question
Context
Generated QDMR & Answers
GT Answer
How many of the per- sonnel were not military specialists? (DROP)
The total number active military per- sonnel ... May 2016, Anned Forces had 16,019 members, which 14,506 were active military personnel and 1,513 civil servants. the 14,506 live military personnel. 3,183 were ficers, ... 520 military specialists.
number personnel: 14,506
13,986
were active military personnel. 2. number of military specialists: military specialists. 3. return the differences 14,506 were active mil- itary personnel and 520 military spe- cialists: 13,986.
How many more people there than housing units? (DROP)
As of the census of 2000, there were 49,129 people.... There were 21,779 housing units 39 per square mile (15/km*)..
return number of people: 49.129 27,350 people. 2. return number of housing unit: 21,779. 3. return the differences of 49,129 people and 21,779: 27,350.
average density of
This cabin built 1925 is located in a park encompasses over million acre and includes parts of two what? (HoTPOTQA)
Lower Park Creek Patrol Cabin. The Lower Park Creek Patrol Cabin Glacier National Park is a rustic backcountry log cabin. Built 1925, the cabin has... Glacier Na- tional Park is a national park The park encompasses over million acre and includes parts of two moun- tain ranges (sub-ranges of the Rocky Mountains), over 130 named lakes,
return park this cabin built in 1925: Glacier National Park. 2. return Glacier National Park encompasses over million acre and includes parts of two what: mountain ranges.
mountain ranges
Bandit was built by which Japanese amusement com- roller
Bandit (Yomiuriland)... Built 1988 by TOGO company... TOGO. TOGO (#k 2 it Kabushiki- gaisha Togo was Japanese amuse- ment company that built roller coasters, giant wheels, carousels, flumes, dark rides, sky cycles and other amusement rides.
return the Japanese amusement company that built roller coast- ers, giant wheels, carousels, flumes, dark rides, sky cycles other amuse- ment rides: TOGO (#% x., 2 it 1 Kabushiki-gaisha Togo 2. return TOGO (#% st, A 2. Kabushiki-
TOGO company
pany that coasters, giant wheels, carousels, flumes, dark rides, sky cycles and other amusement rides? (HotPoTQA)
gaisha Togo built bandit 1988: TOGO.
Table 6: Qualitative analysis of model-generated reasoning chains. We use black bold to mark relevant information in the context.


Reviews:
[{"review_id": "Jr2rz5q0kl", "reviewer": "Reviewer_7ULL", "summary": "", "questions": "1. Were the sample questions and answers shown in Table 3 hand-picked, or random? If hand-picked, how does a random set of generated answers and questions look like? If these were representative illustrative examples, perhaps a table of randomly selected examples should also be added to the appendix.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We are pleased they find CoQ is intuitive, clear, well-motivated, and they acknowledge performance improvements across multiple datasets.\n\n**Q1: Generalization of the authors' method to other datasets (as mentioned in line 583-585, 610-613) is critical to make the paper stronger.**\n\nA1: Thank you for the comments. We tried the following setups to test the generalization abilities on other un-annotated benchmarks using QDMR parser.\n\nNotice that $\\mathcal{D}_\\mathrm{QDMR+}$ is augmented with silver QDMRs generated by a QDMR parser trained with the QDMR annotation of DROP. Instead, we train a QDMR parser with the QDMR annotation of CWP and HotpotQA and generate the bronze QDMR augmentation for DROP. In this way, if we consider DROP as an un-annotated benchmarks, we could still use DROP w/ bronze QDMR to train.\n\n*T5-B Chain-of-Questions results on DROP and DROP-CS*\n\nTrain Data | DROP | DROP-CS\n:---|:---:|:---:\nDROP w/ gold QDMR ($\\mathcal{D}_\\mathrm{QDMR}$) | 46.6 | 44.3\nDROP w/ bronze QDMR | 69.1 | 59.8\nDROP w/ gold+silver QDMR ($\\mathcal{D}_\\mathrm{QDMR+}$) | 74.4 | 63.8\n\nCoQ training with bronze QDMR drops 4-5% F1 on DROP and DROP-CS while it is still much better than the single-run baselines (53.1/45.0), suggesting the potential of CoQ to generalize to un-annotated benchmarks. We view generalizing to other un-annotated benchmarks as an orthogonal research direction, but we will include these results in the camera-ready.\n\n**Q2: Were the sample questions and answers shown in Table 3 hand-picked, or random? If hand-picked, how does a random set of generated answers and questions look like?**\n\nA2: Thank you for the suggestion. They are hand-picked based on the criteria (1) the example must be a multistep QA (2) we look at ~3 examples per dataset satisfying (1) and select one based on how well the context and the relevant information (bold) can be visualized. \n\nA random set of generated answers and questions (we use numpy.random.choice) will look similar as Table 3. A table of 2 random examples from DROP and 2 random examples from HotpotQA is like below. We omit the manually written ground truth QDMR for the dev set and use the ground truth answer instead. We will add a longer table of randomly selected examples in the Appendix in the camera-ready.\n\nQuestion | Context | Generated QDMR & Answers | GT Answer\n:---|:---:|:---:|---:\nHow many of the personnel were not military specialists? (DROP) | The total number of active military personnel .... In May 2016, Armed Forces had 16,019 members, of which **14,506 were active military personnel** and 1,513 civil servants. Of the 14,506 active military personnel, 3,183 were officers, ... **520 military specialists**, ... | 1. return number of personnel: 14,506 were active military personnel. 2. return number of military specialists: 520 military specialists. 3. return the differences of 14,506 were active military personnel and 520 military specialists: 13,986. | 13,986\nHow many more people are there than housing units? (DROP) | As of the census of 2000, there were **49,129 people**, … **There were 21,779 housing units** at an average density of 39 per square mile (15/km2)... | 1. return number of people: 49,129 people. 2. return number of housing unit: 21,779. 3. return the differences of 49,129 people and 21,779: 27,350. | 27,350\nThis cabin built in 1925 is located in a park encompasses over 1 million acre and includes parts of two what? (HotpotQA) | Lower Park Creek Patrol Cabin. **The Lower Park Creek Patrol Cabin in Glacier National Park is a rustic backcountry log cabin.  Built in 1925, the cabin has**... Glacier National Park is a national park ...  **The park encompasses over 1 million acre and includes parts of two mountain ranges (sub-ranges of the Rocky Mountains)**, over 130 named lakes, ... | 1. return park this cabin built in 1925: Glacier National Park. 2. return Glacier National Park encompasses over 1 million acre and includes parts of two what: mountain ranges.  | mountain ranges\nBandit was built in 1988 by which Japanese amusement ride company that built roller coasters, giant wheels, carousels, flumes, dark rides, sky cycles and other amusement rides? (HotpotQA) | Bandit (Yomiuriland)...  **Built in 1988 by the TOGO company**... TOGO. **TOGO (株式会社トーゴ , Kabushiki-gaisha Tōgo ) was a Japanese amusement ride company that built roller coasters, giant wheels, carousels, flumes, dark rides, sky cycles and other amusement rides**. | 1. return the Japanese amusement ride company that built roller coasters, giant wheels, carousels, flumes, dark rides, sky cycles and other amusement rides: TOGO (株式会社トーゴ , Kabushiki-gaisha Tōgo ). 2. return TOGO (株式会社トーゴ , Kabushiki-gaisha Tōgo ) built bandit in 1988: TODO. | TOGO company", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method is intuitive, clear, and results in performance improvements across multiple datasets.\n1. Combines multiple prior directions such as question generation, question decomposition, and multi-step QA to yield a well-motivated method.\n1. The method gives good performance even with simple HardEM based training if a practitioner does not want additional complexity/instability of RL.\n1. The transfer of loss from HardEM to MAPO is well motivated and results in performance improvements", "reasons_to_reject": "Generalization of the authors' method to other datasets (as mentioned in line 583-585, 610-613) is critical to make the paper stronger. \nAs most datasets lack QDMR decomposition, atleast simple transferring of their method using silver predicted data on other datasets should be studied, to make the method more generally applicable.\n", "questions_to_authors": "1. Were the sample questions and answers shown in Table 3 hand-picked, or random? If hand-picked, how does a random set of generated answers and questions look like? If these were representative illustrative examples, perhaps a table of randomly selected examples should also be added to the appendix.", "paper_topic_and_main_contributions": "The authors propose a Question Answering (QA) method for multi-step QA, where the question is first decomposed into sub-questions, which are then solved prior to being used to answer to the original question.\n\nThe authors train a question decomposition model trained on a smaller set of annotated data of decomposing questions, and use this model to generate decompositions for the entire training set. These decomposed questions are then answered by the model, successively, and used to answer the next decomposition until arriving at the answer. The author propose to use HardEM and the RL method MAPO to optimize their model, slowly transitioning from HardEM loss during training to attain better final convergence of RL with the superior initial stability of HardEM.\n\nThe authors test their method on HotPotQA and DROP datasets, and show strong performance improvements in both datasets. The generated questions and answers ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 970 \"model is still hard to generate\" should be reworded to be grammatically correct."}, "extra_text": {}}, {"review_id": "7Yj6m8LPd0", "reviewer": "Reviewer_UiWR", "summary": "", "questions": "A. Is the performance of the BERT-based models reported in Table 1 based on BERT-base or BERT-large?\n\nB. Are the results reported in Tables 1 and 2 the maximum, mean, etc., or just from a single run?\n\nC. What are the existing SOTAs of DROP-CS and HOTPOTQA-ADV?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and are pleased they find CoQ is novel, effective, and shows solid improvements over baseline models.\n\n**Q1: Baseline models are not strong enough.**\n\n**Q1a: The performance of GPT3.5 based models may be underestimated in this paper as the GPT-4 Technical Report reported 64.1 F1 on DROP for 3-shot GPT-3.5.**\n\nA1a: Thanks for pointing this out. We noticed this, and we had tried our best with prompting in Appendix C. The GPT-4 report did not specify their prompt, and we are not able to reproduce their results on DROP. On the other hand, the 64.1 number in the GPT-4 report is still worse than CoQ and other baselines.\n\n**Q1b: DROP baselines, other strong models like QDGAT**\n\nA1b: We tried to run the QDGAT model before to evaluate on DROP-CS, which is our main focus. However, their codebase is not runnable (same for OPERA, others on the top have no code released). Specifically, the QDGAT codebase mentioned “run parse_xml.py” in their README.md, but that file is not in their codebase. They also don't allow posting issues. Notice that TeaBreaC [1], the SotA on DROP-CS as far as we know, also did not compare with these methods.\n\n**Q1c: HotpotQA baselines, leading models at the leaderboard are not considered.**\n\nA1c: As mentioned in the footnote of page 5, we use the two-paragraph version released by MRQA 2019 (Fisch et al, 2019) for efficient training. Most methods on the leaderboard with code released focus on selecting two relevant paragraphs out of ten, such as R3, FE2H [3, 4]. These methods will almost be the same as Longformer on the two-paragraph setup.\n\n**Q2: The \"fast improvement at the beginning and better final convergence\" of the dynamic mixture of Hard-EM and MAPO lacks supporting evidence in the paper.**\n\nA2: Thank you for pointing it out. We will add a convergence figure similar to Fig.2 in MAPO [5], which shows Hard-EM converges faster for a few thousand steps and then is outperformed by Hard-EM+MAPO in the middle.\n\n**Q3: The generalization of the proposed method may be challenging due to the prerequisite of QDMR annotation and task-specific preprocessing.**\n\nA3: We agree that the generalization to arbitrary multistep QA dataset is challenging, and we view generalizing to other un-annotated benchmarks is a research question of its own. \nOn the other hand, we tried the following setups to test the generalization abilities on other un-annotated benchmarks using QDMR parser.\n\nNotice that $\\mathcal{D}_\\mathrm{QDMR+}$ is augmented with silver QDMRs generated by a QDMR parser trained with the QDMR annotation of DROP. Instead, we train a QDMR parser with the QDMR annotation of CWP and HotpotQA and generate the bronze QDMR augmentation for DROP. In this way, if we consider DROP as an un-annotated benchmarks, we could still use DROP w/ bronze QDMR to train.\n\n*T5-B Chain-of-Questions results on DROP and DROP-CS*\n\nTrain Data | DROP | DROP-CS\n:---|:---:|:---:\nDROP w/ gold QDMR ($\\mathcal{D}_\\mathrm{QDMR}$) | 46.6 | 44.3\nDROP w/ bronze QDMR | 69.1 | 59.8\nDROP w/ gold+silver QDMR ($\\mathcal{D}_\\mathrm{QDMR+}$) | 74.4 | 63.8\n\nCoQ training with bronze QDMR drops 4-5% F1 on DROP and DROP-CS while it is still much better than the single-run baselines (53.1/45.0), suggesting the potential of CoQ to generalize to un-annotated benchmarks. We will include these results in the camera-ready.\n\n\n**Q4: Is the performance of the BERT-based models reported in Table 1 based on BERT-base or BERT-large?**\n\nA4: BERT-Calculator and NeRd both use BERT-base.\n\n**Q5: Are the results reported in Tables 1 and 2 the maximum, mean, etc., or just from a single run?**\n\nA5: Single-run. As mentioned in Appendix B.3, experiments on RL take long to train. But we will report the average and std of 3 runs in the camera-ready version.\n\n**Q6: What are the existing SotAs of DROP-CS and HotpotQA-ADV?**\n\nA6: As far as I know, the existing SotA of DROP-CS is TeaBreaC-3B [1], which achieves F1 score 66.8, we use TeaBreaC-L with CoQ and outperform them (68.5). \n\nFor existing SotA of HotpotQA-ADV, note that we mention the setup we are using is 4 paragraphs (as explained in Appendix B.1) instead of 10 paragraphs. Moreover, instead of releasing the dataset, the HotpotQA-ADV paper [2] released a method of creating the dataset. The numbers in their paper are on a dataset created with randomness. In their codebase, they write “in order to recreate the adversarial training data we used in the paper, randomly sample 40% of the adversarial training data generated using this code and combine with the original HotpotQA training set.” Thus, the numbers are not comparable.\n\n[1] Teaching Broad Reasoning Skills for Multi-Step QA by Generating Hard Contexts (Trivedi et al., EMNLP 2022)\n\n[2] Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QA (Jiang and Bansal, ACL 2019)\n\n[3] From Easy to Hard: Two-Stage Selector and Reader for Multi-Hop Question Answering (Li et al. ICASSP 2023) \n\n[4] Rethinking Label Smoothing on Multi-hop Question Answering (Yin et al., 2022)\n\n[5] Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing (Liang et al., 2019)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A novel dynamic mixture of Hard-EM and MAPO for learning with latent answers.\n- CoQ is showing complementary benefiths with TeaBReaC.\n- Solid improvements over single-step run demonstrating the effectiveness of CoQ.", "reasons_to_reject": "- Baseline models are not strong enough. The performance of GPT3.5 based models may be underestimated in this paper as the GPT-4 Technical Report reported 64.1 F1 on DROP for 3-shot GPT-3.5. Other strong models like QDGAT[1], leading models at https://hotpotqa.github.io/ are not considered. \n- The \"fast improvement at the beginning and better final convergence\" of the dynamic mixture of Hard-EM and MAPO lacks supporting evidence in the paper. \n- The generalization of the proposed method may be challenging due to the prerequisite of QDMR annotation and task-specific preprocessing.", "questions_to_authors": "A. Is the performance of the BERT-based models reported in Table 1 based on BERT-base or BERT-large?\n\nB. Are the results reported in Tables 1 and 2 the maximum, mean, etc., or just from a single run?\n\nC. What are the existing SOTAs of DROP-CS and HOTPOTQA-ADV?", "paper_topic_and_main_contributions": "This paper proposes the Chain-of-Questions framework that generates and answers subquestions for answering complex questions that require multiple steps of reasoning. The model is trained with QDMR but treats sub-answers as latent variables and infers them using a novel dynamic mixture of Hard-EM and MAPO. The Chain-of-Questions framework outperforms selected neuro-symbolic methods and GPT-3.5 on DROP, HotpotQA, and their respective contrast and adversarial datasets.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Table 1 caption: mypurple -> purple"}, "extra_text": {"Missing_References": "[1] Kunlong Chen, Weidi Xu, Xingyi Cheng, Zou Xiaochuan, Yuyu Zhang, Le Song, Taifeng Wang,\nYuan Qi, and Wei Chu. Question directed graph attention network for numerical reasoning over\ntext. arXiv preprint arXiv:2009.07448, 2020."}}, {"review_id": "1kMt6zrRIQ", "reviewer": "Reviewer_dfZy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and are glad they acknowledge Chain-of-Questions is an innovative framework apart from existing methods in the field, as well as its effectiveness in challenging scenarios.\n\n**Q1: While the paper highlights impressive F1 gains on specific evaluation sets, a broader range of evaluation metrics would provide a more comprehensive understanding of the framework's performance.**\n\nA1: Thanks for your suggestions. For completeness, we will add EM score in the camera-ready version, the overall result trends and takeaways are the same as F1 scores.\n\n**Q2: While the results demonstrate superior performance on contrast and adversarial sets, the absence of evaluation on real-world datasets may limit the practical application and impact of the proposed framework.**\n\nA2: We would like to point out many previous works on DROP and HotpotQA [1,2,3,4] do not evaluate on real-world datasets. Also, DROP focuses on arithmetic multi-hop reasoning and HotpotQA focuses on entity-centric multi-hop reasoning. They are quite different benchmarks, which already shows the generality of the CoQ approach.\n\nMoreover, we view generalizing the CoQ method to arbitrary QA dataset as an orthogonal contribution to developing it in the first place. \n\n\n\n\n**Q3: Further details or simplifications could assist researchers in adopting the proposed approach more easily.**\n\nA3: We will release the code after the anonymity period ends.\n\n[1] Teaching Broad Reasoning Skills for Multi-Step QA by Generating Hard Contexts (Trivedi et al., EMNLP 2022)\n\n[2] OPERA: Operation-Pivoted Discrete Reasoning over Text (Zhou et al., NAACL 2022)\n\n[3] From Easy to Hard: Two-Stage Selector and Reader for Multi-Hop Question Answering (Li et al. ICASSP 2023)\n\n[4] Is Graph Structure Necessary for Multi-hop Question Answering? (Shao et al., EMNLP 2020)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\nThe Chain-of-Questions framework presents a novel approach to robust multistep question answering, introducing the generation and answering of sub-questions as a key component. This innovative framework sets the paper apart from existing methods in the field.\n\nThe results showcase significant performance gains, with Chain-of-Questions outperforming strong neuro-symbolic methods by 9.0 F1 on a DROP contrast set. Additionally, the framework surpasses GPT-3.5 by a notable margin of 24.3 F1 on a HotpotQA adversarial set, highlighting its effectiveness in challenging scenarios.\n\nAddressing the technical challenge of lacking annotated answers to sub-questions, the paper proposes a solution that treats sub-answers as latent variables and employs a dynamic mixture of Hard-EM and MAPO for inference. This technical innovation enhances the applicability and robustness of the framework.\n\nBy focusing on multistep question answering and providing impressive performance gains across different evaluation sets, the paper demonstrates the robustness and generalizability of the Chain-of-Questions framework.\n\nThe paper compares the framework's performance against strong neuro-symbolic methods and GPT-3.5, providing a comprehensive evaluation that highlights the superiority of Chain-of-Questions.\n", "reasons_to_reject": "While the paper highlights impressive F1 gains on specific evaluation sets, a broader range of evaluation metrics would provide a more comprehensive understanding of the framework's performance.\n\nWhile the results demonstrate superior performance on contrast and adversarial sets, the absence of evaluation on real-world datasets may limit the practical application and impact of the proposed framework.\n\nThe technical solution presented in the paper, involving a dynamic mixture of Hard-EM and MAPO, may introduce complexity during implementation. Further details or simplifications could assist researchers in adopting the proposed approach more easily.\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces the Chain-of-Questions framework, designed to enhance the robustness of multistep question answering by generating and answering sub-questions. While human-annotated question decomposition meaning representation (QDMR) provides supervision for sub-questions, the absence of annotated sub-question answers poses a technical challenge. To address this, the paper presents a novel approach that treats sub-answers as latent variables, employing a dynamic mixture of Hard-EM and MAPO for inference. The effectiveness and robustness of Chain-of-Questions are demonstrated through impressive performance gains, outperforming strong neuro-symbolic methods by 9.0 F1 on a DROP contrast set and surpassing GPT-3.5 by 24.3 F1 on a HotpotQA adversarial set. Overall, this paper introduces the innovative Chain-of-Questions framework for robust multistep question answering. The framework showcases impressive performance gains, surpassing neuro-symbolic methods and GPT-3.5 on various evaluation sets. The paper addresses the absence of annotated sub-question answers through a novel technical solution, further enhancing the framework's robustness. Although there are limitations in terms of evaluation measures and real-world application, the paper's contributions and advancements make it a valuable addition to the field of multistep question answering.\n\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose to split questions that require multiple steps into finer grained questions, answering them and then combining this to generate the final answer. This is done by training a model over QDMR parsings. As QDMR annotation does not have intermediate answers, the authors use hard EM to train the model to answer the questions.

Reviewers overall like this approach and find the methodology convincing.

On the other hand, the reviewers think that the experimental setup in the paper is limiting as the authors does not evaluate on an eval set for which there is no training data. 
I agree with this -- this is indeed limiting as QDMR parsings are only available for a limited amount of dataset, and hard expensive to produce. In general the setup would be that there is no training set for the target domain so it is important to test the generalization of the proposed method.
Decision: Accept-Main