When Do Decompositions Help for Machine Reading?
Authors: Kangda Wei
Dawn Lawrie
Benjamin Van Durme
Yunmo Chen
Orion Weller
Venue: EMNLP 2023 Main (2023)
Keywords: decomposition
machine reading
question answering
TL;DR: Decompositions are only helpful for machine reading when you have a small amount of labeled data; otherwise learn the decomposition implicitly through fine-tuning

Abstract: Answering complex questions often requires multi-step reasoning in order to obtain the final answer. Most research into decompositions of complex questions involves open-domain systems, which have shown success in using these decompositions for improved retrieval. In the machine reading setting, however, work to understand when decompositions are helpful is understudied. We conduct experiments on decompositions in machine reading to unify recent work in this space, using a range of models and datasets. We find that decompositions can be helpful in zero or limited-data settings, giving several points of improvement in exact match. However, we also show that when models are given access to around a few hundred or more examples, decompositions are not helpful (and can actually be detrimental). Thus, our analysis implies that models can learn decompositions implicitly even with limited data.

When Do Decompositions Help for Machine Reading?
Kangda Wei', Dawn Lawrie", Benjamin Van Durme", Yunmo Chen Orion Weller DDpaarmeent of Computer Science and Engineering, Texas A&M University Department of Computer Science, Johns Hopkins University kangda@tamu.een edu, {lawrie, vandurme, yunmo, oweller2j@jhu.ee
Abstract
No Decomposition
Who founded View Askew Productions and produced numerous movies starring Jason Lee?
Answering complex questions often requires multi-step reasoning in order to obtain the nal answer. Most research into decompositions of complex questions involves open-domain systems, which have shown success in using these decompositions for improved retrieval. In the machine reading setting, however, work to understand when decompositions are helpful is understudied. We conduct experiments decompositions in machine reading to unify recent work in this space, using range of mod- els and datasets. We find that decompositions can be helpful zero or limited-data settings, giving several points of improvement in exact match. However, we also show that when mod- els are given access to around a few hundred or not helpful
Explicit Decomposition
Who co-foundeed View Askew Procluctions?
Who produced numerous movies with Jason Les?
3. Who is in both #1 and #2
Final Answer
Implicit Decomposition
SEI INTERSECTION Who co-founded View Askew Productions and produced numerous movies starring Jason Lee?
Figure 1: An instance of HotpotQA in BREAK (Wolf- al., 2020), showing three different decomposition settings: (I) No Decomposition, i.c. regular question answering. (2) Explicit Decompositions that use itera- tive sub-questions, and (3) Implicit Decompositions that prepend the reasoning steps as special tokens.
more examples, decompositions (and can actually be detrimental). Thus, our analysis implies that models can learn decom- positions implicitly even with limited data.
answering (e.g. in the open domain). In that setting, results have consistently shown that question an- swering (QA) models perform better on multi-step questions when they use decomposed questions to help with retrieval (Wolfson et al., 2020; Perez et al., 2020; Geva et al., 2021a). The few works that have used decompositions for machine reading do in limited settings, leaving its effectiveness unclear (Guo et al., 2022: Patel et al., 2022).
1 Introduction
Much past work has examined and improved mod- els' ability to answer complex questions that re- quire multiple steps of reasoning (Welbl et al., 2018; Talmor and Berant, 2018at 8a; Dua et al., 2019a; Wolfson et al., 2020; Weller et al., 2020; Weir and Van Durme, 2022). consistent theme in these works is to break the main complex question down into series of sub-questions to be solved, which is referred to as question decomposition. These meth- human
Therefore, we seek shed light on if and when decompositions are helpful for machine read- ing. To do so, we analyze decomposition methods for several QA models across two multi-step QA datasets. Our results show that decompasitions are only helpful in the low data setting, where there are less than few hundred examples. Using decompo- sitions in anything other than that setting performs the same (or much worse, depending on the strat- egy) as simply training the model end-to-end.
ods generally represent decompositions would, with explicit natural language sub-questions that build together the final answer.
Despite the large amount of research in decom- positions for multi-step question answering, the majority of it has focused on using question decom- position for both information retrieval and question
We publicly release all code at ht ttt::///ggtttt com/ We MKKaaaaa/uuusttt in www-eeooppssiion Joint advising
Thus, overall, decompositions are helpful for question answering when they are used in two main

---

DROP
HotpotQA
0.6
Explkit
0.6
Expliit wiDecomp
Explicit Everything
0.5
Implicit
0.5
Implicit Unordered
0.4
EM Score
Implicit w/Dupi.
EM Score
0.4
Explicit implicit
No Decomp
0.3
0.3
0.2
0.2
0.1
0.0
0.0
bart-base
t5-base Model
unifiedga-v2-t5-larg
bart-base
t5-base Model
unifiedqa-v2-t5--oa
Figure 2: Main results showing the effect of various decomposition strategies. Left shows results DROP, right shows HotpotQA. Runs were done over three random seeds and report the mean; error bars indicate SD.
settings: (1) information retrieval, where the de- compositions help isolate distinct aspects of the question, or (2) in zero low resource settings, where there isn't enough data to implicitly learn the multi-step process through end-to-end training.
The BREAK annotations include the list of de- composed questions that eventually yield the same answer the original question, along with oper- ator assigned each decomposition that represents the type of reasoning for that step (e.g. Boolean, Comparison, etc.). Note that there are no gold la- bels for intermediate decomposition steps; the only ground truth label is for the main question.
2 Experiment Setup
2.1 Data
We use the BREAK (Wolfson et al., 2020) resource which contains annotated decompositions for 10 different benchmarks, including three reading com- prehension benchmarks. BREAK provides human- written decompositions for all dataset splits (e.g. train, test) which we use, thus assuming the oracle decomposition case. We use two of these three datasets (HotpotQA and DROP) as the third, Com- plex WebQuestions (Talmor and Berant, 8b)). does not currently provide training set.² Follow- ing BREAK and other follow-up work (Geva et al., 2021b), train and test our models using the high- level decompositions (also known as QDMRs).
2.2 Models
To explore the effect of decompositions on various types of common NLP models, we employ five dif- ferent models: BART (Lewis et al., 2020), vanilla T5 (Raffel et al., 2020), UnifiedQA-v2 (Khashabi et al., 2020, 2022) which uses a T5 backbone and has been fine-tuned other QA datasets but not HotpotQA (Raffel et al., 2020), LLaMA-7B (Touvron et al., 2023) and Alpaca-7B (Taori al., 2023). This allows us demonstrate the effect of additional fine-tuning (UnifiedQA vs vanilla T5), different architectures (BART vs T5 vs LLaMA), and inrruction-tunnng (LLaMA vs Alpaca). Note that because UnifiedQA-v2 was multi-task trained on DROP, its scores on it are noticeably higher than the other models (Figure 2). However, is compare scores between models, but rather compare scores between different decomposition strategies. Thus, the inclusion this model on DROP shows us that our results hold even if the model was pretrained on it. For more hyperparam- eter and compute details, see Appendix A.
Thus, we use HotpotQA (Yang et al., 2018) and DROP (Dua al., 2019b) in our experiments, using the portions annotated from their train and valida- tion sets. HotpotQA was created from workers on Mechanical Turk who annotated compositional questions from Wikipedia by using the page links (an example can be found in Figure 1). DROP was created test discrete multi-hop reasoning and was also annotated by Mechanical Turk workers who used Wikipedia articles as the context.
purpose
2.3 Decomposition Strategies
NNaee that although the ComplexWebbuuettonne WebQuestions (CWQ) paper initially released the training set the authors have since removed it from the official Dropbox. Furthermore, even if the dataset was available CWQ docs not verify that the questions are answerable from the returned Google search passages.
There are many possible ways to combine decom- position with model fine-tuning. We try a wide variety of techniques (including novel ones) that

---

we group into three categories: (1) no decompo- sition e.g. the baseline QA format, (2) explicit decomposition, and (3) implicit decomposition.
Method
Alpaca
LLaMA
Explicit
(1.8
(3.4
Explicit w/Decomp
53.8(2.0)
(7.7
Explicit Decomposition Explicit decomposi- tions are the most common approach in the decom- position literature, generating the answer iteratively through sub-questions: the model answers the first decomposition step, then replaces placeholders in future decomposition steps with that predicted an- swer, then predicts the second decomposition step, and so forth. Note that using this method (Explicit) naively presents issues with backpropagation, as the model can only backpropagate through the last decomposition step. Variations of strategies this category include giving the model all previous de- composition steps as context (Explicit w/Decomp) or including all decomposition steps and all pre- dicted answers as context (Explicit Everything).
Explicit Everything
58.7(1.3)
(5.5
Implicit
(7.0(1.6)
(0.9
Implicit Unordered
58.3(0.7)
50.9(0.6)
Implicit w/Dupl.
58.5(2.1)
(3.7
Explicit & Implicit
(5.0(
59.3(0.7)
No Decomp
(0.6)
59.8
Table I: Results of using larger LLMs, Alpaca and LLaMA, on HotpotQA. No Decomp still performs the same (within two standard deviations) or better when comparing to methods that use decompositions.
these have issues with training end-to-end, as an Ex- plicit decomposition is not differentiable through all dccomposition steps. Thus, we only end up differentiating through the last step of the explicit decomposition steps, leaving the model unable to learn as effectively. All other approaches to decom- position perform comparably, given random seed variance (e.g. t5-base DROP Implicit Decomp. is 33.7% exact match + 0.9% vs No Decomp 34.0% + 0.4%). In fact, in this full data setting, the No Decomp method performs better or statistically sim- ilar to every other method according to two-sample t-tests with the Bonferroni correction (Weisstein, 2004), across all datasets and models.
Implicit Decomposition Another way to use de- compositions could be to add them implicitly. To do so, we utilize the operators provided in the BREAK annotations which describe the type of reasoning needed, removing duplicate operators and keeping them in their given order. For exam- ple, in Figure the model uses select twice and then intersection in the explicit decomposition rea- soming steps (but we remove the duplicate select). We implement this in practice by adding a new spe- cial token for each operator and prepending them the original question. Although this approach is novel in the context of decompositions, bears similarity to work in the prompting literature, such as soft prompts (Qin and Eisner, 2021; ;; Liu et al., 2022). Variations to this upproach in the same cate- gory include randomizing the order of the special tokens (Implicit Unordered), leaving in duplicate special tokens (Implicit w/Dupl.), or even prepend- ing these operators to the explicit sub- sub-questions in the Explicit Decomposition approach combine the two strategies (Explicit Implicit).
We show the two more recent and larger mod- els (LLaMA and Alpaca) in Table on Hotpot, reporting the mean and standard deviation of the EM score. We find that No Decomp method still performs similarly or outperforms all other decom- position methods (e.g. for Alpaca 59.9 EM with No Decomp vs 54.8 EM for Explicit Decomp). Thus, the conclusion that decompositions are helpful only when there is not enough labeled examples still holds for newer and larger models and even for instruction- tuned models like Alpaca.
Size Experiment Is the No Decomp method al- ways the same or better than the decomposition methods, or can decompositions help in the low- data regime? We answer this question in Figure 3, by varying the amount of training data, comparing the zero-shot Explicit method that typically used (Patel et al., 2022; Dua et al., 2022).3 As we need the zero-shot model to be fine-tuned on
3 Results
Full-Data Experiments We see the main results in Figure 2 with results for DROP on the left and HotpoiQA on the right. Bars are colored-coded according to their method. All bars the mean of three random seeds and error bars indicate the stan- dard deviation. We see that most methods perform nearly the same, except for two that underperform: Explicit and Explicit + Implicit. Note that both of
"Note that finc-tuned Explicir is worse than the zero-shot version in low-data settings (See Figure 3b) We do not evaluate on DROP as UnifiedQAA12 was already trained on DROP.

---

QA to handle the decomposed questions," 4 we use modified version of T5-base (valhalla/lS-base- squad from Huggingface) that was fine-tuned on SQuAD (Rajpurkar et al., 2016). This places the No Decomp method at a disadvantage, as it was not trained on SQuAD and started from T5-base Despite that, however, we see that as the amount of data increases, the fine-tuned models perform better; eventually surpassing the zero-shot method between 100-250 examples for UnifiedQA-v2 and 250-1000 examples for T5.²
0.58
0.56
0.54
Exact Exact Match
0.52
0.50
0
4000
8000
10000
Error Analysis Why does the zero-shot Explicit method perform worse than the fine-tuned No De- comp? In Table 2 we show representative errors from the Explicil method with how often they oc- curred. We randomly sample 50 errors and cate- gorize them into three groups: wrong predictions in the last step, eтrоr propagation from interme- diate steps, and invalid/missing annotations from BREAK (i.e. not the model's fault). We found that the biggest category was predicting an invalid an- notation (42%), i.e. an alias that the dataset did not contain, followed by error propagation (40%) and then wrong last predictions (18%). Thus, compared the non-iterative methods, the iterative process allows error propagation that occurs in roughly 40% of errors, contributing to its lower compar- ative scores (see Appendices and C for other error analyses on No Decomp and cases where No Decomp was better than Explicit).
# of Training Examples
(a) Results from UnifiedQA-v2
Match
No. Decompaaition
Explicit Everything
Explicit Decompasition
0
2000 4000
6000
8000
10000
# of Training Examples
(b) Results from T5-hase
Figure 3: Size Experiments w/HotpotQA. For Figure with UnifiedQA-v2, the red line shows the score of zero- shot Explicit while the blue line is the fine-tuned No Decomp method. For Figure 3b, the red line shows the score of zero-shot Explicit from valhalla/t5-ban squad while the blue lines are fine-tuned TS-base with No Decomp, Explicit Everything, and Explicit methods. Results are over 3 random seeds and show the mean and SD. See Appendix similar results with Alpaca.
4 Related Work
Decompositions in QA Decompositions for QA have long history in complex question answer- ing (Perez et al., 2020; Min al., 2019; Geva et al.. 2021a) with recent interest using them for large language models (Wei et al., 2022; Dua et al., 2022; Zhou et al., 2023; Press et al., 2023). Two of the most related works to ours include Pa- tel et al. (2022), who focus on decompositions in the zero-shot setting only and show improvements (which aligns with our results), and other work (Guo et al., 2022) that shows that decompositions help on the full DROP dataset but which doesn't 't include comparable non-decomposition baseline on the same data. Our analysis complements and
unifies our understanding of decompositions by identifying when decompositions help with respect to dataset size.
Decompositions in other fields Our results for decompositions in textual QA helps to unify results across machine learning, as similar conclusions (e.g. decompositions being less effective then end- to-end methods with large data) can be seen from scores on Computer Vision visual QA leaderboards (Hudson and Manning, 2019; Li et al., 2021).
WW use the same UnifiedQA-v2 model for both zero-shot and fine-tuned it was already trained on QA.
Decomposition Strategies and Prompting De- compositions methods are also related to prompt- ing, where the explicit decompositions can be seen as a hard prompt (Liu et al., 2021: Su et al., 2022)
*We also show several other decomposition methods over various amounts of fine-tuning in Figure 3b, all starting from TS-hase. We see that they perform worse than the No Decomp method at every point.

---

Question
Decompositions
Content (shortened)
Intermediate Predictions
Answer
Error
Who born first, Kwok Kin Pong or Edison Chen?
#1: when was Kwok Kin Pong born?
Edison Koon-hei Chen
#1 30 March
Edison
Wrong Prediction at Last Step (18%)
(hor October 1980)
1987
Chen
#2: when was Edison Chen born?
Kwok Kin Pong
#2. 7 Octooerr
(born 30 March 1987
1980
I1: which is lowest of #1,N??
Hong Kong)... ..
W3: Kwok
Are both Decrhunter and
#1. is Deerhunter a American
Nine lashes
#1. Yes
No
Error
Christian rock band?
American Christian rock band
#2. Yes
propagation (40%)
Nine Lashes
#2. is Nine Lashes a American
#3. Yes
American Christian
Christian rock band?
Deerhunter is an American rock band from Atlanta
rock bands?
#3: if both #1 and #2 true
What actor from "Willow" also starred in "The Usual Suspects"?
#1: who the actor that starred
Kevin Elliot Pollak a role in "Willow". the Usual Suspects stars Kevin Pollak
#1. Kevin
Kevin
Invalid
The Usual Suspects?
Pollak
Elliot
Missing
#2: #1 that was a actor from
#2. Kevin
Pollak
Annotation
Willow?
Pollak
(42%)
Table 2: Error analysis for zero-shot decomposition (i.e. Explicit with fine-tuning, la Patel et al. (2022)) on HotpotQA. Percentages calculated from annotation of 50 instances with representative examples shown. See
Appendix for full data No Decomp or Appendix for when No Decomp succeeded and Explicit failed.
and the implicit decompositions are similar soft prompts (Qin and Fisner, 2021; Liu al., 2022). Other research has looked at developing new prompting methods which either better handle com- plex questions or automatically generate decompo- sitions (as opposed to the human-writted decom- positions in BREAK) (Zhou et al., 2022; Weller et al., 2022, 2023; Press al., 2022). Our work focuses only on human written decompositions, which were shown to be better than automatically generated decompositions in the BREAK paper.
plex question answering datasets that have human- annotated decompositions. However, it is possi- ble that in the future someone could create an- other decomposition--asse dataset that could show slightly different results. We believe this to be un- likely, as our empirical study holds across models and datasets. Another limitation is that do not alternate tasks, such summarization, tagging, etc. as our work focuses on how decompositions work in question answering only, given the large interest using decompositions for QA.
5 Conclusion
References
Our work explored when decompositions are help- ful for machine reading. We showed that decom- positions are helpful when there is limited data available, or when parameters cannot be tuned. However, when enough data exists (empirically around few hundred instances) and parameters can be fine-tuned, it is best to let the model learn the decompositions implicitly through end-to-end training. Furthermore, we show that limitations of not fine-tuned decomposition approaches include еттоr propagation of intermediate steps while also introducing more possibilities for annotator error. We hope that work will help to inform readers they create new datasets and select methods to use for complex question answering.
Dheeru Dua, Shivanshu Gupta, Sameer Singh, and Matt Gardner. 2022. Successive prompting for decom- posing complex questions. In Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pages 1251-1265, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi. Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019a. DROP: reading comprehension benchmark requir- ing discrete reasoning over paragraphs. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Techmologies, Volume (Long and Short Papers), pages 2368-2378. Min- neapolis, Minnesota. Association Computational Linguistics.
6 Limitations
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019b. DROP: reading comprehension benchmark requir- ing discrete reasoning over paragraphs. In Proceed-
Our work has explored the machine reading setting, using what is to our knowledge the only large com-

---

ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human I Language Technologies, Volume I (Long and Short Papers), pages 2368-2378, Min- neapolis. Minnesota. Association for Computational Linguistics.
Vision, ICCV 2021. Montreal, QC, Canada, October 10-17, 2021, pages 14890-14899 IEEE
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang. Hiroaki Hayashi, and Graham Neubig, 2021. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ArXiv preprint, 0bs/2207.115886
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot,
Dan Roth, and Jonathan Berant. 202 Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346- 361.
Xiao Liu, Kaixuan Ji. Yicheng Fu, Weng Tam, Zhengx- iao Du, Zhilin Yang, and Jie Tang. 2022. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 61 16-,, Dublin, Ireland. Association for Computational Lin- guistics.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021b. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346- 361.
Sewon Min, Victor Zhong, Luke Zettlemoyer, and Han- naneh Hajishirzi. 2019. Multi-hop reading compre- hension through question decomposition and rescor- ing, arXiv preprint arXiv: 02916.
Xiao-Y Yu Guo, Yuan-Fang Li, and Gholamreza Haf- fari. 2022. Complex reading comprehension through question decomposition. In Proceedings of the The 20th Annual Workshop of the Australasian Language Technology Association, pages 31-40, Adelaide, Aus- tralia. Australasian Language Technology Associa- tion.
Pruthvi Patel, Swaroop Mishra. Mihir Parmar, and Chitta Baral. 2022. Is question decomposition unit all we need? In Proceedings of the 2022 Confer- ence Empirical Methods in Natural Language Processing, pages 4553-4569, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Drew A. Hudson and Christopher D. Manning. 2019. GQA: A new dataset for real-world visual reason- ing and compositional question answering. In IEEE Conference on Computer Vision and Portern Recogni- tion, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 6700-6709. Computer Vision Founda- tion/ / IEEE.
Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela. 2020. Unsupervised question decomposition for question answering. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8864-8880, Online. Association for Computational Linguisties.
Daniel Khashabi, Yeganeh Kordi, and Hannaneh Ha- jishirzi. 2022. Unifiedqa-v2: Stronger generalization via broader cross-forman training. ArXiv preprint, abs/2202.12352
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. 2022. Mcasuring and narrowing the compositionality gap in language models. arXiv preprini 22X.i:2210.033550
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Han- nanch Hajishirzi. 2020. UNIFIEDQA: Crossing for- mat boundaries with single QA system. In Find- ings of the Association for Computational Linguistics: EMNLP 2020, pages 1896-1907, Online. Association for Computational Linguistics.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models.
Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying LMs with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 5203-5212, Online, Association for Computa- tional Linguistics.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. seguencc-to-sequeeee
BART: Denoising sequence-tor to-sequnnce pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21:140:1-114066
Zhuowan Li. Elias Stengel-Eskin Yixiao Zhang, Cihang Xie, Quan Tran, Benjamin Van Durme, and Alan I.. Yuille. 2021. Calibrating concepts and operations: Towards symbolic reasoning on real images. In 2021 IEEE/CVÉ International Conference on Computer
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of

---

the 2016 Conference on Empirical Methods in Natu- val Language Processing, pages 2383-2392,, Austin, Texas. Association for Computational Linguisties.
Orion Weller, Aleem Khan, Nathaniel Weir, Dawn Lawrie, and Benjamin Van Durme. 2022. Defending against poisoning attacks in open-domain question answering. arXiv preprint 22f1:22222.1002.
Yusheng Su, Xiaozhi Wang. Yujia Qin, Chi-Min Chan,
Yankai Lin, Huadong Wang. Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, Lei Hou, Maosong Sun, and Jie Zhou. 2022. On transferability of prompt tuning for natural language processing. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 3949-390 3969, Seattle, United States. Association for Computational Linguistics.
Orion Weller, Nicholas Lourie, Matt Gardner, and Matthew E Peters. 2020. Learning from task de- scriptions. arXiv preprint arXiv:2011.081/1
Orion Weller, Mare Marone, Nathaniel Weir, Dawn Lawrie, Daniel Khashabi, and Benjamin Van Durme. 2023. according to..." prompting language mod- els improves quoting from pre- pre-training data. arXiv preprint arXiv:2305. 13252.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface's 's transformers: State-of- the-art natural language processing. ArXiv preprint, abs/ abs//1910.05 100.03771.
Alon Talmor and Jonathan Berant. 2018a. The web as a knowledge-base for answering complex ques- tions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistiex: Haman Language Technolo- gies, Volume 1 (Long Papers), pages 641-651, New Orleans, Louisiana. Association for Computational Linguistics.
Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gard- ner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. 2020. Break it down: A question understand- ing benchmark. Transactions of the Association for Computational Linguistics, 8:183-198.
Alon Talmor and Jonathan Berant. 2018b. The web
as a knowledge-base for answering complex ques- tions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Haman Language Technolo- gies, Volume (Long Papers), pages 641 641-651, New Orleans, Louisiana. Association for Computational Linguistics.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D. Manning. 2018. HotpoiQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2369-2380, Brussels, Belgium. Association for Com- putational Linguistics.
Rohan Taori. Ishaan Gulrajani. Tianyi Zhang, Yann Dubois. Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction--ollowee gi github. www. labttsu-labsseeeeee watab//aaafoddaapaa....
llaaa model. https://
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang. Dale Schuurmans. Claire Cui, Olivier Bousquet, Quoc Le, et al. 2022. Least-to-most prompting enables complex reason- ing in large language models. arXiv preprint arXiv:2205.1088
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet. Marie-Aune Lachaux. Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023, Llama: Open and efficient foundation language models.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Lc, and Ed Chi. 2023. Least-to-most prompting enables complex rea- soning in large language models.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Lc, and Denny Zhou, 2022. Chain of thought prompting elicits reasoning in large language models. ArXiv preprint, abs/2201. 1190..
A Hyperparameter and Compute Details
We train our models following the hyperparameters in Wolfson et al. (2020), e.g. a learning rate of le-5 for 5 epochs, using early stopping on a holdout of the training set (5%) to determine the best saved model. The best models were typically ones trained for around 2 epochs. We use T5-base, BART-base, UnifiedQA-V2- V2-LLage,, LLaMA 7B, and Alpaca 7B. For the zero-shot decomposition UnifiedQA-v2 approach (perhaps due to its multi-task pre-training on other QA datasets), we found that using the original question plus the iterative decompositions
Nathaniel Weir and Benjamin Van Durme. 2022. Dy- namic generation of interpretable inference rules in a neuro-symbolic expert system. arXiv preprint ar arXiv:2209.07662.
Eric W Weisstein. 2004.
Bonferroni correction. https:/mathworld. wolfram com/.
Johannes Welbi, Pontus Stenetorp. and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics, 6:287-
302.

---

added solid boost final performance and hence we use those results it provides the strongest zero-shot baseline. Each zero-shot decomposition run took approximately 15-30 minutes to evaluate.
0.5
Compute time ranged from 1 hr for the shortest jobs with smaller data sizes and non non-iterative train- ing to around 18 hours for iterative decompositions methods with UnifiedQA-v2-latt --aragge 1 RTX 6000 GPU. For LLaMA and Alpaca,
Exact/Match
0.4
0.3
one 80GB
A100 GPU with runs taking between 12-24 hours.
No Desompasition
Explicil Decompasition
Data was prepared building off of the original BREAK authors' code. Models were accessed via the Huggingface repository (Wolf et al., 2019).
4000
6000
8000
# of Training Examples
Note that our reported no- oo-decompositionn results DROP with T5 show comparable or greater per- formance Guo al. (2022) when using only the BREAK dataset.
Figure 4: Size Experiments w/HotpotQA using Alpaca. Explicit and No Decomp results are shown. Results are over random seeds and show and SD.
sults to the error analysis in Table 2 where Error Propagation causes the most errors.
Error Analysis for Full-Data No Decomp
D Size Experiment for Alpaca
We also do an error analysis for the full-data ver- sion the No Decomp method to compare with the Explicit error analysis done in the main paper. As error propagation not a possible category for this model, since it is end-to-end, there are only two er- ror categories we use: (1) Wrong Prediction (54%) and (2) Invalid or Missing Annotation (46%). We that missing annotation/alias was the cause of the incorrect answer 46% of the time, which is comparable to the 42% of the time the zero-shot Explicit method had an error due to an alias. For the other 54%, the model would output "yes" when it should be "no" or extract the wrong string from the passage leading to an incorrect prediction, etc.
We also performed the size experiment in Section 3 with Alpaca-7B Shhavinooalppcc-natvve and show the results in Figure Note that Alpaca is roughly 10 points better than T5-base (approximately 60 EM vs 51 EM). The No Decomp method performs similarly or better than the Explicit method in all cases with varying training data size. Our conclu- sion still remains across models and decomposition strategies: No Decomp is better than or similar the Decomp models, when given enough training examples.
C Error Analysis With No Decomp succeeded and zero-shot Explicit Decomp failed
Based on Table 2, the Invalid Missing Anno- tation takes up the largest percentage of error, which could happen to both decomposition and nonedeecomnosition method. In order to further strengthen our point that question decompositions cause eтror propagation, thus underperforming the No Decomp method overall, we also conduct an error analysis where the fine-tuned No Decomp suc- ceeded and the zero-shot Explicit Decomp failed, with 20 sampled errors using UnifiedQA-v2. The new results are: (1) Error Propagation (45%), (2) Wrong Prediction at Last Step (35%), and Invalid or Missing Annotation (20%). We see similar re-


Reviews:
[{"review_id": "xu4O0myEbZ", "reviewer": "Reviewer_SCmL", "summary": "", "questions": "A. For fine-tuning with decomposition, are the test example also converted into corresponding decomposition format used in the training?\n\nB. During training/evaluation, are context/supporting facts also passed to the model or only the questions are used?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and for recognizing that our work has “main claims [that] are clear and can be routed to the supporting evidences easily” and “a good number of decomposition strategies are attempted”!\n\n---\n\n> While I’m not expecting the author to conduct experiments on more powerful models\n\nPrevious work (“Is a Question Decomposition Unit All We Need?” by Patel et. al 2022 and others cited in our paper) have claimed to show the positive effects of decomposition for even standard size T5 models. Thus our experiments help shed more light on when decompositions are useful, using the same models previously studied. \n\nHowever, we agree that newer and bigger models are always helpful, so we have added Llama and Alpaca-Llama experiments on HotpotQA, which look similar to the T5 results. We can’t include a figure nicely in OpenReview, but here are the results in Table form (again, it has the same conclusion as the T5 results, e.g. No Decomp is better in all cases or within 2 SD otherwise). **This shows that even for newer models our findings hold: decompositions are helpful only when there are not enough labeled examples.**\n\nScores are the mean and standard deviation of EM, like in the paper. Note that they are also better absolute scores than UnifiedQA-v2, despite only seeing the training set of HotpotQA (as opposed to the many QA datasets UnifiedQA-v2 trained on before further fine-tuning on HotpotQA train).\n\nAlpaca:\n|Decomposition Method|Scores            |\n|--------------------|------------------|\n|Explicit            |54.8 +- 0.4       |\n|Explicit w/Decomp   |53.8 +- 2.0         |\n|Explicit Everything |58.7 +- 1.3       |\n|Implicit            |57.0 +- 1.6         |\n|Implicit Unordered  |58.3 +- 0.7       |\n|Implicit w/Dupl.    |58.5 +- 2.1       |\n|Explicit & Implicit |55.0 +- 0.7         |\n|No Decomp           |59.9 +- 0.6       |\n\nLlama:\n|Decomposition Method|Scores            |\n|--------------------|------------------|\n|Explicit            |53.4 +- 3.5       |\n|Explicit w/Decomp   |57.7 +- 0.3       |\n|Explicit Everything |55.9 +- 0.5       |\n|Implicit            |60.9 +- 0.9       |\n|Implicit Unordered  |50.9 +- 0.6       |\n|Implicit w/Dupl.    |53.7 +- 0.6       |\n|Explicit & Implicit |59.3 +- 0.7       |\n|No Decomp           |59.8 +- 1.2       |\n\nWe agree with unlimited time and compute, larger models (like 65B) would be better, but due to computational resources we could only do 7B models. We will add these tables to our appendix to further strengthen our claims.\n\n---\n> A. For fine-tuning with decomposition, are the test example also converted into corresponding decomposition format used in the training?\n\nYes, during test time, questions are also asked in the decomposition format (e.g. gold decompositions at test time). This provides the strongest baseline to compare to the No Decomp model, and despite it they are still similar or worse than No Decomp. \n\n> B. During training/evaluation, are context/supporting facts also passed to the model or only the questions are used?\n\nDuring training/evaluation, both contexts and questions are used, following previous work in MRC. \n\n---\n\n**References**: we will make sure to add the missing reference. Thank you for pointing that out!\n\n**Presentation**: thank you for the suggestions, we will update the paper and add additional example input-output pairs (beyond Figure 1) to the appendix", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is concise, the main claims are clear and can be routed to the supporting evidences easily.\n- A good number of decomposition strategies are attempted, which further reinforces the claim of the paper.", "reasons_to_reject": "- While I’m not expecting the author to conduct experiments on more powerful models, as the paper is making claim on the universal usefulness of decomposition, it’s good noting that for larger model the decomposition can work better (see Missing References). Performing error analysis on fine-tuning with decomposition can be helpful to understand which model capacity is essential for decomposition to work.", "questions_to_authors": "A. For fine-tuning with decomposition, are the test example also converted into corresponding decomposition format used in the training?\n\nB. During training/evaluation, are context/supporting facts also passed to the model or only the questions are used?", "paper_topic_and_main_contributions": "The paper aims to study whether or not question decomposition is helpful for machine reading comprehension tasks.\n\nThe paper conducted experiments on a few scenarios: explicit decomposition fine-tuning, implicit decomposition fine-tuning, and zero-shot explicit decomposition, compared against fine-tuning with varying amount of training data with no decomposition. The result from three models (bart, t5, unifiedqa-v2) on two datasets (DROP, HotpotQA) are collected.\n\nThe paper shows that incorporating decomposition in the fine-tuning process is either not helpful or detrimental. The zero-shot decomposition is outperformed by no decomposition fine-tuning with a few hundreds examples.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- For zero-shot decomposition, works like https://arxiv.org/abs/2205.10625 and https://arxiv.org/abs/2210.03350 have proposed prompting strategies that encourages the model to automatically decompose the question and answer itself, which are good to include in the related works.", "Typos_Grammar_Style_and_Presentation_Improvements": "- In table 1, I think the error is wrong prediction at the first step instead of the last step?\n- Showing an input-output pair for the decomposition fine-tuning strategies will be helpful for readers to understand what learning signal each method is contributing to the model."}}, {"review_id": "91bslPWBe3", "reviewer": "Reviewer_mgx8", "summary": "", "questions": "See second point above.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and for recognizing that our work “addresses an under-explored area in the QA/decomposition field”!\n\n---\n> What is the added value of studying specifically machine reading questions?\n\nThe added value of studying machine reading (as opposed to ODQA) is that previous work has shown that decomposition helps in ODQA for improving retrieval. However, in MRC there is no retrieval so it is unclear if decomposition would help. We show that decompositions only helps when there’s not much data. \n\nThus, we specifically do not study ODQA because we do not want to confound our experimental results with retrieval as that has been well-studied. We instead focus on studying the LMs and QA. **We agree this is important for the paper and do already include this discussion (of this motivation and definitions) in our paper on lines 33-45**.\n\n---\n\n> where do the decompositions come from at test time? Do the unified-QA/t5 models perform this decomposition in a zero-shot setting without seeing any such example? \n\n**All decompositions, train and test, are provided by BREAK and are written by humans**. Thus, no model ever has to perform the decomposition as the decomposition is provided. We note that this means it is the oracle decomposition setting (e.g. gold decomposition at test time) and provides the strongest baseline for us to compare against. **Since No Decomp is still better or similar to them, this strengthens our claim that decompositions are not needed when you have enough training data**.\n\nSince the models just need to do regular QA (and not provide the decomposition themselves) it is fine for them to do standard QA and is a fair comparison. We note that the BREAK paper showed that model generated decompositions are worse than these human decompositions so we are using the best decompositions available. \n\nWe will update lines 65-58 to make this more clear.\n\n---\n\n> it would be more informative to examine only the set of errors where the fine-tuned succeeded and the zero-shot did not. \n\nWe believe that our current error analysis shows the point we were hoping to make (that decompositions cause error propagations). **However, upon your suggestion we also ran your suggested error analysis and will add it to the appendix**. \n\nThe new error analysis (where the fine-tuned No Decomp succeeded and the zero-shot Explicit Decomp failed, with 20 sampled errors) using UnifiedQAv2:\n\n| Reason               | Percent |\n|----------------------|---------|\n| Error Propagation    | 45%    |\n| Wrong Prediction at Last Step | 35%    |\n| Invalid or Missing Annotation   | 20%    |\n\n**We see that the results are similar to the previous error analysis in the paper**, as the No Decomp model was better than the Explicit case because of the lack of error propagation roughly 45% of the time (the current error analysis in the paper says 40% so they are similar). \n\nWe hope this helps assuage your concern and will add this to the appendix to help future readers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper addresses an under-explored area in the QA/decompositions field.", "reasons_to_reject": "* Motivation - what is the inherent difference of machine learning vs open-domain questions? What is the added value of studying specifically machine reading questions? For example, the question presented in Figure ` from HotpotQA can be both an open-domain question and a machine reading question (i.e. depending on whether context is included) -- I think an additional discussion explaining the different characteristic of the required reasoning would be helpful.\n* Experiment details - I am not sure I understand the zero-shot setup in the size experiment as it is not explicitly stated, where do the decompositions come from at test time? Do the unified-QA/t5 models perform this decomposition in a zero-shot setting without seeing any such example? As this is a main result for the paper (full-data results with decompositions did not improve accuracy) this seems like a crucial point.\n* Error analysis - since biggest category has invalid annotations, errors which are probably exhibited for both models, it would be more informative to examine only the set of errors where the fine-tuned succeeded and the zero-shot did not. At the moment it is hard for me to tell why does the zero-shot method perform worse.", "questions_to_authors": "See second point above.", "paper_topic_and_main_contributions": "The goal of this work is to understand if and when decompositions are useful in machine learning, as most research thus far has investigated open-domain.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oSMoREl8Vn", "reviewer": "Reviewer_dFMJ", "summary": "", "questions": "- In lines 165-171, the authors explained why the Explicit and Explicit + Implicit methods didn't work well (problem of Explicit method). Then, couldn't we use variation (Explicit w/Decomp or Explicit Everything) instead of Explicit method in Explicit + Implicit method?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and for recognizing that our work “addresses an under-explored area in the QA/decomposition field”!\n\n---\n> I would like to raise a concern on the size experiment of Figure 3 ... In my opinion, this experiment should have been based on the pretrained LMs rather than the SQuAD fine-tuned QA models\n\n**We agree with this sentiment and have already done so!** We used two different models to conduct the size experiment: one is UnifiedQA-V2, which is fine-tuned on SQuAD (and many others), but the other one is base T5, which is a pretrained LM (which is what you’re suggesting). We showed that our conclusion holds in both cases (with the LM fine-tuned on QA datasets and not fine-tuned on QA).\n\n---\n\n> In order for this paper’s conclusion to be more meaningful, it would be good to add an analysis of the cause ... does an ability to decompose a question arise within a QA model given sufficient training data?\n\nWe definitely agree that the decomposition provides an inductive bias and that it is helpful to have an analysis of the cause - in fact **we provide this in the Size Experiments** (Figure 3). These charts show how the scores improve when given N training examples of multi-hop QA instances, just as you suggest (both for UnifiedQA which has seen QA and for T5-base which has not).\n\nIf you mean that our result is not surprising, we would agree but mention that the community does not agree, see *“Is a Question Decomposition Unit All We Need?”* by Patel et. al. 2022 and others cited in our paper that suggest question decompositions are better. Hence why we wrote this paper to inform the community since others don’t agree and to provide evidence.\n\n---\n> couldn't we use variation (Explicit w/Decomp or Explicit Everything) instead of Explicit method in Explicit + Implicit method?\n\nYou are correct that this variation could be run, as there are many combination approaches that could be tried. However, it does not likely to change the outcome (since each of Explicit w/Decomp and Implicit do not perform better than No Decomp) and we already tried many novel approaches in our paper.  **Due to the combinatorial explosion of potential combinations of methods, the fact that no one uses those combinations, and the difficulty of displaying that many results, we did not include more than one combination approach.**\n\n---\n\n**Presentation**: thank you for the feedback. We will update the colors and remove the word concurrent.\n\n**References**: We will add the reference, thank you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The writing is generally neat, and the paper is easy to read/understand.\n\n- The conclusions presented in the paper can be used as a guideline for other researchers which can be helpful to the community.\n\n- Experiments are conducted on three models and two datasets to show that their claims are not limited to a specific model or dataset.", "reasons_to_reject": "- I would like to raise a concern on the size experiment of Figure 3. The authors use QA models fine-tuned on SQuAD (single-hop QA dataset) to compare zero-shot Explicit method and No Decomp method. However, as the question decomposition methods decompose a multi-hop question into single-shop questions, it is rather natural that the zero-shot Explicit method is advantageous when the training dataset is small. In my opinion, this experiment should have been based on the pretrained LMs rather than the SQuAD fine-tuned QA models, comparing the performance of the Explicit method (not zero-shot) and the No Decomp method according to the number of training samples.\n\n- (Suggestion rather than a reason to reject) Since the question decomposition itself provides an additional inductive bias, it can be expected that the training sample efficiency will naturally occur. In order for this paper’s conclusion to be more meaningful, it would be good to add an analysis of the cause. For example, does an ability to decompose a question arise within a QA model given sufficient training data (multi-hop QA instances)? If so, does this ability correspond to the inductive bias provided by the Explicit/Implicit methods?\n\n- (Minor) The presentation of some figures needs to be improved (Refer to Presentation Improvements below).", "questions_to_authors": "- In lines 165-171, the authors explained why the Explicit and Explicit + Implicit methods didn't work well (problem of Explicit method). Then, couldn't we use variation (Explicit w/Decomp or Explicit Everything) instead of Explicit method in Explicit + Implicit method?", "paper_topic_and_main_contributions": "This paper addresses the question of whether decomposition is helpful in complex question answering (multi-step/hop QA), and if so, under what circumstances. The authors claim that question decomposition is useful when only handful of training samples (multi-hop QA instances) are given, and unhelpful or even hindering when sufficient data are available.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Min, Sewon, et al. \"Multi-hop Reading Comprehension through Question Decomposition and Rescoring.\" Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.\n\n- The above paper explains the types of multi-hop questions (especially in HotpotQA), so it may be helpful to understand Implicit decomposition.", "Typos_Grammar_Style_and_Presentation_Improvements": "- In Figure 2, the meaning of each bar can be figured out in the legend order, but the colors are too similar. It would be nice to have more color separation for clarity.\n\n- In the case of UnifiedQA-v2 in Figure 3, it is hard to see a better range for the zero-shot Explicit method. Can you provide data or adjust the x-axis of the figure to show it clearly?\n\n- (Line 220) Guo et al. (2022) was published in December 2022, and it may be difficult to call it concurrent due to the time difference."}}]

Author rebuttal:

Meta-review: Thanks for your analysis. To me what you showed is kind of trivially true but there is a significant part of the research community who will find your results surprising. Therefore, I believe your paper can help to advance the state of the art in Question Answering.
Decision: Accept-Main