Decomposed Prompting: A Modular Approach for Solving Complex Tasks
Authors: Tushar Khot
Harsh Trivedi
Matthew Finlayson
Yao Fu
Kyle Richardson
Peter Clark
Ashish Sabharwal
Venue: ICLR 2023 poster (2023)
Keywords: prompting
decomposition
in-context learning
reasoning
few-shot prompts
multi-step reasoning

Abstract: Few-shot prompting is a surprisingly powerful way to use Large Language Models (LLMs) to solve various tasks. However, this approach struggles as the task complexity increases or when the individual reasoning steps of the task themselves are hard to learn, especially when embedded in more complex tasks. To address this, we propose Decomposed Prompting, a new approach to solve complex tasks by decomposing them (via prompting) into simpler sub-tasks that can be delegated to a library of prompting-based LLMs dedicated to these sub-tasks. This modular structure allows each prompt to be optimized for its specific sub-task, further decomposed if necessary, and even easily replaced with more effective prompts, trained models, or symbolic functions if desired.
We show that the flexibility and modularity of Decomposed Prompting allows it to outperform prior work on few-shot prompting using GPT3. On symbolic reasoning tasks, we can further decompose sub-tasks that are hard for LLMs into even simpler solvable sub-tasks. When the complexity comes from the input length, we can recursively decompose the task into the same task but with smaller inputs. We also evaluate our approach on textual multi-step reasoning tasks: on long-context multi-hop QA task, we can more effectively teach the sub-tasks via our separate sub-tasks prompts; and on open-domain multi-hop QA, we can incorporate a symbolic information retrieval within our decomposition framework, leading to improved performance on both tasks. Datasets, Code and Prompts available at https://github.com/allenai/DecomP.

Published as a conference paper at ICLR 2023
Decomposed Prompting: A MODULAR APPROACH FOR SOLVING COMPLEX TASKS
Tushar Khot*, Harsh Trivedi Matthew Finlayson*. Yao Fu Kyle Richardson 4 Peter Clark Ashish Sabharwal
Allen Institute for AI Stony Brook University tushark @ Peallnnaiorg hjtrivedi www.es.tssssssoeeeeeeoo edu, matthewf @ealenai.org, yao.fu@ rao.fu@ed.ac.uk, kyler@ allenai.org, peterc @ allenai.org, ashishs @ @allenai.org
University of Edinburgh
ABSTRACT
Few-shot prompting is a surprisingly powerful way to use Large Language Models (LLMs) to solve various tasks. However, this approach struggles as the task com- plexity increases when the individual reasoning steps of the task themselves are hard to learn, especially when embedded in more complex tasks. To address this, we propose Decomposed Prompting, a new approach to solve complex tasks by decomposing them (via prompting) into simpler sub-tasks that can be delegated to shared library of prompting-basee LLMs dedicated to these sub-tasks. This modular structure allows each prompt to be optimized for its specific sub-task, further decomposed necessary, and even easily replaced with more effective prompts, trained models, symbolic functions if desired.
We show that the flexibility and modularity of Decomposed Prompting allows it to outperform prior work few-shot prompting using GPT-3, On symbolic reason- ing tasks, we can further decompose sub-tasks that are hard for LLMs into even simpler solvable sub-tasks. When the complexity comes from the input length, we can recursively decompose the task into the same task but with smaller in- puts. We also evaluate our approach long-context multi-hop QA, we can more effectively teach the sub-tasks via our separate sub-tasks prompts; and on open-domain multi-hop QA, we can easily incorporate symbolic information retrieval module within our decomposition framework, leading to improved performance on both tasks
textual multi-step reasoning tasks: on
INTRODUCTION
Large Language Models (LLMs) such GPT-3 (Brown et al. 2020) have been shown to solve various tasks given only few examples as prompts, also referred to as in-context learning. These models can even perform more complex reasoning tasks when shown the sequence of simple rea-
soning steps needed to perform the complex task as a prompt (Wei et al. 2022; Nye et al. 2021). In essence, the sequence of reasoning steps, such in Chains-of-Thouge et al. 2022), demonstrates how to decompose the complex task as well as how each reasoning step should be performed. However, as tasks become more complex, few demonstrations of the complex task aren't 't sufficient for current models to learn to perform all necessary reasoning steps. E.g.. few- shot demonstrations of concatenating the th letter of words in string insufficient for GPT-3 to learn to extract the 1tth letter, or learn to answer hard single-hop questions when only provided a few demonstrations multi-hop questions. Additionally, is unclear whether tasks such as document retrieval and integration, for knowledge-intensive tasks, can even be done by few-shot prompts.
(CoT) prompting (Wei
To address these limitations, we propose Decomposed Prompting (DECOMP), a new approach to solve complex tasks by instead decomposing them into simpler sub-tasks and delegating these to sub-task specific LLMs, with both the decomposer and the sub-task LLMs (henceforth, sub-task handlers) having their own few-shot prompts. Fig illustrates our approach. The decomposer
Work done during internship at Allen Institute for AT Datasets, Code and Prompts available at
/Decomp

---

Published as a conference paper at ICLR 2023
Standard
Chain-of- Chain-of-Thought
Decomposer Prompt
Suh-Task Hundlers
Prompting
Prompting
Decomposed Prompting
Figure 1: While standard approaches only provide labeled examples (shown as grey input box with green label box), Shain-of-Thoughle prompting also describes the reasoning steps to arrive at the answer for every example in the prompt. Decomposed Prompting, on the other hand, uses the decomposer prompt to only describe the procedure to solve the complex tasks using certain sub- tasks. Each sub-task, indicated here with A, and C is handled by sub-task specific handlers which can vary from standard prompt (sub-task A), further decomposed prompt (sub-task B) or symbolic function such as retrieval (sub-task C)
prompt only describes sequence of sub-tasks (A, B, and C) needed solve the complex tasks, in- dicated with the dashed lines. Each sub-task is then delegated to the corresponding sub-task handler shown on the right.
Using a software engineering analogy, the decomposer defines the top-level program for the com- plex task using interfaces to simpler, sub-task functions. The sub-task handlers serve modular, debuggable, and upgradable implementations of these simpler functions, akin to software library. If particular sub-task handler, say the one for identifying the A letter or retrieving document, is not performing well enough, we can debug this handler in isolation, explore alternative prompts or implementations, and seamlessly plug the improved module back into the overall system, as a systematic way to try to improve performance on the complex end-task.
This approach has several advantages over prior work (as also shown the figure). The sub-task handlers can be shown a broader and richer set of examples (of the simpler task) than the specific ones needed for the complex task prompt (task A). decomposed into simpler sub-tasks (task B). Similar to software libraries, these sub-task handlers can be shared across multiple tasks; e.g., here tasks A and C are reused in the model for task B. As noted above, sub-task handler can be easily swapped with improved implementation without any change to the rest of the system. Few-shot prompt based LLMs can be even replaced with a symbolic system for tasks more suited for non-neural methods; e.g., task C uses a symbolic retrieval system such as Elasticsearch that can handle very large-scale corpora. Lastly, we can even improve upon prior work by simply adding an crror-correcting tub-task handler as a post-processing step.
sub-task is too complex, it can be further
To illustrate these advantages of DECOMP, we empirically evaluate it against prior work on eight challenging datasets using GPT3 models: (1) On task of concatenating the teth letter, we show that our approach of factoring out each sub-task allows us to more effectively teach the sub-problem of extracting the 1 letter(specifically, by decomposing it into even easier sub-tasks).
(2) On task of reversing a list, we show that DECOMP allows us to extend the capabilities of weaker model and build a scale-invariant system by recursively decomposing the task into reversal of smaller and smaller lists. (3) On task of long-context QA (Khot et al. 2022). our approach allows each sub- task handler to accommodate more examples than feasible with CoT prompting leading to better QA performance. (4) On three multi-hop open-domain QA datasets (Yang et al. 2018 Ho et al. 2020 Trivedi et al. 2022). we can incorporate a symbolic retrieval (ElasticSearch) API as the hand er for the retrieval sub-task leading better results than COT. (5) On two Math QA datasets (Cobbe al.) 2021 Roy & Roth 2015),, we can post-process CoT to easily fix frequent formatting errors, resulting surprisingly high improvement of 14-17 pts.
2 RELATED WORK
Few-shot Prompts for Multi-Step Reasoning Large-scale Language models (LLMs) have been shown to learn various NLP tasks given just few examples as prompts (Brown et al. 2020). Recently, they have also been successfully applied to various multi-step reasoning tasks by providing the intermediate reasoning steps, i.e. Chain-of- needed to arrive at the answer. An alternate approach has been to compose multiple LLMs or
(Wei et al. 2022 Chowdhery al. 2022
with symbolic functions to perform multi-step reasoning (Jung et al. 2022 Creswell et al. 2023
2

---

Published as a conference paper at ICLR 2023
Press et al.) 2022; Parisi et al. 2022 Gao et al. 2022 Schick et al. 2023 inter alia). We view these prior works as specialized systems with a pre-defined decomposition structure.
The closest works to our approach are the ideas of least-to-most prompting (Zhou et al. 2023) and successive prompting (Dua et al. 2022) where one prompt/model is used to generate the sub- questions needed to answer complex question and second prompt/model sequentially answers these sub-questions. In contrast, our approach allows for diverse decomposition structures including recursion and other non-linear decomposition structures. E.g.. by definition, least-to-most asks ques- tions from easiest to the hardest and requires an LLM to eventually answer the complete question ("most" in least-to-most) whereas we have no such restriction. Additionally, we iteratively generate new questions based on previous answers (similar to successive prompting) and can explicitly assign different prompts or symbolic systems to answer each sub-question.
Modular Approaches for Multi-Step Reasoning Our work follows a long literature in NLP on neural modular modeling architectures (Andreas et al., 2016 Talmor & Berant 2018 Min et al.
2019 Jiang Bansal 2019 Gupta al.][ Perez et al. 2020; Khot al.j [021]] Levine al.
and other tasks. We take particular inspiration from the Text Modular Networks approach of Khot al. (202T), whereby problem decomposition consists of a learned next question generator trained to generate questions in the language of collection of textual and symbolic agents. Best-first search strategy was used to explore the space of possible decompositions during inference. In contrast this work, which largely centered around supervised training of the next-question generator given existing agents, we leverage the power and recent successes of few- shot LLMs to build both the decomposer and the sub-task agents that best fit the ideal decomposition. This has the advantage of obviating the need for specialized supervised training data that may not always be available for all sub-tasks key bottleneck of this prior work.
2022 for
DECOMPOSED PROMPTING
As with conventional few-shot prompting, the goal is to teach an LLM to find an answer A to a query Q using a small set of in-context examples D = {E,,.... ..., Di }- The answer A is obtained from the underlying distribution p(A D. (Dohan et al., 2022 In the most basic few-shot setup, examples take the form E, = (Q,, Aj). In the case of CoT-style prompting, the goal is to obtain answers by first generating sequence chain of intermediate reasoning steps "thoughts" T, and then deriving the final answer based on T. To teach this ability, one uses more sophisticated in-context examples that take the form E, (Q,: (Ty,1, Ag))
In DECOMP, the core is a decomposer LLM that tries to solve a complex task by generating a prompting program P for it. Each step of P directs a simpler sub-query auxiliary set of sub-task functions F available to the system. Given a query Q whose answer is A. the program P is a sequence of the form ((f1. Q1, A1)..., (fr, Qk. Ax)) where Aş is the final answer predicted by P and Q, is sub-query directed to the sub-task function f: E F. P is executed by a high-level imperative controller, which passes the inputs and outputs between the decomposer and sub-task handler until stopping condition in P is met and the final output obtained.
function in an
To teach the decomposer LLM in few-shot prompting manner, we use in-context examples that take the form E;- = ((Qj, (Ji.1, Qj.1, A,. (Jj.,,, Qj.ky, Aj,x,)) )) where Aj.s, A, is the final answer for Q, and (Q). Qz.,,) is decomposition of Q2- Each sub-task function f. in turn, is operationalized via a sub-task handler as an in-context prompting LLM (e.g., separate CoT-style prompt or additional prompting program dedicated to that sub-task), or any other symbolic or learned function (e.g., a calculator or specialized supervised trained model).
3.1 DECOMPOSED PROMPTS
To illustrate this with an example, consider a multi-step task such as "Concatenate the first letter of every word in str using a space". We can solve this task by decomposing it into sequence of three simple sub-tasks: 1) Collect the list of words in the str; 2) For each word, extract the third letter; 3) Concatenate the extracted letters using space as the separator. Fig. 2 shows an example decomposition prompt for this task. Much like a conventional structured program, the top-level decomp prompt provides example program F, using three sub-task functions: fi :split that
3

---

Published as a conference paper at ICLR 2023
QC: Concatenate the first letter of word in "Jack
Q: What words Tesla*?
Ryan" using spaces
A: ["Elon", "Musk", "Tesla"]
Q1: [split] What the words "Jack Ryan"?
Q: What are letters in "C+*"? A: [C, "4", "+"
#1: Jack". "Ryan")
02: (foreach) [str_ poo] What the first letter #1?
split
#2: ["J", 'R']
Q3: [merge] Concatenate #2 with spaces
Concatenate ["r",, "c"]
#3: "J
A: 'nic"
Q4: [EOQ)
@: Concatonate ["hi, ",, "c", "e"" using spaces
A: 'nice"
decomp
mergo
Figure 2: Prompts for the decomposer and the split and merge sub-tasks used by the decom- poser. The decomposer specifies the sequence of questions and corresponding sub-tasks (within square braces). The sub-task prompts can be written independent of the complex task examples and can even capture generalizations, e.g., letters in word (split) and no delimiter (merge).
splits words in an input string, f2 : pos that finds character positions in strings and fa merge that concatenates characters. In this case, we operationalize each sub-task function as separate in-context prompt (e.g., using a standard prompting approach for split and merge on the right side), each containing a set of in-context examples that are independent of the original complex task.
In addition to the three functions described above, additional control structure is included, such as the symbolic function foreach, which iterates over arrays and references to previous answers such as #1. We note that such helper function is not strictly necessary (e.g., we could directly generate "Q2": What is the first letter of Jack?" and "Q3': What the first letter Ryan?" instead of Q2 in the figure) and is added to reduce the manual effort needed to specify the decomposition and also reduce potential errors during decomposition. In our experiments we use two of the compositional operators defined by Khot et al. (2022) (see appendix for details), although it is capable of using all their operators (which also capture the QDMR operators from Wolfson et al. (2020).. ).
2C: Concalenntle the second letler every word John Smith" using spaces
Smith// A: [Jonn" "Emin")
spit
+Q1 [split]
words "John Smithh? #1: l'Johnn, "Smith")
What
"
st pos
"o"
X22: (oomacc) [str_pos] [oo] What the second lettor #17
C: What the sccond lettor "Smtt?? A
['o", 'm]]
"n"
+23: [merge] Concatenate #2 with spaces
C: Concatinate (oo 'wf] with sparess A
merge
"am""
04 (E0O]
A:oo
Figure 3: The inference procedure in DECOMP iteratively calls the decomposer prompt generate the next question and sub-task at each step, given the current history of question and answers. The generated question is then routed to the assigned sub-task handler (with some handling of special operators, when needed). When the special ond-of-questions [EOQ] marker is generated, the previ- ous answer returned as the final prediction.
3.2 PROMPT EXECUTION AND INFERENCE
Given a new question and a set of background in-context examples D, the inference (i.e., the pro- gram construction and execution) process illustrated in Fig. 3 The new complex question is fed to the decomposer prompt to get the first sub-question to be asked to the split prompt. With the help of our symbolic controller, the answer generated from this prompt then appended to the decom- poser prompt to get the second sub-question, Q2. Due to the foreach operator in the generated question, Q2 results two questions (one for each word in # 1) to be fed to the str-pos prompt. The answers are combined into array to get the answer #2. The entire decomposition history is used to generate Q3 and passed to the merge prompt to get the final answer. Since the task has been solved, the decomposition prompt produces the special end-of-sequena cof-sequence marker([EOQ)) and the last answer is returned as the final answer. Formally, performing inference involves finding the best an- swer A query Q which in the simplest form involves computing the MAP answer using the
4

---

Published as a conference paper at ICLR 2023
LLMs predictive distribution for A, i.e., A arg m p(A D, Q. (Dohan et al. 2022). For practicality, such computations are approximated using greedy search in our experiments.
3.3 DECOMP CAPABILITIES
Hierarchical Decomposition Certain sub-tasks, even when given many examples, are not solvable with few-shot prompting. E.g., we found identifying the 4 letter of a string to be challenging for the GPT3 text- text-davinci-o00 model. In such scenario, prompt further, to first identify the letters and their position and then select the h element this array (see Fig- 4 We also re-use existing sub-task prompts in framework. E.g., the split prompt can be reused since it was developed for the general task of splitting strings
can decompose the sub-task
QC: What is the letter at position 1 of the word "Google"? QS: [split] What are the letters and their positions in "Google"?
Q: What is position 1 "("Coloriess" 1), ("green', 2), ('Ideas", 3). ("sleep", 4), ("furiously", 5]]"?
A: "Colorless"
A: 'I[G, 1). (o, 2). (o, 3). (g. 4), (l. 5), (e. (e.6)l
QS: [arr pos] What is at position in #1?
Q: What position 2 "[GG. (o. 2). (o. 3). (g. 4). 5). (e. 6) A="o"
"G"
QS: [EOQ]
str ___
arr_pos
Figure 4: Since identifying the hthh character is challenging for GPT3 davinci- -00 model, we further decompose it into two simpler sub-tasks: split the word into its letters (using the shared sub- task split) and then return the t item of this list using the arr-pos prompt.
Recursive Decomposition Some problems can be naturally broken down into one or more smaller problems of the same form. Recursive algorithms such merge sort this idea to solve large problems efficiently, using a succinctly described method. We apply this same principle in DECOMP by allowing the decomposer prompt recursively call itself, as shown in Fig- for the task of list reversal. By using recursion, we are able to generalize any base prompting approach (CoT in this figure) to much longer lists by breaking the input into smaller and smaller lists till we reach list length where the model is highly accurate. Such recursive approaches can not be described by current methods such as CoT and standard prompting. Least-to-most prompting (Zhou et al.) 2023) also proposes similar solution but differs in two key aspects (a) it has to identify all the sub- problems in one-shot instead of our iterative top-down decomposition (b) has learn to identify the relevant answers from the previous solutions which we get for free from our decomposition.
QC: Reverse the sequence "newspaper, glasses, laptop, bottle".
Base Case (using CoT)
QS: [ist spli]] Split the sequence "... "newspaper, glesses"
QC: Reverse the sequence "laptop, photo, clip". [exiract]
Taatop. bottle"
laplop.. Second photo. Third cli. Now to reverse, change the order to: Third is Second photo. First laptop. answer 'clp, photo, laptop".
QS: [reverse] Reverse the sequence "newspaper, glasses'
A: "glasses, newspaper"
Recursive calls [reverse] Reverse sequence "laptop, bottle"
A: "bottle, laptop"
"laplap, phoko, clip' QS: [EO0]
[merge] Concalenate "bottle, laploy" and "glasses, newspaper" using a comma "bottle, laptop, glasses, newspaper* QS: [EOQ]
reverse
Figure 5: Sample prompt for recursive decomposition for reversing lists. Each list is split into two halves and each half is reversed and concatenated in the reverse order. We can recursively split a list till we hit the base case (lists of length here) where existing approaches such as CoT are accurate.
External API Calls In certain cases, the sub-tasks may not be feasible to solve using only a LLM. E.g., retrieving knowledge from KB large corpus. Such sub-tasks, however, can be easily solved using existing systems such as retrieving documents using an Elasticsearch index or webpages using Google search (Lazaridou et al. 2022). Fig. 6 shows how DECOMP can easily use such system to retrieve the relevant documents and answer single-hop open-domain question.
A Appendix G Containn the complete splie prompt which has examples for questions such as "Q: What are the letters and their positions in "Mathison"?".
5

---

Published as a conference paper at ICLR 2023
QC: Which company manufactured Lost Gravity?
retrieve
QS:
[refrieve] Which company manufactured Lost Gravity?
A:
['Lss Gravity (roller coaster)" "The Roller Coaster" ...]
Wikipedia Titl: Mack Rides
QS:
[singlehop_ reqa] Titles: ["Lost Gravity (roller coaster)".
Mack Rides GmbH & Co
Question: Which company manufactured Lost Gravity?
Q: Which company manufactured The
("titles": ("Lost Gravity (roller coaster)", ...] "answer": "Mack Rides")
Lost Gravity?
QS: [EOQ]
retrieve aadqa
"Mack Rides"
singlehop roqa
Figure 6: A Decomposed Prompt to answer open-domain questions using Elasticsearch- re-
trieval. Full usage of this prompt for open-domain multihop questions is given in Fig. 11
4 CASE STUDIES
We showcase DECOMP's strengths through four tasks; two symbolic manipulation tasks similar to those investigated by Wei et al. (2022) and two existing textual multi-hop reasoning tasks. Unless specified, we use text-dav iniiiniiii 002 InstructGPT3s model (Ouyang et al. (022) the LLM and report the Exact Match (EM) numbers, following prior work. For order-independen list an- swers, evaluate equality as EM. We compare our approach to CoT rather than each specific decomposition structure used in prior work. See App. G for the complete prompts for all our tasks.
4.1 ht LETTER CONCATENATION (HIERARCHICAL DECOMPOSITION
We compare DECOMP to CoT prompting for concatenating letters at the h position. All prompts contain examples of concatenating letters in position
and last position of strings with 3 words. We create three different prompts for all our baselines and present the average to account for variance due to the choice of examples following Perez et al. (202I). We use the decomp, split, str.pos (further decomposed as shown in Fig. 4. and me rge prompts for decomposition prompting. We adapt the CoT for last letter concatenation from prior work (Wei al. 2022) for this task as shown below. In addition, we consider rolled out version of our decomposition prompts in terms of CoT, i.e., we describe the entire decomposition process (identify words, split each word into letters, take 4th letter and concatenate) single COT. e.g, for the question "Take the letters position 4 of the words "Herbert Alexander Simon" and concatenate them using a space.", we use the CoT:
Chain Shain-OO-Thoor Ot-Tuuught
Chain-Or-Thouppe (rolled out)
The letter at position 4 of "Herbert" "b". The letter at position 4 of "Alexander" is "x". The letter posi- tion of "Simon" "o". Concatenating "b", "x", "o" using space leads to "b x o". So, "Herbert Alexan- der Simon" outputs o"....
The words in "Herbert Alexander Simon" are "Her- bert", "Alexander", and "Simon". The letters and their positions "Herbert" are "[(H. 1), (e, 2). (r. 3), (b, 4), (c, 5), (r, 6), (t, 7)1". The letter position 4 this sequence is "b". outputs x o"...
We similarly adapt the teast-to-moss prompt (Zhou et al. 2023) to include rollout. (see App. We compare these four prompting techniques 4 datasets evaluate generalization along axes: new letter position k 3 (2) longer inputs, #words=4 and 5; (3) new delimiter "!.". The words in the examples come from list of most popular first and last names All evaluation datasets have 100 examples. We present results on space as delimiter averaged across three prompts in Fig.
DECOMP outperforms chain-of-thought and least-to-most prompting, even when the prompt uses the same reasoning procedure the rolled out decomposition. This shows that the separate prompts are more effective at teaching hard sub-tasks than a single CoT prompt.
DECOMP generalizes perfectly to longer sequences. As the length of the input sequence increases, our approach continues to achieve close to 100% accuracy on this task [1 The CoT-based approaches drop noticeably in their scores with longer input lengths, widening the performance gap.
Note that none of the sub-task prompts contain examples for this position.
Forebears wtt/eett//ea/eeeee hatartth/ooo worenaee//eeeeeeseeeeaaaaaa and forebears wwt///att/ssuraae
We obtain similar results with semi-colon shown in Fig. 22 the appendix.
NNoe that report aggregate metrics for DECOMP too hut the std. dev is zero here

---

Published as a conference paper at ICLR 2023
CoT
w/ rollout
L2M w/ rellout
DecomP
100
05.0
96.8
w/
DecomP
86
80-
64
11
53
40
40-
24
20
120
6.0
20
16
5.5
3,0
6.5
4.5
M-3(1ID)
N-4(00D)
N-5(OOD)
N=I(ID)
N=6(00D)
N=3(00D)
N=10(00D)
Figure 7: EM Results on the th letter concate- nation task (k=3) using space as delimiter with different number of words in the input. DE- COMP outperforms and generalizes better than CoT as well as Least-to-most prompting.
Figure 8: EM results reversing sequences. Incorporating CoT in DECOMP greatly in- creases the ability of the model to generalize to new sequence lengths.
4.2 LIST REVERSAL (RECURSIVE DECOMPOSITION)
We use the task of reversing lists of word to show how recursive DECOMP enables length general- ization. We adapt the relevant CoT prompt from Wei et al. 2022), and integrate it in a decomposed prompt. As control, we also compare to CoT version w/ rollout of our decomposed prompt. All prompts contain the same 3 examples of reversing word sequences with 3-5 items. We evaluate all prompts for generalization 4, 6, 8, and 10-item sequences, Here we use davinci- -001 show that DECOMP enables a weaker model approach davinci-C0s performance (which does solve this task). We use the strategy from Fig. 5 and provide our prompts in App. G Fig. B shows the results of the prompting strategies on different input lengths.
DECOMP improves the length generalization of few-shot prompting. While our base CoT prompt does not generalize at all to longer sequences, our approach can recursively decompose the problem and achieve better length generalization. Moreover, the CoT version of our decomposi- tion strategy fails because the unrolled prompt becomes too long and convoluted without the ability to abstract away sub-modules.
4.3 LONG-CONTEXT QUESTION ANSWERING
We next evaluate on the CommaQA-E dataset (Khot et al.) 2022) under the reading comprehension setting. The dataset consists of synthetically generated entities (e.g. Erowid award), facts ("Wether- ality was an actor in the movie Dewbar.)) and multi-hop questions (e.g.. "What awards have the actors of the Erowid winning movies received?"). Due to the presence of many distractors and, result, longer context, this dataset has been shown to be hard for standard LMs even when fine-tuned.
What awards have movies produced by people born in 1910 won?
What swards have movies produced by people born in 1910 won?
QS: [qa] Who born the year 1910?
QS: [simp_ga] Who boon the year 1910?
A: ( ["Tesplmmale" "Muntari")
A: ["Toaplemole", 'Muntanil"]
QS: (foreach_mene) merge) [qa] For which movies was #1 the producer?
QS: (foreach_ (ooreach_mergg) [pos_ [8] For which movies was #1 the producer?
[Featsaw". 'Zalate". "Premercy"]
A: ["Featsaw" 'Zalate', "Premercy"]
QS: twworeec_meereeee (08) Which awards were given to #2?
QS: (foreach, mmege))eereee) [aw_ cal Which awards were given to #2?
[Zorgian", "Chowwuret", "Halloweock"]
['Zorgion,, "Chowwurst, "Hallowcock]]
QS: [EOQ]
QS: (E0O]
coarse
fine
Figure 9: Sample prompts used for the CommaQA dataset. On the left, the coarse- -rarined decompo- sition defines a single QA sub-task with all single-hop questions being delegated to a single sub-task handler. On the right, the fine-grained decomposition assigns questions to three different sub-tasks (see App. G for their prompts) depending on the question type. This allows us to provide more examples for each question type allowing the model to learn the sub-task more effectively.
To fit these questions within GPT3's context limit (2049 tokens), we generate smaller version of the CommaQA-E dataset and of the compositional generalization split such that we can fit at least
We use the vocabulary from
Wei et al.
2022 https: http:::/wwwwwccaablo com/1 st./ 89583

---

Published as a conference paper at ICLR 2023
four examples in the context for CoT prompts. The CoT prompts describe the sequence of facts needed to arrive at the answer (see App. G for all the prompts).
separate the task of de- composition (independent of the context) from the sub-tasks of single-hop question answering. As shown in Fig. 9 we provide examples of the context-independent decomposition in the decom- poser prompt and use the separate sub-task prompts to teach the QA skill over the given context. Addi- tionally, we can choose the granularity of decompo- sition to trade off human effort for increased accu- racy. For example, we could have single QA prompt to handle all the questions or create QA prompts for different classes of questions. In our experiments, each sub-task prompt contains QA examples (2 questions/para). We evaluate three different prompts and report the average results in Fig. 10
For DECOMP,
CoT
DacomP
DecomP((ine)
100
GPT3 (tet dovino 022)
10
59.7
50.7
50
42.0
33.5
30
IID
Comp. Can
Figure 10: EM results on the CommaQA-F datasets. DECOMP always outperforms CoT, with fine-grained marginally out- out- out-performing coarse-grained decomposition.
We make three observations on CommaQA. DE- COMP is more accurate than CoT irrespective of the granularity of decomposition split. Finer grained decomposition can help im- prove task performance by providing more examples for each class of questions, which in turn increases single-hop QA accuracy. DECOMP generalizes to new compositions such as the compo- sitional generalization split of CommaQA, which tests models on unseen compositions of relations observed in the training set. While CoT has a drop in score, both decomposition based approaches actually get a small bump (the subset of relations used in this split are easier for our QA models).
the evaluation
4.4 OPEN-DOMAN QUESTION ANSWERING
Next, we demonstrate the ability of approach integrate external API calls the task of open-domain multihop question answering. We evaluate our approach on three datasets: (1) 2Wiki- MultihopQA (Ho et al. 2020) (2) MuSiQue (Trivedi et al. 2022) (3) HotpotQA (Yang al. 2018).
We describe the open- domain versions of these datasets in more detail in App.(A) [ We use the Codex coode-avinni-O00 model here since it can fit the much longer contexts needed. We also evaluate the impact of model scale on DECOMP by using models from the Flan-T5 family: Flan-T5-L.arge (O.7B), Flan-T5-XI (3B), and Flan- man-T5-XXI, (11B)
Wikipedia Title: Lost Gravity (roller Lost Gravity is steel roller coaster Wikipedia Tille: Mack Rides Mack Rides GmbH & Co ...
QC: what country was Lost Gravity manufactured?
QS:
[retrieve_ odga] Which company manufactured Lost Gravity?
A:
("titles: ["Lost Gravity (ralier coaster)" ... 1. "answer": ['Mack Rides"])
Qs: fretrieve, odqa] The company Mack Rides is from which country?
A: ("titles:: ["Mack Rides", ...]. "answer": ["Germany'.
Q: In what country was Lost Gravity manufactured? "Germany"
QS: [multihop reqa] Tilles: ["Lost Gravity (roller coaster)" .... 'Mack Rides ...].
Question: In what country Lost Gravity manufactured?
decomposer
multihop roqa
Figure 11:
The prompt used to answer open-domain multihop questions using Elasticsearch-based retrieval. The reteriee_on prompt is given in Fig.
Fig. shows the decomposition prompt we use. The decomposer generates (singlehop) sub- questions and delegates them retrieve.odqa (described in Fig. 6). As we showed earlier, this module retrieves relevant documents then uses an RC model to answer. retri retrieve riieveoodqa re- turns both the answer and the documents, allowing subsequent sub-questions to use the answers (e.g. "Mack Rides") and the mult t ihop_ cccop_rco reqa model to use the documents. The final mult t ihopproqa model is prompted to produce the answer directly or using CoT given K paragraphs.
We still use GiT3-sized models for decomposition since only these models are reliably able to produce the required structured outputs.
8

---

Published as a conference paper at ICLR 2023
We compare our approach against two baselines: A. No Context (No-Ctxt), A closed- setting baseline where the model must rely only on its parametric knowledge. B. NoDecomp Context (NoDecomp-Ctate A simple retrieval baseline where we retrieve K paragraphs using the multi-hop question as the input and use that as context. For both NoDecomp-Ctxt and Decomp-Ctxt, K is selected by hyperparameter tuning (App. Aj. We manually annotate CoT's and decompositions for 20 training set questions, and sample 3 prompts of 15 questions each for all approaches. The detailed prompts are given the Appendix G
We evaluate on 300 held-out dev questions in each dataset.
No-Caol QA
NeDecomp- Ctxt QA
Decomp-Ctxt QA
No-Cast QA
NoDecomp- Ctxt. QA
Decomp- Coxt QA
.01
coce davino
T5-XXL
70-
64
70-
63,3
60
502
40
50
49.0
49.0
32.3
29.3
25.4
23,3
24.8
18.3
20-
15.1
MuSiQue
HotpotQA
2WilMMulthhpAA
MuSiQue
Hotpot
2WikMMulthhpQA
Figure 12: Answer FiPon three open-domain QA datasets using two base LMs: Codex (left) and Flan-T5-XX0 (right) with direct prompting. Decomp-Ctee models (ours) significantly outperforms the No-Ctxt models (no retrieval) in all settings and also outperforms our strong retrieval baseline (NoDecomp-Cte QA), with the exception of Codex HotpotQA where it is comparable. See App. A.3 for results on smaller Flan-T5 models and CoT prompting.
We present results on all three datasets with direct QA prompts in Fig. 12 with other results in App. A The Decomp-Ctxt models performs significantly better than No-Ctxt models in all the set- tings showing that external knowledge can be leveraged to improve few-shot models on open-domain mulithop QA. Furthermore, we show that our Decomp-Ctx: models outperform the strong retrieval baseline (NoDecomp- -tttt) in all settings except one (Codex with HotpotQA). Finally, we show that even with the much smaller Flan-T5 T5-X-L model, Decomp-Ctxt outperforms all the baselines and can even achieve scores comparable to the Codex-only systems.
4.5 ADDITIONAL. RESULTS
Post-processing CoT for error correction DECOMP also allows us to create targeted sub-task handler to focus on the source of error in any system. For example, CoT for arithmetic reasoning often rely on patterns (answer is .)) to extract answers but the CoT does not always fit this pattern. Instead, we can assign the answer extraction to better sub-task handler (GPT3) and reduce these types of errors. This results in a 17 pt improvement on MultiArith (78 + 95) and 14 pt improvement on GSM8K (36 > 50.6) compared to CoT prompting (details in App. B).
While DECOMP outperforms the baselines in aggregate, we also the gains of DECOMP are consistent across prompt choices (see App. D and decompasition schemes (see App. E
5 CONCLUSION
We proposed new approach, Decomposed Prompting, to solve complex tasks using few-shot prompts, by decomposing them into a prompting program built out of simpler sub-tasks. Drawing inspiration from software libraries, our decomposer and shared sub-tasks are designed in a modular fashion: they use their own few-shot prompts, allowing one to independently optimize each prompt, decompose sub-task further if necessary, even seamlessly replace it with symbolic system. We show that Decomposed Prompting outperforms prior work on four different tasks and generalization settings, establishing it as an effective few-shot paradigm for solving complex tasks.
Answer F1 is computed by treating prediction and ground truth answer as bags of tokens and computing their precision and recall (Rajpurkar al. 2016).. See HotpotQA (Yang al. 2018) for details.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
We thank members of the Aristo team at the Allen Institute for AI (AI2) for their constructive feedback and the reviewers for their invaluable suggestions. This work supported in part by the National Science Foundation under grants IIS2007290.
REFERENCES
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In CVPR, 2016.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- ciates, Inc., 2020. URL httpss://procesdiingg neur weeerrrpss.. cccppapperr/222//11ll
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. PaLM: Scaling language modeling with pathways. arXiv preprini arXiv:2204.03311 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Relichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
Antonia Creswell, Murray Shanahan, and Irina Higgins. language models for interpretable logical reasoning. In ICIR, 2023. URL https:/
Selection-inference Exploiting large
openreview http:/ne//eete. fort//oormmmaa Sateruummidd3PP33ggggoo3
David Dohan, Winnie Xu, Aitor Lewkowycz. Jacob Austin, David Bieber, Raphael Gontijo Lopes, Yuhuai Wu, Hemryk Michalewski, Rif A Saurous, Jascha Sohl-dickstein, et al. Language model cascades. arXiv preprint arXiv:2207.1033: 2022.
Dheeru Dua, Shivanshu Gupta, Sameer Singh, and Matt Gardner. Successive prompting for de- composing complex questions. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1251-1265, Abu Dhabi, United Arab Emirates, Decem- ber 2022. Association for Computational Linguistics. URL URL.ttps://ccaaatoooog org/
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. PAL: Program- aided language models. ArXiv, abs/2211.10435, 2022.
Nitish Gupta, Kevin Lin, Dan Roth, Sameer Singh, and Matt Gardner. Neural module networks for reasoning over text. In ICLR, 2020.
Xanh Ho. A. Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop dataset for comprehensive evaluation of reasoning steps. COLING, 2020.
Yichen Jiang and Mohit Bansal. Self-assembling modular networks for interpretable multi-hop
reasoning. In EMNLP, 2019.
Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. Maieutic prompting: Logically consistent reasoning with recursive explanations. In EMNLP, 2022.
Tushar Khot, Daniel Khashabi, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Text modular networks: Learning to decompose tasks in the language of existing models. In NAACI, 2021.
10

---

Published as a conference paper at ICLR 2023
Tushar Khot, Kyle Richardson, Daniel Khashabi, and Ashish Sabharwal. Hey AI, you solve complex tasks by talking to agents? In Findings of ACL, 2022.
Angeliki Lazaridou, Elena Gribovskaya, Wojciech Stokowiec, and Nikolai Grigorev. Interne- augmented language models through few-shot prompting for open-domain question answering. ArXiv, abs/2203.05115, 2022.
Yoav Levine, Itay Dalmedigos, Ori Ram, Yoel Zeldes, Daniel Jannai, Dor Muhlgay, Yoni Osin, Opher Lieber, Barak Lenz. Shai Shalev-Shwartz, et al. Standing on the shoulders of giant frozen language models. arXiv preprint arXiv:2204.10119 2022.
Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi. Multi-hop reading compre- hension through question decomposition and rescoring. In ACL, 2019.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Au- gustus Odena. Show your work: Seratchpads for intermediate computation with language models. ArXiv, abt.2112.0011.. 14, 2021.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke E, Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. In NeurlPS, 2022.
Aaron Parisi, Yao Zhao, and Noah Fiedel. TALM: Tool augmented language models. arXiv preprint 12Xiv:2205.12255. 2022.
Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela. Unsupervised ques- tion decomposition for question answering. In EMNLP, 2020.
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. In NeurlPS, 2021.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350, 2022.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100, 000+ questions for machine comprehension of text. In EMNLP, 2016.
Subhro Roy and Dan Roth. Solving general arithmetic word problems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1743 1752, 2015. doi: 10.18653/v1/DI5--1000
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. ArXiv, 0bss2200.00466 2023.
Alon Talmor and Jonathan Berant. The web as a knowledge-base for answering complex questions. In NAACL, 2018
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multi- hop questions via single-hop question composition. TACL, 2022.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, and Denny Zhou. Self-consistency improves chain thought reasoning in language models. In ICLR, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi. Quoc Le. and Denny Zhou. Chain of thought prompting elicits reasoning large language models. In NeurIPS, 2022.
Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. Break it down: A question understanding benchmark. TACL. 2020.

---

Published as a conference paper at ICLR 2023
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP, 2018.
Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schu- urmans, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models, In ICLR, 2023.
12

---

Published as a conference paper at ICLR 2023
A OPEN DOMAIN QA DETAILS
A.I RETRIEVAL CORPUSES FOR OPEN DOMAIN QA
We use HotpotQA in the fullwiki setting where it comes with the associated Wikipedia corpus for open-domain QA. 2WikiMultihopQA and MuSiQue, however, originally reading comprehension datasets. Questions in 2WikiMulühopQAA and MuSiQue are associated with 10 and 20 paragraphs respectively. To turn these datasets into open-domain QA datasets, dataset by combining all the paragraphs in the train, dev and test questions. As a result we get corpus size of 430,225 paragraphs for 2WikiMultihoQQA and 139,416 for MuSiQue.
create a corpora for each
A.2 HYPERPARAMETER TUNING FOR OPEN DOMAIN QA
We treat the number of paragraphs to retrieve (K) in NoDecomp-Ctxt and Decomp-Ctxt models as a hyperparameter. We select it based on grid search on set of values to maximize performance on held out set of 100 questions for each dataset. For NoDecomp- Ctxt, search K {6, 8, 10) for GPT3 models and K 2, 4, 6, 8 for Flan-T5-* models. For Decomp-Ctxr Ctxt, we search K E {2, 4, 6} for GPT3 and Flan-T5-* models. Note that the ranges are different between GPT3 and Flan- as GPT3 can fit in more number of tokens. The ranges are different for NoDecomp-Ctxt and Decomp- Cixt as K refers number of paragraphs retrieved in each round of retrieval, and NoDecomp-Cll has only one step of retrieval whereas Decomp-Ctx usually has multiple retrieval steps.
A.3 ADDITIONAL. RESULTS
No-Cbxt QA
NoDeccmp-Crat QA
Decomp-Crxt QA
No-Coxt QA
NoDecamp-Chs QA
Decomp-Che Chxt QA
10-
-0
MuSiQuue
MuSiQuee
30-
29.7
30.0
43-
30-
26:8
33-
20-
22.
26.4
23-
22:
23.3
21.
19.1196
18.8
20-
20-
15.1
15-
15-
10-
6.4
10
5
Flan- TS-Large
Flan-15.x- T5-XL
Flan- T5-XXL
code-davinc-002
Flan- TS-Large
Flan TS-XL
Flan-T5-XXL
code-davinc-002
(a) CoT QA Prompt
(b) Direct QA Prompt
Figure 13: Results on MuSiQue dataset
No- Cb-C QA
NoDecomp Coxt QA
Decomp Cbxt QA
No: NoxC QA
NoDecomp Cbxt QA
M
Decomp:
Sexomp-Ctxt.ti Ctxt QA
60
(olpott
47.8
49
51.6
51.8
54.
03
HorpanQAA
40.0
50.740.
4277
40
40
36 6
32
33
24.8
21.3
10
10
Flan- L5-L
T5.XL
Flan- T5-XLT
codo-davinc--022
Flan- T5-Large
Flan-TS-XL
Flan-T5.XXL
code-davinci-002
(a) CoT QA Prompt
(b) Direct QA Prompt
Figure 14: Results on HotpotQA dataset
13

---

Published as a conference paper at ICLR 2023
No-Ctet
NoDecomp-Crs QA
Decomp-Cax QA
No-Coxt
NcDecomp-Cox QA
Decomp-Ctxt
60
99
70.8
70-
70-
04.
60-
52
58.4
01
56.7
50
46.8
90
36
40.9
10
90
51.4
30
70-
70-
20
10
0-
Flan Ta-Laage
Flax-T5-XL
Flan-T5-XX
code code-davinci-000
Flan- La-Lag
Flan-TS-X TS-XL
Flan-T5-XXL code davinei-002 002
(a) CoT QA Prompt
(b) Direct QA Prompt
Figure 15: Results on 2WikiMultthopQA dataset
A.3.1 MUSIQUE
We present all the results on the MuSiQue dataset in Fig. Across all settings, we can see that retrieval helps substantially (large gains No-Ctxt QA) with further improvements achieved by our DecomP- based Decomp-Ctxt QA model.
A.3.2 HOTPOTQA
We present all the results the HotpotQA dataset in Fig. 14 On this dataset too, we can see large gains by incorporating retrieval but the gains from using DecomP are mostly seen in the smaller models.
A.3.3 2WIKIMULTIHOPQA
We present all the results on the 2WikiMultihopOA dataset in Fig. 15 large gains by incorporating retrieval and also observe substantial gains by incorporating DecomP (as compared to NoDecomp-Ctx)..
On this dataset, we can see
B MATH QA
We apply Decomposed Prompting to two math QA datasets: GSM8K Cobbe et al. (2021) and Mul- tiArith Roy & Roth (2015). For laain-of-thought, used the original prompts for math reason- ing Wei et al. (2022 For For example:
Q: There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will be 21 trees. How many trees did the grove workers plant today? A: There 15 trees originally. Then there were 21 trees after some more were planted. So there must have been -5 = 6. The answer is 6.
Most CoT systems Wei et al. (2022): Wang et al. 2023) rely on extracting the answer by finding the number following "answer is" However, this may not always be accurate. For example, the following CoT would be unanswerable by relying on simple patterns.
Parker chews 4 pieces of gum day. There are 15 pieces of gum pack. So he will need 30 15 8 packs of gum to last him 30 days.
Rather than relying on patterns with limited generalization, we can use language model to extract the answer reliably. Specifically, first identifying the chain-of-thought reasoning and then using a second GPT3-based sub-module to extract the answer from the COT. We show examples of our prompts here (full prompt in App. G
Decomposed Prompting to decompose the task into
Example from the Decomposition Prompt
14

---

Published as a conference paper at ICLR 2023
QC: There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they done, there will be 21 trees. How many trees did the grove workers plant today?
QS: [cot] There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will be 21 trees. How many trees did the grove workers plant today?
A: There are 15 trees originally. Then there were 21 trees after some more were planted. So there must have been 21 15 = trees planted.
QS: Igpt_ans [pp_aan]] There are 15 trees originally. Then there were 21 trees after some more were planted.
So there must have been 21 15 6 trees planted.
A: 6
QS: [EOQ]
Example from the gpt ans prompt
Q: There are 15 trees originally. Then there 21 trees after some more were planted. So there must have been 21 15 = 6 trees planted.
A:
CoT
DecomP (coarse)
DecomP (fine)
CoT
DecomP
80
78.0
56
50.7
40-
70-
36.0
30
10
20
1rat-curin-001 tect-daainni-001
tect-davinci-n -002
Figure 17: As the models become weaker (davinci-001) and smaller (curic-001), the performance of all the models drop. DE- COMP still outperforms CoT till the perfor- mance reaches close to zero with curie.
MultiArith
Figure 16: Our simple decomposition results in 14-17 pts on two MathQA datasets: GSM8k and MultiArith.
We present results in Fig. 16 On the GSM8K data se outperform CoT by 14 points. On the MultiArith ""t [I] we achieve a 17 pt improvement compare to COT. While this is a simple change, it illustrates the possibility of using DECOMP for other complex answer types, e.g. extractive answer generation from chain-of-thoughtss
C EFFECT OF SCALE ON COMMAQA
We evaluate text-curie-001, text-davinci-OU0 and text-davinci-002 on the CommAQA dataset. Since the curie-001 and davinci-001 have smaller context window size, we further reduced our prompts fit within their context windows (2048 tokens). As shown in Fig. I7 both CoT and DECOMP are effected by the model size.
D RESULTS ON ALL PROMPTS
D.1 PER-PROMPT RESULT ON LETTER CONCATENATION
We present the results of the letter concatenation task (with space delimiter) for different values of N in Fig. 18 Our results are stable across the different prompts (PI, P2 and P3) and always outperform
CoT and Least-to-Most prompting.
We randomly sample 300 examples from the test set due to costs with API usage
We randomly sample 200 examples from the test due with API usage
15

---

Published as a conference paper at ICLR 2023
10
COT
COT w/ rollout
L2M rolicul
Decemp
Nmd
rollout
L2M wolloutt
100
75
96
96
96
50
58
60
64
455
+42
25
GOT
GOT wt roliout
L2M es rollout
Decomp
100
EM
S1
P1
Figure 18: Across values of and different prompts (PI, P2 and P3), DECOMP outperform chain-of-thoughl reasoning and even ceas-ttomoost prompting.
D.2 PER-PROMPT RESULTS ON COMMAQA
We also present the results of all the prompts on the CommAQA dataset in Fig. can observe that DRCOMP outperforms CoT on each prompt set.
19
Here too, we
Decomp (cparse)
Decomp (fre)
Decomp (ccarse)
Decomp (fin)
80
70
60
59
52
54
BM
40
40
39
34
20
0
P1
P2
P3
P1
P2
P3
Test Set
(b) Comp. Gen. Set
Figure 19: Results of different prompts on the CommAQA dataset.
EFFECT OF DECOMPOSITION SCHEME
To evaluate the effect of the decomposition scheme, we experiment with two other simple decom- position structures for the letter concatenation and reversal tasks.
Letter Concatenation For letter concatenation, we consider an alternate scheme where we use GPT3 to generate each question rather than loop the answers, e.g.,
16

---

Published as a conference paper at ICLR 2023
QC: Take the last letters of the words in "Augusta Ada King" and concatenate them using space.
QS: [split] What the words in "Augusta Ada King"?
A: I"Augusta", "Ada", "King")
QS: [str.position] What is the last letter in "Augusta"?
A: "a"
QS: [str.ppo sstr.position] What is the last letter in "Ada"?
A: "a"
QS: [str_ sstr_position] What is the last letter "King"?
A: "g"
QS: [merge] Concatenate ["a", "a", "g"] using a space.
A: "a g"
QS: [EOQ]
By using the decomposer prompt model to generate the sub-questions, we can more robust to formatting issues in the output answers, e.g., sub-questions even the first answer not a valid array. However, the generated sub-questions may not correctly use all the elements of the list (change in order, missed element, repeated elements, etc).
expect GPT3 to still generate the appropriate
List Reversal For list reversal, instead of splitting into halves, we take the tail the list, reverse and then concatenate it the head. i.e. reverse(list) reverse([is([1:) list[O]. This requires more GPT3 calls (O(n) compared to the original approach of splitting the list into halves (O(log(n)).
In both these cases, noticed that the performance did not drop shown in Fig. 20 and Fig. 21 On the letter concatenation task, the results were exactly the same. The new reversal decomposition schema was actually stronger on longer inputs at the cost of more calls GPT3 (O(n(n)) using binary splits vs O(n) one element at a time). Both these decomposition schemes are still better than COT.
(Tail Sporr)
Decomp (Loop)
Decomp (Generate))
10C
98
98
75
74
6-1
42
N
N=3
N-44
N-5
Figure 21: Recursively reversing the tail of list is more stable longer lengths but comes at the cost of more calls to GPT3.
Figure 20: Both decomposition schemes for the letter concatenation task have the same scores.
F ERROR ANALYSIS
F.1 LETTER CONCATENATION
F.1.1 DECOMP
We analyzed the errors in DECOMP on the letter concatenation task and only found errors in the sub-task execution.
17

---

Published as a conference paper at ICLR 2023
CoT
CoT w/ rollout
DecomP
97
94
82
70
12
7
N 3(IID)
N 40000(
N 5(00D)
Figure 22: EM Results on the hthh letter concatenation task (k=3) using semi-colon as delimiter with different values for N. the number of words in the input. DECOMP always outperforms and generalizes better than COT.
Q: Take the letters at position 3 of the words in "Nancy Samina Abbas Caudhari Bano" and concatenate them using space.
A: b n
Prediction: m nun
Error: Incorrect letter extraction (Sub-task)
What is at position in "[(N, 1), (a, 2), (n. 3), (c, 4), (y. 5)]"? = "c"
Q: Take the letters at position 3 of the words in "Orlando Stephen Cho Teixeira Pierre" and concatenate them using space.
A: e leii
Prediction: leoie
Error:
Incorrect concatenation (Sub-task)
Concatenate ["I", "e". "o", "i", "e"] using a space. = "leoie" "
F.1.2 COT w/ ROLLOUT
We analyzed the errors in CoT on the letter concatenation task and found similar errors during the generation of CoT. But the frequency of these errors was higher than DECOMP, as it is not possible to effectively teach each sub-task with COT.
Q: Take the letters at position 3 of the words in "Sheila Nicolas Verma Sha Sousa" and concatenate them using space.
A: e ererral
Pred: icrau
Error: Incorrect letter extraction
...The letters and their positions "Sheila" "[(S, 1), (h, 2), (e, 4), (1, 5), (a, 6))". The
letter at position in this sequence "i"
Q:Take the letters at position of the words in "Shobha Kailash Nakamura Peter Benitez" and concatenate them using space.
A: o ookktn tn
Pred: olkti
Error: Incorrect letter extraction ...Benitez" are "[(B, 1), (e, 2), (n, 3), (i, 4), (1. 5), (e, 6), (z. 7))". The letter at position this sequence is "i"...
18

---

Published as a conference paper at ICLR 2023
F.2 COMMAQA
Similarly in CommaQA, the emrors mostly due sub-task errors, which in this dataset correspond to answering single-hop questions. CoT also makes the same types of errors but they are more frequent since this QA sub-task can not be delegated specialized prompt in COT. Since all emros are this type, we show only one example here.
Q: What awards have movies written by people born in 1933 won?
A: ("Hydrallium", "Pompasole"]
Pred: ["Pompasole"]
Error: Incorrect sub-question answer
Sub-Q: -.bb-:hee What movies has Haldron written?
Sub-A: { ("Polyternaflurrmetttnne" "Skia", "Warpstone"]
Pred: ("Skia", "Warpstone"]
G TASK PROMPTS
We have provided the task prompts for all the datasets for COT and our Decomposed Prompting approach.
CoT Since CoT methods also perform 2-step reasoning: first generate the chain-of-thought and second extract the answer from the CoT, we use the same decomposition bbasdd framework for COT baselines too. For example, consider the following example our COT prompt:
QC: Take the letters position 1 of the words in "Alan Mathison Turing" and concatenate them using a space.
QS: [extract] The letter position 1 of "Alan" is "A" The letter at position of "Mathison" is
"M". The letter at position of "Turing" "T" Concatenating "A" "M", "T" using a space leads to "A M T". So," " Alan Mathison Turing" outputs "A M T".
A: "A T
QS: [EOQ]
GPT3 generates the chain-of-thoughh during the "decomposition" step and regex-based answer extractor ract ('.* outputs " In some cases, the module name is skipped in the prompt (the CoT is sent to the extractor by default).
then takes this CoT and generates the answer.
Operators In this work, we use the same operators as defined by Khot et al. Their sel select index with its answer. When
operator is just the basic operator that replaces references to not specified, select is assumed to be the default operator. addition, we consider two operators our experiments: project.valuse and
project_valuee This operator takes list answer #i X and iterates over it to generate new questions by replacing mentions of #i i.e. Q = repeecaace x) for e XJ. The answer to each question is simply concatenated to get the final answer i.e. [model(g) for qE Q]. We refer to this foreach for simplicity in the main text.
project values flat uni queq This operator performs the same steps as project_va veeu but then additionally flattens the list and only returns the unique enti- ties in the flattened list. refer to this as foreach.merge in the main text for simplicity.
G.1 LETTER CONCATENTATION
We show one the prompts used for experiments here. The entire set of prompts is provided as supplemetary material.
19

---

Published as a conference paper at ICLR 2023
G.1.1 DECOMPOSED PROMPTING
decomp
QC: Take the last letters of the words in "Augusta Ada King" and concatenate them using space.
QS: [split] What are the words in "Augusta Ada King"?
A: I" Auuuustta" "Ada", "King"]
QS: (project_values) [str sooiitiot What is the last letter in "#1"?
A: I"a", "aa"""""""" "a"."g" "g"]
QS: [merge] Concatenate #2 using space.
A: "a a g"
QS: [EOQ]
QC: Take the letters at position 1 of the words in "Alan Mathison Turing" and concatenate them using space.
QS: [split] What are the words in "Alan Mathison Turing"? A: I"Alan", "Mathison", "Turing"]
QS: (project_values) [str_position What is the letter at position in "#1"?
A: ["'"" "M", "M"."T"
QS: [merge] Concatenate #2 using space.
A: "A M T
QS: [EOQ]
QC: Take the letters at position 4 of the words in "Herbert Alexander Simon" and concatenate them using space.
QS: [split] What are the words in "Herbert Alexander Simon""?
A: ["Herbert", "Alexander", "Simon"]
QS: (project values) [str position] What is the letter at position 4 in "#1"?
(project.values)
A: ["bb,, "x"]""""
QS: [merge] Concatenate #2 using space.
A: "b x o"
QS: [BOQ]
split
Q: What are the words "Alan Mathison Turing"?
A: ["Alan", "Mathison", "Turing"]
Q: What are the letters in "Alan"?
A: ["A", "a" "a"'"""]
Q: What are the letters and their positions in "Mathison"?
A: "[(M, 1), 2), 3), (h, 4), (i. 5), (s. 6), (o, 7), (n, 8)]"
Q: What are the words and their positions in "Herbert Alexander Simon"?
A: "[Heerbrrt, 1), (Alexander, 2), (Simon, 3)
st str.posit iissitionn
QC: What the letter position of the word " "Augusta"?
QS: (select) [split] What are the letters and their positions in "Augusta"?
A: "[(A, 1), (u. 2), (g. 3), (u, 4), (s, 5), 6), (a, 7)]"
QS: (select) [arr position] What is at position in #1?
A: "A"
QS: [EOQ]
QC: What is the last letter of the word "Mathison"?
QS: (select) [split] What are the letters and their positions in "Mathison"?
A: "I(M, 1), (a, 2), (t, 3), (h, 4), (i, 5), (s, 6). (o, 7), (n, 8)]"
QS: (select) [arr position] What is the last letter in #1?
A: "n"
20

---

Published as a conference paper at ICLR 2023
QS: [EOQ]
QC: What is the word at the position 4 in "Colorless green ideas sleep furiously"?
QS: (select) [split] What are the words and their positions in "Colorless green ideas sleep furiously
"?
A: "[(Colorless, 1), (green, 2), (ideas, 3), (sleep, 4), (furiously, 5)]"
QS: (select) [arr- porr-tosiion] What the position 4 in #1?
A: "sleep"
QS: [EOQ]
merge
Q: Concatenate ("A", "I", "a", "n"].
A: "Alan"
Q: Concatenate ["b", "x", "o"] using space.
A: "b x o"
Q: Concatenate ["a", "a", "g"] using a comma.
A: "a.a.g"
Q: Concatenate ("Alan", "Mathison", "Turing"] using space.
A: "Alan Mathison Turing"
Q: Concatenate ("Allen", "Institute"].
A: "AllenInstittte"
arr per.posiiion
Q: What is position 4 in "[C"Colorless", 1), ("green", 2), ("ideas", 3), ("sleep", 4), ("furiously", 5) A: "sleep"
Q: What is position in "[(M, 1), (a. 2), 3), 4), (. 5), (s. 6). (o. 7), (n, 8)]"?
A: "M"
Q: What is at the last position in "[(A, 1), (u, 2), (g. 3), (u, 4), (s. 5). (t, 6), (a, 7))"?
A: "a"
Q: What is at position in "[[Herbert, 1), (Alexander, 2), (Simon, 3)]"?
A: "Herbert"
Q: What is last position in "[(Allen, 1), (Institute, 2), (for, 3), (Artificial, 4), (Intelligence, ")?"
A: "Intelligence"
Q: What is at position 4 in "[(A, 1), (1, 2), (e, 3), (x, 4), (a. 5), (n. 6), (d, 7), (e. 8), (r. 9)'"
A: "x"
G.1.2 COT WITH ROLLOUT
COT rollout
QC: Take the last letters of the words in "Augusta Ada King" and concatenate them using space. QS: The words in " "Augusta Ada King" are "Augusta", "Ada" and "King". The letters and their positions in "Augusta" are "[(A, 1), (u, 2), (g. 3), (u, 4), (s, 5). (t. 6). (a. 7))". . The last letter in this sequence is "a" The letters and their positions in "Ada" are "[(A, 1), (d, 2), (a, 3) The last letter in this sequence is "a". The letters and their positions in "King" are "[(K, 1), 6i. 2), n. 3). (g, 4)]". The last letter in this sequence is "g". Concatenating "a", "a", "g" using space leads to "a g". So, "Augusta Ada King" outputs "a "aag".
A: a
OS: [EOQ]
21

---

Published as a conference paper at ICLR 2023
QC:
Take the letters at position 1 of the words in "Alan Mathison Turing" and concatenate them using space.
QS: The words in "Alan Mathison Turing" are "A "Alan", "Mathison", and "Turing". The letters and their positions in "Alan" are "[(A, 1), (1, 2), (a, 3), (n, 4)]" The letter position in this sequence is "A". The letters and their positions in "Mathison" are [(M, 1), (a. 2). (t, 3), (h, 4),
(i, 5), (s. 6), (o, 7), (n, 8)]". The letter at position I in this sequence is "M". The letters and their positions in "'Turin" are "[(T, 1), (u. 2), (r. 3), (i, 4), (n, 5), (g. 6)1". The letter at position in this sequence is "T". Concatenating "A", "M", "T" using a space leads to "A M T", So, "Alan Mathison Turing" outputs A M T
A: "A T
QS: [FOQ]
QC:
Take the letters at position 4 of the words in "Herbert Alexander Simon" and concatenate them using a space.
QS:
The words in "Herbert Alexander Simon" are "Herbert", " "Alexander". and "Simon". The letters and their positions in "Herbert" are "[(H, 1), (e. 2), (r. 3), (b, 4), (e. 5), (r, 6). (t. 7)]". The letter position 4 in this sequence is "b". The letters and their positions in "Alexander" are "[(A, 1), (1, 2), (e. 3), (x, 4), (a, 5), (n, 6), (d. 7), (e. 8), (r, 9)]". : The letter at position 4 in this sequence is "x". The letters and their positions in "Simon" are "[(S. 1). (i, 2), (m, 3), 4), (n, 5)]". The letter at position 4 in this sequence is "o". Concatenating "b", "x", "o" using
space leads "b o". So, "Herbert Alexander Simon" outputs "b A: "b o
QS: [EOQ]
G.1.3 COT
COT
QC: Take the last letters of the words in "Augusta Ada King" and concatenate them using space. QS: The last letter of "Augusta" is "a". The last letter of "Ada" is " The last letter of "King" is g". Concatenating "a", "a", "g" using space leads "a a ag". So, "Augusta Ada King" outputs "a ag"
A: "a a g"
QS: [EOQ]
QC: Take the letters position 1 of the words in " "Alan Mathison Turing" and concatenate them using space.
OS:
The letter position of "Alan" is "A" The letter at position of "Mathison" is "M". The letter at position of "Turing" is "T". Concatenating "A", "M", "T" using space leads to "A M T". So, "Alan Mathison Turing" outputs "A M T".
A: "A T
QS: [EOQ]
QC:
Take the letters at position 4 of the words in "Herbert Alexander Simon" and concatenate them using space.
QS: The letter at position of "Herbert" is "b". The letter at position "Alexander"" is "x". The letter at position 4 of "Simon" is "o". Concatenating "b", "x", "o" using space leads to "b o". So, "Herbert Alexander Simon" outputs "b x o".
A: "b o
QS: [EOQ]
G.1.4 LEAST-TO-MOST w/ ROLLOUT
Least-to-most Decomp
QC: Take the last letters of the words in "Augusta Ada King" and concatenate them using space. QS: [12m] Take the last letters of the words in "Augusta Ada" and concatenate them using space.
A: The words in "Augusta Ada" are "Augusta" and "Ada". The letters and their positions Augusta" are "[(A, 1), (u, 2), (g. 3), (u, 4), (s. 5), (t, 6), (a, 7))". The last letter in this sequence
22

---

Published as a conference paper at ICLR 2023
is "a". The letters and their positions in "Ada" are "I(A, 1), (d, 2), (a, 3)1" The last letter in this sequence is "a". Concatenating "a", , "a" using a space leads to "a So, "Augusta Ada" outputs "a a".
QS:
[12m] Take the last letters of the words in "A "Augusta Ada King" and concatenate them using a space.
A: Augusta Ada" outputs "a The letters and their positions in "King" are "[(K, 1), (i. 2), (n, 3), (g. 4) The last letter in this sequence is "g". Concatenating "a a", "g" using a space leads to "a "aag.. So, "Augusta Ada King". outputs "aag" QS: [ans_ext] So, "Augusta Ada King" outputs "a a ag".
A: "a a gg"
QS: [EOQ]
QC: Take the letters at position 1 of the words in "Alan Mathison Turing" and concatenate them using space.
OS:
[12m] Take the letters at position of the words in "Alan Mathison" and concatenate them using space.
A: The words in Alan Mathison" are "Alan" and "Mathison". The letters and their positions in " Alan" are "[(A. 1). (1, 2). (a, 3). (n, 4)]" The letter position in this sequence is "A", The letters and their positions in "Mathison" are [(M, 1), (a. 2), (t. 3), (h, 4), (i. 5), (s. 6). (o. 7), (n, 8)]". The letter at position 1 in this sequence is "M". Concatenating "A", "M" using space
leads to "A M So, Allan Mathison" outputs "A M
QS: [12m] Take the letters at position of the words "Alan Mathison Turing" and concatenate
them using space.
A: "Alan Mathison" outputs "A M The letters and their positions in "Turing" are "[(T, 1), (u, 2), (r, 3), (i, 4), (n, 5), (g. 6)]". The letter at position 1 in this sequence is "T" Concatenating "A M", "T" using a space leads to "A M T", So, "Alan Mathison Turing" outputs "A M T".
QS: [ans_ext] So. "Alan Mathison Turing" outputs "A M T".
A: "A T
QS: [HOQ]
QC: Take the letters at position 4 of the words in "Herbert Alexander Simon" and concatenate them using space.
QS:
[12m] Take the letters at position 4 of the words in "Herbert Alexander" and concatenate them using space.
A: The words in "Herbert Alexander" are "Herbert" and "Alexander". The letters and their positions in "Herbert" are "[(H, 1), (e, 2), (r, 3), (b, 4), (e, 5), (r, 6). (t, 7))" The letter at position 4 this sequence is "b". The letters and their positions in " "Alexander" are "[(A, 1), 1, 2), (c, 3). (x, 4), (a, 5). (n, 6), (d, 7), (e. 8), (r. 9)1". The letter position in this sequence is "x". Concatenating "b", "x" using a space leads to "b x". So, "Herbert Alexander" outputs bx"
QS: [12m] Take the letters at position 4 of the words in "Herbert Alexander Simon" and concatenate them using space.
A: "Herbert Alexander" outputs "b x". The letters and their positions in "Simon" are "[(S, 1), (i, 2) (m, 3), (o, 4), (n, 5)]" The letter at position 4 in this sequence is "o". Concatenating "b x", o" using space leads to "b x o", So. "Herbert Alexander Simon" outputs "b x o".
QS: [ans ext] So. "Herbert Alexander Simon" outputs "b o
A: "b x o"
QS: [BOQ]
Least-to-most COT (12m)
Q: Take the last letters of the words in " "Augusta Ada" and concatenate them using a space.
A: The words in "Augusta Ada" are "Augusta" and "Ada". The letters and their positions in Augusta" are "[(A, 1), (u, 2), (g. 3), (u, 4), (s. 5), (t, 6), (a, 7)J", The last letter in this sequence is "a". The letters and their positions in "Ada" are "[(A, 1), (d, 2), (a, 3)1". The last letter in this sequence is "a". Concatenating "a", "a" using a space leads to a". So, "Augusta Ada" outputs "a a".
Q: Take the last letters of the words in "Augusta Ada King" and concatenate them using space.
23

---

Published as a conference paper at ICLR 2023
A: "Augusta Ada" outputs "a a". The letters and their positions in "King" are "I(K, 1), (i. 2). (n, 3), (g. 4)]" . The last letter this sequence is "g". Concatenating "a a", "g" using a space leads to "a ag.". So, "Augusta Ada King" outputs "a ag".
Q: Take the letters at position 1 of the words in "Alan Mathison" and concatenate them using space.
A: The words in " "Alan Mathison" are "Alan" and "Mathison". The letters and their positions in Alan" are "I(A, 1), (I, 2), (a, 3). (n, 4)]". The letter at position in this sequence is "A". The letters and their positions in "Mathison" are [(M, 1), (a. 2), (t. 3), (h, 4), (i. 5), (s. 6), (o. 7), (n, 8)]" The letter at position 1 in this sequence is "M". Concatenating "A", "M" using space leads A M". So, " "Alan Mathison" outputs "A M".
Q: Take the letters at position 1 of the words "Alan Mathison Turing" and concatenate them
using space.
A: "Alan Mathison" outputs "A M The letters and their positions "Turing" are "I(T, 1), (u, 2), (r, 3), (i, 4), (n, 5), (g, 6)]" The letter at position 1 in this sequence is "T"". Concatenating "A
M", "T" using a space leads to "A M T"". So, " "Alan Mathison Turing" outputs "A M T".
Q: Take the letters at position 4 of the words in "Herbert Alexander" and concatenate them using space.
A: The words in "Herbert Alexander" are "Herbert" and "Alexander", The letters and their positions "Herbert" are "[(H, 1), (e, 2), (r, 3), (b, 4), (e, 5), (r, 6), (t, 7))". The letter at position 4 in this sequence is "b". The letters and their positions in "Alexander" are "I(A, 1), 2), (c, 3), (x, 4), (a, 5), (n, 6), (d, 7), (e, 8), (r, 9)I". The letter at position in this sequence is "x". Concatenating "b", "x" using space leads to "b x", So, "Herbert Alexander" outputs bx"
Q: Take the letters at position 4 of the words "Herbert Alexander Simon" and concatenate them using space.
A: "Herbert Alexander" outputs "b x". The letters and their positions in "Simon" are "[(S, 1), (i, 2) (m, 3), (o, 4), (n, 5)]" The letter at position 4 in this sequence is "o". Concatenating "b x"," o" using a space leads to "b x o", So, "Herbert Alexander Simon" outputs "b o",
G.1.5 ALT DECOMP SCHEEM (GENERATE EACH SUB-QUESTION)
decomp
QC: Take the last letters of the words in "Augusta Ada King" and concatenate them using space. QS: [split] What are the words in "Augusta Ada King"?
A: [""uuusta", "Ada". "King"l
QS: [str [osstiotioon] What is the last letter in "Augusta"?
A: "a"
QS: [str [str_position] What is the last letter in "Ada"?
A: "a"
QS: [str position] What is the last letter in "King"?
A: "g"
QS: [merge] Concatenate ["a", "a", "g"] using space.
A: "a a g"
QS: [EOQ]
QC: Take the letters at position 1 of the words in "Alan Mathison Turing" and concatenate them using space.
QS: [split] What are the words in "Alan Mathison Turing"?
A: [" ["Alln",, "Mathison", "Turing")
QS: Istr_position What is the letter at position I in "Alan"?
A: "A"
QS: [str-poster cot--position] What is the letter at position 1 in "Mathison"?
A: "M"
QS: [str position] What is the letter at position 1 in "Turing"?
A: "T"
24

---

Published as a conference paper at ICLR 2023
QS: [merge] Concatenate ["A", "M", "T""] using space.
A: "A T
QS: [EOQ]
QC:
Take the letters position 4 of the words in "Herbert Alexander Simon" and concatenate them using space.
QS: [split] What the words in "Herbert Alexander Simon"?
A: ["Herbert", "Alexander", "Simon"]
QS: [ssriiosit What is the letter at position 4 in "Herbert"?
A: "b"
QS: |str position] What is the letter at position 4 "Alexander"?
A: "x"
QS: Istr-position What is the letter at position 4 in "Simon"?
A: "o"
QS: [merge] Concatenate ["b", "x", "o"] using a space.
A: "b x
QS: [BOQ]
G.2 SEQUENCE REVERSAL
G.2.1 SPLIT REVERSAL
The prompts in this section implement Algorithm
!
Algorithm recursive reversal strategy that splits the sequence in half, reverses each half, and concatenates them. Runs in Oflog n) calls to the LM where is the number of items in the sequence.
procedure SPLITREVERSF(2)
The reversal of >> The base case The reversed sequence b The inductive case Half the length of
if [x| < 4 then
return T|2| :
else
n xx|/2
f +-++0 I1,..., In
P. The first half of The reversed first half D The second half of The reversed second half The concatenated reversed halves
G8 + SPLITREVERSE())
r Intl.... ......................... T|sl
R SPLITREVERSE
return oft ..... pit
: git
end if end procedure
reverse
QC: Reverse the sequence "driving license, button, packet, identity card, shoe".
QS:
[extract] The sequence is "1. driving license, 2. button, 3. packet, 4. identity card, 5. shoe", The sequence is 5 items long, which more than the minimum length 4, so we split Half of 5 is 5 2 2.5. Dropping the decimal, get that the first half will be items long, ending in "2. button". The first half (2 items) is "1. driving license, 2. button".
A: "1. driving license, 2. button"
QS: [extract] The first half of the sequence ends with "2, button",, so the second half starts after "2. button" with "3. packet", The full sequence items long, and the first half is items long, so the second half will be 5- - 2 = items long. The second half of the sequence (3 items) is "3. packet, identity card, shoe".
A: "3. packet, 4. identity card, 5. shoe"
QS:
Remove the numbers from #1.
A: "driving license, button"
QS: gremove_nubber Remove the numbers from #2.
A: "packet, identity card, shoe"
QS: [reverse] Reverse the sequence #3.
A: "button, driving license"
25

---

Published as a conference paper at ICLR 2023
QS: [reverse] Reverse the sequence #4.
A: "shoe, identity card, packet"
QS: [join] #6 #5
A: "shoe, identity card, packet, button, driving license"
QS: [FOQ]
QC: Reverse the sequence "laptop, photo, clip".
QS:
[extract] The sequence is "1. laptop. 2. photo, 3. clip". The sequence is 3 items long, which is
less than the minimum length of so we don't need split it. All we need to do is reverse laptop, photo, clip".
A: "laptop, photo, clip"
QS: [cot] Reverse the sequence #1.
A: "clip. photo, laptop"
QS: [EOQ]
QC: Reverse the sequence "newspaper, glasses, laptop, bottle".
QS: [extract] The sequence is "1. newspaper, 2. glasses, 3. laptop, 4. bottle". The sequence is 4 items long, which is equal the minimum length of 4, we split it. Half of 4 is 4 / 2.0. Dropping the decimal, we get that the first half will be 2 items long. The first half (2 items) of the sequence is "1. newspaper. 2. glasses".
A: "I. newspaper, 2. glasses"
QS: [extract] The first half of the sequence ends with "2. glasses", so the second half starts after "2. glasses" with "3. laptop". The full sequence is 4 items long and the first half is 2 items long, so the second half will be - items long, ending in "2, glasses". The second half the sequence (2 items) is "3. laptop, 4. bottle".
A: "3. laptop, 4. bottle"
QS: [nmmmeeennmbers Remove the numbers from #1.
A: "newspaper, glasses"
QS: [remove numbers] Remove the numbers from #2.
A: "laptop, bottle"
QS: [reverse] Reverse the sequence #3.
A: "glasses, newspaper"
QS: [reverse] Reverse the sequence #4.
A: "bottle, laptop"
QS: [join] #6 #5
A: "bottle, laptop, glasses, newspaper"
Q [EOQ]
remove.numbers
Q: Remove the numbers from "4. bottle, 3. laptop, 2. glasses, 1. newspaper".
A: "bottle, laptop, glasses, newspaper"
Q: Remove the numbers from "1. identity card, 2. packet, 3. button"..
A: "identity card, packet, button"
Q: Remove the numbers from "I. player, 2. passport, umbrella, 4. radio".
A: "player, passport, umbrella, radio"
join
Q: "bottle, laptop" "glasses, newspaper".
A: "bottle, laptop, glasses, newspaper"
Q: "identity card, packet, button" "magazine, notebook, glasses".
A: "identity card, packet, button, magazine, notebook, glasses"
Q: "passport, umbrella, radio, mobile phone, photo" "player".
A: passportt umbrella, radio, mobile phone, photo, player"
26

---

Published as a conference paper at ICLR 2023
Q: "mirror, case" "toothbrush, alarm clock".
A: "mirror, case, toothbrush, alarm clock"
Q: "light bulb, clip, umbrella" "driving
"driving licence, watch". A: "light bulb, clip, umbrella, driving licence, watch"
cot
QC: Reverse the sequence "newspaper, glasses, laptop, bottle".
QS: First is newspaper. Second glasses. Third is laptop. Fourth is bottle. Now to reverse, change the order to: Fourth is bottle. Third is laptop. Second is glasses. First is newspaper. So the answer is "bottle, laptop, glasses, newspaper".
A: "bottle, laptop, glasses, newspaper"
QC: Reverse the sequence "laptop, photo, clip".
QS: First is laptop. Second is photo. Third is clip. Now to reverse, change the order to: Third is clip
Second is photo. First laptop. So the answer is "clip, photo, laptop".
A: "clip, photo, laptop"
QC: Reverse the sequence "driving license, button, packet, identity card, pineapple".
QS: First is driving license. Second is button. Third packet. Fourth is identity card. Fifth pineapple. Now to reverse, change the order to: Fifth is pineapple. Fourth is identity card. Third is packet. Second is button. First is driving license. So the answer is "pineapple, identity card, packet, button, driving license".
A:
"pineapple, identity card, packet, button, driving license"
unrolled led unccoopdddeccmp
QC: Reverse the sequence "driving license, button, packet, identity card, shoe".
QS: The sequence is "1. driving license, 2. button, 3, packet, 4. identity card, 5. shoe". The sequence
items long, which is more than the minimum length 4, so we split it. Half of get that the first half will be items long, ending in "2. button". The first half (2 items) "1. driving license, 2. button". The first half of the sequence ends with "2. button", so the second half starts after "2. button" with "3. packet". The full sequence 5 items long, and the first half is items long. so the second half will be
5 is 5 2 2.5. Dropping the decimal,
5 5-21 2 =3 items long. The second half of the sequence (3 items) is "3. packet, 4. identity card, 5. shoe". Removing the numbers from "1. driving license, 2. button", we get "driving license, button", Removing the numbers from "3. packet, identity card, 5. shoe". we get "packet, identity card, shoe". Reversing the sequence "driving license, button", ", First is driving license. Second is button. Now to reverse, change the order to: Second is button. First is driving license. So the answer is "button, driving license". Reversing the sequence "packet, identity card, shoe", First is packet. Second is identity card. Third shoe. Now to reverse, change the order to: Third is shoe. Second is identity card. First is packet. So the answer is "shoe, identity card, packet". Joining "shoe, identity card, packet" and "button, driving license", the answer is "shoe, identity card, packet, button, driving license".. A: "shoe, identity card, packet, button, driving license" OS: [EOQ]
QC: Reverse the sequence "laptop, photo, clip".
QS:
The sequence is "1 laptop. 2. photo, 3. clip". The sequence 3 items long, which is less than the minimum length of so we don't need split it. All we need to do is reverse "laptop, photo, clip"" First
is First laptop. Second is photo. Third is clip. Now to reverse, change the order to Third clip. Second is photo. First is laptop. So the answer is "clip, photo, laptop".
A: "clip. photo, laptop"
QS: [EOQ]
QC: Reverse the sequence "newspaper, glasses, laptop. bottle".
QS: The sequence is "1. newspaper, 2. glasses, 3. laptop, 4. bottle". The sequence is 4 items long, which is equal to the minimum length of 4, so we split it. Half of 4 is 4/ = 2.0. Dropping the decimal, get that the first half will be 2 items long, ending in "2. glasses". The first half
27

---

Published as a conference paper at ICLR 2023
(2 items) is "1. newspaper, 2. glasses". The first half of the sequence ends with "2. glasses", so the second half starts after "2. glasses" with "3. laptop". The full sequence is 4 items long, and the first half is items long, the second half will be 4 - = items long. The second half of the sequence (2 items) is "3. laptop, 4. bottle". Removing the numbers from "1. get "newspaper, glasses". Removing the numbers from "3. laptop, 4. bottle", we get "laptop, bottle". Reversing the sequence "newspaper, glasses", First is newspaper. Second glasses. Now reverse, change the order to: Second is glasses. First newspaper. So the answer is "glasses, newspaper". Reversing the sequence "laptop, bottle", First laptop. Second bottle. Now reverse, change the order to: Second is bottle. First is laptop. So the answer is "bottle, laptop". Joining "bottle, laptop" and "glasses, newspaper", the answer "bottle, laptop, glasses, newspaper". "bottle, laptop, glasses, newspaper"
newspaper, 2. glasses",
A:
OS: [BOQ]
reverse (tail)
QC: Reverse the sequence "driving license, button, packet, identity card, shoe".
Q1: [extract] The sequence is "1. driving license, button, packet, identity card shoe". The sequence is 5 items long, which is more than the minimum length of 4, so split it. We take the first element in the sequence which is "I. driving license".
#1: "1. driving license"
Q2: [extract] The full sequence 5 items long, so the remaining sequence will be 5 - items long. The tail of the sequence with 4 items "2. button, 3. packet, 4. identity card, 5, shoe". #2: "2 button, 3. packet, 4. identity card, 5. shoe
Q3: gremove_numbert Remove the numbers from #1.
#3: "driving license"
Q4: [remove numbers] Remove the numbers from #2,
#4: "button, packet, identity card, shoe"
Q5: [reverse] Reverse the sequence #4.
#5: "shoe, identity card, packet, button"
Q6: [join]
#6: "shoe, identity card, packet, button, driving license"
Q7: [EOQ]
QC: Reverse the sequence "laptop. photo, clip".
Q1: [extract] The sequence is "1. laptop, photo, 3. clip". The sequence is 3 items long, which is less than the minimum length of 4, so we don't need to split it. All we need to do is reverse laptop, photo, clip".
#1: "laptop, photo, clip"
Q2: [cot] Reverse the sequence #1.
#2:
"clip, photo, laptop"
Q3: [EOQ]
QC: Reverse the sequence "newspaper, glasses, laptop. bottle".
QI: [extract] The sequence is "1. newspaper, 2. glasses, 3. laptop, 4. bottle". The sequence is items long, which is more than the minimum length of 4, so we split it. We take the first element in the sequence which is "1. newspaper".
#1: "1. newspaper"
Q2:
[extract] The full sequence 4 items long, the remaining sequence will be 4- 1 items long. The tail of the sequence with 3 items "2 glasses, 3. laptop, 4. bottle".
#2: "2. glasses, 3. laptop. bottle"
Q3: [remove numbers] Remove the numbers from #1. "newspaper"
Q4: [remove_numberdd Remove the numbers from #2.
#4: "glasses, laptop, bottle"
Q5: [reverse] Reverse the sequence #4.
#5: "bottle, laptop, glasses"
Q6: [join] #5 #3
#6: "bottle, laptop, glasses, newspaper"
Q7: [FOQ]
28

---

Published as a conference paper at ICLR 2023
G.3 LONG-DOCUMENT QA
We show one of the prompts used for CommaQA experiments here. The entire set of prompts is provided as supplemetary material.
G.3.1 DECOMPOSED PROMPTING: (COARSE)
decomp
What awards have movies produced by people born in 1910 won?
QS: (select) [qa] Who were born in the year 1910?
A: ["Teeplemole", "Muntaril"]
QS: (project_ values. faautsssstttuuu wwt.tllaattaaaaa aatquuniqe [qa] For which movies was #1 the producer?
A: ["Featsaw", "Zalate", "Premercy"]
QS: (project_ wwweoeaaaleesseaiiggeeieeee [qa] Which awards were given to #2?
A: I'"Zorion", "Chowwurst", "Hallowcock"]
QS: [EOQ]
QC: What movics have people from the country Stridery acted in?
QS: (select) [qa] Who is from the country Stridery?
A: ["Gastrat"]
OS: prroject_valuee flat flaquun [qa] Which movies has #1 been actor in?
A: "PPatteershipaakr",, "Nilitude", "Warpstone"]
QS: [EOQ]
QC: What awards have the actors of the Erowid winning movies received?
QS: (select) [qa] Which movies were given the Erowid award?
A: ["Dewbar", "Caudacite")
QS: ponoject_valuee faaesssffat__unnaa _latuuuiuuuee [qa] Who are the actors in the movie #1?
A: ["Wetherality". "Lougerière", "Gigabut")
QS: aproject_values_ffat...
[qa] Which awards were given to #2?
A: [" ""Anioonder", 'TTifogation"]
QS: [EOQ]
QC: What awards did the movies directed by the Modiparity winners receive?
QS: (select) [qa] Who has been awarded the Modiparity award?
A: ["Bioperatology"]
QS: (project_vaal values flat (nique) [qa] Which movies has #1 directed?
-aroject_vvauus....
A: ["Pestok". "Vitrilateral"]
QS: (prooectt.values_fffe fitttues_fat_uuquueee [qa] Which awards were given to #2?
A: I"Gutney", "Antipositive"]
QS: [EOQ]
QC: What awards have movies written by people born in 1935 won?
QS: (select) [qa] Who were born the year 1935?
A: ["Selerocybin", "Zayage"]
QS: (projecttvaluute ___faat_uuuu flat_unique) [qa] What movies has #1 written?
A: ["Noenometer", "Tayenne", "Pneumodendron")
QS: awrojeec_values_faatuuuugg [qa] Which awards were given to #2?
A: ["Brownbeard", "Goosehead", "Handt"]
QS: [EOQ]
QC: What movies have the directors from Legault directed?
QS: (select) [qa] Who from the country Legault?
A: ["Metatoun", "Sapien"]
QS: (project. fttuuessfaa.uuiquue [qa] What movies has #1 been the director of?
A: ("Coacheship", "Misapportionnent]]
QS: [EOQ]
29

---

Published as a conference paper at ICLR 2023
movie:
Premercy; ; directed by: Muntaril. movie: Skirtsicine ; director: Teeplemole. movie: Featsaw
directed by: Monsterscar. movie: Zalate; ; director: Monsterscar. movie: Zalate; awarded:
Hallowcock. movie: Featsaw: awarded: Zorgion. movie: Premercy: award: Chowwurst.
movie: Skirtsicine; award: Hallowcock. award: Goatfly; winner: Teeplemole. person:
Monsterscar ; award: Glodome, person: Muntaril award: Goatfly. movie: Featsaw release
year: 1973. movie: Zalate; release year: 1964. movie: Skirtsicine release year: 1973. movie: Premercy: year: 1961. Teeplemole was an actor in the movie Skirtsicine. Muntaril was an actor in the movie Skirtsicine. Monsterscar was an actor in the movie Premercy. Muntaril was an actor in the movie Featsaw. Teeplemole was an actor in the movie Zalate. Muntaril was born in the year 1910. Teeplemole was born 1910. Monsterscar was born in 1942.
Teeplemole is from the country of Piperfish. Monsterscar is from the country of Piperfish. Muntaril is from the country of Clony. Muntaril produced the movie Skirtsicine with others. Monsterscar was one of the producers of the movie Featsaw. Monsterscar produced the movie Premercy with others. Monsterscar produced the movie Zalate with others. Teeplemole was one of the producers of the movie Featsaw. Teeplemole produced the movie Zalate with others. Muntaril produced the movie Premercy with others. Monsterscar wrote for the movie Premercy. Muntaril one of the writers for the movie Zalate. Muntaril wrote for the movie Featsaw. Teeplemole wrote for the movie Featsaw. Monsterscar was one of the writers for the movie Zalate. Teeplemole was one of the writers for the movie Skirtsicine. Q: For which movies was Teeplemole the producer?
A:
["Featsaw", "Zalate"]
Q: Which awards were given to Featsaw?
A: ["Zorgion""
movie:
Misgendery directed by: Wetherality. movie: Dewbar: director: Gigabut. movie:
Caudacite director: Lougerière. movie: Tayenne: directed by: Lougerière. movie:
Misgendery: awarded: Microsouenesis. movie: Dewbar awarded: Erowid. movie: Tayenne;
awarded: Cockspit. movie: Caudacite award: Erowid. award: Aniconder winner:
Wetherality. award: Aniconder: winner: Lougerière. person: Gigabut; ; award: Trifogation.
movie: Dewbar; release year: 1991. movie: Tayenne: year: 2013. movie: Caudacite release
year: 2008. movie: Misgendery year: 1991. Wetherality was an actor in the movie Dewbar.
Gigabut was an actor in the movie Tayenne. Lougerière was an actor in the movie Tayenne. Lougerière acted in the movie Caudacite. Lougerière acted in the movie Misgendery. Gigabut was an actor in the movie Caudacite. Wetherality was an actor in the movie Misgendery. Wetherality was born in the year 1917. Lougerière was born in 1926. Gigabut was born in the year 1917. Gigabut grew up in the nation of Triclops. Lougerière is from the country of Tatkin Wetherality grew up in the nation of Tatkin. Lougerière produced the movie Dewbar with others. Gigabut produced the movie Tayenne with others. Gigabut produced the movie Dewbar with others. Lougerière was one of the producers of the movie Misgendery. Wetherality was one of the producers of the movie Caudacite. Gigabut was one of the producers of the movie Caudacite. Wetherality produced the movie Misgendery with others. Wetherality produced the movie Tayenne with others. Wetherality wrote for the movie Tayenne. Gigabut wrote for the movie Misgendery. Lougerière was one of the writers for the movie Caudacite. Wetherality wrote for the movie Misgendery. Gigabut wrote for the movie Tayenne. Gigabut wrote for the movie Dewbar. Lougerière wrote for the movie Dewbar. Wetherality wrote for the movie Caudacite.
Q: Who are the actors in the movie Caudacite? A: ["Lougeriére, "Gigabut"]
Q: Which movics were given the Erowid award?
A: ("Dewbar", "Caudacite")
movie:
Pastillobox ; directed by: Firmline. movie: Clenestration ; directed by: Carblock. movie:
Pestok ; directed by: Bioperatology. movie: Vitrilateral ; director: Bioperatology. movie: Vitrilateral ; award: Antipositive. movie: Clenestration awarded: Handt. movie: Pastillobox ;
awarded: Handt. movie: Pestok awarded: Gutney. movie: Pestok writer: Firmline. movie:
30

---

Published as a conference paper at ICLR 2023
Clenestration written by: Carblock. movie: Pastillobox: written by: Bioperatology. movie:
Pestok; writer: Bioperatology. movie: Clenestration: written by: Firmline. movie: Vitrilateral
writer: Bioperatology. movie: Pastillobox writer: Carblock. movie: Vitrilateral written by Carblock. movie: Pestok release year: 1986. movie: Clenestration ; year: 1986. movie:
actor in the movie Pastillobox. Firmline was an actor in the movie Vitrilateral. Bioperatology was an actor in the movie Clenestration. Firmline acted in the movie Pastillobox. Carblock was actor in the movie Clenestration. Bioperatology was actor in the movie Pestok. Firmline was born in the year 1904. Bioperatology was born in the year 1935. Carblock was born in 1935. Carblock grew up in the nation of Knoppock. Firmline grew up in the nation of Tatkin.
Vitrilateral year: 1999. movie: Pastillobox release year: 1984. Carblock
Bioperatology grew up the nation of Tatkin. Bioperatology won the Modiparity award. Halfbill was awarded to Firmline. Halfbill was awarded to Carblock. Bioperatology was one of the producers of the movic Pestok. Bioperatology produced the movie Vitrilateral with others. Firmline produced the movie Pastillobox with others. Firmline produced the movie Clenestration with others. Carblock was one of the producers of the movie Pastillobox. Carblock produced the movie Vitrilateral with others. Carblock produced the movie Clenestration with others. Firmline was one of the producers of the movie Pestok.
Q: Who has been awarded the Modiparity award?
A: "Bioperatology")
Q: Which movies has Bioperatology directed??
A: ["Pestok", "Vitrilateral")
movie:
Nohit director: Mimicocycle. movie: Noenometer ; director: Mimicocycle, movie:
Tayenne: directed by: Zayage. movie: Pneumodendron. director: Sclerocybin. movie:
Tayenne; awarded: Goosehead. movie: Nohit ; awarded: Handt. movie: Pneumodendron:
award: Handt. movie: Noenometer awarded: Brownbeard. movie: Nohit writer:
Mimicocycle. movie: Noenometer; written by: Sclerocybin. movie: Tayenne ; writer:
Sclenocybin. movie: Pneumodendron written by: Zayage. movie: Tayenne writer: Zayage.
movie: Pneumodendron ; written by: Mimicocycle. movie: Noenometer release year: 1991.
movie: Tayenne year: 2013. movie: Nohit year: 2005. movie: Pneumodendron. year: 2005. Mimicocycle was an actor in the movie Tayenne. Zayage acted in the movie Pneumodendron. Zayage was an actor in the movie Nohit. Sclerocybin was an actor in the movie Nohit.
Sclerocybin was an actor in the movie Tayenne. Mimicocycle was an actor in the movie Noenometer. Zayage was born 1935. Sclerocybin was born 1935. Mimicocycle was born in 1930. Mimicocycle is from the country of Calderita. Sclerocybin grew up in the nation of Calderita. Zayage from the country of Obility. Quinion was awarded to Zayage. Fannyxist awarded Sclerocybin. Famnyxist the movie Nohit with others. Zayage was one of the producers of the movie Nohit.
awarded to Mimicocycle. Mimicocycle produced
Sclerocybin one of the producers of the movie Tayenne. Sclerocybin produced the movie Pneumodendron with others. Zayage produced the movie Pneumodendron with others. Mimicocycle was one of the producers of the movie Tayenne. Sclerocybin one of the producers of the movie Noenometer. Zayage produced the movie Noenometer with others
Q: What movies has Sclerocybin written? A: [" "Noenometer", "Tayenne"]
Q: Who were born in the year 1935?
A: ""Selerocybin", '"aaygee"]
G.3.2 DECOMPOSED PROMPTING: (FINE)
decomp
What awards have movies produced by people born in 1910 won?
QS: (select) [simp-qal Who were born in the year 1910?
A: ["Teeplemmole" "Muntaril"]
QS: saaplecttvalus.lte flat-uniquue [pos_qa] I For which movies was #1 the producer?
A: ["Featsaw". "Zalate", "Premerey")
QS: (project values flat unique) [aw qa] Which awards were given to #2?
A: ["Zorgion", "Chowwurst". "Hallowcock")
31

---

Published as a conference paper at ICLR 2023
QS: [EOQ]
QC: What movies have people from the country Stridery acted in?
QS: (select) [simp qa] Who is from the country Stridery?
A: ["Gastrat"]
QS: _oagject_valus.... flat__niquu) [pos_qa] Which movies has #1 been an actor in?
A: ""Partterrhhimmake",, "Nilitude", "Warpstone"]
QS: [EOQ]
QC: What awards have the actors of the Erowid winning movies received?
QS: (select) [aw qa] Which movies were given the Erowid award?
A: ["Dewbar", "Caudacite")
QS: (project_values__iee uiiqqqe [pos_qa] Who are the actors in the movic #1?
A: ["Wetherallit", "Lougerière", "Gigabut"]
QS: (project values..auue..... flat unique) [aw qa] Which awards were given to #2?
A: ["Annconder", "Trifogation"]
QS: [BOQ]
QC: What awards did the movies directed by the Modiparity winners receive?
QS: (select) [aw-qa] Who has been awarded the Modiparity award?
A: ["Bioperatology")
Q (project values flat unique) [pos qa] Which movies has #1 directed?
A: ["Pestok", "Vitrilateral"]
QS: (project_values. satpecevvvleesefttuuuuiiiii [aw _w] Which awards were given to #2?
A: ["Gutney", "Antipositive"]
QS: [EOQ]
QC: What awards have movies written by people born in 1935 won?
QS: (select) [simp qa] Who were born in the year 1935?
A: ["Sclerocybin", "7ayyge")
QS: (project_values.... www.aaalllattuiiqulele
[pos_qa] What movies has #1 written?
A: ["Noenometer", "Tayenne", "Pneumodendron")
QS: (project wawues
wawuuesffattuuiiee (laquu) [aw [aw-qa] Which awards were given to #2?
A: ["Brownbeard", "Goosehead", "Handt"]
QS: [BOQ]
QC: What movies have the directors from Legault directed?
QS: (select) [simp-qa] Who is from the country Legault?
A: ["Metatoun", "Sapien"]
OS: (project values flat unique) [pos. What movies has #1 been the director of?
A: I"Coacheship", "Misapportionment"]
QS: [BOQ]
aw_ga
movie: Premercy: directed by: Muntaril. movie: Skirtsicine director: Teeplemole. movie: Featsaw
directed by: Monsterscar. movie: Zalate director: Monsterscar. movie: Zalate awarded:
Hallowcock. movie: Featsaw awarded: Zorgion. movie: Premercy award: Chowwurst.
movie: Skirtsicine award: Hallowcock. award: Goatfly winner: Teeplemole, person:
Monsterscar award: Glodome, person: Muntaril award: Goatfly. movie: Featsaw release
year: 1973. movie; Zalate release year: 1964. movie: Skirtsicine release year: 1973. movie:
Premercy year: 1961. Teeplemole was an actor in the movie Skirtsicine. Muntaril was an actor the movie Skirtsicine. Monsterscar was an actor in the movie Premercy. Muntaril was an actor in the movie Featsaw. Teeplemole was an actor in the movie Zalate. Muntaril was born in the year 1910. Teeplemole was born 1910. Monsterscar was born in 1942.
Teeplemole is from the country of Piperfish. Monsterscar is from the country of Piperfish. Muntaril is from the country of Clony. Muntaril produced the movie Skirtsicine with others. Monsterscar
of the producers of the movie Featsaw. Monsterscar produced the movie Premercy with others. Monsterscar produced the movie Zalate with others. Teeplemole was one of the producers of the movie Featsaw. Teeplemole produced the movie Zalate with
32

---

Published as a conference paper at ICLR 2023
others. Muntaril produced the movie Premerey with others. Monsterscar wrote for the movie Premercy. Muntaril was one of the writers for the movie Zalate. Muntaril wrote for the movie Featsaw. Teeplemole wrote for the movie Featsaw. Monsterscar one of the writers for the movie Zalate. Teeplemole was one of the writers for the movie Skirtsicine.
Q: Which awards were given to Zalate?
A:
["Hallowcock"]
Q: Which awards were given to Featsaw? ["Zorgion"]
A:
movie:
Misgendery ; directed by: Wetherality. movie: Dewbar; director: Gigabut. movie:
Caudacite: director: Lougerière. movie: Tayenne; directed by: Lougerière. movie:
Misgendery: awarded: Microsouenesis. movie: Dewbar awarded: Erowid. movie: Tayenne;
awarded: Cockspit. movie: Caudacite: award: Erowid. award: Aniconder: winner:
Wetherality. award: Aniconder: winner: Lougerière. person: Gigabut: ; award: Trifogation.
movie: Dewbar; release year: 1991. movie: Tayenne ; year: 2013. movie: Caudacite release
year: 2008. movie: Misgendery year: 1991. Wetherality was an actor in the movie Dewbar. Gigabut was an actor in the movie Tayenne. Lougerière
an actor in the movie Tayenne. Lougerière acted in the movie Caudacite. Lougerière acted in the movie Misgendery. Gigabut was an actor in the movie Caudacite. Wetherality was an actor in the movie Misgendery. Wetherality was born in the year 1917. Lougerière was born in 1926. Gigabut was born in the year 1917. Gigabut grew up in the nation of Triclops. Lougerière is from the country of Tatkin Wetherality grew up in the nation of Tatkin. Lougerière produced the movie Dewbar with others. Gigabut produced the movie Tayenne with others. Gigabut produced the movie Dewbar with others. Lougerière was one of the producers of the movie Misgendery. Wetherality was one of the producers of the movie Caudacite. Gigabut was one of the producers of the movie Caudacite. Wetherality produced the movie Misgendery with others. Wetherality produced the movie Tayenne with others. Wetherality wrote for the movic Tayenne. Gigabut wrote for the movie Misgendery. Lougerière was of the writers for the movie Caudacite. Wetherality wrote for the movie Misgendery. Gigabut wrote for the movie Tayenne. Gigabut wrote for the movie Dewbar. Lougerière wrote for the movie Dewbar. Wetherality wrote for the movie Caudacite.
Q: Which movies were given the Erowid award? A: [ "Debarr", "Caudacite")
Q: Which awards were given to Wetherality? A: ("(" ["Aniconder"]
movie: Pastillobox ; directed by: Firmline. movic: Clenestration ; directed by: Carblock. movie:
Pestok; directed by: Bioperatology. movie: Vitrilateral ; director: Bioperatology. movie:
Vitrilateral ; award: Antipositive. movie: Clenestration awarded: Handt. movie: Pastillobox
awarded: Handt. movie: Pestok: awarded: Gutney. movie: Pestok writer: Firmline. movie:
Clenestration written by: Carblock. movie: Pastillobox written by: Bioperatology. movie:
Pestok; writer: Bioperatology. movie: Clenestration written by: Firmline. movie: Vitrilateral
; writer: Bioperatology. movie: Pastillobox writer: Carblock. movie: Vitrilateral written by Carblock. movie: Pestok release year: 1986. movie: Clenestration ; year: 1986. movie: Vitrilateral year: 1999. movie: Pastillohox release year: 1984. Carblock was an actor in the movie Pastillobox. Firmline was an actor in the movie Vitrilateral. Bioperatology was an actor in the movie Clenestration. Firmline acted in the movie Pastillobox. Carblock was actor in the movie Clenestration. Bioperatology was an actor in the movie Pestok. Firmline was born in the year 1904. Bioperatology was born in the year 1935. Carblock was born in 1935.
Carblock grew up in the nation of Knoppock. Firmline grew up in the nation of Tatkin.
Bioperatology grew up in the nation of Tatkin. Bioperatology won the Modiparity award. Halfbill was awarded to Firmline. Halfbill was awarded to Carblock. Bioperatology was one of the producers of the movie Pestok. Bioperatology produced the movie Vitrilateral with others. Firmline produced the movie Pastillobox with others. Firmline produced the movie Clenestration with others. Carblock was one of the producers of the movie Pastillobox. Carblock produced the movie Vitrilateral with others. Carblock produced the movie Clenestration with others. Firmline was one of the producers of the movie Pestok.
33

---

Published as a conference paper at ICLR 2023
Q: Who has been awarded the Modiparity award?
A: "Bioperatology"
Q: Which awards were given to Pestok?
A:
["Gutney"]
movie:
Nohit director: Mimicocycle. movie: Noenometer director: Mimicocycle. movie:
Tayenne; directed by: Zayage. movie: Pneumodendron. director: Sclerocybin. movie:
Tayenne; ; awarded: Goosehead. movie: Nohit ; awarded: Handt. movie: Pneumodendron;
award: Handt. movie: Noenometer: awarded: Brownbeard. movie: Nohit writer:
Mimicocycle. movie: Noenometer: written by: Sclerocybin. movie: Tayenne: ; writer:
Sclerocybin. movie: Pneumodendron written by: Zayage. movie: Tayenne writer: Zayage.
movic: Pncumodendron ; written by: Mimicocycle. movie: Noenometer release year: 1991.
movie: Tayenne; year: 2013. movie: Nohit year: 2005. movie: Pneumodendron. year: 2005. Mimicocycle was an actor the movie Tayenne. Zayage acted in the movie Pneumodendron Zayage actor in the movie Nohit. Sclerocybin Sclerocybin was an actor in the movie Tayenne. Mimicocycle was an actor in the movie Noenometer. Zayage was born in 1935. Sclerocybin was born in 1935. Mimicocycle was born in 1930. Mimicocycle is from the country of Calderita. Sclerocybin grew up in the nation of Calderita. Zayage from the country of Obility. Quinion was awarded to Zayage. Fannyxist was awarded to Sclerocybin. Fannyxist was awarded to Mimicocycle. Mimicocycle produced the movie Nohit with others. Zayage was one of the producers of the movie Nohit.
actor in the movie Nohit.
Sclerocybin was one of the producers of the movie Tayenne. Sclerocybin produced the movie Pneumodendron with others. Zayage produced the movie Pneumodendron with others. Mimicocycle was one of the producers of the movie Tayenne. Sclerocybin was one of the producers of the movie Noenometer. Zayage produced the movie Noenometer with others
Q: Which awards were given to Noenometer? A: "rrowneaard"]
Q: Which awards were given to Pneumodendron?
A: ["Handt"]
pos_ga
movie:
Premercy: directed by: Muntaril. movie: Skirtsicine director: Teeplemole. movie: Featsaw
: ddirected by: Monsterscar. movie: Zalate: ; director: Monsterscar. movie: Zalate awarded:
Hallowcock. movie: Featsaw awarded: Zorgion. movie: Premercy award: Chowwurst.
movie: Skirtsicine award: Hallowcock. award: Goatfly winner: Teeplemole, person:
Monsterscar award: Glodome. person: Muntaril award: Goatfly. movie: Featsaw release
year: 1973. movie: Zalate release year: 1964. movie: Skirtsicine release year: 1973. movie: Premercy: year: 1961. Teeplemole was an actor in the movie Skirtsicine, Muntaril was an actor in the movie Skirtsicine. Monsterscar was actor in the movie Premercy. Muntaril was an actor in the movic Featsaw. Teeplemole was an actor in the movie Zalate. Muntaril was born in the year 1910. Teeplemole was born in 1910. Monsterscar was born in 1942.
Teeplemole is from the country of Piperfish. Monsterscar is from the country of Piperfish. Muntaril is from the country of Clony. Muntaril produced the movie Skirtsicine with others. Monsterscar was one of the producers of the movie Featsaw. Monsterscar produced the movie Premercy with others. Monsterscar produced the movie Zalate with others. Teeplemole was one of the producers of the movie Featsaw. Teeplemole produced the movie Zalate with others. Muntaril produced the movie Premercy with others. Monsterscar wrote for the movie Premercy. Muntaril was one of the writers for the movie Zalate. Muntaril wrote for the movic Featsaw. Teeplemole wrote for the movie Featsaw. Monsterscar was one the writers for the movie Zalate. Teeplemole was one of the writers for the movie Skirtsicine.
Q: For which movies was Teeplemole the producer?
A: ["Featsaw". "Zalate"]
Q: For which movies was Muntaril the producer? A: ["Premercy]
34

---

Published as a conference paper at ICLR 2023
movie: Misgendery: directed by: Wetherality. movie: Dewbar; director: Gigabut. movie:
Caudacite; director: Lougerière. movie: Tayenne; directed by: Lougerière. movie:
Misgendery; awarded: Microsouenesis, movie: Dewbar; awarded: Erowid. movie: Tayenne:
awarded: Cockspit. movie: Caudacite award: Frowid. award: Aniconder: winner:
Wetherality. award:
Aniconder winner: Lougerière. person: Gigabut ; award: Trifogation.
movie: Dewbar ; release year: 1991. movie: Tayenne: year: 2013. movie: Caudacite release
year: 2008. movie: Misgendery year: 1991. Wetherality was an actor in the movie Dewbar.
Gigabut was an actor in the movie Tayenne. Lougerière was an actor in the movie Tayenne. Lougerière acted in the movie Caudacite. Lougerière acted in the movie Misgendery. Gigabut was an actor in the movie Caudacite. Wetherality was an actor in the movie Misgendery. Wetherality was born the year 1917. Lougerière was born in 1926. Gigabut was born in the year 1917. Gigabut grew up in the nation of Triclops. Lougerière is from the country of Tatkin Wetherality grew up in the nation of Tatkin. Lougerière produced the movie Dewbar with others. Gigabut produced the movie Tayenne with others. Gigabut produced the movie Dewbar with others. Lougerière was one of the producers of the movie Misgendery. Wetherality was one of the producers of the movie Caudacite. Gigabut producers of the movie Caudacite. Wetherality produced the movie Misgendery with others. Wetherality produced the movie Tayenne with others. Wetherality wrote for the movie Tayenne. Gigabut wrote for the movie Misgendery. Lougerière was one of the writers for the movie Caudacite. Wetherality wrote for the movie Misgendery. Gigabut wrote for the movie Tayenne. Gigabut wrote for the movie Dewbar. Lougerière wrote for the movie Dewbar. Wetherality wrote for the movie Caudacite.
of the
Q: Who are the actors in the movie Dewbar?
A: ["Wetherality"]
Q: Who are the actors in the movie Caudacite?
A: ["Lougerière, "Gigabut"]
movie:
Pastillobox: ; directed by: Firmline. movie: Clenestration ; directed by: Carblock. movie:
Pestok directed by: Bioperatology. movie: Vitrilateral ; director: Bioperatology. movie:
Vitrilateral ; award: Antipositive. movie: Clenestration awarded: Handt. movie: Pastillobox ;
awarded: Handt. movie: Pestok awarded: Gutney, movie: Pestok writer: Firmline. movie:
Clenestration written by: Carblock. movie: Pastillobox written by: Bioperatology. movie: Pestok; writer: Bioperatology. movie: Clenestration written by: Firmline. movie: Vitrilateral
: writer: Bioperatology. movie: Pastillobox : writer: Carblock. movie: Vitrilateral written by Carblock. movie: Pestok release year: 1986. movie: Clenestration year: 1986. movie:
movie Pastillobox. Firmline Pastillobox ; release year: 1984. Carblock was an actor in the actor in the movie Vitrilateral. Bioperatology
Vitrilateral : year: 1999. movie:
actor in the movie Clenestration. Firmline acted in the movic Pastillobox. Carblock was an actor in the movie Clenestration. Bioperatology was actor in the movie Pestok. Firmline born in the year 1904. Bioperatology was born in the year 1935. Carblock was born in 1935.
Carblock grew up in the nation of Knoppock. Firmline grew up in the nation of Tatkin. Bioperatology grew up in the nation of Tatkin. Bioperatology won the Modiparity award. Halfbill was awarded Firmline. Halfbill was awarded Carblock. Bioperatology was one of the producers of the movie Pestok. Bioperatology produced the movie Vitrilateral with others. Firmline produced the movie Pastillobox with others. Firmline produced the movie Clenestration with others. Carblock was one of the producers of the movie Pastillobox. Carblock produced the movie Vitrilateral with others. Carblock produced the movie Clenestration with others. Firmline was one of the producers of the movie Pestok.
Q: Which movies has Bioperatology directed?
A: ["Pestok". "Vitrilateral")
Q: Which movies has Carblock directed?
A: ["Clenestration")
movie: Nohit director: Mimicocycle. movie: Noenometer ; director: Mimicocycle. movie: Tayenne directed by: Zayage. movie: Pneumodendron director: Sclerocybin. movie: Tayenne; awarded: Goosehead. movie: Nohit awarded: Handt. movie: Pneumodendron:
award: Handt. movie: Noenometer: awarded: Brownbeard. movie: Nohit writer:
35

---

Published as a conference paper at ICLR 2023
Mimicocycle. movie: Noenometer: written by: Sclerocybin. movie: Tayenne: writer:
Sclerocybin. movie: Pneumodendron ; written by: Zayage. movie: Tayenne writer: Zayage.
movie: Pneumodendron ; written by: Mimicocycle. movie: Noenometer release year. 1991.
movie: Tayenne: year: 2013. movie: Nohit year: 2005. movie: Pneumodendron ; year: 2005. Mimicocycle was actor in the movie Tayenne. Zayage acted in the movie Pneumodendron. Zayage was an actor in the movie Nohit. Sclerocybin was an actor in the movie Nohit.
Sclerocybin was an actor in the movie Tayenne. Mimicocycle was an actor in the movie Noenometer. Zayage was born in 1935. Sclerocybin was born in 1935. Mimicocycle was born in 1930. Mimicocycle is from the country of Calderita. Sclerocybin grew up the nation of Calderita. Zayage is from the country of Obility. Quinion was awarded to Zayage. Fannyxist was awarded to Sclerocybin. Fannyxist was awarded to Mimicocycle. Mimicocycle produced the movie Nohit with others. Zayage was one of the producers of the movie Nohit.
Sclerocybin was one of the producers of the movie Tayemne. Sclerocybin produced the movic Pneumodendron with others. Zayage produced the movie Pneumodendron with others. Mimicocycle was one of the producers of the movie Tayenne. Sclerocybin was one of the producers of the movie Noenometer. Zayage produced the movie Noenometer with others
Q: What movies has Sclerocybin written? A: ["Noenometrr" "Tayenne"]
Q: What movies has Zayage written?
A: "Pneumodendron", "Tayenne"]
s simp-ga
movie: Premercy: directed by: Muntaril. movie: Skirtsicine director: Teeplemole. movie: Featsaw
directed by: Monsterscar. movie: Zalate; ; director: Monsterscar. movie: Zalate; awarded:
Hallowcock. movie: Featsaw awarded: Zorgion. movie: Premercy award: Chowwurst.
movie: Skirtsicine; award: Hallowcock. award: Goatfly winner: Teeplemole, person:
Monsterscar : award: Glodome. person: Muntaril award: Goatfly. movie: Featsaw release
year: 1973. movie: Zalate release year: 1964. movie: Skirtsicine release year: 1973. movie: Premercy: year: 1961. Teeplemole was an actor in the movie Skirtsicine. Muntaril was an actor in the movie Skirtsicine. Monsterscar an actor in the movie Featsaw. Teeplemole was an actor in the movie Zalate. Muntaril was born in the year 1910. Teeplemole was born in 1910. Monsterscar was born in 1942.
actor in the movie Premercy. Muntaril was
Teeplemole is from the country of Piperfish. Monsterscar is from the country of Piperfish. Muntaril is from the country of Clony. Muntaril produced the movie Skirtsicine with others. Monsterscar was one of the producers of the movie Featsaw. Monsterscar produced the movic Premercy with others. Monsterscar produced the movie Zalate with others. Teeplemole was one of the producers of the movie Featsaw. Teeplemole produced the movie Zalate with others. Muntaril produced the movie Premercy with others. Monsterscar wrote for the movie Premercy. Muntaril was one of the writers for the movie Zalate. Muntaril wrote for the movie Featsaw. Teeplemole wrote for the movie Featsaw. Monsterscar was one the writers for the movie Zalate. Teeplemole was one of the writers for the movie Skirtsicine.
Q: Who were born in the year 1910? A: "Teeplemoll", "Muntaril"]
Q: From which country is Monsterscar? A: ["Piperfish"]
movie: Nilitude: ; director: Monsterscar. movie: Dewbar ; directed by: Metatoun. movie: Warpstone
directed by: Gastrat, movie: Partnershipmaker director: Metatoun. movie: Dewbar ; award:
Tachychronograph. movie: Partnershipmaker awarded: Tachychronograph. movie: Nilitude:
award: Paleodactyl. movie: Warpstone award: Sabonade, person: Gastrat ; award:
Trifogation. award: Polyquadrase winner: Monsterscar. award: Trifogation winner:
Metatoun. movie: Warpstone release year: 1956. movie: Dewbar release year: 1984, movie:
Nilitude year: 1984. movie: Partnershipmaker: year: 1962. Gastrat was actor in the movie Partnershipmaker. Metatoun was an actor in the movie Partnershipmaker. Metatoun an actor in the movie Nilitude. Gastrat acted in the movie Nilitude. Monsterscar was actor in the movie Dewbar. Gastrat acted in the movie Warpstone. Metatoun acted in the movie Warpstone. Metatoun born in 1939. Gastrat born in the year 1933.
36

---

Published as a conference paper at ICLR 2023
Monsterscar was born in 1933. Metatoun grew up in the nation of Moulole. Gastrat is from the country of Stridery. Monsterscar grew up in the nation of Moulole. Monsterscar produced the movie Nilitude with others. Monsterscar was one of the producers of the movie Warpstone Metatoun was one of the producers of the movie Warpstone, Gastrat was one of the producers of the movie Nilitude. Metatoun produced the movie Partnershipmaker with others. Metatoun produced the movie Dewbar with others. Monsterscar was one of the producers of the movie Partnershipmaker. Gastrat produced the movie Dewbar with others. Metatoun wrote for the movie Partnershipmaker, Gastrat wrote for the movie Warpstone. Gastrat was one of the writers for the movie Dewbar. Monsterscar was one of the writers for the movie Nilitude. Metatoun wrote for the movie Warpstone.
Q: Who from the country Stridery?
A: ["Gastrat"]
Q: Which movies were released in 1984? ("Dewbar", "Nilitude"]
A:
movie:
Nohit director: Mimicocycle. movie: Noenometer ; director: Mimicocycle. movie:
Tayenne: directed by: Zayage. movie: Pneumodendron director: Sclerocybin. movie:
Tayenne; awarded: Goosehead. movie: Nohit awarded: Handt. movie: Pneumodendron:
award: Handt. movie: Noenometer awarded: Brownbeard. movie: Nohit writer:
Mimicocycle. movie: Noenometer; written by: Sclerocybin. movie: Tayenne writer:
Sclerocybin. movie: Pneumodendron ; written by: Zayage. movie: Tayenne: writer: Zayage.
movie: Pneumodendron written by: Mimicocycle. movie: Noenometer release year: 1991.
movie: Tayenne: year: 2013. movie: Nohit year: 2005. movie: Pneumodendron, ; year: 2005. Mimicocycle was actor in the movie Tayenne. Zayage acted in the movie Pneumodendron. Zayage was an actor in the movie Nohit. Sclerocybin was an actor in the movie Nohit. Sclerocybin
actor in the movie Tayenne. Mimicocycle Noenometer. Zayage was born in 1935. Sclerocybin was born in 1935. Mimicocycle was born in 1930. Mimicocycle is from the country of Calderita. Sclerocybin grew up in the nation of Calderita. Zayage is from the country of Obility. Quinion was awarded to Zayage. Fannyxist was awarded Sclerocybin. Fannyxist was awarded Mimicocyele. Mimicocycle produced the movie Nohit with others. Zayage was one of the producers of the movie Nohit.
an actor in the movie
Sclerocybin was one of the producers of the movie Tayenne. Sclerocybin produced the movie Pneumodendron with others. Zayage produced the movie Pneumodendron with others. Mimicocycle was one of the producers of the movie Tayenne. Sclerocybin was one of the producers of the movie Noenometer. Zayage produced the movie Noenometer with others
Q: Who were born the year 1935? A: ("Selerocybin", "Zayage"]
Q: When the movie Nohit released?
A: ["2005"]
movie: Coacheship director:
Metatoun. movie: Assamplifier: ; director: Kapod. movie:
Misapportionmeent : director: Sapien. movie: Quinsid: director: Kapod. movie: Assamplifier
award: Zorgion. movie: Quinsid; awarded: Airpipe. movie: Coacheship award: Electrodesal.
movie: Misapportionment award: Airpipe. movie: Coacheship, written by: Metatoun.
movie: Misapportionment written by: Kapod. movie: Coacheship: written by: Kapod. movie
Quinsid ; writer: Sapien. movie: Misapportionment written by: Metatoun. movie:
Assamplitier; written by: Kapod. movie: Assamplifier: written by: Sapien. movie:
Assamplifier: release year: 2000. movie: Coacheship year: 2001. movie: Quinsid year:
2005. movie: Misapportionment year: 2005. Sapien was an actor in the movie
Misapportionment. Sapien acted in the movie Assamplifier. Kapod acted in the movie Quinsid Sapien acted in the movie Coacheship. Metatoun was actor in the movie Quinsid. Kapod acted in the movie Misapportionment. Metatoun acted in the movie Coacheship. Kapod acted in the movie Assamplifier. Sapien was born in the year 1910. Metatoun was born in 1928.
Kapod was born in the year 1910. Metatoun is from the country of Legault. Kapod grew up in the nation of Tatkin. Sapien is from the country of Legault. Malwarp was awarded Sapien. Metatoun won the Monkeynote award. Kapod won the Monkeynote award. Kapod was one of the producers of the movie Quinsid. Metatoun was one of the producers of the movie
37

---

Published as a conference paper at ICLR 2023
Misapportionment Metatoun of the producers of the movie Quinsid. Sapien of the producers of the movie Assamplifier. Sapien produced the movic Coacheship with others. Metatoun was of the producers of the movie Assamplifier. Kapod one of the producers of the movie Misapportionment. Kapod was one of the producers of the movie Coacheship.
Q: Who is from the country Legault?
A: ["Metatoun", "Sapien"]
Q: When was Kapod born?
A: ['1910"]
G.3.3 COT
COT
movie:
Premercy: directed by: Muntaril. movie: Skirtsicine director: Teeplemole. movie: Featsaw
directed by: Monsterscar. movie: Zalate ; director: Monsterscar. movie: Zalate awarded:
Hallowcock, movie: Featsaw awarded: Zorgion. movie: Premercy award: Chowwurst.
movie: Skirtsicine: award: Hallowcock. award: Goatfly winner: Teeplemole, person:
Monsterscar ; award: Glodome, person: Muntaril award: Goatfly. movie: Featsaw release
year: 1973. movie: Zalate release year: 1964. movie: Skirtsicine release year: 1973. movie: Premercy: year: 1961. Teeplemole was an actor in the movie Skirtsicine. Muntaril was an actor in the movie Skirtsicine. Monsterscar an actor in the movie Featsaw. Teeplemole was an actor in the movie Zalate. Muntaril was born in the year 1910. Teeplemole was born in 1910. Monsterscar was born in 1942.
actor in the movie Premercy. Muntaril was
Teeplemole is from the country of Piperfish. Monsterscar is from the country of Piperfish. Muntaril is from the country of Clony. Muntaril produced the movic Skirtsicine with others. Monsterscar was one of the producers of the movie Featsaw. Monsterscar produced the movie Premercy with others. Monsterscar produced the movie Zalate with others. Teeplemole was one of the producers of the movie Featsaw. Teeplemole produced the movie Zalate with others. Muntaril produced the movie Premercy with others. Monsterscar wrote for the movie Premercy. Muntaril one of the writers for the movie Zalate. Muntaril wrote for the movie Featsaw. Teeplemole wrote for the movie Featsaw. Monsterscar was one of the writers for the movie Zalate. Teeplemole
one of the writers for the movie Skirtsicine.
QC: What awards have movies produced by people born in 1910 won?
OS: The people born in 1910 were Muntaril and Teeplemole. Teeplemole produced the movies Featsaw and Zalate. Muntaril produced the movie Premercy, Featsaw was awarded the Zorgion award. Premercy was awarded the Chowwurst award. Zalate was awarded the Hallowcock award. So the answer is ["Zorgion", "Chowwurst", "Halloweock"). A: I"Zorgion", "Chowwurst", "Hallowcock"]
QS: [EOQ]
movie: Misgendery directed by: Wetherality. movie: Dewbar: director: Gigabut. movie:
Caudacite ; director: Lougerière. movie: Tayenne directed by: Lougerière. movie:
Misgendery awarded: Microsouenesis. movie: Dewbar awarded: Erowid. movie: Tayenne:
awarded: Cockspit. movie: Caudacite award: Erowid. award: Aniconder: winner:
Wetherality. award: Aniconder winner: Lougerière. person: Gigabut; ; award: Trifogation.
movie: Dewbar ; release year: 1991. movie: Tayenne: year: 2013. movie: Caudacite release
year: 2008. movie: Misgendery year: 1991. Wetherality was an actor in the movie Dewbar.
Gigabut was an actor in the movie Tayenne. Lougerière was an actor in the movie Tayenne. Lougerière acted the movie Caudacite. Lougerière acted in the movie Misgendery. Gigabut was an actor in the movie Caudacite. Wetherality was an actor in the movie Misgendery. Wetherality was born in the year 1917. Lougerière was born in 1926. Gigabut was born in the year 1917. Gigabut grew up in the nation of Triclops. Lougerière is from the country of Tatkin Wetherality grew up the nation of Tatkin. Lougerière produced the movie Dewbar with others. Gigabut produced the movie Tayenne with others. Gigabut produced the movie Dewbar with others. Lougerière was one of the producers of the movie Misgendery. of the
Wetherality one of the producers of the movie Caudacite. Gigabut producers of the movie Caudacite. Wetherality produced the movie Misgendery with others. Wetherality produced the movie Tayenne with others. Wetherality wrote for the movie
38

---

Published as a conference paper at ICLR 2023
Tayenne. Gigabut wrote for the movie Misgendery. Lougerière was one of the writers for the movie Caudacite. Wetherality wrote for the movie Misgendery. Gigabut wrote for the movie Tayenne. Gigabut wrote for the movie Dewbar. Lougerière wrote for the movie Dewbar. Wetherality wrote for the movie Caudacite.
QC: What awards have the actors of the Erowid winning movies received?
QS: The movies that won the Erowid award are Dewbar and Caudacite. Wetherality was an actor in Dewbar. Lougerière and Gigabut acted in Caudacite. Wetherality won the Aniconder award. Lougerière also won the Aniconder award. Gigabut won the Trifogation award. So the answer is I"Aniconder", "Trifogation"]. I"Aniconder", "Trifogation"] QS: [BOQ]
A:
movic: Pastillobox ; directed by: Firmline. movic: Clenestration ; directed by: Carblock. movie:
Pestok; directed by: Bioperatology. movie: Vitrilateral ; director: Bioperatology. movie:
Vitrilateral ; award: Antipositive. movie: Clenestration awarded: Handt. movie: Pastillobox; ;
awarded: Handt. movie: Pestok; awarded: Gutney. movie: Pestok; writer: Firmline. movie:
Clenestration written by: Carblock. movie: Pastillobox written by: Bioperatology. movie:
Pestok; writer: Bioperatology. movie: Clenestration written by: Firmline. movie: Vitrilateral
; writer: Bioperatology. movie: Pastillobox writer: Carblock. movie: Vitrilateral written by Carblock. movie: Pestok release year: 1986. movie: Clenestration ; year: 1986. movie:
Vitrilateral year: 1999. movie: Pastillobox release year: 1984. Carblock was an actor in the movie Pastillobox. Firmline was an actor in the movie Vitrilateral. Bioperatology was an actor in the movie Clenestration. Firmline acted in the movie Pastillobox. Carblock was an actor in the movie Clenestration. Bioperatology was an actor in the movie Pestok. Firmline was born in the year 1904. Bioperatology was born in the year 1935. Carblock was born in 1935.
Carblock grew up in the nation of Knoppock. Firmline grew up in the nation of Tatkin. Bioperatology grew up in the nation of Tatkin. Bioperatology won the Modiparity award. Halfbill was awarded to Firmline. Halfbill was awarded to Carblock. Bioperatology was one of the producers of the movie Pestok. Bioperatology produced the movie Vitrilateral with others. Firmline produced the movie Pastillobox with others. Firmline produced the movie Clenestration with others. Carblock was one of the producers of the movie Pastillobox. Carblock produced the movie Vitrilateral with others. Carblock produced the movie Clenestration with others. Firmline was one of the producers of the movie Pestok.
QC: What awards did the movies directed by the Modiparity winners receive? QS: The Modiparity winners are Bioperatology. Bioperatology has directed the movies Pestok and Vitrilateral. Pestok has won the Gutney award. Vitrilateral has won the Antipositive award. So the answer l"Gutney"" " "Antipositive"). ["Gutney", "AAtipositive"]
A:
QS: [BOQ]
movie:
Nohit director: Mimicocycle. movie: Noenometer ; director: Mimicocycle. movie: Tayenne: ; directed by: Zayage. movie: Pneumodendron director: Sclerocybin. movie:
Tayenne: ; awarded: Goosehead. movie: Nohit ; awarded: Handt. movie: Pneumodendron:
award: Handt. movie: Noenometer; awarded: Brownbeard. movie: Nohit writer:
Mimicocycle. movie: Noenometer; written by: Sclerocybin. movie: Tayenne writer:
Sclerocybin. movie: Pneumodendron : written by: Zayage. movie: Tayenne writer: Zayage.
movie: Pneumodendron written by: Mimicocycle. movie: Noenometer release year: 1991.
movie: Tayenne: year: 2013. movie: Nohit year: 2005. movie: Pneumodendron ; year: 2005. Mimicocycle was an actor in the movie Tayenne. Zayage acted in the movie Pneumodendron. Zayage was an actor in the movie Nohit. Sclerocybin was an actor in the movie Nohit. Selerocybin
actor in the movie Tayenne. Mimicocycle Noenometer. Zayage was born in 1935. Sclerocybin was born in 1935. Mimicocycle was born in 1930. Mimicocycle is from the country of Calderita. Selerocybin grew up in the nation of Calderita. Zayage is from the country of Obility. Quinion was awarded to Zayage. Fannyxist was awarded Sclerocybin. Fannyxist was awarded Mimicocycle. Mimicocycle produced the movie Nohit with others. Zayage was one of the producers of the movie Nohit. Sclerocybin was one of the producers of the movie Tayenne. Sclerocybin produced the movie Pneumodendron with others. Zayage produced the movie Pneumodendron with others.
an actor in the movie
39

---

Published as a conference paper at ICLR 2023
Mimicocycle was one of the producers of the movie Tayenne. Sclerocybin one of the producers of the movie Noenometer. Zayage produced the movie Noenometer with others What awards have movies written by people born in 1935 won?
QC:
QS:
The people born in 1935 were Sclerocybin and Zayage. Sclerocybin has written the movies Noenometer and Tayenne. Zayage has written the movies Pneumodendron and Tayenne. Noenometer has won the Brownbeard award. Tayenne has won the Goosehead award. Pneumodendron has won the Handt award. So the answer is ("Brownbeard", "Gooschead", Handt"].
A: B"Brownbeard", "Goosehead", "Handt"]
QS: [EOQ)
G.4
SHORTER PROMPTS FOR SMALLER CONTEXT WINDOWS
qa (small)
movie:
Premercy: directed by: Muntaril. movie: Skirtsicine director: Teeplemole. movie: Featsaw
directed by: Monsterscar. movie: Zalate ; director: Monsterscar. movie: Zalate awarded:
Hallowcock, movie: Featsaw awarded: Zorgion. movie: Premercy award: Chowwurst.
movie: Skirtsicine; award: Hallowcock. award: Goatfly winner: Teeplemole, person:
Monsterscar ; award: Glodome, person: Muntaril award: Goatfly. movie: Featsaw release
year: 1973. movie: Zalate release year: 1964. movie: Skirtsicine release year: 1973. movie: Premercy: year: 1961. Teeplemole was an actor in the movie Skirtsicine. Muntaril was an actor in the movie Skirtsicine. Monsterscar an actor in the movie Featsaw. Teeplemole was an actor in the movie Zalate. Muntaril was born in the year 1910. Teeplemole was born in 1910. Monsterscar was born in 1942.
actor in the movie Premercy. Muntaril was
Teeplemole is from the country of Piperfish. Monsterscar is from the country of Piperfish. Muntaril is from the country of Clony. Muntaril produced the movic Skirtsicine with others. Monsterscar was one of the producers of the movie Featsaw. Monsterscar produced the movie Premercy with others. Monsterscar produced the movie Zalate with others. Teeplemole was one of the producers of the movie Featsaw. Teeplemole produced the movie Zalate with others. Muntaril produced the movie Premercy with others. Monsterscar wrote for the movie Premercy. Muntaril one of the writers for the movie Zalate. Muntaril wrote for the movie Featsaw. Teeplemole wrote for the movie Featsaw. Monsterscar was one of the writers for the movie Zalate. Teeplemole one of the writers for the movie Skirtsicine. Q: Which awards were given to Zalate? ["Hallowcock"]
A:
Q: For which movies was Muntaril the producer?
A: ["Premercy]
Q: Who are the actors in the movie Premercy?
A: ["Monsterscar"]
movie: Nilitude; ; director: Monsterscar, movie: Dewbar: directed by: Metatoun. movie: Warpstone
directed by: Gastrat. movie: Partnershipmaker ; director: Metatoun. movie: Dewbar award:
Tachychronograph. movie: Partnershipmaker awarded: Tachychronograph. movie: Niliude
award: Paleodactyl. movie: Warpstone award: Sabonade. person: Gastrat award:
Trifogation. award: Polyquadrase winner: Monsterscar. award: Trifogation winner:
Metatoun. movie: Warpstone release year: 1956. movie: Dewbar release year: 1984. movie:
Nilitude; year: 1984, movie: Partnershipmaker: year: 1962. Gastrat was an actor in the movie Partnershipmaker. Metatoun was an actor in the movie Partnershipmaker. Metatoun was an actor in the movie Nilitude. Gastrat acted in the movie Nilitude. Monsterscar was an actor in the movie Dewbar. Gastrat acted in the movie Warpstone. Metatoun acted in the movie Warpstone. Metatoun was born in 1939. Gastrat was born in the year 1933.
Monsterscar was born in 1933. Metatoun grew up in the nation of Moulole. Gastrat is from the country of Stridery. Monsterscar grew up in the nation of Moulole. Monsterscar produced the movie Nilitude with others. Monsterscar was one of the producers of the movie Warpstone Metatoun one of the producers of the movie Warpstone. Gastrat one of the producers of the movie Nilitude. Metatoun produced the movie Partnershipmaker with others. Metatoun produced the movie Dewbar with others. Monsterscar one of the producers of
40

---

Published as a conference paper at ICLR 2023
the movie Partnershipmaker. Gastrat produced the movie Dewbar with others. Metatoun wrote for the movie Partmershipmaker. Gastrat wrote for the movie Warpstone. Gastrat was one of the writers for the movie Dewbar. Monsterscar one of the writers for the movie Nilitude. Metatoun wrote for the movie Warpstone.
Q: Which movies has Gastrat been actor in?
A: [
"Partnershipmaker, "Nilitude", "Warpstone"]
Q: Who is from the country Stridery?
A: ["Gastrat"]
Q: Which movies were given the Tachychronograph award?
A: ("Dewbar", "Partnershipmaker")
aw qa (smalll 11)
movie: Premercy: directed by: Muntaril. movie: Skirtsieine director: Teeplemole. movie: Featsaw
directed by: Monsterscar. movie: Zalate ; director: Monsterscar. movie: Zalate awarded:
Hallowcock, movie: Featsaw awarded: Zorgion. movie: Premercy: award: Chowwurst.
movie: Skirtsicine: award: Hallowcock. award: Goatfly winner: Teeplemole, person:
Monsterscar ; award: Glodome, person: Muntaril award: Goatfly. movie: Featsaw release
year: 1973. movie: Zalate release year: 1964. movie: Skirtsicine release year: 1973. movie: Premercy: year: 1961. Teeplemole was an actor in the movie Skirtsicine. Muntaril was an actor in the movie Skirtsicine. Monsterscar an actor in the movie Featsaw. Teeplemole was an actor in the movie Zalate. Muntaril was born in the year 1910. Teeplemole was born in 1910. Monsterscar was born in 1942.
actor in the movie Premercy. Muntaril was
Teeplemole is from the country of Piperfish. Monsterscar is from the country of Piperfish. Muntaril is from the country of Clony. Muntaril produced the movic Skirtsicine with others. Monsterscar was one of the producers of the movie Featsaw. Monsterscar produced the movie Premercy with others. Monsterscar produced the movie Zalate with others. Teeplemole was one of the producers of the movie Featsaw. Teeplemole produced the movie Zalate with others. Muntaril produced the movie Premercy with others. Monsterscar wrote for the movie Premercy. Muntaril one of the writers for the movie Zalate. Muntaril wrote for the movie Featsaw. Teeplemole wrote for the movie Featsaw. Monsterscar was one of the writers for the movie Zalate. Teeplemole was one of the writers for the movie Skirtsicine.
Q: Which awards were given to Zalate?
A:
["Hallowcock")
Q: Which awards were given to Premercy?
A: ["Chowwurst"]
movie:
Misgendery directed by: Wetherality. movie: Dewbar: director: Gigabut, movie:
Caudacite director: Lougerière. movie: Tayenne directed by: Lougerière. movie:
Misgendery: awarded: Microsouenesis. movie: Dewbar awarded: Erowid. movie: Tayenne:
awarded: Cockspit. movie: Caudacite award: Erowid. award: Aniconder winner:
Wetherality, award: Aniconder: winner: Lougerière. person: Gigabut award: Trifogation.
movie: Dewbar release year: 1991. movie: Tayenne : year: 2013. movie: Caudacite release
year: 2008. movie: Misgendery year: 1991. Wetherality
an actor in the movie Dewbar.
Gigabut was an actor in the movie Tayenne. Lougerière was an actor in the movie Tayenne. Lougerière acted in the movie Caudacite. Lougerière acted in the movie Misgendery. Gigabut was an actor in the movie Caudacite. Wetherality was an actor in the movie Misgendery. Wetherality was born in the year 1917. Lougerière was born in 1926. Gigabut was born in the year 1917. Gigabut grew up in the nation of Triclops. Lougerière is from the country of Tatkin Wetherality grew up in the nation of Tatkin. Lougerière produced the movie Dewbar with others. Gigabut produced the movie Tayenne with others. Gigabut produced the movie Dewbar with others. Lougerière was one of the producers of the movie Misgendery. Wetherality was one of the producers of the movie Caudacite. Gigabut was one of the producers of the movie Caudacite. Wetherality produced the movie Misgendery with others. Wetherality produced the movie Tayenne with others. Wetherality wrote for the movie Tayenne. Gigabut wrote for the movie Misgendery. Lougerière was one of the writers for the movie Caudacite. Wetherality wrote for the movie Misgendery. Gigabut wrote for the movie
41

---

Published as a conference paper at ICLR 2023
Tayenne. Gigabut wrote for the movie Dewbar. Lougerière wrote for the movie Dewbar. Wetherality wrote for the movie Caudacite.
Q: Which movies were given the Erowid award? A: ("Dewbar", "Caudacite")
Q: Which awards were given to Wetherality?
A: [" ["Aniconder"]
pos (small)
movie: Premercy: directed by: Muntaril. movie: Skirtsicine ; director: Teeplemole. movie: Featsaw
directed by: Monsterscar. movie: Zalate ; director: Monsterscar. movie: Zalate awarded:
Hallowcock. movie: Featsaw: awarded: Zorgion. movie: Premercy award: Chowwurst.
movie: Skirtsicine award: Hallowcock. award: Goatfly winner: Teeplemole, person:
Glodome, person: Muntaril award: Goatfly. movie: Featsaw release year: 1973. movie: Zalate; release year: 1964. movie: Skirtsicine release year: 1973. movie: Premercy: year: 1961. Teeplemole was an actor in the movie Skirtsicine. Muntaril was an actor the movie Skirtsicine. Monsterscar was an actor in the movie Premercy. Muntaril was an actor in the movie Featsaw. Teeplemole was actor in the movie Zalate. Muntaril was born in the year 1910. Teeplemole was born 1910. Monsterscar was born in 1942.
Monsterscar ; award:
Teeplemole is from the country of Piperfish. Monsterscar is from the country of Piperfish. Muntaril is from the country of Clony. Muntaril produced the movie Skirtsicine with others. Monsterscar was one of the producers of the movie Featsaw. Monsterscar produced the movie Premercy with others. Monsterscar produced the movie Zalate with others. Teeplemole was one of the producers of the movie Featsaw. Teeplemole produced the movie Zalate with others. Muntaril produced the movie Premercy with others. Monsterscar wrote for the movie Premercy, Muntaril was one of the writers for the movie Zalate. Muntaril wrote for the movie Featsaw. Teeplemole wrote for the movie Featsaw. Monsterscar was one of the writers for the movie Zalate. Teeplemole was one of the writers for the movie Skirtsicine.
Q: For which movies was Teeplemole the producer?
A:
["Featsaw", "Zalate"]
Q: For which movies was Muntaril the producer?
A: ["Premercy]
movie: Nilitude ; director: Monsterscar. movie: Dewbar directed by: Metatoun. movie: Warpstone
directed by: Gastrat. movie: Partnershipmaker ; director: Metatoun. movie: Dewbar ; award:
Tachychronograph. movie: Partnershipmaker awarded: Tachychronograph. movie: Nilitude
award: Paleodactyl. movie: Warpstone: award: Sabonade, person: Gastrat ; award:
Trifogation. award: Polyquadrase winner: Monsterscar. award: Trifogation winner:
Metatoun. movie: Warpstone: release year: 1956. movie: Dewbar release year: 1984, movie:
Nilitude: year: 1984. movie: Partnershipmaker: year: 1962. Gastrat was an actor the movie Partnershipmaker. Metatoun was an actor in the movie Partnershipmaker. Metatoun was an actor in the movic Nilitude. Gastrat acted in the movie Nilitude. Monsterscar was an actor in the movie Dewbar. Gastrat acted in the movie Warpstone. Metatoun acted in the movie Warpstone. Metatoun was born in 1939. Gastrat was born in the year 1933.
Monsterscar was born in 1933. Metatoun grew up in the nation of Moulole. Gastrat is from the country of Stridery. Monsterscar grew up in the nation of Moulole. Monsterscar produced the movie Nilitude with others. Monsterscar was one of the producers of the movie Warpstone Metatoun was one of the producers of the movie Warpstone. Gastrat was one of the producers of the movie Nilitude. Metatoun produced the movie Partnershipmaker with others. Metatoun produced the movie Dewbar with others. Monsterscar was one of the producers of the movie Partnershipmaker. Gastrat produced the movie Dewbar with others. Metatoun wrote for the movie Partnershipmaker, Gastrat wrote for the movie Warpstone. Gastrat was one of the writers for the movie Dewbar. Monstersear was one the writers for the movie Nilitude. Metatoun wrote for the movie Warpstone.
Q: Which movies has Gastrat been an actor in? A: Partnershipmaker", "Nilitude", "Warpstone"]
Q: Which movies has Monsterscar been actor in?
42

---

Published as a conference paper at ICLR 2023
A: ["Dewbar"]
simp-qa (small)
movie: Premercy: directed by: Muntaril. movie: Skirtsicine director: Teeplemole. movie: Featsaw
directed by: Monsterscar. movie: Zalate ; director: Monsterscar. movie: Zalate awarded:
Hallowcock. movie: Featsaw awarded: Zorgion. movie: Premercy award: Chowwurst.
movie: Skirtsicine; award: Hallowcock. award: Goatfly; winner: Teeplemole, person:
Monsterscar award: Glodome, person: Muntaril award: Goatfly. movie: Featsaw release
year: 1973. movie: Zalate release year: 1964. movie: Skirtsicine release year: 1973. movie:
Premercy year: 1961. Teeplemole an actor in the movic Skirtsicine. Muntaril was actor in the movie Skirtsicine. Monsterscar was an actor in the movie Premercy. Muntaril was actor in the movie Featsaw. Teeplemole was actor in the movie Zalate. Muntaril born in the year 1910. Teeplemole was born 1910. Monsterscar was born in 1942.
Teeplemole is from the country Piperfish. Monsterscar from the country of Piperfish. Muntaril is from the country of Clony. Muntaril produced the movie Skirtsicine with others. Monsterscar was one of the producers of the movie Featsaw. Monsterscar produced the movie Premercy with others. Monsterscar produced the movie Zalate with others. Teeplemole was one of the producers the movie Featsaw. Teeplemole produced the movie Zalate with others. Muntaril produced the movie Premercy with others. Monsterscar wrote for the movie Premercy. Muntaril was one the writers for the movie Zalate. Muntaril wrote for the movic Featsaw. Teeplemole wrote for the movie Featsaw. Monsterscar one of the writers for the movie Zalate. Teeplemole was one of the writers for the movie Skirtsicine. Q: Who were born in the year 1910? A: ("Teeplemole,, "Muntaril"]
Q: Who is from the country Piperfish? A: I ["Teeplemole", "Monsterscar"]
movie: Nilitude ; director:
Monsterscar. movie: Dewbar directed by: Metatoun. movie: Warpstone
directed by: Gastrat. movie: Partnershipmaker director: Metatoun. movie: Dewbar award:
Tachychronograph. movie: Partnershipmaker awarded: Tachychronograph. movie: Nilitude
award: Paleodactyl. movie: Warpstone award: Sabonade, person: Gastrat award:
Trifogation. award: Polyquadrase : winner: Monsterscar. award: Trifogation winner:
Metatoun. movie: Warpstone release year: 1956. movie: Dewbar release year: 1984. movie:
Nilitude; year: 1984. movie: Partnershipmaker: year: 1962. Gastrat was an actor in the movie Partnershipmaker. Metatoun was an actor in the movie Nilitude. Gastrat acted in the movie Nilitude. Monsterscar was an actor in the movie Dewbar. Gastrat acted in the movie Warpstone. Metatoun acted in the movie Warpstone. Metatoun was born in 1939. Gastrat was born in the year 1933.
actor in the movie Partnershipmaker. Metatoun
Monsterscar was born in 1933. Metatoun grew up in the nation of Moulole. Gastrat is from the country of Stridery. Monsterscar grew up in the nation of Moulole. Monsterscar produced the movie Nilitude with others. Monsterscar was one of the producers of the movie Warpstone Metatoun was one of the producers of the movie Warpstone. Gastrat was one of the producers of the movie Nilitude. Metatoun produced the movie Partnershipmaker with others. Metatoun produced the movie Dewbar with others. Monsterscar the movie Partnershipmaker. Gastrat produced the movie Dewbar with others. Metatoun wrote for the movie Partnershipmaker. Gastrat wrote for the movie Warpstone. Gastrat was one of the writers for the movie Dewbar. Monsterscar was one the writers for the movie Nilitude. Metatoun wrote for the movie Warpstone.
one of the producers of
Q: Who is from the country Stridery?
A: ["Gastrat"]
Q: Who were born in the year 1939?
A: ["Metatoun"]
COT (small)
movie: Premercy: directed by: Muntaril. movie: Skirtsicine director: Teeplemole. movie: Featsaw
directed by: Monsterscar. movie: Zalate director: Monsterscar. movie: Zalate awarded:
43

---

Published as a conference paper at ICLR 2023
Hallowcock. movie: Featsaw awarded: Zorgion. movie: Premercy award: Chowwurst.
movie: Skirtsicine award: Hallowcock. award: Goatfly; winner: Teeplemole, person:
Monsterscar award: Glodome. person: Muntaril award: Goatfly. movie: Featsaw release
year: 1973. movie: Zalate release year: 1964. movie: Skirtsicine release year: 1973. movie: actor in the movie Skirtsicine. Muntaril
Premercy year: 1961. Teeplemole
actor in the movie Skirtsicine. Monsterscar was an actor in the movie Premercy. Muntaril was an actor in the movie Featsaw. Teeplemole was an actor the movie Zalate. Muntaril was born in the year 1910. Teeplemole was born in 1910. Monsterscar was born in 1942.
Teeplemole from the country Piperfish. Monsterscar is from the country of Piperfish. Muntaril is from the country of Clony. Muntaril produced the movie Skirtsicine with others. Monsterscar was one of the producers of the movie Featsaw. Monsterscar produced the movie Premercy with others. Monsterscar produced the movie Zalate with others. Teeplemole was one of the producers the movie Featsaw. Teeplemole produced the movie Zalate with others. Muntaril produced the movie Premercy with others. Monsterscar wrote for the movie Premercy. Muntaril was one the writers for the movie Zalate. Muntaril wrote for the movie Featsaw. Teeplemole wrote for the movie Featsaw. Monsterscar
one of the writers for the movie Zalate. Teeplemole was one of the writers for the movie Skirtsicine.
QC: What awards have movies produced by people born in 1910 won?
QS: The people born in 1910 were Muntaril and Teeplemole. Teeplemole produced the movies Featsaw and Zalate. Muntaril produced the movie Premercy. Featsaw was awarded the Zorgion award. Premercy was awarded the Chowwurst award. Zalate was awarded the Halloweock award. So the answer ("Zorgion", "Chowwurst", "Hallowcock"). A: I. I"Zorgion", "Chowwurst", "Hallowcock"] [BOQ]
QS:
movie: Nilitude; ; director: Monsterscar. movie: Dewbar directed by: Metatoun. movie: Warpstone
directed by: Gastrat. movie: Pattnershipmaker ; director: Metatoun. movie: Dewbar ; award:
Tachychronograph. movie: Partnershipmaker awarded: Tachychronograph. movie: Nilitude
award: Paleodactyl. movie: Warpstone award: Sabonade. person: Gastrat ; award:
Trifogation. award: Polyquadrase ; winner: Monsterscar. award: Trifogation winner:
Metatoun, movie: Warpstone release year: 1956. movie: Dewbar relcase year: 1984. movie:
Nilitude: year: 1984, movie: Partnershipmaker year: 1962. Gastrat was an actor in the movie Partnershipmaker. Metatoun was an actor in the movie Partnershipmaker. Metatoun was an actor in the movie Nilitude. Gastrat acted in the movie Nilitude. Monsterscar was an actor the movie Dewbar. Gastrat acted the movie Warpstone. Metatoun acted the movie Warpstone. Metatoun was born in 1939. Gastrat was born in the year 1933.
Monsterscar was born 1933. Metatoun grew up in the nation of Moulole. Gastrat is from the country of Stridery. Monsterscar grew up in the nation of Moulole. Monsterscar produced the movie Nilitude with others. Monsterscar was one of the producers of the movie Warpstone of the
Metatoun one of the producers of the movie Warpstone. Gastrat producers of the movie Nilitude. Metatoun produced the movie Partnershipmaker with others. Metatoun produced the movie Dewbar with others. Monsterscar was one of the producers of the movie Partnershipmaker. Gastrat produced the movie Dewbar with others. Metatoun wrote for the movie Partnershipmaker. Gastrat wrote for the movie Warpstone. Gastrat was one of the writers for the movie Dewbar. Monsterscar was one of the writers for the movie Nilitude. Metatoun wrote for the movie Warpstone.
QC: What movies have people from the country Stridery acted in? QS: The people born in Stridery are Gastrat. Gastrat acted in Partenrshipmaker, Nilitude and Warpstone. So the answer is ["Partnershipmaker"" "Nilitude", "Warpstone"]. "'Partnershipmakrr,, "Nilitude", "Warpstone"]
A:
QS: [EOQ]
G.5
MATH QA
The decomposer here deterministically calls cot to generate the CoT and then calls gpt.ans to extract the answer.
cot
44

---

Published as a conference paper at ICLR 2023
Q: There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they done, there will be 21 trees. How many trees did the grove workers plant today? A: There are 15 trees originally. Then there were 21 trees after more were planted. So there must have been 21 15 = 6. The answer is 6.
Q: If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot
A: There are originally 3 cars. 2 more cars arrive. 3 2 = 5. The answer is 5.
Q: Leah had 32 chocolates and her sister had 42. If they ate 35, how many pieces do they have left in total?
A: Originally, Leah had 32 chocolates. Her sister had 42. So in total they had 32 42 = 74. After cating 35, they had 74 - 35 39. The answer is 39.
Q: Jason had 20 lollipops. He gave Denny some lollipops. Now Jason has 12 lollipops. How many lollipops did Jason give Denny?
A: Jason started with 20 lollipops. Then he had 12 after giving some to Denny. So he gave Denny 20 12 8. The answer is 8.
Q: Shawn has five toys. For Christmas, he got two toys each from his mom and dad. How many toys does he have now?
A: Shawn started with toys. he got toys each from his mom and dad, then that is more toys.
5 + 4 9. The answer is 9.
Q: There were nine computers in the server room. Five more computers were installed each day, from monday to thursday. How many computers are now the server room?
A: There were originally 9 computers. For each of 4 days, 5 more computers were added. So 5
= 20 computers were added. 9 20 is 29. The answer is 29.
Q: Michael had 58 golf balls. On tuesday, he lost 23 golf balls. On wednesday, he lost 2 more.
How many golf balls did he have the end of wednesday?
A: Michael started with 58 golf balls. After losing 23 on tuesday, he had 58 23 = 35. After losing
more, he had 35 2 33 golf balls. The answer is 33.
Q: Olivia has $23. She bought five bagels for $3 each. How much money does she have left? A: Olivia had 23 dollars. 5 bagels for 3 dollars each will be 5 x 3 15 dollars. So she has 23 15 dollars left. 23 15 is 8. The answer is 8.
gpt_ans
Q: There are 15 trees originally. Then there were 21 trees after some more were planted. So there must have been 21 15 = 6 trees planted.
A:
Q: There are originally cars. 2 more cars arrive. + 5. The answer is 5.
A:
Q: Originally, Leah had 32 chocolates. Her sister had 42. So in total they had 32 +2 4 After eating 35, they had 74 - 35 = 39. The answer is 39.
A: 39
Q: Jason started with 20 lollipops. Then he had 12 after giving some Denny. he gave Denny 20 - 12 8 apples.
A:
Q: Shawn started with toys. he got 2 toys each from his mom and dad, then that is more toys. 5 + 4 9. The answer is 9.
A:
45

---

Published as a conference paper at ICLR 2023
Q: There were originally 9 computers. For each of 4 days, 5 more computers were added. So 5 = 20 computers were added. 9 20 is 29. The answer is 29.
A: 29
Q: Michael started with 58 golf balls. After losing 23 tusdayy, he had 58 23 = 35. After losing
2 more, he had 35 2 33 golf balls.
A: 33
Q: Olivia had 23 dollars. 5 bagels for 3 dollars each will be 3 15 dollars. So she has 23 15
8 dollars left.
A:
G.6 OPEN DOMAIN QA
The prompts in this section implement Decomposed Prompting approach to open-domain multihop QA. For brevity we' ve've included prompts for 5 of 20 randomly sampled questions. The full prompts are attached with the submission and will also be released with the code. Note that we selected set of 100 questions from the development set to tune the hyperparameter (number of paragraphs to retrieve for all of the retrieval-based approaches).
hotpot decomp
QC: In which country did this Australian who was detained in Guantanamo Bay detention camp and published "Guantanamo: My Journey" receive para-military training? (select) [retrieve odqa] Who is the Australian who was detained in Guantanamo Bay detention camp and published "Guantanamo: My Journey"?
QS:
A: ("titles": ["Guantanamo: My Journey', "Bismullah v. Gates", "Guantanamo Bay detention camp"".""nsweet"" "answer": ("David Hicks"]}
QS: (select) [retrieve_odqa] In which country did David Hicks receive his para-military training?
A:
("titles": ["John Adams Project". "Camp Echo (Guantanamo Bay)", "Guantanamo Bay
Museum of Art and History", ", "David Hicks"], "answer": [" ["Afghanistan"]]
QS: (select) [multihop titleqa] Titles: ["Guantanamo: My Journey". "Bismullah v. Gates", Guantanamo Bay detention camp", "John Adams Project", "Camp Echo (Guantanamo Bay)", "Guantanamo Bay Museum of Art and History", "David Hicks"). Question: In which country did this Australian who was detained in Guantanamo Bay detention camp and published Guantanamo: My Journey" receive para-military training?
A: ["Afghanistan"]
QS: [EOQ]
QC: How many awards did the "A Girl Like Me" singer win at the American Music Awards of 2012?
QS: (select) [retrieve odqa] Who is the singer of A Girl Like Me"?
A:
("titles": ["American Music Award", "Rihanna"), "answer": ["Rihanna"]]
OS:
(select) [retrieve_odqa] How many awards did Rihana won in the American Music Awards of 2012?
A: ("titles": ['American Music Awards of 2012", "APRA Music Awards of 2012", "List of awards
and nominations received by TLC", "Native American Music Awards"). "answer": ["one"]] QS: (select) [multihop_titlege Titles: ["American Music Award". "Rihanna", "American Music Awards of 2012". "APRA Music Awards of 2012", "List of awards and nominations received by TLC", "Native American Music Awards"]. Question: How many awards did the "A Girl Like Me" singer win at the American Music Awards of 2012?
A: ["one"]
QS: [BOQ]
QC: who is older Jeremy Horn or Renato Sobral ?
QS: (select) [retrieve odqa] When Jeremy Horn born?
46

---

Published as a conference paper at ICLR 2023
A: ("titles"; ["Zaza Tkeshelashvili", "Jeremy Horn", "Jeremy Horn (singer)", "Ricardo Arona"],
answer": ["August 25, 1975"]}
QS:
(select) [retrieve odqa] When Renato Sobral born?
A:
("titles": ["Brian Warren", "Renato Sobral"), "answer": ("September 7, 1975"]]
QS:
(select) paaultihhop_tttleqpee Titles: I"Zaaa Tkeshelashvili", "Jeremy Horn", "Jeremy Horn
singer)", "Ricardo Arona", "Brian Warren", "Renato Sobral"). Question: who is older Jeremy
Horn Renato Sobral
A: [" ["Jeremy Horn"]
QS: [EOQ]
QC: What was the 2014 population of the city where Lake Wales Medical Center is located? QS: (select) [retrieve_odqa] Lake Wales Medical Center is located what city? ("titles": ["Baylor College of Medicine", "Lake Wales Medical Center", "Tufts University School of Medicine", "Hanford Community Medical Center"), "answer": ("Polk County, Florida"])
A:
QS: (select) [retrieve_odge _oqqa] What was the population of Polk County in 2014?
A:
("titles": ["Banner University Medical Center Tucson", "Lake Wales, Florida"), "answer": ["15,140"]]
QS:
(select) [multihop_tiileee _tilleq]]] Titles: ["Baylor College Medicine", "Lake Wales Medical Center ""Tuuts University School of Medicine", "Hanford Community Medical Center", "Banner University Medical Center Tucson", "Lake Wales, Florida"). Question: What was the 2014 population of the city where Lake Wales Medical Center is located?
A: ["15,140"]
QS: [BOQ]
QC: Nobody Loves You was written by John Lennon and released on what album that was issued by Apple Records, and was written, recorded, and released during his 18 month separation from Yoko Ono?
QS:
(select) [rddqi What album was issued by Apple Records, and written, recorded, and released during John Lennon's 's 18 month separation from Yoko Ono? A: ("titles": ["John Lennon/Plastic Ono Band", "Milk and Honey (album)", "Walls and Bridges"], "answer": ("Walls and Bridges"]]
QS: (select) [retrieve _odaa] Nobody Loves You was written by John Lennon on what album?
A: ("titles": ["John Lennon Museum", "Nobody Loves You (When You' re Down and Out)"), answer": ["Walls and Bridges"]]
QS: (select) moultihooptitlegeee Titles: ["John Lennon/Plastic Ono Band", "Milk and Honey (album) "Walls and Bridges".. "John Lennon Museum", "Nobody Loves You (When You' rre Down and Out)"). Question: Nobody Loves You was written by John Lennon and released on what album that was issued by Apple Records, and was written, recorded, and released during his 18 month separation from Yoko Ono?
A: [ ["Walls and Bridges")
QS: [EOQ]
hotpotqa: retrieve _odqa
QC: Who is the Australian who was detained in Guantanamo Bay detention camp and published " Guantanamo: My Journey""?
QS: (select) [retrieve] Who is the Australian who was detained in Guantanamo Bay detention camp and published "Guantanamo: My Journey"?
A: ["Guantanamo: My Journey". "Bismullah v. Gates", "Guantanamo Bay detention camp"]
QS: (select) [singlehop titleqa] Titles: ["Guantanamo: My Journey"" "Bismullah v. Gates", Guantanamo Bay detention camp"]. Question: Who is the Australian who was detained in
Guantanamo Bay detention camp and published "Guantanamo: My Journey"?
A: ("titles": ["Guantanamo: My Journey", "Bismullah Gates", "Guantanamo Bay detention camp"], "answer": ("David Hicks"]}
QS: [EOQ]
47

---

Published as a conference paper at ICLR 2023
QC: In which country did David Hicks receive his para-military training?
QS: (select) [retrieve] which country did David Hicks receive his para-military training?
A: ["John Adams Project", "Camp Echo (Guantanamo Bay)", "Guantanamo Bay Museum of Art and History". "David Hicks"]
QS: (select) (singlehop_titkee _titleqa] Titles: ["John Adams Project", "Camp Echo (Guantanamo Bay)", Guantanamo Bay Museum of Art and History", "David Hicks"]. Question: In which country did David Hicks receive his para-military training?
A:
("titles": ['John Adams Project". "Camp Echo (Guantanamo Bay)", "Guantanamo Bay Museum of Art and History", "David Hicks"], "answer": ["Afghanistan")]
QS: [EOQ]
QC: Who is the singer "A Girl Like Me"?
QS: (select) [retrieve] Who is the singer of "A Girl Like Mc"?
A: ["American Music Award", "Rihanna"]
QS:
(select) haggleoop_itleegee. Titles: ["American Music Award", "Rihanna"]. Question: Who is the singer "A Girl Like Me"?
A: ("titles": [" ["American Music Award", "Rihanna"), "answer": ["Rihanna"]]
QS: [EOQ]
QC: How many awards did Rihana won in the American Music Awards of 2012?
QS: (select) [retrieve] How many awards did Rihana won in the American Music Awards of 2012? A: [" A'Amrriaan Music Awards of 2012", "APRA Music Awards of 2012", "List awards and nominations received by TLC", "Native American Music Awards"]
QS: (select) [singlehop.titlege Titles: [" A'merican Music Awards of 2012", "APRA Music Awards of 2012", "List of awards and nominations received by TLC", "Native American Music Awards"]. Question: How many awards did Rihana won in the American Music Awards of 2012?
A: ("titles": ["American Music Awards of 2012", " "APRA Music Awards of 2012", "List of awards and nominations received by TLC", "Native American Music Awards"J, "answer": ["one"]]
OS: [EOQ]
QC: When was Jeremy Horn born?
QS: (select) [retrieve] When was Jeremy Horn born?
A: I'Zaza Tkeshelashvill", "Jeremy Horn", "Jeremy Horn (singer)", "Ricardo Arona"]
QS: (select) [singlchop.titkee _itleeaattttttt Titles: l"Zaza Tkeshelashvili", "Jeremy Hom"", "Jeremy Horn
singer)", "Ricardo Arona"]. Question: When was Jeremy Horn born? ("titles": ["Zaza Tkeshelashvili", "Jeremy Horn", "Jeremy Horn (singer)", "Ricardo Arona"]," answer": ["August 25, 1975"]}
A:
QS:
[BOQ]
QC: When was Renato Sobral born?
QS:
(select) [retrieve] When was Renato Sobral born?
A: ["Brian Warren", "Renato Sobral"]
QS:
(select) singglehopttillqqul Titles: ["Brian Warren", "Renato Sobral"]. Question: When was
Renato Sobral bomn?
(" ("titles": ["Brian Warren", "Renato Sobral"], "answer"; ["September' 7, 1975"]] QS: [EOQ]
A:
QC: Lake Wales Medical Center is located in what city?
OS:
(select) [retrieve] Lake Wales Medical Center is located in what city? A: ["Baylor College of Medicine", "Lake Wales Medical Center", "Tufts University School of Medicine". "Hanford Community Medical Center"]
48

---

Published as a conference paper at ICLR 2023
QS:
(select) Saangglhho..ttllqpee Titles: ["Baylor College of Medicine", "Lake Wales Medical Center", "Tufts University School of Medicine", "Hanford Community Medical Center"]. Question: Lake Wales Medical Center is located in what city?
A: ("titles": ["Baylor College of Medicine", "Lake Wales Medical Center", "Tufts University School of Medicine", "Hanford Community Medical Center"), "answer": ["Polk County, F]orida"]} [EOQ]
QS:
QC: What was the population of Polk County in 2014?
QS: (select) [retrieve] What was the population Polk County 2014?
A:
Banner University Medical Center Tucson", "Lake Wales, Florida"]
QS:
(select) [singlchop_titlca] Titles: ["Banner University Medical Center Tucson", "Lake Wales,
Florida"]. Question: What was the population of Polk County in 2014?
A: ["titles": ["Banner University Medical Center Tucson", "Lake Wales, Florida"), "answer": ["15,140"]]
OS:
[BOQ]
QC:
What album was issued by Apple Records, and written, recorded, and released during John Lennon's ' 18 month separation from Yoko Ono?
QS:
(select) [retrieve] What album was issued by Apple Records, and written, recorded, and released during John Lennon's 's 18 month separation from Yoko Ono?
A: ["John Lennon/Plastic Ono Band", "Milk and Honey (album)", "Walls and Bridges"]
QS:
(select) [singlehop_titles
Titles: ["John Lennon/Plastic Ono Band", "Milk and Honey (album )" "Walls and Bridges"]. Question: What album was issued by Apple Records, and written, recorded, and released during John Lennon's ss 18 month separation from Yoko Ono?
A:
("titles": ["John Lennon/Plastic Ono Band", "Milk and Honey (album)". "Walls and Bridges"]. "answer": ["Walls and Bridges"]]
OS:
[FOQ]
QC: Nobody Loves You was written by John Lennon on what album?
QS:
(select) [retrieve] Nobody Loves You was written by John Lennon on what album?
A:
["John Lennon Museum", "Nobody Loves You (When You're Down and Out)"]
QS: (select) [singlehop __ttleqaa]]]a Titles: ["John Lennon Museum". "Nobody Loves You (When You' re Down and Out)"). Question: Nobody Loves You was written by John Lennon on what album?
A: ("titles": ["John Lennon Museum", "Nobody Loves You (When You're re Down and Out)"J,
answer": ["Walls and Bridges"]] [EOQ]
QS:
hotpotqa:
singlehop.titlega
Wikipedia Title: David Hicks <hidden for brevity>>
Wikipedia Title: Guantanamo: My Journey <hidden for brevity>
Wikipedia Title: Murat Kurnaz hidden for brevity.>
Q: Who is the Australian who was detained in Guantanamo Bay detention camp and published Guantanamo: My Journey"?
A: ["David Hicks"]
Q: In which country did David Hicks receive his para-military training?
A: [" ["AAfhhaistta"]]
49

---

Published as a conference paper at ICLR 2023
Wikipedia Title: American Music Award <hidden for brevity>>
Wikipedia Title: Rihanna <hidden for brevity.>
Wikipedia Title: American Music Awards of 2012 <iideen for brevity>
Q: Who the singer AA Girl Like Me"?
A: ["Rihanna"]
Q: How many awards did Rihana won in the American Music Awards of 2012?
A: ["one"]
Wikipedia Title: Renato Sobral <hidden for brevity
Wikipedia Title: Zaza Tkeshelashvili <hidden for brevity>
Wikipedia Title: Jeremy Horn <hidden for brevity>>
Q: When Jeremy Horn born?
A: ["August 25, 1975"]
Q: When was Renato Sobral born?
A: ["September 1975"]
Wikipedia Title: Lake Wales Medical Center <hidden for brevity>
Wikipedia Title: Lake Wales, Florida <hidden for brevity>
Wikipedia Title: Tufts University School of Medicine <hidden for brevity>
Q: Lake Wales Medical Center is located in what city?
A: ["Polk County, Florida"]
Q: What was the population of Polk County in 2014?
A: ["15,140"]
Wikipedia Title: Nobody Loves You (When You're re Down and Out) <hidden for brevity>>
Wikipedia Title: Walls and Bridges <hidden for brevity>
Wikipedia Title: Mother (John Lennon song) <hidden for brevity
50

---

Published as a conference paper at ICLR 2023
Q: What album issued by Apple Records, and written, recorded, and released during John Lennon's s 18 month separation from Yoko Ono?
A: ["Walls and Bridges"]
Q: Nobody Loves You written by John Lennon what album?
A: ("Walls and Bridges"]
hotpotqa: multihop.titleqa (direct)
Wikipedia Title: David Hicks <hidden for brevity>
Wikipedia Title: Guantanamo: My Journey <hidden for brevity>
Wikipedia Title:
Murat Kurnaz
chidden for brevity>>
Q: In which country did this Australian who was detained in Guantanamo Bay detention camp and published "Guantanamo: My Journey" receive para-military training?
A: ["Afghanistan"]
Wikipedia Title: American Music Award <hidden for brevity>
Wikipedia Title: Rihanna <hidden for brevity>
Wikipedia Title: American Music Awards of 2012 <hidden brevity>
Q: How many awards did the "A Girl Like Me" singer win at the American Music Awards of 2012?
A: ["one"]
Wikipedia Title: <hidden for brevity>
Renato Sobral
Wikipedia Title: Zaza Tkeshelashvili <hidden for brevity>>
Wikipedia Title: Jeremy Horn chidden for brevity>>
Q: who is older Jeremy Horn or Renato Sobral ? A: ["Jeremy Horn"]
Wikipedia Title: Lake Wales Medical Center <hidden for brevity.>
Wikipedia Title: Lake Wales, Florida <hidden for brevity>>
Wikipedia Title: Tufts University School of Medicine <hidden for brevity>>
Q: What was the 2014 population of the city where Lake Wales Medical Center is located?
51

---

Published as a conference paper at ICLR 2023
A: ["15,140"]
Wikipedia Title: Nobody Loves You (When You're Down and Out) <hidden for brevity>
Wikipedia Title: Walls and Bridges <hidden for brevity>
Wikipedia Title: Mother (John Lennon song) <hidden for brevity>
Q: Nobody Loves You was written by John Lennon and released on what album that was issued by Apple Records, and was written, recorded, and released during his 18 month separation from Yoko Ono?
A: ["Walls and Bridges"]
hotpotqa: miltihop.titlega (cot)
Wikipedia Title: David Hicks <hidden for brevity>
Wikipedia Title: Guantanamo: My Journey <hidden for brevity>
Wikipedia Title:
Murat Kurnaz hidden for brevity
Q: In which country did this Australian who was detained in Guantanamo Bay detention camp and published "Guantanamo: My Journey" receive para-military training?
A: The Australian who was detained Guantanamo Bay detention camp and published "
Guantanamo: My Journey" is David Hicks. David Hicks received his para-military training in Afghanistan. So the answer is: Afghanistan.
Wikipedia Title: American Music Award <hidden for brevity>>
Wikipedia Title: Rihanna <hidden for brevity>>
Wikipedia Title: American Music Awards 2012
chidden for brevity>
Q: How many awards did the "A Girl Like Me" singer win at the American Music Awards of 2012?
A: The singer of AA Girl Like Me" singer is Rihanna. In the American Music Awards of 2012, Rihana won one award. So the answer is: one.
Wikipedia Title: Renato Sobral <hidden for brevity.>
Wikipedia Title: Zaza Tkeshelashvili <hidden for brevity
Wikipedia Title: Jeremy Horn < hidden for brevity>>
Q: who is older Jeremy Horn Renato Sobral ?
52

---

Published as a conference paper at ICLR 2023
A: Jeremy Horn born August 25, 1975. Renato Sobral was born September 7, 1975. Thus, Jeremy Horn older. So the answer is: Jeremy Horn.
Wikipedia Title: Lake Wales Medical Center <hidden for brevity.>
Wikipedia Title: Lake Wales, Florida <hidden for brevity>
Wikipedia Title: Tufts University School of Medicine <hidden for brevity>
Q: What was the 2014 population of the city where Lake Wales Medical Center is located? A: Lake Wales Medical Center is located in the city of Polk County, Florida. The population of Polk County in 2014 was 15,1 10. So the answer is: 15,140.
Wikipedia Title: Nobody Loves You (When You't re Down and Out) <hidden for brevity"
Wikipedia Title: Walls and Bridges <hidden for brevity>
Wikipedia Title: Mother (John Lennon song) <hideen for brevity>
Q: Nobody Loves You was written by John Lennon and released on what album that was issued by Apple Records, and written, recorded, and released during his 18 month separation from Yoko Ono?
A: The album issued by Apple Records, and written, recorded, and released during John Lennon's s 18 month separation from Yoko Ono is Walls and Bridges. Nobody Loves You was written by John Lennon on Walls and Bridges album. So the answer is: Walls and Bridges.
2wi kilkkimlltttom hit.mmiiihhhpppp..o.m decomp
QC: Do director of film Coolie No. (1995 Film) and director of film The Sensational Trial have the same nationality?
QS:
(select) [retrieve_odee odqaa] Who is the director of the film Coolie No. 1 (1995 film)?
("titles": ["Hanro Smitsman", "Coolie No. (1995 film)"], "answer": ["David Dhawan"]] QS: (select) [retrieve_ode odqa] Who is the director of the film The Sensational Trial?
A:
A: ("titles": ["Rachel Feldman". "TThe Sensational Trial"), "answer": ("Karl Freund"]]
QS: (select) [retrieve_odqa] What is David Dhawan's 's nationality?
A: {"titles": ["David Dhawan", "Brian Johnson (special effects artist)"], "answer": ["India"]]
QS: (select) [retrieve_ode [dqaa] What is Karl Freund's nationality?
A: ("titles"" ['Karl Freund",, "lan Barry (director)"). "answer": ["Germany"]]
OS: (select) [multihop _tilleqaitaaaattaae Titles: ["Hanro Smitsman", "Coolie No. (1995 film)", "Rachel Feldman", "The Sensational Trial", "David Dhawan", "Brian Johnson (special effects artist)", "Karl Freund", "lan Barry (director)"). Question: Do director of film Coolie No. (1995 Film and director of film The Sensational Trial have the same nationality?
A: ["no"]]
QS: [EOQ]
QC: Which film has the director died later, The Gal Who Took the West or Twenty Plus Two? QS: (select) [retrieve_odqa] Who is the director of the film Twenty Plus?
A: ("titles": ["Riki Gal", "Twenty Plus Two"], "answer": ["Joseph M. Newman"]]
QS: (select) [retrieve_odqa] Who is the director of the film The Gal Who Took the West?
A: ("titles": ["Querelle", "The Gal Who Took the West"], "answer": ["Frederick Cordova"]]
[" OS: (select) [retrieve odqa] When did Joseph M. Newman die?
53

---

Published as a conference paper at ICLR 2023
A: ("titles"; ["Joseph M. Newman", "Thulasi (1987 film)"], "answer": ("January 23, 2006"]]
QS: (select) Iretrieve_odqa) When did Fred de Cordova die?
("titles": ["Fred de Cordova", "Le Masque de la Meduse"), "answer": ["September 15, 2001"]} OS: (select) [multihop titleqa] Titles: ("Riki Gal", "Twenty Plus Two", "Querelle", "The Gal Who
A:
Took the West", "Joseph M. Newman", "Thulasi (1987 film)", "Fred de Cordova", "Le Masque de la Meduse"). Question: Which film has the director died later, The Gal Who Took
the West or Twenty Plus Two?
A: [ ("Teenty Plus Two"]
QS: [EOQ]
QC: Who is the grandchild of Krishna Shah (Nepalese Royaly?
QS: (select) [retrieve_odqa] Who is the child of Krishna Shah?
A:
("titles": ["Ana Gruzinsky-Golitsyn "Krishna Shah (Nepalese royal)", "Diana Weston","
Albina du Boisrouvray"), "answer": ("Rudra Shah"]}
OS: (select) [retrieve_odee _odqa] Who is the child of Rudra Shah?
A:
("titles": ["Jim Ramel Kjellgren", "Constance Anne Herschel", "Rudra Shah"], "answer"; [* Prithvipati Shah"]}
QS: (select) Imultihop.titlees _itleqa] Titles: ["Ana Gruzinsky-olltsyyn, "Krishna Shah (Nepalese royal)
"Diana Weston", "Albina du Boisrouvray"", "Jim Ramel Kjellgren", "Constance Anne Herschel", "Rudra Shah"]. Question: Who is the grandchild of Krishna Shah (Nepalese Royal }?
A: P"Piithvipati Shah"]
QS: [BOQ]
QC: Where did the director of film Maddalena (1954 Film) die?
QS: (select) [retrieve odqa] Who is the director the film Maddalena?
A:
("titles": ["Ian Barry (director)". "Maddalena (1954 film)", "Brian Kennedy (gallery director) "I, "answer": ("Augusto Genina"]}
QS: (select) [retrieve_ode _odqa] Where did Augusto Genina die?
A:
("titles": ["Dana Blankstein", "Augusto Genina", "Peter Levin", "Olav Aaraas"], "answer": [" Rome"]]
OS:
(select) [multihop -ttleeaatttaaat Titles: ['lan Barry (director)", "Maddalena (1954 film)", "Brian Kennedy (gallery director)", "Dana Blankstein". "Augusto Genina", "Peter Levin", "Olav Aaraas"]. Question: Where did the director of film Maddalena (1954 Film) die?
A:
["ome"]
QS:
[EOQ]
QC: Who was born first out of Martin Hodge and Ivania Martinich? (select) [retrieve_ode _odva] When Martin Hodge born? A: ("titles": ["Greg A. Hill (artist)", "Martin Hodge", "John Allen (Oxford University cricketer)") "] "answer": ["4 February 1959"]}
QS:
QS: (select) cettrevee_dgga When was Ivania Martinich born?
A:
("titles": ['Tom Dickinson". "Ivania Martinich", "Wesley Barresi", "Henry Moore (cricketer)
"I. "answer""" |'25 July 1995"]}
QS: (select) muultihop_titleqaa Titles: ["Greg Hill (artist)",, "Martin Hodge", "John Allen ( Oxford University cricketer)", "Tom Dickinson", "Ivania Martinich", "Wesley Barresi", Henry Moore (cricketer)"), Question: Who was born first out Martin Hodge and Ivania Martinich?
A: [" ["Martin Hodge"]
OS: [EOQ]
2w 2wikimultihopqaa
retrieve odqa
QC: Who is the director of the film Coolie No. I (1995 film)?
QS: (select) [retrieve] Who is the director of the film Coolie No. (1995 film)?
A: ["Hanro Smitsman". "Coolie No. (1995 film)"]
54

---

Published as a conference paper at ICLR 2023
QS:
(select) Isinglehop.titlen Titles: ["Hanro Smitsman", "Coolie No. (1995 film)"]. Question:
Who is the director of the film Coolie No. (1995 film)?
A:
("titles": ("Hanro Smitsman", "Coolie No. (1995 film)"], "answer": ("David Dhawan"]]
QS:
[FOQ]
QC: Who is the director of the film The Sensational Trial?
QS:
(select) [retrieve] Who is the director of the film The Sensational Trial?
A:
["Rachel Feldman". "The Sensational Trial"]
QS:
(select) [singlehop tttleqaattaaaa Titles: ("Rachel Feldman", "The Sensational Trial"). Question:
Who is the director of the film The Sensational Trial?
A:
("titles": ("Rachel Feldman", "The Sensational Trial"), "answer": ["Karl Freund"]]
QS: [BOQ]
QC: What is David Dhawan's ss nationality?
QS: (select) [retrieve] What is David Dhawan's nationality?
A:
["David Dhawan", "Brian Johnson (special effects artist)"]
OS:
(select) Jsinglehop.titlege
Titles: ("David Dhawan", "Brian Johnson (special effects artist)"].
Question: What is David Dhawan's nationality?
A: ("titles:: ("David Dhawan", "Brian Johnson (special effects artist)"], "answer"; ["India"]]
QS: [EOQ]
QC: What is Karl Freund's nationality?
QS: (select) [retrieve] What is Karl Freund's s nationality?
A:
["Karl Freund", "lan Barry (director)")
QS:
(select) [singlchop titleqa] Titles: ["Karl Freund", "lan Barry (director)"). Question: What is
Karl Freund's nationality?
A:
("titles": ("Karl Freund"', "lan Barry (director)"), "answer": ("Germany"]]
QS: [EOQ]
QC: Who is the director of the film Twenty Plus?
QS:
(select) [retrieve] Who the director of the film Twenty Plus?
A: ["Riki Gal", "Tweety Plus Two"]]
QS: (select) (singlehop_titles
Titles: ("Riki Gal", "Twenty Plus Two"]. Question: Who is the director of the film Twenty Plus?
("titles": ['Riki Gal", "Twenty Plus Two"], "answer": ["Joseph M. Newman"]] OS: [EOQ]
A:
QC: Who is the director of the film The Gal Who Took the West?
QS:
(select) [retrieve] Who is the director of the film The Gal Who Took the West? ["Querelle", "The Gal Who Took the West"]
A:
QS: (select) [singlehop titleqa] Titles: ("Querelle", "The Gal Who Took the West"]. Question:
Who is the director of the film The Gal Who Took the West? ("titles"" ["Querelle", "The Gal Who Took the West"), "answer": ["Frederick de Cordova"])
QS: [EOQ]
QC: When did Joseph M. Newman die?
OS: (select) [retrieve] When did Joseph M. Newman die?
A:
["Joseph M. Newman". '"Thlasi (1987 film)")
OS:
(select) sagglehop_titlegee Titles: ["Joseph M. Newman", "Thulasi (1987 film)"). Question:
When did Joseph M. Newman die?
A: ("titles": ["Joseph M. Newman". "Thulasi (1987 film)"], "answer": ["January 23, 2006"]}
OS: [EOQ]
55

---

Published as a conference paper at ICLR 2023
QC: When did Fred de Cordova die?
QS:
(select) [retrieve] When did Fred de Cordova die?
A:
["Fred de Cordova", "I.e Masque de la Meduse"]
QS:
(select) Isinglehop_tiee [ Titles: ["Fred de Cordova", "Le Masque de la Meduse"). Question:
When did Fred de Cordova die?
A:
("titles": ["Fred de Cordova", "Le Masque de Meduse"), "answer": ("September 15, ]]
QS: [EOQ]
QC: Who the child of Krishna Shah?
QS: (select) [retrieve] Who is the child of Krishna Shah?
A: ["Ana Gruzinsky-Golitsnn" "Krishna Shah (Nepalese royal)", "Diana Weston", "Albina
Boisrouvray"]
OS:
(select) [singlehop.titee titleqa] Titles; ["Ana Gruzinsky-Golitsyn" "Krishna Shah (Nepalese royal )" "Diana Weston". "Albina du Boisrouvray"]. Question: Who is the child of Krishna Shah? ("titles": [" |nAna
A:
"Krishna Shah (Nepalese royal)", "Diana Weston"," Albina du Boisrouvray"). "answer": ("Rudra Shah"]}
OS:
[EOQ]
QC: Who is the child of Rudra Shah?
QS:
(select) [retrieve] Who is the child of Rudra Shah?
A: ["Jim Ramel Kjellgren". "Constance Anne Herschel", "Rudra Shah"]
QS:
(select) haaglehho__itleeqee Titles: ["Jim Ramel Kjellgren", "Constance Anne Herschel", Rudra Shah"]. Question: Who the child of Rudra Shah?
A:
("titles": ["Jim Ramel Kjellgren", "Constance Anne Herschel", "Rudra Shah"], "answer": [" Prithvipati Shah"]}
OS: [FOQ]
QC: Who is the director of the film Maddalena?
QS: (select) [retrieve] Who is the director of the film Maddalena?
A: ["lan Barry (director)", "Maddalena (1954 film)", "Brian Kennedy (gallery director)"]
QS: (select) [singlehop_titkee
Titles: ["lan Barry (director)", "Maddalena (1954 film)", "Brian Kennedy (gallery director)"). Question: Who is the director of the film Maddalena?
A: ("titles"): ["lan Barry (director)". "Maddalena (1954 film)", "Brian Kennedy (gallery director)
"I. "answer": ["Augusto Genina"]}
QS: [BOQ]
QC: Where did Augusto Genina die?
QS: (select) [retrieve] Where did Augusto Genina die?
A:
["Dana Blankstein". "Augusto Genina". "Peter Levin", "Olav Aaraas"]
OS:
(select) Säinglhho_titleque Titles: ("Dana Blankstein", "Augusto Genina", "Peter Levin"," Olav Aaraas"]. Question: Where did Augusto Genina die?
A: ("titles": ["Dana Blankstein", "Augusto Genina", "Peter Levin", "Olav Aaraas"], "answer": [* Rome"]] [EOQ]
QS:
QC: When was Martin Hodge born?
OS: (select) [retrieve] When Martin Hodge born?
A: ["Greg A. Hill (artist)", "Martin Hodge", "John Allen (Oxford University cricketer)")
OS: (select) saingleho__ttlegee Titles: ["Greg A. Hill (artist)", "Martin Hodge". "John Allen
Oxford University cricketer)"). Question: When was Martin Hodge born?
A: ("titles": ["Greg A. Hill (artist)", "Martin Hodge", "John Allen (Oxford University cricketer)"), "answer": ["4 February 1959"]}
QS: [EOQ]
56

---

Published as a conference paper at ICLR 2023
QC: When was Ivania Martinich born?
QS: (select) [retrieve] When was Ivania Martinich born?
A: ["Tom Dickinson", "Ivania Martinich", "Wesley Barresi", "Henry Moore (cricketer)"
QS: (select) [singlchop_title [itppt] Titles: ["Tom Dickinson", "Ivania Martinich", "Wesley Barresi", Henry Moore (cricketer)"). Question: When was Ivania Martinich born?
A: ("titles": ["Tom Dickinson". "Ivania Martinich", "Wesley Barresi", "Henry Moore (cricketer)
"I. "ansseer: ["25 July 1995"]}
QS: [EOQ]
wwikimultihoper
singlehop.titlega
Wikipedia Title: David Dhawan <hidden for brevity>
Wikipedia Title: Howard W. Koch <hidden for brevity>
Wikipedia Title: The Sensational Trial <hidden for brevity>
Wikipedia Title: Coolie No. I (1995 film) <hidden for brevity>
Wikipedia Title: Karl Freund <hidden for brevity
Q: Who the director of the film Coolie No. 1 (1995 film)?
A: ["David Dhawan"]
Q: Who is the director of the film The Sensational Trial?
A: ["Karl Freund"]
Q: What is David Dhawan's ss nationality?
A: ["India")
Q: What is Karl Freund's nationality?
A: ["Germany"]
Wikipedia Title: Fred Cordova <hidden for brevity>
Wikipedia Titl: Thulasi (1987 film) <hidden for brevity>>
Wikipedia Title: Joseph M. Newman <hidden for brevity>>
Wikipedia Title: The Gal Who Took the West <hidden for brevity.>
Wikipedia Title: Twenty Plus Two <hidden for brevity>>
Q: Who the director the film Twenty Plus?
A: ["Joseph M. Newman"]
Q: Who is the director of the film The Gal Who Took the West?
57

---

Published as a conference paper at ICLR 2023
A: ["Frederick de Cordova"]
Q: When did Joseph M. Newman die?
A: ["January 23, 2006")
Q: When did Fred de Cordova die?
A: ["September 15, 2001"]
Wikipedia Title: Ana Gruzinsky-oolisyn <hidden for brevity>
Wikipedia Title: Rudra Shah <hidden for brevity.>
Wikipedia Title: Krishna Shah (Nepalese royal) <hidden for brevity>
Q: Who is the child of Krishna Shah?
A: ["Rudra Shah"]
Q: Who the child of Rudra Shah?
A: ("Prithvipati Shah"]
Wikipedia Title: Augusto Genina <hidden for brevity
Wikipedia Title: Ian Barry (director) <hidden for brevity>
Wikipedia Title Maddalena (1954 film) <hidden for brevity>
Q: Who director the film Maddalena?
A: I" Augusto Genina"]
Q: Where did Augusto Genina die?
A: ["Rome"]
Wikipedia Titl: Martin Hodge <hidden for brevity>>
Wikipedia Title: Ivania Martinich <hidden for brevity'>
Wikipedia Title: Tom Dickinson chidden for brevity
Q: When Martin Hodge born?
A: ["4 February 1959"]
Q: When was Ivania Martinich born?
A: 2"25 July 1995")
2wik Pwikimultihopen multihop titleqa (direct)
Wikipedia Title: David Dhawan <hidden for brevity>
58

---

Published as a conference paper at ICLR 2023
Wikipedia Title: Howard W. Koch <hidden for brevity>>
Wikipedia Title: The Sensational Trial <hidden for brevity.>
Wikipedia Title: Coolie No. I (1995 film) <hidden for brevity>
Wikipedia Title: Karl Freund chidden for brevity>>
Q: Do director of film Coolie No. I (1995 Film) and director of film The Sensational Trial have the same nationality?
A: ["no"]
Wikipedia Title: Fred de Cordova <hidden for brevity"
Wikipedia Title: Thulasi (1987 film) <hidden for brevity>
Wikipedia Title: Joseph M. Newman <hidden for brevity>>
Wikipedia Title: The Gal Who Took the West <hidden for brevity>
Wikipedia Title: Twenty Plus Two <hidden for brevity.>
Q: Which film has the director died later, The Gal Who Took the West or Twenty Plus Two? A: ("Twenty Plus Two"]
Wikipedia Title Ana Gruzinsky-Golitsyn <hidden for brevity>
Wikipedia Title: Rudra Shah <hidden for brevity>
Wikipedia Title: Krishna Shah (Nepalese royal) <hidden for brevity'>
Q: Who is the grandchild of Krishna Shah (Nepalese Royal)?
A: ("Prithvipati Shah"]
Wikipedia Title: Augusto Genina <hidden for brevity>>
Wikipedia Title: Ian Barry (director) <hidden for brevity>>
Wikipedia Title: Maddalena (1954 film) <hidden for brevity>
59

---

Published as a conference paper at ICLR 2023
Q: Where did the director of film Maddalena (1954 Film) die?
A: ["Rome"]
Wikipedia Title: Martin Hodge <hidden for brevity.>
Wikipedia Title: Ivania Martinich <hidden for brevity>
Wikipedia Title: Tom Dickinson <hidden for brevity>
Q: Who was born first out of Martin Hodge and Ivania Martinich? A: ("Martin Hodge"]
2wikimultihopga mitthhp.titlega (cot)
Wikipedia Title: David Dhawan <hidden for brevity
Wikipedia Title: Howard W. Koch <hidden for brevity>
Wikipedia Title: The Sensational Trial <hidden for brevity
Wikipedia Title: Coolie No. I (1995 film) <hidden for brevity>
Wikipedia Title: Karl Freund hidden for brevity>>
Q: Do director of film Coolie No. 1 (1995 Film) and director of film The Sensational Trial have the same nationality?
A: Coolie No. 1 (1995 film) directed by David Dhawan. The Sensational Trial was directed by Karl Freund. David Dhawan's 's nationality is India. Karl Freund's nationality is Germany. Thus, they do not have the same nationality. So the answer is: no.
Wikipedia Title: Fred de Cordova <hidden for brevity>>
Wikipedia Title: Thulasi (1987 film) <hidden for brevity>
Wikipedia Title: Joseph M. Newman <hidden for brevity>>
Wikipedia Title: The Gal Who Took the West <hidden for brevity'>
Wikipedia Title:
Twenty Plus Two <hidden for brevity>
Q: Which film has the director died later, The Gal Who Took the West or Twenty Plus Two?
A: The film Twenty Plus Two was directed by Joseph M. Newman. The Gal Who Took the West directed by Frederick de Cordova. Joseph M. Newman died on January 23, 2006. Fred de Cordova died on September 15, 2001. Thus, the person to die later from the two Twenty Plus Two. So the answer is: Twenty Plus Two.
60

---

Published as a conference paper at ICLR 2023
Wikipedia Title: Ana Gruzinsky-Golitsyn <hidden for brevity>
Wikipedia Title: Rudra Shah <hidden for brevity>
Wikipedia Title: Krishna Shah (Nepalese royal)
<hidden for brevity>
Q: Who is the grandchild of Krishna Shah (Nepalese Royal)? A: Krishna Shah has child named Rudra Shah. Rudra Shah has a child named Prithvipati Shah. Thus, Krishna Shah has grandchild named Prithvipati Shah. So the answer is: Prithvipati Shah.
Wikipedia Title: Augusto Genina <hidden for brevity
Wikipedia Title: lan Barry (director) <hidden for brevity>
Wikipedia Title: Maddalena (1954 film)
<hidden for brevity>
Q: Where did the director of film Maddalena (1954 Film) die?
A: The film Maddalena directed by Augusto Genina. Augusto Genina died Rome. So the answer is: Rome.
Wikipedia Title: Martin Hodge <hidden for brevity>
Wikipedia Title: Ivania Martinich <hidden for brevity>
Wikipedia Title: Tom Dickinson <hidden for brevity
Q: Who was born first out of Martin Hodge and Ivania Martinich?
A: Martin Hodge was born February 1959, Ivania Martinich born on 25 July 1995, Thus, Martin Hodge was born first. So the answer is: Martin Hodge.
musique ans: decomp
QC:
When did the first large winter carnival take place in the city where CIMI-FM is licensed to broadcast?
QS:
(select) [edqa) What city is CIMI-FM licensed to broadcast in?
A:
("titles": ["KWMZ-FM". "CIMI-FM", "KSAO (FM)", "WTTL-FM", "KOLU", "WORW", WLRX (FM)"), "answer": ["Quebec City"]}
QS: (select) [retrieve odqa] When did the first large winter carnival in Quebec City take place?
A: ("titles": ["WRQY", "Quebec Winter Carnival", "KBCR-FM"), "answer": ["1894"]]
OS: (select) imulthhopttilequa Titles: ["KWMZ-FM". "CIMI-FM", "KSAO (FM)", "WTTL-FM
" """OOU", "WORW" "WLRX (FM)", "WRQY", "Quebee Winter Carnival", "KBCR-FM "J. Question: When did the first large winter carnival take place in the city where CIMI-FM is licensed to broadcast?
A: ['1894"]
QS: [EOQ]
61

---

Published as a conference paper at ICLR 2023
QC: What the headquarters for the organization who sets the standards for ISO 21500?
QS:
(select) [retrieve odqa] Who sets the standards for ISO 21500?
A:
("titles": ["ISO 1666-2MM", "ISO 3166-2:15", "ISO 3166-2:1T", "ISO 3166-2:BB", "ISO 21500", "ISO 3166-2:ET", "ISO 3166-2:AT", "ISO 3166-2:CG", "ISO 22000"]. "answer".
["International Organization for Standardization"))
QS: (select) [retrieve_ode _odqa] Where is the headquarters for International Organization for Standardization?
A: ("titles": ["International Organization for Standardization"" "ISO 4031". "ISO 3166-2:CN",
Unicode", "ISO 3166-2:GH", "ISO 3166-2:A0", "ISO/TC 68", "ISO 7001", "ISO 3307")." answer": ["Geneva"]]
QS:
(select) [multihop_titleqa]] Titles: ["ISO 3166-2:BM", "ISO 3166-2:18", "ISO 3166-2:1T",
ISO 3166-2:BB", "ISO 21500", "ISO 3166-2:ET", "ISO 3166-2:AT", "ISO 3166-2:CG", ISO 22000", "International Organization for Standardization" "ISO 4031". "ISO 3166-2:CN "Unicode". "ISO 3 3166-2:GH", "ISO 3166-2:A0", "ISO/TC 68", "ISO 7001", "ISO 3307"]. Question: What is the headquarters for the organization who sets the standards for ISO 21500?
A: ["Geneva"] QS: [EOQ]
QC:
How long is the US border with the country that borders the state where Finding Dory takes place?
QS:
(select) [retrieve_odge _odqa] In which state does Finding Dory take place?
A:
("titles": ["Finding Dory", "Latvia", "List of countries that border only other country",
Removal of Hungary's s border fence with Austria"], "answer": ["California"]] QS: (select) [retrieve_odqa] Which country shares border with Califoria?
A:
("titles": ["Mexico-United States border", "Kingdom of Gera", "Pesticide", "Currie Cup"],"
answer": ["Mexico"]]
QS: (select) [retrieve_odqa] What the length of the US border with Mexico?
("titles": ["Piscataqua River border dispute", "Share Coke", "Mexico-United States border", A:
"Hotel Arbez"), "answer": 1"1,989 mi"]]
QS: (select) cuultihop_titleqaee Titles: ("Finding Dory", "Latvia", "List of countries that border only one other country". "Removal of Hungary's 's border fence with Austria", "Mexico- United States border", "Kingdom of Gera", "Pesticide", "Currie Cup", "Piscataqua River border dispute", "Share Coke", "Mixico-United States border", "Hotel Arbez"]. Question: How long the US border with the country that borders the state where Finding Dory takes place?
["1,989 mi"] OS: [EOQ]
A:
QC: When was Neville A. Stanton's 's employer founded? (select) [retrieve_odqa] Who is the employer of Neville Stanton?
QS:
A: ("titles": ["Harriot Stanton Blatch", "Robichaud Canada (Treasury Board)", "The Peggy Neville Show". "Stanton Township. Champaign County, Illinois", "Women's 's suffrage the United States", "Neville A. Stanton"), "answer"" ["University of Southampton"))
QS: (select) fretrieve_odqe When was the University of Southampton founded?
A: ("titles": ["Southampton" "Presley Neville"J. "answer": ["1862"]]
QS: (select) [multihop [multihop_útllqq]] Titles: ("Harriot Stanton Blatch", "Robichaud v Canada (Treasury Board)", "The Peggy Neville Show". "Stanton Township. Champaign County, Illinois".. Women's s suffrage the United States", "Neville A. Stanton", "Southampton", "Presley Neville"). Question: When was Neville A. Stanton's 'ss employer founded?
A: ["1862"]
OS: [EOQ]
QC: In which county was the birthplace of the Smoke in tha City performer?
QS: (select) [retrieve_odee _oda]] Who the performer Smoke in the City?
62

---

Published as a conference paper at ICLR 2023
A: ("titles": ["Nicotine marketing", "Khabarovsky District", "Beyonce", "Asthma", "Smoke in tha City"]. "answer": ["MC Eiht"]}
QS: (select) [retrieve odqa] What is the birthplace of MC Eiht?
A:
("titles": ["Cherokee City, Arkansas", "MC Fiht", "Biysky District"), "answer": ["Compton"])
QS: (select) [retrieve_odqa] Compton is located in which county?
A: ("titles: ["Vilnius County", "Biblioteca Ayacucho", "Compton, California"), "answer": ["Los
Angeles County"]]
QS: (select) [multihop_titles titleqa] Titles: ("Nicotine marketing", "Khabarovsky District", "Beyonce". "Asthma". "Smoke in tha City", "Cherokee City, Arkansas", "MC Eiht", "Biysky District", Vilnius County", "Biblioteca Ayacucho". "Compton, California"). Question: In which county was the birthplace the Smoke in tha City performer?
A: LoL Angeles County"]
QS: [BOQ]
musiquelans: retrieve.odga
QC: What city is CIMI-FM licensed to broadcast in?
QS: (select) [retrieve] What city is CIMI-FM is licensed broadcast in?
A: ["KWMZ-FM", "CIMI-FM", "KSAO (FM)", "WTTL-FM",, "KOLU", "WORW", "WLRX FM)"]
QS: (select) [singlehop_tiee titlepaattteeaat Titles: ["KWMZ-FM", "CIMI-FM", "KSAO (FM)". "WTTL-FM
""KOLU", "WORW", "WLRX (FM))). Question: What city CIMI-FM is licensed
broadcast in?
A: ("titles": ["KWMZ-FM", "CIMI-FM", "KSAO (FM)", "WTTL-FM", "KOLU", "WORW"," WLRX (FM)"), " "answer": ["Quebec City"]} QS: [EOQ]
QC: When did the first large winter carnival in Quebec City take place?
QS: (select) [retrieve] When did the first large winter carnival Quebec City take place?
A: ["WRQY", "Quebec Winter Carnival", "KBCR-FM")
QS: (select) [singlehop.titleg] Titles: ("WRQY", "Quebec Winter Carnival", "KBCR-FM"), Question: When did the first large winter carnival in Quebec City take place?
A: ("titles": ("WRQY", "Quebec Winter Carnival", "KBCR-FM"), "answer": ["1894"]]
OS: [HOQ]
QC: Who sets the standards for ISO 21500?
QS: (select) [retrieve] Who sets the standards for ISO 21500?
A:
["ISO 3166-2:BM". "ISO 3166-2:1S", "ISO 3166-2:1T", "ISO 3166-2:BB", "ISO 21500","
ISO 3166-2:E7", "ISO 3166-2:AT" "ISO 3166-2:CG". "ISO 22000"]
QS: (select) [singlehop_titles]
Titles: ['ISO 3166-2:BM", "ISO 3166-2:1S", "ISO 3166-2:1T".
ISO 3166-2:BB". "ISO 21500", "ISO 3166-2:ET", "ISO 3166-2:AT",
31666166-
ISO 22000"]. Question: Who sets the standards for ISO 21500?
A: ("titles": ['ISO 3166-2:BM". "ISO 3166-2:1S". "ISO 3166-2:1T". "ISO 3166-2:BB" "ISO 21500", "ISO 3166-2:ET". "ISO 3166-2:ATT" "ISO 3166-2:CG", "ISO 22000"), "answer":
["International Organization for Standardization"))
QS: [EOQ]
QC: Where is the headquarters for International Organization for Standardization?
QS: (select) [retrieve] Where the headquarters for International Organization for Standardization
( ["International Organization for Standardization" "ISO 4031", "ISO 3166-2:CN", "Unicode".
A:
"ISO 3166-2:GHH', "ISO 3166-2:A0", "ISO/TC 68", "ISO 7001". "ISO" 3307"] QS: (select) [singlehop_titee -itleeqa Titles: ("International Organization for Standardization", "ISO 4031". "ISO 3166-2:CN". "Unicode". "ISO 3166-2:GH", "ISO 3166-2:A0", "ISO/TC 68",
"ISO 7001". "ISO 3307"]. Question: Where the headquarters for International Organization for Standardization?
63

---

Published as a conference paper at ICLR 2023
A: ("titles"; ["International Organization for Standardization" "ISO 4031". "ISO 3166-2:CN". Unicode", "ISO 3166-2:GH", "ISO 3166-2:A0", "ISO/TC 68". "ISO 7001", "ISO 3307"],
answer": ["Geneva"]]
QS: [FOQ)
QC: In which state does Finding Dory take place?
QS: (select) [retrieve] In which state does Finding Dory take place?
A: ["Finding Dory". "Latvia", "List of countries that border only one other country". "Removal of Hungary's s border fence with Austria"]
QS: (select) (singlehop titleqa] Titles: ["Finding Dory", "Latvia", "List countries that border only one other country", "Removal of Hungary's is border fence with Austria"]. Question: In which state does Finding Dory take place?
A:
("titles": ("Finding Dory". "Latvia", "List of countries that border only other country"," Removal Hungary's s border fence with Austria"), "answer": ["California"]] [EOQ]
OS:
QC: Which country shares a border with California?
QS: (select) [retrieve] Which country shares border with California?
A: ["Mexico-United States border", "Kingdom of Gera", "Pesticide", "Currie Cup"]
QS: (select) [singlehop titleqa] Titles: ""Mexico-United States border", "Kingdom Gera", Pesticide". "Currie Cup"). Question: Which country shares border with California? A: ("titles": ["Mexico-United States border", "Kingdom of Gera", "Pesticide", "Currie Cup"), answer": ["Mexico"]]
QS: [EOQ]
QC:
What is the length of the US border with Mexico?
QS:
(select) [retrieve] What is the length of the US border with Mexico?
A:
["Piscataqua River border dispute", "Share Coke", "Mexico-United States border", "Hotel Arbez"]
QS:
(select) haagllhhop_itlleeee Titles: ["Piscataqua River border dispute", "Share a Coke", Mexico-United States border", "Hotel Arbez"]. Question: What is the length of the US border with Mexico?
A:
("titles": ["Piscataqua River border dispute", "Share Coke", "Mexico-United States border". "Hotel Arbez"], "answer": ["1,989 mi"]]
QS: [EOQ]
QC: Who is the employer of Neville A. Stanton?
QS: (select) [retrieve] Who is the employer of Neville A. Stanton?
A: ("Harriot Stanton Blatch", "Robichaud v Canada (Treasury Board)", "The Peggy Neville Show
"Stanton Township, Champaign County, Illinois". "Women's 's suffrage in the United States",
"Neville A. Stanton"]
QS: (select) [singlchop titleqa] Titles: ["Harriot Stanton Blatch", "Robichaud Canada (Treasury Board)", "Thh Peggy Neville Show", "Stanton Township. Champaign County, Illinois", Women's suffrage the United States", "Neville A. Stanton"]. Question: Who the
employer of Neville A. Stanton?
A: ("titles") ["Harriot Stanton Blatch", "Robichaud Canada (Treasury Board)", "The Peggy Neville Show", "Stanton Township. Champaign County. Illinois", "Women' suffrage in the United States", "Neville A. Stanton"), "answer": ["University Southampton"]]
QS: [EOQ]
QC: When was the University of Southampton founded?
QS: (select) [retrieve] When was the University of Southampton founded?
A: ""Southampton", "Presley Neville"]
64

---

Published as a conference paper at ICLR 2023
QS: (select) Isinglehop.titlee Titles: ("Southampton", "Presley Neville"]. Question: When the University of Southampton founded?
A:
("titles": ("Southampton", "Presley Neville"), "answer": ["1862"]]
QS: [EOQ]
QC: Who is the performer Smoke in the City?
QS: (select) [retrieve] Who is the performer of Smoke in the City?
A: ["Nicotine marketing", "Khabarovsky District", "Beyonce", "Asthma". "Smoke in tha City"]
QS: (select) sagglehop_titeeppe Titles: ("Nicotine marketing", "Khabarovsky District", "Beyonce", "Asthma", "Smoke tha City"). Question: Who the performer of Smoke the City? A: ("titles": ["Nicotine marketing", "Khabarovsky District", "Beyonce", "Asthma", "Smoke in tha City"], "answer": ["MC Eiht"]} QS: [EOQ]
QC: What is the birthplace of MC Eiht?
QS: (select) [retrieve] What is the birthplace of MC Biht?
A: ["Cherokee City, Arkansas", "MC Eiht", "Biysky District"]
QS: (select) Isinglchop_title [ Titles: ["Cherokee City, Arkansas", "MC Eiht", "Biysky District"), Question: What is the birthplace of MC Eiht?
A: ("titles": ["Cherokee City, Arkansas", "MC Eiht", "Biysky District"), "answer": ["Compton"]]
OS: [EOQ]
QC: Compton is located in which county?
QS:
(select) [retrieve] Compton is located in which county?
A: ["Vilnius County", "Biblioteca Ayacucho". "Compton, California")
OS:
(select) (singlehop titleqa] Titles: ["Vilnius County", "Biblioteca Ayacucho", "Compton, Californaa"). Question: Compton is located which county?
A: ("titles": ["Vilnius County", "Biblioteca Ayacucho", "Compton, California"], "answer": ["Los Angeles County"]}
QS: [EOQ]
musi ansiqueans:: singlehop.titlege
Wikipedia Title: CIMI-FM <hidden for brevity>
Wikipedia Title: Quebec Winter Carnival <hidden for brevity>
Wikipedia Title: KELD-FM
<hidden for brevity>
Q: What city is CIMI-FM is licensed to broadcast in?
A: ("Quebec City"]
Q: When did the first large winter carnival in Quebec City take place? A: ["1894"]
Wikipedia Title: ISO/TC 68 <hidden for brevity>
Wikipedia Title: ISO 21500 <hidden for brevity>>
Wikipedia Title: ISO 3166-2:GH <hidden for brevity>
65

---

Published as a conference paper at ICLR 2023
Q: Who sets the standards for ISO 21500?
A: IIInternational Organization for Standardization")
Q: Where is the headquarters for International Organization for Standardization?
A: ["Geneva"]
Wikipedia Title: Mexico-United States border <hidden for brevity>
Wikipedia Title: Mexico-United States border <hidden for brevity
Wikipedia Title: Finding Dory <hidden for brevity>
Wikipedia Title: India
<hidden for brevity>
Q: In which state does Finding Dory take place?
A: ["California"]
Q: Which country shares a border with California?
A: ["Mexico"]
Q: What is the length of the US border with Mexico? A: ["1,989 mi")
Wikipedia Title: Southampton <hidden for brevity.>
Wikipedia Title: Jonathan Stanton <hidden for brevity>>
Wikipedia Title: Neville A. Stanton hidden for brevity>
Q: Who is the employer of Neville A. Stanton?
A: ("University of Southampton"]
Q: When was the University of Southampton founded? A: ["1862"]
Wikipedia Title: Compton, Califoria <hidden for brevity>
Wikipedia Title: Smoke in tha City <hidden for brevity>>
Wikipedia Title: MC Eiht <hidden for brevity>
Wikipedia Title: Eastern Bengal and Assam <hidden for brevity
Q: Who the performer of Smoke in the City?
66

---

Published as a conference paper at ICLR 2023
A: ["MC Eiht"]
Q: What is the birthplace of MC Eiht?
A: ["Compton"]
Q: Compton is located in which county?
A: ["Los Angeles County"]
musique_ans
mittihp..tilleaa (direct)
Wikipedia Title: CIMI-FM <hidden for brevity>
Wikipedia Title: Quebec Winter Carnival <hidden for brevity'>
Wikipedia Title: KELD-FM
<hidden for brevity>
Q: When did the first large winter carnival take place the city where CIMI-FM is licensed to broadcast?
A: ["1894"]
Wikipedia Title: ISO/TC 68 <hidden for brevity.>
Wikipedia Title: ISO 21500 <hidden for brevity>
Wikipedia Title: ISO 3166-2:GH < <hidden for brevity>
Q: What is the headquarters for the organization who sets the standards for ISO 21500? A: ["Geneva"]
Wikipedia Title: Mexico-United States border <hidden for brevity>
Wikipedia Title: Mexico-United States border <hidden for brevity>
Wikipedia Title: Finding Dory <hidden for brevity>>
Wikipedia Title: India
<hidden for brevity>>
Q: How long is the US border with the country that borders the state where Finding Dory takes place? A: ["1,989 mi"]
Wikipedia Title: Southampton <hidden for brevity>
Wikipedia Title: Jonathan Stanton chidenn for brevity
67

---

Published as a conference paper at ICLR 2023
Wikipedia Title: Neville A. Stanton <hidden for brevity*>
Q: When was Neville A. Stanton's 's employer founded?
A: ["1162")
Wikipedia Title: Compton, California <hidden for brevity>
Wikipedia Title: Smoke in tha City <hidden for brevity>
Wikipedia Title: MC Eiht <hidden for brevity>
Wikipedia Title: Eastern Bengal and Assam <hidden for brevity>
Q: In which county was the birthplace of the Smoke tha City performer? A: ["Los Angeles County"]
musique.ans: multihop.titlega (cot)
Wikipedia Title: CIMI-FM <hidden for brevity>
Wikipedia Title: Quebec Winter Carnival <hidden for brevity>
Wikipedia Title: KELD-FM
hidden for brevity>
Q: When did the first large winter carnival take place in the city where CIMI-FM is licensed to broadcast?
A: CIMI-FM is licensed broadcast in Quebec City. The first large winter cariiall in Quebec City took place in 1894. So the answer is: 1894.
Wikipedia Title: ISO/TC 68 <hidden for brevity>
Wikipedia Title: ISO 21500 <hidden for brevity>
Wikipedia Title: ISO 3166-2:GH chidden for brevity>
Q: What is the headquarters for the organization who sets the standards for ISO 21500? A: The standards for ISO 21500 were set by International Organization for Standardization. The International Organization for Standardization has headquarters in Geneva. So the answer Geneva.
Wikipedia Title: Mexico-United States border <hidden for brevity>
Wikipedia Title: <hideen for brevity>>
Mexico-United States border
68

---

Published as a conference paper at ICLR 2023
Wikipedia Title: Finding Dory <hidden for brevity*
Wikipedia Title: India <hidden for brevity>>
Q: How long is the US border with the country that borders the where Finding Dory takes place?
A: Finding Dory is supposed take place in California. The country that shares a border with California is Mexico. The length of the border with Mexico is 1,989 mi. So the answer is: 1,989 mi.
Wikipedia Title: Southampton <hidden for brevity>
Wikipedia Title: Jonathan Stanton <hidden for brevity>
Wikipedia Title: Neville A. Stanton <hidden for brevity
Q: When was Neville A. Stanton's s employer founded?
A: The employer Neville A. Stanton University of Southampton. The University Southampton was founded in 1862. So the answer is: 1862.
Wikipedia Title: Compton, California <hidden for brevity>
Wikipedia Title: Smoke in tha City <hidden for brevity>>
Wikipedia Title: MC Eiht <hidden for brevity
Wikipedia Title: Eastern Bengal and Assam hiddeen for brevity>>
Q: In which county was the birthplace of the Smoke in tha City performer?
A: The performer of Smoke tha City MC Eiht. MC Eiht's 's birthplace is Compton. Compton located in the county of Los Angeles County. So the answer is: Los Angeles County.
69


Reviews:
[{"review_id": "ovtKnehIoy", "reviewer": "Reviewer_Pqnp", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A framework called DecomP for solving multi-step reasoning problems with large language models is proposed. A task is \"decomposed\" into a sequence of smaller subtasks, where each subtask calls a reusable submodule (itself a LLM call or an external knowledge retrieval); a simple syntax for control flow allowing substitution of past subtask results and iteration over lists is introduced. Experiments on two string and list operation tasks and two question-answering tasks show that DecomP compares favourably with chain-of-thought approaches.", "summary_of_the_review": "The paper is quite well written and adds some interesting ideas to our understanding of multi-step reasoning with LLMs. On the other hand, given all the recent findings that go in a similar direction, the strong results with this particular method did not surprise me very much. This isn't a reason not to publish the paper, of course, but I would hope to see something that would make it stand out more, such as results on a broader (more diverse) set of tasks that illustrate more variety of subtasks and interesting ways to combine them. ~Therefore, I am giving a cautious \"marginally below\", but am willing to be convinced to raise the score.~ See post-rebuttal comment below.", "strength_and_weaknesses": "Strengths:\n- (++) I like the approach of separating controller and subtask modules. This work contributes to an increasing realization that linear attention -- where the entire reasoning sequence is a single chain of tokens -- could be insufficient for hard reasoning problems; the chosen comparisons with CoT illustrate this well. The ability, even if only hypothetical, to teach a controller to query external structured retrieval systems, execute programs, etc., is an exciting prospect.\n- (+) Interesting combination of few-shot prompting (to teach the decomposer/controller) with question decomposition along the lines of [Perez et al., 2020; Wolfson et al., 2020].\n\nWeaknesses / questions:\n- (--) The breadth of applications is a little unsatisfying given how general the proposed approach can be. The examples currently given are either ones that can be turned into exact programs (list and string operations) or ones that use a chain of questions for knowledge retrieval (QA). But where can decomposed prompting be applied beyond these cases? The most interesting tasks may be the ones that require a mix of formal steps  (that a simple program could do) / querying of symbolic systems and knowledge generation / retrieval.\n- (-) The following would be important to illustrate:\n  - How does direct prompting -- no CoT -- performs on these tasks?\n  - How sensitive is the system to the model size and prompt format? Can GPT-2 XL or the smallest variant of GPT-3 be used for (a) the decomposer and (b) the submodules, and if so, which ones? (From my own experience playing with CoT-like approaches, I would expect large differences even among davinci, text-davinci-001, and text-davinci-002 variants of GPT-3 for some problems.)\n- When the submodules are expected to produce structured output, such as a list of strings, how reliably do they generate text in the correct format? (What would happen if a submodule outputs something not of the form \"[ item1, item2, ... ]\", but the following call contains \"foreach\"?)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "i4XnGrAqTM_", "reviewer": "Reviewer_H95Z", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a novel prompting algorithm based on a decomposition or divide-and-conquer idea. The idea is instead of feeding long CoT into the LLM, it decomposes the difficult task into easier subtask. The advantage is the smaller task can either be solved better with LLM or can be handled by external knowledges. Several experiments have been conducted to prove the effectiveness of the proposed method.", "summary_of_the_review": "The paper provides a novel idea, and several experiments to support its claim. It would be even better to experiment on challenging task, such as GSM8K as suggested. ", "strength_and_weaknesses": "- Following some predecessors, solving a difficult task by devide-and-conquer, decomposing them into simpler one, is quite novel to me. \n\n- The potential of the proposed is on solving challenging tasks, such as arithmetics.  I would encourage the authors try on challenging GSM8K dataset. It's not only because it's a dataset that CoT haven't solved it well, but I believe it could be a great fit to demonstrate the strength of the proposed algorithm. Many complex  arithmetics problems can be decomposed into simple sub-problems. I would expect each subproblem can be solved well by LLM, so as the whole problem under the decomposition scheme. \n\n- I have a few questions.  In practice, how do we decide which decomposition scheme to use? also, how do  we decide the question of each subproblem? is it required human design to tailor for each application like CoT? if so, how sensitive the results is to the design of the questions and decomposition? \n\n- How does this model work under a smaller LLM? In CoT, an interesting finding is it works better with larger LLM. How about the proposed one?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8uwgDGTv31y", "reviewer": "Reviewer_AJBU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes Decomposed Prompting, a new prompting method to decompose a task into subtasks and handle them by modular prompts. This allows hierarchical and recursive decomposition, as well as calling external APIs. Across symbolic manipulation (k-th letter concatenation, list reversal) and question answering (CommaQA-E, HotpotQA) tasks, Decomposed Prompting is shown to outperform Chain-of-thought prompting.", "summary_of_the_review": "I believe Decomposed Prompting is a natural next step for prompting, and results here look good. We probably need more analysis and insights so that people can design decomposed prompts for their tasks.", "strength_and_weaknesses": "- Strength: the method seems a natural step in the direction of prompting, and empirical results across different tasks are good. \n\n- Weakness: experiments seem to only focus on establishing better results than baselines (e.g. chain-of-thought), but I feel we need better analysis about the method itself. A key thing missing is failure modes and limitations -- I'd argue such insights are more valuable for practitioners interested in adopting Decomposed Prompting. \n\nAlso, there is some recent and related work around modular use of LLMs, such as Socratic Models [1]. Although motivation and tasks are not the same, some discussion is needed.\n\n[1] Zeng et al., Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0L_iJTKi7g", "reviewer": "Reviewer_WUGg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed to use decomposed prompting to solve complex NLP tasks. The idea behind decomposed prompting is very simple: divided and conquer. For a given complex task, they first decompose it into simpler sub-tasks via task prompting and then solve each sub-task by prompting-based LLMs dedicated to the sub-task. They conduct experiments on four different tasks including $k^{th}$ LETTER CONCATENATION, LIST REVERSAL, LONG-CONTEXT QUESTION ANSWERING, and OPEN-DOMAIN QUESTION ANSWERING. The empirical results not only show that the proposed framework improves the few-shot performance over CoT prompting but also demonstrate flexibility: it is easy to incorporate symbolic operations and information retrieval systems into the framework.\n", "summary_of_the_review": "Although the idea is not brand new, the experiments are well-designed and demonstrate how the proposed approach could be used in different scenarios. I think the general community would find this useful and many researchers would benefit from the design choices for different tasks. Personally, I don't feel quite excited after reading this paper. It is a solid extension of the previous literature, but I expect more from an ICLR publication. Therefore, I lean toward a weak rejection. \n", "strength_and_weaknesses": "Strengths:\n- The proposed approach is well-motivated.\n- Strong empirical results across different datasets.\n- Comprehensive experiments to show the flexibility of the proposed framework.\n\nWeaknesses:\n- To me, the main weakness of this paper is the lack of novelty. It is more of an extension of the least-to-most prompting (Zhou et al., 2022). The least-to-most prompting can also generate the decompositions iteratively based on the outputs from previous steps. What's really new in this paper is that instead of solving the decompositions all at once by a pretrained language model, the authors show that solving them by dedicated LLMs or other systems (e.g., external API calls) is even better. \n- The scope of the experiments is not broad enough given the flexibility of the proposed framework. It would be better if the authors could test it in more challenging settings. There are two possible settings I think might be interesting: \n    - Answering the \"why\" questions in long-form questions answering (Fan et al., 2019). For example, to answer \"why were the main predators in New Zealand large birds, compared to other places in the world?\", we could possibly decompose it into several sub-questions to gather relevant information and synthesize them:\n        1.  What are some of the main predators in the world?\n        2.  Why New Zealand doesn't have *results from #1* as the main predators?\n    - Fact-checking real-world complex claims through decomposition (Chen et al., 2022). For example, to check a claim like \"Joe Biden stated on August 31, 2020, in a speech: \"When I was vice president, violent crime fell 15% in this country. ... The murder rate now is up 26% across the nation this year under Donald Trump.\" We may need to know the answers to the following sub-questions and synthesize them to produce final veracity for the claim:\n        1. Did the crime rate fall by 15% during Joe Biden's presidency?\n        2. Did the murder rate in 2020 increase by 26% from 2019?\n        3. Is Biden comparing crime rates from the same time interval in his statement?\n        4. Is the violent crime rate and murder rate directly comparable?\n\n    It would be better to show to what extent the decomposed prompts work in such real-world scenarios. Can you generate the decompositions iteratively? It is possible to synthesize the final answer by the decompositions?\n\nReferences:\n - Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: Long Form Question Answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558–3567, Florence, Italy. Association for Computational Linguistics.\n - Chen, Jifan, Aniruddh Sriram, Eunsol Choi, and Greg Durrett. \"Generating Literal and Implied Subquestions to Fact-check Complex Claims.\" arXiv preprint arXiv:2205.06938 (2022).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster