Ask Me Anything: A simple strategy for prompting language models
Authors: Simran Arora
Avanika Narayan
Mayee F Chen
Laurel Orr
Neel Guha
Kush Bhatia
Ines Chami
Christopher Re
Venue: ICLR 2023 notable top 25% (2023)
Keywords: large language models
prompt-engineering
in-context learning

Abstract: Large language models (LLMs) transfer well to new tasks out-of-the-box simply given a natural language prompt that demonstrates how to perform the task and no additional training. Prompting is a brittle process wherein small modifications to the prompt can cause large variations in the model predictions, and therefore significant effort is dedicated towards designing a painstakingly crafted "perfect prompt" for a task. To mitigate the high degree of effort, we instead ask whether collecting multiple decent, yet imperfect, prompts and aggregating them can lead to a high quality prompting strategy. Our observations motivate our proposed method, Ask Me Anything (AMA). We first develop an understanding of the effective prompt formats, finding question-answering (QA) prompts, which encourage open-ended generation ("Who went to the park?") tend to outperform those that restrict the model outputs ("John went to the park. True or False?"). AMA recursively uses the LLM to transform task inputs to the effective QA format. AM generates multiple questions per input and applies these prompts to collect several noisy "votes" for the input's true label. We find the prompts have varying accuracies and dependencies and thus propose to use weak supervision, a procedure for combining the noisy predictions, to produce the final predictions. We evaluate AMA across open-source model families (EleutherAI, BLOOM, OPT, and T0) and sizes (125M-175B parameters), demonstrating an average performance lift of 10.2\% over the few-shot baseline. This simple strategy enables the open-source GPT-J-6B model to match and exceed the performance of few-shot GPT3-175B  on 15 of 20 popular benchmarks. Averaged across these tasks, the GPT-J-6B model outperforms few-shot GPT3-175B. We release our code here: https://github.com/HazyResearch/ama_prompting.

Published as a conference paper at ICLR 2023
ASK ME ANYTHING: A SIMPLE STRATEGY FOR PROMPTING LANGUAGE MODELS
Simran Arora², Avanika Narayan* Mayee Chen, Laurel Orr, Neel Guha, Kush Bhatia, Ines Chami, Christopher Ré
{s isimare timarora avanika, mfchen, lorrl, nguha, kushb, chri serressree (cs 1 (ines. chami Jeumbersstationn
stanford.edd
ABSTRACT
Large language models (LLMs) transfer well tasks out-of-the-box sim- ply given a natural language prompt that demonstrates how to perform the task and additional training. Prompting is brittle process wherein small modi- fications to the prompt can cause large variations in the model predictions, and therefore significant effort is dedicated towards designing a painstakingly crafted perfect prompt for a task. To mitigate the high degree of effort, we instead ask whether collecting multiple decent, yet imperfect, prompts and aggregating them can lead to high quality prompting strategy. Our observations motivate our pro- posed method, ASK ME ANYTHING PROMPTING (AMA). first develop an understanding of the effective prompt formats, finding qmestion-answeriig prompts, which encourage open-ended generation ("Who went to the park?") tend outperform those that restrict the model outputs ("John went to the park. True or False?"), AMA recursively uses the LLM to transform task inputs to the effective QA format. AMA generates multiple questions per input and applies these prompts to collect several noisy votes for the input's ' true label. We find the prompts have varying accuracies and dependencies and thus propose use weak supervision, a procedure for combining the noisy predictions, to produce the final predictions. We evaluate AMA across open-source model families (EleutherAL. BLOOM, OPT, and TO) and sizes (125M-175B parameters), demonstrating erage performance lift of 10.2% over the few-shot baseline. This simple strategy enables the open-source GPT-J-6B model to match and exceed the performance of few-shot GPT3-175B on 15 of 20 popular benchmarks. Averaged across these tasks, the GPT-J-6B model outperforms few-shot GPT3-175B. We release our code here: htttps://githae www///oo//a/ae/eeeeeeeee htt///azyesseerrree ama aaaaapproppppaaa inng. ing.
(QA)
av-
INTRODUCTION
Large language models (LLMs) are bringing us closer to the goal of task-agnostic machine learning (Brown et al., 2020; Bommasani al., 2021). Rather than training models for new tasks, LLMs applied to new tasks out-of-the box with no additional training. In this paradigm, termed in-context learning, LLMs are controlled through user-provided natural language specifications of the task, or prompts, which illustrate how to complete a task. A prompt is defined by template which contains placeholders for in-context demonstrations of the inputs and outputs for the task.
Recent work has evaluated LLM prompting performance on a broad set of tasks and finds the process be brittle small changes to the prompt result large performance variations (Zhao et al., 2021; Holtzman et al., 2021). The performance further varies depending on the chosen LLM family (Ouyang et al., 2022; Sanh et al., 2022, inter alia.) and model size (Wei et al., 2022c; Lampinen et al., 2022). To improve reliability, significant effort is dedicated towards designing painstakingly perfect prompt. For instance, Mishra et al. (2021) and Wu et al. (2022) recommend that users manually explore large search-spaces of strategies to tune their prompts on task- task-by-task basis.
This work instead considers aggregating the predictions of multiple effective, yet imperfect prompts to improve prompting performance over broad set of models and tasks. Given task input, each prompt produces a vote for the input's 's true label, and these votes are aggregated to produce a final prediction. In pursuit of high quality prompting via aggregation, we face the following challenges:
1. Effective prompts: High quality prompts are a precursor to improvements from aggregation. We take the original prompts which yield near-random performance using the GPT-3 model in

---

Published as a conference paper at ICLR 2023
AMA PROMPTING
Run a collection of prompt--chains.o where the generate inputs to question and answer
Combine the neisy enswers using weak supervision
Input Example
Fabe given the context??
the following claim True
Fawept Chales
Frange challg
Answer the question from context
map to output
Write the claim a yexino question.
Context: John and
Context: Joe's birthday was yesterday
F False
friends went theater
camped with
Question: Was Joe's birthday yesterday? Answer: yes
Claim: John went to the park. Answer:
and saw Jurassic Park.
Claim: the test not hard
Question: Wes the test hard?
Question: Did lack camp with Mark?
Context: John and his friends went to theater and saw Jurassic
True
False
Claim: John went to the park. Question:
Question: Did John go to the park?
Model Input
Answer:
False
Prompt Chain
final prediction
Did John go the park?
Model Output
question() prompt
answert prompt
Figure 1: AMA uses the LLM itself' to reformat task inputs to more effective formats. AMA creates multiple reformatted prompts per input. The LLM predictions from the prompts aggregated using weak supervision.
Brown al. (2020) for two SuperGLUE tasks (CB, RTE). Generating multiple prompts in the same format and taking majority vote prediction across prompts has a minor effect (+4% for CB) and can even hurt performance versus the average prompt performance (-2% for RTE). Many proposals for improved prompts focus on a single task type and evaluate on a single model- family and/or size (Wei et al., 2022c; Jung et al., 2022). We need a structure for prompting that improves quality across tasks and models.
2. Scalable collection: After identifying effective prompt formats, we need to obtain such prompts at scale. The original format of a task varies widely, and prior works manually rewrite each task imput new formats (Mishra et al., 2021; Wu et al., 2022), which is challenging to scale. Generating multiple prompts per input increases the difficulty.
3. Prompt aggregation: Using the prompts above (for CB and RTE), we see 9.5% average variation in accuracy and that the Jaccard index over errors is 69% higher than if prompt errors were i.i.d., suggesting highly correlated prompt outputs. Majority vote (MV) is the primary unsupervised aggregation strategy in prior prompting work (Jiang et al., 2020; Schick & Schütze, 2021), but it does not account for either property, making it unreliable. We need a strategy that accounts for the varying accuracies and dependencies.
We propose ASk ME ANYTHING PROMPTING (AMA), a simple approach that enables open-source LLMs with 30x fewer parameters to exceed the few-shot performance of GPT3-175B. In AMA:
1. We identify properties of prompts that improve effectiveness across tasks, model types, and model sizes. We study standard prompt-formats categorized by prior work (Brown et al., 2020) and find prompts that encourage open-ended answers ("Where did John go?") to be more effective than prompts that restrict the model output to particular tokens (e.g. "John went to the park. Output True False?"). For instance, converting three SuperGLUE tasks (CB, RTE, WSC) from the original restrictive formats in (Brown et al., 2020) to open-ended formats provides a 72% performance improvement (Section 3.2). Given a task input, we find that simple structure of (1) forming questions based on the input and (2) prompting the LLM to answer the questions applies quite generally and improves performance across diverse benchmark tasks.
2. We propose a strategy for scalably reformatting task inputs to the effective formats found in (1). We propose transform task inputs to the effective open-ended question-answe answering format by recursively using the LLM itself in a task-agnostic two step pipeline. We first use question() ()- prompts, which contain examples of how transform statements to various (e.g., yes-no, cloze) questions and second use answer() ()-rrmmpts that demonstrate ways of answering questions (c.g., concise or lengthy answers). Applying prompt-chains- answer (nswer(question()) prediction for the input 2.. Chains are (1) reused across inputs and (2) different pairs of functional prompts can be combined to create variety. We apply the varying functional prompt-chains to an input to collect multiple votes for the input's true label.
gives a final
3. We propose the use of weak supervision (WS) to reliably aggregate predictions. We find that the errors produced by the predictions of different chains can be highly varying and correlated. While majority vote (MV) may do well on certain sets of prompts, it performs poorly in the above cases. AMA accounts for these cases by identifying dependencies among prompts and using WS, procedure for modeling and combining noisy predictions without any labeled data (Ramner et al,, 2017; Varma et al., 2019). We apply WS to prompting broadly for the first time in this work,
We draw inspiration from Wu et al. (2022) and focus on task-agnostic and scalable prompt-chains.
2

---

Published as a conference paper at ICLR 2023
showing it improves the reliability of prompting with off the-the-shel LLMs and no further training. We find that AMA can achieve up to 8.7 points of lift over MV and recovering dependencies boosts performance by up to 9.6 points.
We apply our proposed prompt-aggregation strategy, AMA, to 20 popular language benchmarks and 14 open-source LLMs from 4 model families (Neo, BLOOM, OPT, and TO) spanning 3 orders- of-magnitude (125M-
parameters). The simple strategy provides an improvement over the few-shot (A: = 3) baseline by an average of 10.2% + 6.1% absolute (21.4% + 11.2% relative) lift across models. We find the largest gains are on tasks where the knowledge required to complete the task found in the provided context and comparatively less on closed-book tasks (e.g.. factual recall). Most excitingly, ASK ME ANYTHING PROMPTING enables an open-source LLM, which is 30x parameters smaller, to match or exceed the challenging GPT3-175B few-shot baseline results in Brown et al. (2020) on 15 of 20 benchmarks. We hope AMA helps address painpoints in widely applying in-context learning (Arora & Ré, 2022; Narayan et al., 2022) by improving the ability to proceed with tesssthaa--rffeee prompts and encouraging the use of small, open-source LLMs.
2 RELATED WORK
Several existing works seek to improve the zero-to-fee sero-to-few-shot tusk-transfer abilities of LLMs.
Training based strategies Prior works have improved prompting performance by training larger models over more or curated data, and for longer (Kaplan et al., 2020; Chowdhery et al., 2022) or by explicitly fine-tuning LMs over prompts (Wang al., 2022b; Wei et al., 2022a; Sanh et al., 2022: Ouyang et al., 2022). We complementarily aim to improve the prompting performance of off-the-shelf language models with no additional fine-tuning.
Prompt-engineering troiit-engineering is the process of designing natural language specifica- tions of task, which are used to condition the LLM at inference time. Prior work finds that the prompt format changes the model behavior and proposes particular formats. Some formats are designed-for or evaluated-on a narrow task type, model type, or model size (Wei et al., 2022c; Jung et al., 2022). Others require users to manually rewrite task-inputs to the prescribed formats on a beample-by-example basis in a task-specific manner (Mishra et al., 2021; Wu et al., 2022). Our recursive use of the LLM is similar to Jung et al. (2022), which focuses on commonsense reasoning. We draw inspiration from these lines of work and investigate a broader set of tasks and model sizes.
Complementary work investigates how to simplify complex tasks (e.g., logical, compositional, and multi-hop reasoming). to achieve better performance in the prompting paradigm. Creswell et al. (2022); Wu et al. (2022); Zhou et al. (2022); Yang al. (2022) explicitly decompose the complex tasks into steps, which are each handled in a separate inference-pass. However, these methods draw distinction between the explicitly compositional tasks which can be naturally decomposed into multiple steps and "single-step" language tasks. These prior works do not support the single-step tasks such as classification, NLU, QA, which are the focus of our work.
Prompt aggregation Prior works note the sensitivity of prompting under slight modifications and propose strategies improve the performance of single prompts (Zhao et al., 2021; Liu al., 2021). Complementing this, we aggregate the results of multiple prompts. Shi et al. (2022) observes that different prompt example selections yield different results, and suggests combining the results of different prompts as an exciting future direction. Prompt aggregation has been applied in several prior works. Many works train models to perform the aggregation and/or to achieve strong results with small LMs (Jiang et al., 2020; Schick & Schütze, 2021: Cobbe et al., 2021; Zelikman et al., 2022, inter alia.). Self-Consistency Wang et al. (2022a), which requires no training. does not report improvements for small LMs (<10B parameters). We also compare AMA to Self-Consistenee in Appendix B. The unsupervised aggregation strategy used in prior works is Majority Vote- we are the first to use Weak Supervision for unsupervised prompt aggregation.
Weak supervision (WS) WS is a powerful framework that learns the accuracies and correlations of multiple noisy sources and aggregates them to produce weak labels for training data (Ratner et al., 2017: 2016; 2018; Varma et al., 2019; Fu et al., 2020). WS has been applied to prompting in the context of distilling LLM by aggregating the outputs of hand-curated prompts into a labeled dataset and training a smaller model on (Smith et al., 2022). In contrast, we aim to use aggregation to improve out-of-the-box L.MM performance reliably, which has not previously been explored.
3

---

Published as a conference paper at ICLR 2023
Prompt Style
Relative Gain
(6.78 vs 175B)
0.4
Prompt Format
CB
WSC
RTE
Restricted 8-shot
22.0
50.0
53.0
0.2
+ Calibration
58.9
59.2
Cloze-QA Format
48.2
56.0
62.5
0.0
Standard- -A Format
83.3
69.2
62.0
QA
Cloze Restrictive
Figure 2: Relative lift with model scale using results and prompt-styles reported Brown et al. (2020) (Left).
Ablating the prompt-style style using the GPT-J-6B model. We include calibration results Zhao et al. (2021) and the "_" indicates the method cannot be applied to the task (Right).
3 ASK ME ANYTHING PROMPTING
We propose ASK ME ANYTHING PROMPTING (AMA), a prompting approach that uses multiple imperfect coampts-rathen than one painstakingly crafted perfect prompt-and reliably aggregates their outputs, We describe and motivate AMA's 's prompt format (Section 3.2), how AMA scalably produces collections of prompts (Section 3.3), and AMA's 's aggregation method (Section 3.4).
3.1 PRELIMINARIES
We consider supervised tasks, (X, where x X' the example and y E y is the output. We have an unlabeled dataset D = {2r [= for which we wish to predict each y/s We apply LLMs to this task by using a prompt-a a natural language prefix that demonstrates how to complete a task. A prompt consists of prompt template, with placeholders for (1) zero more in-context task demonstrations and (2) for the inference example x as shown in Figure 3. Given a prompt p. we use p X' y to refer the output of the prompted LLM which produces prediction ý = p(x)). Specifically, the LLM runs inference on with x substituted for the placeholder in the template.
We denote collection of prompts P P2, ..... Pml] Given input D, we (1) apply collec- tion P to each x E DD and (2) aggregate their predictions, denoted as (fx) [p (r),.... Pm Pm(#)) (3)]. using an aggregator function e )m + y to produce outputs on each æ. We can thus express the procedure via two key components we aim to understand, the prompts P and aggregator ø.
Running examples For the motivating observations in the rest of this section, we use three Su- perGLUE (Wang et al., 2019) tasks Conksmm-ommm (CB), Recognizing Textual Entailement (RTE), and Winograd Schema Challenge (WSC)- and the DBPedia and AGNews classification tasks (Zhang et al., 2015). We evaluate over the GPT-J-6B model. CB and RTE require determining the validity of a statement given some context (as in Figure 1), WSC requires outputting the subject corresponding to given pronoun, and DBPedia and AGNews contain 14 and 4 classes, respectively. We use as a running example: determine if the statement "John went to the park" is valid, given the context "John invited Mark to watch Jurassic Park with his family at the theater".
Simple baseline We take the prompts proposed in (Brown et al., 2020) for GPT-3 and produce P with five prompts for each task by using different sets of in-context examples. Comparing majority vote (MV), the unsupervised aggregation strategy used in prior work, to the average performance of the prompts, MV gives 39.3% (+2.2%) for CB and 54.5% (-2%) for RTE. The delta from aggregating is minor and in the worst case, harmful. The need for effective prompts and a reliable aggregation strategy motivate our study. We provide similar results on additional tasks in Appendix B.
3.2 EFFECTIVE PROMPT FORMATS
First, we explore what makes an effective prompt format towards improving the quality of P(x).
Standard prompt formats We ground our analysis in three standard categories of prompts used in prior work including Brown et al. (2020); Sanh et al. (2022, inter alia.): (1) questions that restrict the model output particular tokens ("John invited Mark to come watch Jurassic Park. Output True or False?"); (2) cloze-questions which ask the model to fill in the remaining text ("John invited Mark to come watch Jurassic and using the LLM to fll-the-blank "Park"); and (3) traditional (yes-no, Wh) free-form questions ("Where did John invite Mark?"). Compare these three formats, we see:
1. Open-ended prompts appear to outperform restrictive-prompts. We first group the results in Brown et al. (2020) based on the format used for the task, along the above categorizations (see Figure 2). When scaling from GPT3-6.7B to GPT3-175B. we find that the relative gain is far lower on open-ended (cloze and traditional QA) formats restricted formats.
4

---

Published as a conference paper at ICLR 2023
Next, CB, RTE, and WSC are originally formatted with restrictive -eromttt in Brown et al. (2020), and we form copies of the tasks in the open-ended question (cloze and free-form QA) formats. This improves the performance of the small model on average from 41.7% to 71.5% (+72%). Intuitively, the task of answering open-ended questions is aligned with the next-token prediction language modeling objective. We observe that more precise questions give larger lifts. For WSC the restrictive prompt form is: "The pronoun 'his' refers to "Mark" in the context. True or False?", given the context "Mark went to the park with his dog.". Reformatting to "What does 'his' refer to?" and evaluating whether the answer is "Mark" provides 38% lift (69.2% accu- racy). Yet, further extracting the portion of the context that mentions the pronoun ("his dog"), reformatting ("Whose dog?") and prompting with precise questions gives 49.4% lift (74.7%).
2. The use of open-ended questions over restrictive-prompts can increase the difficulty of map- ping open-ended answers to valid output classes. For tasks with output spaces that are likely observed during pretraining (yes-no questions, sentiment classification), naturally generates valid ÿ E y For tasks with specialized output classes (i.e. multi-class clas- sification), we need to map the answer the open-ended question (e.g.. "What is the document about?") to a valid output class. For example, given "Personality and Mental Health is a quarterly peer-reviewed academic journal published by ...". we observe that the LLM typically outputs semantically correct summaries of the document topic, e.g. "journal". We find that in- serting a step for the LLM to map the open-ended output "journal" to valid category via the prompt "A 'journal' maps to category: written work" enables a 33.3% and 11.1% lift over the few-shot baseline on DBPedia (14-way classification) and AGNews (4-way) respectively.
that the LLM
Why is the QA prompt format effective? We analyze the LM pretraining corpus to better under- stand why the proposed QA prompt template may be effective. The EleutherAl models are trained on The Pile corpus Black et al. (2021); Wang & Komatsuzaki (2021); Gao et al. (2021). Over 2% random sample of the 200B token Pile data, we find that open-ended QA structures (i.e., which ask the model "Is .Is..." p", "Who.. ... ?"") appear on the order of 1000 more frequently than the restrictive- prompt structures (i.e., which instruct the model to output "True False", "Yes or No") (see Table 10). Further, when applying the few-shot restrictive prompts, we observe large imbalances in the Fl-scores for different classes (Table 12) and question whether answering the restrictive prompts is challenging due to biases acquired during pretraining. We find that there are large imbalances in Pile between the frequencies of "yes" vs. "no", and "True" vs. "False". Details are in Appendix H.
AMA's prompt format Motivated by the two observations above, we proceed in AMA with a two-step prompting pipeline: (1) generating questions based the input and (2) prompting the LLM to answer the generated questions. These prompts are effective, and to further improve perfor- mance we next turn to generating and aggregating over multiple prompt-outputs for each input. For intuition, different questions (with our running example: "Who went to the park?", "Did John go the park?". "Where did John go?")) emphasize different aspects of the input and can provide com- plementary information towards reasoning about the answer. Manually generating multiple prompts per input is challenging, and so we study how to do this at scale in the following section.
3.3 CREATING PROMPT COLLECTIONS AT SCALE
Our goal is to produce a collection of prompts, P, that can be applied to tasks at scale. To produce prompts in the open-ended chain of functional prompts, referred to as prompt()-chain. We describe these prompts as functional because they apply task-agnostic QA prompting template to all inputs in the tasks. We describe the two functional prompts used in AMA below. We use Figure 1 as running example.
format, we recursively apply the LLM itself using
(a) question():
q generates a question q (such "Did John go to the park?") from an input x ("John went to the park."). question() prompts simply contain demonstrations of how a statement can be transformed to an open-ended question.
(b) answer(): q + a applies the question generated by (a) to the context of x to produce interme- diate answers (such as "No" or "theater"). The answer() prompts contain demonstrations of how to answer a question (optionally) given some input context.
To create P for aggregation, AMA constructs different prompt()-chaig where each unique prompt()-chain is different view of the task and can emphasize different aspects of æ. Inspired by Sanh et al. (2022) and Liu et al. (2021), we also vary chains through two key levers- -th in- context demonstrations and the style of prompt questions- as shown in Figure 3. To vary the style
5

---

Published as a conference paper at ICLR 2023
base question0 prompt
vary in-context demonstrations
vary question style to Wh
place- holder dimonstration
Write the claim as yesino question.
Write the claim as yes/ne question.
Write the claim es question.
Claim: lack camped with Mark
Claim: the balloon papped
Claim: Jack camped with Mark
Question: Did lack camp with Mark?
Question: Did the balloun pop?
Question Who did lack camp with?
Claim: the test was not hard
Claim: the sun sets 7pm
Claim: the test was not hand
hard?
Question: Does sun set ypm?
Question: What was
Claim:
Kinput_ x>
Claim: <im::<imm <input x>
Cleim:
<input x>
Question:
Question:
Question:
Figure 3: Example prompt with the in-context demonstrations and placeholder (Left) with two different prompt
variations (Right) created by changing demonstrations and question style.
of open-ended prompt questions, we construct question() and answer() prompts that produce and answer either Yes/No, Wh, multiple-choice, or cloze- questions.
3.4 PROMPT AGGREGATION
To aggregate the prompt predictions (z)) into outputs ý reliably, we apply tools from weak super- vision, a powerful approach for learning high-quality models from weaker sources of signal without labeled data (Ratner et al., 2017). We first describe properties of P(x)) that illustrate when the sim- ple baseline of majority vote may perform poorly. We then describe our aggregator øws, which explicitly identifies and then accounts for these properties.
Baseline observations To understand how to aggregate Pxx), we present a set of observations on CB, RTE, and WSC. For each, we compare two baselines for constructing P: (1) Pr: varying the prompt template with no overlap in the in-context examples, and (2) PE: varying the in-context examples for fixed prompt template, all with P = 5. We observe the following properties on P:
1. Varied overall accuracies: While prompts PF may seem more similar than those in PT. the gap between the best and worst p: E P is large in both cases - 1% for PF. and 9.6% for PT.
2. Varied class-conditional accuracies (Zhao et al., 2021): Beyond overall prompt accuracy, the average variance of class-conditional prompt accuracies is 9.7% across the tasks and baselines.
Highly-correlated outputs: Prompt predictions have dependencies among each other. The Jaccard index over sets averaged across tasks is 42.2 for PE and 39.9 for Pr. For reference, two prompts that produce i.i.d. errors and have 60% accuracy each would have a score of 0.25.
The three observations present challenges in aggregating predictions via simple approaches like MV. MV tends to do better than using one prompt, but it weights all prompts equally and treats them independently. Such an aggregation method may be sufficient over certain collections of prompts but is not reliable across general P that may exhibit the three properties we have observed.
AMA Aggregation Given the varied accuracies and dependencies among prompt (rompt()-chaiil) (>chhiins we draw on recent work in weak supervision (Ratner al., 2017), which can aggregate outputs while accounting for the accuracy and dependency properties without relying on labeled data. We learn probabilistic graphical model on Pro,e(y: P(z)) and define the aggregator as dww( arg max Pra.e (yy P(x)). G = (V, E') is dependency graph where V = {y, P(z) and E is an edgeset where (pi()), (x)) E E iff p1(2) and p3 (I) are conditionally dependent given y. 0 are the accuracy parameters for P (a)). Since lack labeled data y, we cannot estimate G or 0 directly from D. Our procedure is as follows:
1. We use structure learning approach from Varma et al. (2019) recover the dependency struc- ture G using (az) applied D.
2. We use G, D, and (zz) to learn the accuracies of the prompts P from Ratner et al. (2018).
3. We compute Pro.a(yl P(r)) and aggregate our predictions.
The key insight that the inverse covariance matrix on V, Σ-1, is graph-structured, meaning that Σ,j 0 iff px( and (z) are conditionally independent given y. This property yields systems of equations on V from which we can recover dependencies and accuracies.
Algorithms summarizing the end-to-end AMA procedure are in Appendices D and E respectively.
4 INFORMATION FLOW IN AMA
Before evaluating end-to-end quality, we look at simple information theoretic metric to understand the contributions of the individual components and Φ in the prompting procedure.

---

Published as a conference paper at ICLR 2023
Conditional Entropy with Model Scale
Conditional Entropy with Aggregation Method 0.70
0.70
HrY/ | |Yhhat)
0.65
[-hhtt
0.65
0.60
0.60
L-0 shol
k=2
0.55
kod
AMA (951
0.50
HY Prompts]
0.50
HIY| Prorph))
NV
8.0
8.5
9.5 10.0
Model Parameters (Log-Scale)
Number prompts
Figure 4: (yll)) as function of model size E {125M, 1.3B, 6B, 20B} and prompts with A: = {0, 2, 4, 8} in-context examples (Left), and the aggregation strategy over the prompts, majority vote (MV) and weak su- pervision (WS) with the 6B model (Right). (=)((=)) is measured in both plots. Plots over RTE with GPT-J-6B; each k-shot point is the average of seeds. Plots for BLOOM models are in Appendix C.
Information flow metric Specifically, we examine the conditional entropy. Hy ū), which mea- sures the amount of uncertainty remaining in the true label y given a prediction ÿ. Intuitively, H (vl)) will be low when I encodes information relevant to y. In our setting, ğ - (P(()) is dependent on the two components of the prompting procedure, the prompts P and aggregator Φb. The following simple decomposition of H (vll) enables studying the contribution of each component:
H (vl
H(y)P()) + Hlyl)) ( P(x))
Controlled by P prompi quality Controlled by aggrogation method &
The first term (y) (xx). Hy () depends on the quality and quantity of the individual prompts in P(r) (since Hy P(r)) < H (vlp(r))) A set of prompts that contains relevant information for contributes to a low (ylg). The second term H(yl0) H(yl P(z)) shows that H Hyll) depends on how the aggregation step compresses the information in ((r) to predict ÿ. An aggregator & that more accurately matches the true Pr(yl P(z)) reduces the information loss in the compression step.
Evaluation We use (1) to evaluate our proposed solution AMA. First considering H (y) P(x)), in Figure 4 (Left) we observe AMA outperforms k-shot baselines with expected scaling in terms of both individual prompt()
quality (as shown by AMA No Agg) and their quantity.
Next we consider the gap term H (y) () - (y) P (z)). It enables us to understand why MV is insuffi- cient: it compresses information from P according to a specific construction of Pr(y, P(x)), for which (x)) 1 (z(2)) y for all j, E [m]. and Pr(p.(z) = cly = c) for e E y is a single better-than- random constant across i and c. When the true distribution is vastly different- as is common-thin results in a large gap between the optimal H (y) P(=)) and Hyy (Mv)) in Figure 4 (Right). WS can improve & over the standard MV baseline reduce the information loss // (v)ÍAMA)
(y) P(x)).
In addition to empirical measurements, we can provide a theoretical characterization for the infor- mation flow. In Appendix F, we express H (yLÍAMA)) in terms of the individual prompt accuracies under the standard weak supervision model (i.e., Ising model on y and P(x) (Ratner et al., 2018)).
There has been recent interest in how LLMs improve primarily along the three axes of parameter scale, training data, and compute (Kaplan et al., 2020; Hoffmann et al., 2022; Wei et al., 2022b). In Figure 4, as we increase the number of prompts to be aggregated, the conditional entropy reduces. Prompt aggregation may be another useful axis for understanding LLM scaling performance.
5 RESULTS
We evaluate AMA on 20 popular language benchmarks used in Brown et al. (2020); Sanh et al. (2022). We report results across 14 unique LMs including 4 model familics (Nco (Black et al., 2021), OPT (Zhang et al., 2022), BLOOM, and TO (Sanh et al., 2022)) spanning 125M-175B parameters. We aim to validate whether AMA provides consistent lift across diverse tasks (Section 5.1) and model families (Section 5.2), and reliably aggregates the predictions across prompts (Section 5.3).
Experimental details In Table 1, we compare to the www-shot-prompter GPT3-175B LM using the numbers published in Brown et al. (2020); Zhao et al. (2021), given the model's popularity and strong off-the-shed quality. Brown et al. (2020) uses A E [32..70] and Zhao et al. (2021) uses A 8 depending on the task, providing a challenging baseline. We evaluate using the same tasks on which GPT-3 was originally evaluated: SuperGLUE (Wang et al., 2019), NLI (Mostafazadeh et al., 2017; Nie et al., 2020), classification (Zhang et al., 2015; Socher et al., 2013; He & McAuley, 2016), and QA tasks (Kasai al., 2022; Kwiatkowski et al., 2019; Berant et al., 2013; Dua et al., 2019).
7

---

Published as a conference paper at ICLR 2023
Model
Neo Few-Shot
Neo (QA)
Neo (QA WS)
GPT-3 Few-Shot 175B
Fine-Tuned
# Params
6B
6B
6B
Natural Language Understanding
BoolQ
64.8(a)
64.8
67.2+0.0
77.5 77.5(u2)
91.0
CB
(3)2(3)
83.3
83.940.0
82.1(32)
96.9
COPA
(9.0(3)
58.2
84.0+0.0
(32)
94.8
MultiRC
60.8(3)
58.8
63.8
74.8(32)
62.3
ReCoRD
75.5(3)
74.5
74.410.0
89.0(
92.5
RTE
(8.0(3)
61.4
74.710.0
72.9(32)
92.5
StoryCloze
(6.0
76.3
87.8
87.7
91.8
WIC
48.8(3)
59.0
61.340.22
(323
76.1
WSC
50.0(3)
74.7
77 77.8+0.0
75.0
93.8
Natural I. Language Inference
ANLIRI
32.3(3)
34.5
37.8
36.8(50)
73.8
ANLI R2
33.
35.4
37.940.2
34.0
50.7
ANLI R3
33.4(3)
37.0
40.940.5
40.2(50)
48.3
Classification
AGNews
74.5(3)
83.7
86.420.0
79.1
94.3
Amazon
53.1(a)
67.0
68.620.0
41.9(s)
90.5
DBPedia
50.7(a)
81.4
83.940.0
83.2(s)
99.0
SST
94.5(3)
94.5
70.0
95.6(8)
97.5
Question Answering
DROP
32.3(3)
51.0
51.640.0
36.5/20)
88.0
NQ
13.7(3)
19.7
19.640.0
29.9(64)
44.5
RealTimeQA
31.T())
34.7
36.0.0.0
35.4(1)
WehQs
31.3())
41.2
44.140.0
41.5
45.5
Table I: AMA results for the GPT-J-6B parameter model compared the few-shot GPT3-175B. The GPT- reported Brown al. (2020); Zhao
175B numbers examples is in parentheses. Note prompts can abstain from predicting, which can lead to lower average numbers for QA, such as on COPA. For QA tasks and ReCoRD, we report the majority vote score, as applying WS is complex for open-domain tasks. Results for the BLOOM 7. parameter model are in Appendix 2.
(2021), where the numbers in-context
For AMA use 3-6 prompt()-chaing to generate predictions per input. We model the dependen- cies and accuracies of each compp-preeiction per task, without using any labeled training data, to obtain the final aggregated prediction per input via weak supervision (WS). We report both the erage performance over the prompt()-chaine (QA) and with AMA's WS aggregation (QA + WS). We report QA WS across random seeds for the model. Further details in the Appendix.
5.1 MAIN RESULTS
We report benchmark results in Table comparing the open-source GPT-J-6B and few-shot (k E (32..70) GPT3-175B. We find that the open-source 6B parameter model exceeds the average few-shot performance of the GPT3-175B model on 15 of 20 benchmarks. Overall, AMA gives a 37.6% improvement over the parameter model's few-shot (k: 3) performance achieve this.
We find that AMA provides the most lift on tasks where all requisite knowledge is included in the task input (e.g., reading comprehension) and that largely rely on model's NLU abilities. The lift lower tasks that rely the LMs memorized knowledge (e.g. closed-book QA). However, on the closed-book WebQ task where the answers are likely seen during pretraining, we find that prompting the LM generate relevant context, and then answer the original question using the generated context is effective. That being said, the closed-book NQ task shows there are limitations.
We similarly see limitations when tasks cannot rely the latent knowledge. We observe small performance gap between model sizes on RealTimeQA, which includes questions that have tempo- rally changing answers that are less likely be memorized. Similarly, for tasks requiring domain knowledge, e.g. the "Amazon Instant Video" class in the Amazon task, all model-sizes achieve near-0 performance. We provide extended error analysis Table results in Appendix
5.2 EVALUATION ACROSS MODELS
Benchmark results We evaluate the lift from AMA over out-of-the-bot few-shot performance across different sizes of four open-source LMS (Neo, OPT, BLOOM, and TO) across 7 tasks (4
We do not rank mlsscsaasiiiiattoo scoring, which is commonly used (Brown et al. (2020); Sanh et (2022)) reduce task complexity, harring explicitly multiple-choice tasks (ReCORD, StoryCloze and COPA).
8

---

Published as a conference paper at ICLR 2023
chmakk Results
Selection
Extraction
Text Generation
Question Generation
Absolite AMA Lit
0.20
Dignostic Score
0.8
0.6
0.30
0.4
0.05
0.2
0.0
18
10:2081758
<16
T0.38
T0_38<16
Size (Parametors)
Size
Size
Size
Size
<18
18
6-78
T0_38
(a) Absolute lift vs. model sizc (parameters).
(b) Average diagnostic score vs. model size (parameters).
Figure 5: Evaluation across model sizes for diagnostics and benchmarks. report the absolute lift from AMA over fow-shot (& 3) performance, averaged over 7 tasks with 95% confidence intervals (Left). Diagnostic plots ordered by the amount of lift models of the size-category see the benchmarks (Right).
NLU, NLI, I classification). In this analysis, we want to understand the effectiveness of AMA's prompt reformattings across models and report the average prompt performance the 3-6 promit()-chains used per task. Excitingly, the AMA prompt ()-chains apply quite generally. We observe 10.2% 6. absolute lift across models (21.4% = 11.2% relative lift) on average across models and tasks, as shown in Figure 5a (a). We observe the absolute lift increases with model size and levels out. The average absolute (relative) lift by model family (across tasks and sizes) is 11.0% (24.4%) for Neo, 11.0% (23.4%) for BLOOM, and 11.9% (22.7%) ) for OPT, though only 2.9% (8.3%) for TO. TO is popular open-source (non-GPT) LM, which was fine-tuned on prompt-output pairs and transfers well to new tasks in zero-shot fashion.
Diagnostics for understanding AMA lift We next provide a set of diagnostics to better under- stand the model reasoning skills that correspond to the different degrees lift from AMA, including TO's limited benefit from AMA's 's prompt()-chiins The diagnostics measure four basic operations required in AMA question generation, answer generation, answer selection, and extraction. For each operation, we create 1-3 tasks each with 50 manually-laheled samples (See Appendix G).
We measure the average performance across each operation across different sizes of 4 model types (Neo, OPT, BLOOM, and TO). We group models and sizes into four buckets of TO (3B parameters) and GPT models (< IB, 1B, and 6 7B parameters). Figure 5b shows results where the buckets are ordered by their average AMA lift across the tasks from Section 5.2, meaning TO (3B) sees the least lift while 6 7B GPT models realize the most lift. We find that overall, models with higher performance across the four operations see more lift with AMA. TO performs poorly on the generative tasks, indicating the importance of text and question generation for AMA.
5.3 EVALUATION AGAINST OTHER AGGREGATION METHOOSS
We compare our WS aggregation approach with the standard unsupervised approach, majority vote (MV), on prompt()-chains. We find that AMA can achieve up to 8.7 points of lift over MV, and matches or outperforms MV on 16 out of 20 tasks. On the remaining tasks, we perform worse than MV by at most 1.0 points. We also examine the effect of modeling dependencies in WS. We find that on 9 tasks, our approach recovers dependencies the data (rather than assuming conditionally independent (z)), which improves performance by up to 9.6 points and an average of 2.2 points. We provide more details and evaluation against labeled data baselines in Table 7 (Appendix B.5).
Next, we evaluate TO on zero-shot prompts from the public PromptSource (Bach et al., 2022), which are better aligned with how this model has been trained. Using the off-the-shelf prompts from PromptSource for 4 NLU tasks which TO heldout during training, we see average lift of 6.1 points when applying weak supervision over these prompts. Details are in Appendix B.3.
CONCLUSION
In this work we introduce ASK ME ANYTHING PROMPTING which scalably obtains multiple effec- tive prompts given a task input and combines the intermediate answers to these prompts using weak supervision to provide the final answer. The improvements of AMA stem from observations on the effectiveness of open-ended questions over restrictive prompts, and the ability to model the vary- ing accuracies and dependencies across collection of prompts using veak-supervision. We hope this work highlights the importance of the prompt structure and encourages future work to improve the capabilities of small and open-source models.

---

Published as a conference paper at ICLR 2023
7
REPRODUCIBILITY STATEMENT
We release prompts and code for reproducing all benchmark results for few-shot and AMA prompt- ing, and our diagnostic evaluation splits here: https: https:///itth/i /ttp//gggtttttteeee hwwp/Haayeeeeaccce ama prompting.
8
ETHICS STATEMENT
We intend for AMA aid practitioners in their exploration and of LLMs-especially smaller, open-source LLMs. However, we recognize that AMA could be used to perform harmful or uneth- ical tasks. AMA is a proof-of-concept; has error-modes and we recognize the inherent risks to using LLMs. Detailed discussions of these risks are in Bommasani et al. (2021); Weidinger et al. (2021).
ACKNOWLEDGEMENTS
The computation required this work was provided by Together Computer (https: // together, /). We are grateful to the Numbers Station (https: / crmberrstatton ion. ai/), Snorkel (https: (ittps::/snorke)) /ttp//oooooo.ee/ee/oeoeeee.eeeet aii)), Stanford Center for Research on Foundation Models (https: //tf:///////m//mm/ anford. edu/), and Stanford HAI (attps: http::/hhaaaaaaa.....ee stanfsstaoorrd edu/) organizations for the resources that supported this work. We thank Fred Sala, Karan Goel, Maya Varma, Joel Johnson, Sabri Eyuboglu, Kawin Ethayarajh, Niladri Chatterji, Neha Gupta, Alex Rat- ner, Percy Liang, and Rishi Bommasani for their helpful feedback and discussions. We grate- fully acknowledge the support of NIH under No. U54EB020405 (Mobilize), NSF under Nos. CCF1763315 (Beyond Sparsity), CCF1563078 (Volume to Velocity), and 1937301 (RTML); US DEVCOM ARL under No. W911NF-21-2-0255 (Interactive Human-Al Teaming); ONR under No. 1100141112266 (Unifying Weak Supervision); ONR 400014-20- 1-2400: Understanding and Ap- plying Non-Fuclidean Geometry in Machine Learning; NO001 4201 2275 (NEPTUNE); NXP, Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASE, Accenture, Erics- son, Qualcomm, Analog Devices, Google Cloud, Salesforce, Total, the HAI-GCP Cloud Credits for Research program, the Stanford Data Science Initiative (SDSI), and members of the Stanford DAWN project: Facebook, Google, and VMWare. SA is supported by a Stanford Graduate Fellowship. LO supported by an Intelligence Community Postdoctoral Fellowship. The U.S. Government is autho- rized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions recommendations expressed in this material are those the authors and do not necessarily reflect the views, policies, or endorsements, either expressed implied, of NIH, ONR, the U.S. Government.
REFERENCES
Simran Arora and Christopher Ré. Can foundation models help us achieve perfect secrecy?, 2022.
URL peetts://arxive org/ abs 2205 13722.
Stephen H Bach, Victor Sanh, Zheng-Xin Yong. Albert Webson, Colin Raffel, Nihal V Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, et al. Promptsource: An in- tegrated development environment and repository for natural language prompts. arXiv preprint arXiv:2202.01279, 2022.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantie parsing Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1533-1544, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL wwtph:tt///// www. ac wewweb.... org/ argganthhloog//13 1160.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autore- gressive Language Modeling with Mesh- TeessoToos,r,,o, March 2021. URL ht kttps: : /oi.. org/ 10. 5281 zenodo. 5297715. If you use this software, please cite it using these metadata.
10

---

Published as a conference paper at ICLR 2023
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:218..07258 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Emmanuel J. Candès, Xiaodong Li. Yi Ma, and John Wright. Robust principal componenn analysis? ACM, 58(3), jun 2011. ISSN 0004-5411. doi: 19114//99703922299799959 URL https: ///doi.org/// 1145/1 970392. .00.009..1990000 1970395.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.0m 023111, 2022.
Christopher Clark, Kenton Lee, Ming-Wor Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2924-2936, 2019.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv:2110.14166V2 2021. URL https: //arxv/aavvvv//aa///aeeeaeeaaav wtg/pdf////eoooooooo///poppppppp pdtppdd////// 22100211.... 14168.1. pdf.
Antonia Creswell, Murray Shanahan, and Irina Higgins. salectioniinfcrence Exploiting large lan- guage models for interpretable logical reasoning. arXiv preprint arXiv:2205.097//, 2022.
Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmenthank: In- vestigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pp. 107-124, 2019.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proc. of NAACL, 2019.
Kawin Ethayarajh. How contextual are contextualized word representations? comparing the geom- etry of bert, elmo, and gpt-2 embeddings. EMNLP, 2019.
Daniel Fu, Mayee Chen, Frederic Sala, Sarah Hooper, Kayvon Fatahalian, and Christopher Re. Fast and three-rious:
Speeding up weak supervision with triplet methods. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 3280-3291. PMLR, 13-18 Jul 2020. URL https: http:s:///poocecinggs.oo mlr
html.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2021. URL https:://axxt /ttp//aaivvvrrr/.e/e. abs/ / 22011.1110.. 00027.
Ruining He and Julian McAuley, Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In proceedings of the 25th international conference on world wide web, pp- 517, 2016.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hen- nigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen. Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optima large language models, 2022. URL https: / //////aaa /arxivv org/ abs/ 2203.15 15556.
11

---

Published as a conference paper at ICLR 2023
Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi Choi, and Luke Zettlemoyer. Surface form competition: Why the highest probability answer isn't 't always right. arXiv:2104.08315, 2021.
HuggingFace, Nov 2021. URL https: hettpps://ugggingfac..... co/
htt.://nnna.e....cdeee...... models.
Zhengbao Jiang, Frank F, Xu, Jun Araki Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics (TACL), 2020.
Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. Maieutic prompting: Logically consistent reasoning with recursive explanations, 2022. URL https: arxiv. org/ abs
Jared Kaplan, Sam McClandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv:2001.088. 003601, 2020.
Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah Smith, Yejin Choi, and Kentaro Inui. Realtime qa: What's the an- swer right now? arXiv preprinl arXiv:2207 22t.:2332..3.33,,, 2022.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Look- ing beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 252-262, 2018.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Al- berti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wa Wei Chang. Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics, 2019.
Andrew K. Lampinen, Ishita Dasgupta, Stephanie C. Y. Chan, Kory Mathewson, Michael Henry Tessler, Antonia Crwswell, James L. McClelland, Jane X. Wang, and Felix Hill. Can language models learn from explanations in context?, 2022.
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. Thir- teenth international conference the principles of knowledge representation and reasoning. 2012.
Chin-Yess Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. URL https: / //aalaaantologggoo.. org/ / W04 1003..
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-37, 2021.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannanch Hajishirzi. Reframing instructional prompts to gptk's ' language. arXiv preprint arXiv:2109.00 07830, 2021.
Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James Allen. Lsdsem 2017 shared task: The story cloze test. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, pp. 46-51, 2017.
Avanika Narayan, Ines Chami, Laurel Orr, and Christopher Ré. Can foundation models wrangle your data? arXiv preprint XiX.:22005009911. 2022.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Ad- versarial nli: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2020.
OpenAI, Nov 2021. URL https:/
12

---

Published as a conference paper at ICLR 2023
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint WrX.i:2203.0215.. 2022.
Mohammad Taher Pilehvar and Jose Camacho-Collados, Wic: the word-in-context dataset for eval- uating context- meaning representations. arXiv preprint arXiv: 2018.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't 't know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784-789, 2018.
Alexander Ratner, Stephen H. Bach, Henry Ehrenberg. Jason Fries, Sen Wu, and Christopher Ré Snorkel: Rapid training data creation with weak supervision. Proceedings of the VLDB Endowment, 1163):29--22,, nov 2017. doi: 10.4477831577943315797, URL https:
1tt.oooooo11117778833335757300 317777
Alexander Ratner, Braden Hancock, Jared Dunnmon, Frederic Sala, Shreyash Pandey, and Christo- pher Ré. Training complex models with multi-task weak supervision, 2018. URL https:
02840.
Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data pro- gramming: Creating large training sets, quickly. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, Cur- ran Associates, Inc., 2016. URL https: / https:://tocceeeeee http://proeeennggee///eee/ neurips. ccpppp/r//pppp11.. file/ 9e1666799e86455555 (eeeee22226ccccee6225555 wawp:e2e...aaaaaer..p..p
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI spring symposium: logical formaliza- tions of commonsense reasoning, pp. 90-95, 2011.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, al. Multitask prompted training enables zero-shot task generalization. In The Tenth International Conference on Learning Representa- tions, 2022.
Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. arXiv:2009.07//812, 2021.
Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida I. Wang. Natu- ral language to code translation with execution. arXiv:2204.114442, 2022. URL https:
/ //arxivv.ott org. http//f/2222/ 21t.//24..1111 11444 pdf.
Ryan Smith, Jason A. Fries, Braden Hancock, and Stephen H. Bach. Language models in the loop: Incorporating prompting into weak supervision. UrXiv:2205.023181) 2022.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642 Seattle, Washington, USA, October 2013. Association for Computa- tional Linguistics. URL https: www. aclweb. org/ ant /D13 D- 1170.
Paroma Varma, Frederic Sala, Ann He, Alexander Ratner, and Christopher Re. Learning depen- dency structures for weak supervision models. In Kamalika Chaudhuri and Ruslan Salakhutdi- nov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 6418-6427. PMLR, 09-15 Jun 2019. URL htttss://proceeciing
mlr wollr.presse v97 vataaaaaallll...aamaaaaaa. 9a .tmll
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. SuperGI.UE: A stickier benchmark for general- purpose language understanding systems. arXiv preprint 1905.00537, 2019.
Ben Wang and Aran Komatsuzaki. Gpt-j-6b: 6 billion parameter autoregressive language model, 2021.
13

---

Published as a conference paper at ICLR 2023
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le Le, Ed H. Cho, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. 2022a.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganch Kordi, et al. Benchmarking generalization via in-context instructions on 1,600+ language tasks. arXiv, 2022b.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan
Du, Andrew M. Dai, and Le Quoc V. Finetuned language models are zero-shot learners.
1.x.i:2200.01652, 2022a.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang. Jeff Dean, and William Fedus. Emergent abilities of large language models, 2022b. URL wttpt:tpp//o /arxivv.v..... org/ abs/ 2206. 07682.
Jason Wei, Xuezhi Wang, Dale Schourmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022c.
Laura Weidinger, John Mellor. Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, al. Ethical and social risks of harm from language models. arXiv preprint arXiv:21 2021.
Tongshuang Wu, Michael Terry, and Carrie Jun Cai. Ai chains: Transparent and controllable human- ai interaction by chaining large language model prompts. In CHI Conference Human Factors in Computing Systems, pp. 1-22, 2022.
Jingfeng Yang, Haoming Jiang, Qingyu Yin, Danqing Zhang, Bing Yin, and Diyi Yang. Se- qzero: Few-shot compositional semantic parsing with sequential prompts and zero-shot models. arXiv:2205.07381v.. 2022. URL https: /arxivv. worooorrtt.e http/dd//2220 0ww00.33111000 .df.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: Self-taught reasoner bootstrap- ping reasoning with reasoning. arXiv:2203. 14465v2, 2022. URL https://p/// /ttp::////a/v////////
pdf/2205 14.65.46.....
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. ReCoRD: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint 1810. 1 12885, 2018
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettle- moyer. Opt: Open pre-trained transformer language models, 2022. URL https://arxis: org/abs 2205.2052222 01068.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS, 2015.
Tony Z. Zhao, 1 Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In arXiv:2102.0069002, 2021. URL
https:/ http/rxivvv....a org. wwtp://p//////////////////////// 09690. pdf.
Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schu- urmans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-most prompting en- ables complex reasoning in large language models. arXiv:2205.1022 2022. URL https: /20.:///...../........22 10625.
APPENDIX
A EXPERIMENT DETAILS
We A100 NVidia GPUs all experiments.
14

---

Published as a conference paper at ICLR 2023
A.1 MODELS
We evaluate over 4 model families: TO, BLOOM. Neo, OPT, and GPT3. In evaluations, the following model family variants: Neo (GPT-Neo- 125M, GPT-Neo-I.3B. GPT-Neo-6B, GPT- Neo-20B), BLOOM (BLOOM-560M, BLOOM-1.77, 7B, BLOOMM7.1BB, BL.0MM-176B) OPT(PPT- 125M, OPT-1.3B, OPT-6.7B, OPT-13B, OPT-175B), TO (TO-3B), and GPT-3 (davinci). We down- load TO, BLOOM, OPT, and Neo models from the HuggingFace Model Hub (HuggingFace, 2021). All inference calls to the OpenAl Davinci model were made using the OpenAl API davinci end- point (OpenAl, 2021), the original GPT-3 175B parameter model used in (Brown et al., 2020). We access these models by passing our input prompts to the endpoint for per-sample fee.
A.2 METRICS
For RealTimeQA, the reported GPT-3 performance in Kasai et al. (2022) is reported over the text www--ddvviiiiiieeeee 002 API endpoint. Given that all our GPT-3 evaluations are over davinci, we re-evaluate the GPT-3 performance on RealTimeQA using the davinci endpoint and the few- shot prompt from RealTimeQA."
We follow the metrics used in Brown et al. (2020). All tasks are scored using matching accuracy except for DROP/RealTimeQA that use text fl, WehQ/NQ that use span overlap accuracy, and MultiRC that uses fl accuracy.
A.3 WEAK SUPERVISION
For each task, unlabeled samples from the training set. The additional data used less noisy parameter esti- mation, although in section B.5 we find that this data may not be necessary. We run the structure learning part of the weak supervision algorithm (for G) with the default parameters from Varma et al. (2019). If the recovered sparse matrix has all entries greater than 1, we pass in an empty edgeset to the next step of learning 0 (e.g., data too noisy to learn structure from); otherwise, we pass in the edge with the highest value the sparse matrix.
an unlabeled dataset constructed from the test set as well as 1000 additional
B ADDITIONAL RESULTS
B.1 BLOOM MODEL. RESULTS
In Table 2, we provide results using the BL00M-7.1B parameter model over all 20 benchmarks. We observe consistent lift over few-shot performance using AMA, though the performance remains below that of the comparably sized GPT-J-6B parameter model reported in Table 5.1. We note that the few-shot results for BLOOM 7. IB also often lower than the few-shot results for GPT-J-6B,
B.2 LARGE MODEL RESULTS
In Table 3, we provide results using the large open-source language models including the 175B parameter BLOOM and OPT models. We observe that even at the large model scale, AMA provides performance improvements over the few-shot baseline.
B.3 TO COMPARISON
While we observe that TO performs poorly on synthetic tasks correlated with the prompt()- (see Figure 5), we find that aggregating over zero-shot instructions in Prompt-Sourt provides lift Bach et al. (2022). Specifically, when evaluating over unique prompts for CB, WIC, WSC and RTE respectively, we find that aggregating with MV yields an average lift of 3.7 accuracy points and aggregating with WS gives an average lift of 6.15 accuracy points (see Table 5).
Phttps://gi thub. cwm. real meqa/ rea wttpa//eeaattmmmia.ttm timega_puble
15

---

Published as a conference paper at ICLR 2023
Model
BLOOM Few-Shot
BLOOM (QA)
BLOOM (QA WS)
GPT-3 Few-Shot 175B
# Params
7.1B
7.1B
7.1B
Natural Language
Understanding
BoolQ
61.9 (3)
66.0
67. 07.9
77.5 7725
CB
41.1(3)
67.3
77 40.9
82.1(39)
COPA
65.0:3)
57.8
74.0+0.5
92.0 (32)
MultiRC
21.3:3)
56.9
59. 39.7
8328
ReCoRD
71.8(3)
70.0
69.8
89.0 (920((2)
RTE
53.1 (3)
61.2
67.5 67.5110.0
72.9 (32)
StoryCloze
(9.4(3)
70.4
80.8
87.7 87.7
WiC
50.8:)
57.8
60.0
55.3
WSC
61.5
62.2
60.640.0
75.0
Natural Language Inference
ANLIRI
(4.9(3)
32.9
34.60
36.8(50)
ANLI
33.6(3)
34.1
34.640.0
35)0
ANLI R3
32.3(3)
35.0
36.640.7
40.2
Classification
AGNews
68.3(3)
74.4
81.7
79.1
Amazon
58.1(3)
62.8
65. 20.0
41.9(s)
DBPedia
72.3(3)
62.8
69.8
83.2 (e)
SST2
56.4(3)
90.2
91.040.0
95. 6,8)))))
Question Answering
DROP
30.5;3)
67.9
67 67.9
36.5(20)
NQ
12.1(3)
15.2
15. 15.1
(9.9(64)
RealTimeQA
21.8(3)
27.7
29.010.0
35.4(1)
WebQs
28.0(3)
34.8
34.8
41.5 (145(66)
Table 2: AMA results for the BLOOM-7.18 parameter model compared to the few-shot GPT3-175B. The GPT- 175B numbers as reported in Brown et al. (2020), where the numbers shots parentheses, and the classification task baselines are from Zhao et al. (2021). DBPedia results are on randomly subsampled test set of 10k points.
Model
BLOOM Few-Shot
BLOOM AMA
OPT Few-Shot 175B
OPT AMA
Fine-Tuned
# Params
175B
175B
175B
Natural Language
Understanding
CB RTE WSC
64.3
82.1
96.9
57.4
66.5
53.8
67.7
92.5
68.3
71.8
49.0
76.0
93.8
WIC
51.7
59.4
51.9
58.4
76.1
Natural Language Inference
ANLI RI
37.3
37.4
37.9
37.9
73.8
ANLI R2
34.4
36.1
37.2
37.2
50.7
Classification
SST
45.6
95.2
55.6
95.8
97.5
Table 3: AMA results for the large open source models. These are raw results corresponding Figure
B.4 ABLATING THE PROMPT REFORMATTING AND AGGREGATION COMPONENTS OF AMA
Here we study the degree to which both prompt re- formatting and aggregation are required to achieve high quality, extending the observations Section We produce few-shot prompts that follow the same template by varying the & == 3 in-context prompt examples. In this process, the proposed AMA prompt reformatting is not applied. We apply each of the few-shot prompts and aggregate the results using majority vote MV and WS. We observe aggregation alone provides lift over the average prompt performance in Table 6. However the gap to AMA performance remains large. Aggregation and forrrmatt are both critical and complementary the end-to-end solution.
B.5 WEAK SUPERVISION ABLATIONS
Comparison to other aggregation baselines Table compares AMA's aggregation method against several other baselines for aggregating prompt()-chinn including majority vote. We com- pare against weighted majority vote (WMV), where use labeled data weight according each
16

---

Published as a conference paper at ICLR 2023
Model
TO TO 3B AMA
PP AMA
CB
50.0
44.6
RTE
51.8
58.7
WSC
66.0
62.5
WIC
49.7
51.9
SST
47.4
47.2
ANLI-RI
33.3
32.1
ANLI-R2
32.9
33.6
Table 4: AMA results for TO models.
Task
CB
WIC
WSC
RTE
TO (3B) Mean
45.4
50.7
65.1
64.6
TO (3B) 10 Formats MV
60.7
50.5
68.3
60.6
TO (3B) AMA MV
50.0
49.5
64.4
49.5
TO (3B) 10 Formats WS
60.7
50.8
69.2
69.7
TO (3B) AMA WS
50.0
51.4
66.4
59.2
Table 5: Performance of TO as reported in Sanh et al. (2022) compared to majority vote (MV) and weak supervision (WS) over 10 different prompt formats in Prompt-Source. When using the Prompt-Source prompts, the average lift across tasks is 3.6 points for MV and 6.1 points for WS.
prompt's accuracy by constructing Φwwv (xx)) E exp(- (p:(2) y)- E; is the error of prompt p: on training set of 1000 examples, and 1j is a temperature hyperparameter, for which we perform sweep over [0.25, 0.5, 1, 4, 8, 16, 32] using 20% validation split. We also compare against a simple strategy using the prompt that performs the best on labeled set of data (Pick Best). Finally, AMA (no deps) is method when pass in empty edgeset to the algorithm in Ratner et al. (2018).
Varying amount of additional data We study the effect of varying the amount of additional unlabeled training data that is used in learning the probabilistic graphical model on y. Pxx). On three tasks (RTE, WSC, and AGNews) averaged over 5 runs, we run AMA with 100%, 50%, 20%, 10%, and 0% of the additional dataset while still evaluating on the fixed test dataset. Figure shows AMA's 's accuracy versus the amount of additional unlabeled data used. We find that even without any of the additional data, average accuracy does not decrease WSC AGNews, and only decreases by 0.4 points on RTE, still outperforming GPT3-175B few-shot. This suggests that the additional data is not necessary for AMA's performance.
Latency of Weak Supervsion Over RTE, WSC, and AGNews, we find that WS (both learning the graphical model and aggregating outputs) takes an average 13.0 seconds when dependencies are not modeled. When dependencies are modeled in RTE (as dependencies are ignored in WSC and AGNews because they both exhibit dense recovered structured matrices), the algorithm takes average of 84.3 seconds to As point of comparison, we include Table 8 which shows the time seconds for running inference with the GPT-J-6B model on the same tasks. The latency introduced by running weak supervision is comparatively low.
B.6 ADDITIONAL AMA BASELINES
Here we compare AMA Self Calf-Consistency Wang et al. (2022a), which is particularly relevant in that it also aggregates over multiple prompt outputs without requiring any additional supervised training. Self-Consistency builds on Chain-of-Though prompting Wei et al. (2022c), which pro- poses to guide the LM generate reasoning paths in addition to the final prediction. We use the exact prompts and overlapping benchmark tasks provided in the Appendix using GPT-J-6B and report the results in Table 9. For Self-Consistency, sampling as discussed in Wang et al. (2022a), using temperatures E (0.0, 0.3, 0.5, 0.7.
Wang et al. (2022a), temperature based
Overall, observe AMA outperforms Belf-Consistency at this model scale. This agrees with the results in Wang et al. (2022a) and Wei et al. (2022c), which report limited performance improve- ments for small LMs (<10B).
17

---

Published as a conference paper at ICLR 2023
Model
GPT-J Few-Shot
GPT-J Few-Shot
GPT-J Few-Shot Weak Supervision
GPT-J AMA Weak Supervision
Aggregation
Average
Majority Vote
Natural Language Understanding
CB
23.8
17.9
50.0
83.9
RTE
53.5
53.1
54.2
75.1
WSC
46.2
38.5
38.5
77.9
COPA
80.0
81.0
81.0
84.0
Natural Language Inference
ANLI R1
33,4
33.5
33.5
37.8
ANLI
33.2
32.9
32.2
37.9
ANLI R3
35.4
36.5
34.6
40.2
Classification
AGNews
70.3
70.7
75.0
86.4
Amazon
61.9
62.4
62.5
68.2
Table 6: Results from applying prompt aggregation via Majority Vote and Weak Supervision random few-shot (I: 3) prompts. Here we apply no prompt reformatting to the proposed AMA QA template.
# Prompts
Avg
MV
WMV
Pick Best
AMA (no dep)
AMA (WS)
No labels:
Natural Language Understanding
WSC
3
74.7
77.8
77.8
75.0
77.8
77.8 00.0
WiC
5
59.0
61.3
60.9
60.0
60.8
61.3 61.3
RTE
5
61.4
66.0
71.4
62.0
65.1
74.7
CB
3
83.3
82.1
82.1
83.9
82.140.0
83.9
MultiRC
3
58.8
63.8
63.4
63.4
63.7
63.8
BoolQ
5
64.9
65.9
67.2
68.3
65.9
67.2
CОРА
4
58.3
85.0
82.0
82.0
84.0
84.0
Natural Language Inference
ANLIRI
5
37.6
36.1
36.8
37.411.0
37.8
ANLI
5
35.4
36.3
36.0
36.0
38.7
37.9
ANLIRI R3
5
37.0
39.0
38.4
38.4
39.610.9
40.9
StoryCloze
6
76.3
87.9
81.8
81.8
82.240.0
87.8+p.0
Classification
DBPedia
3
81.4
84.1
83.9
82.2
83.9
83.9
SST2
3
94.5
95.7
95.7
95.2
95. 95.740..
95. 90.7
Amazon
3
67.0
68.6
68.6
67.3
68 68.61.0.
68.620.0
AGNews
3
83.7
86.5
84.2
83.8
86.4
86.4
Table 7: AMA Aggregation method ablation for the GPT-Neo-6B parameter model, as well as the number of prompt() -chaiss used for each task. For ReCoRD, and QA tasks (DROP, WebQs, RealTimeQA, NQ), 3 prompts each use majority vote as our aggregation strategy reported the (QA + WS) columns Table and Table
Task
Number of Examples
Total Inference Cost (seconds)
RTE
277
8310
WSC
104
3141
AGNews 7600
53200
Table 8: Total inference cost in applying the AMA prompt chains to achieve the results in Table 5.1, using the GPT-J-6B model.
C CONDITIONAL ENTROPY ABLATIONS
In addition to the plots for the Neo models which we present in Section 4, we measure the conditional entropy metric across 4 BLOOM models (see Figure (Left)) and for aggregation the BLOOM 7.1B model (Right). We evaluate the (560M, 1.3B, 7.1B, 175B) parameter BLOOM models on RTE and the k-shot points are each the average of 4 random seeds.
18

---

Published as a conference paper at ICLR 2023
WSC
0.77
AMA GPT:-4B1
0.80
Accuosy on Dood test catazet
0.88
GP13-1258 few Shot
0.79
GP13-1758 Shot
9.76
0.96
0.78
0.74
0.75
0.77
0.84
0.52
GPT3-1758 few Shot
AMA GBT
Acourayy
0.72
0.73
0.75
0.74
0.50
0.71
0.73
0.78
200
4.00
600
300
1000
100
200
500
400
500
200
4.00
600
300
1000
Additional untabieeeddee datapoints used
Additioral unlabeled dotaponnts used
Addiional uniabeled catapoints used
Figure 6:
Performance on RTE, WSC, and AGNews averaged over 5 runs when using varying amounts of
additional unlabeled training data for estimating Pr(y, P())) WS.
Baseline
Self-Consistency with Chain-of- Thouggt Ask Me Anything
Aggregation # Outputs
5
RTE
47.3
75.1
BoolQ
63.1
67.2
ANLI-RI
33.4
37.8
Table 9: Comparison between Self-Consistency Wang al. (2022a) and AMA using GPT-J-6B and the same number of prompts.
D AMA PROCEDURE
D.1 USER INPUTS TO AMA
To use AMA, the requirement from the user is to provide question() and answer() prompts. How- in this work, we provide many pre-written prompts for users to adopt off-the-shelf. We also note that many tasks reuse the exact same prompts (with the exact same in-context examples) to achieve the reported results. Because there is large search space in natural language prompt de- sign, we hope by providing users with the QA template, AMA helps reduce the required effort.
D.2 STEPS
Here we summarize the AMA procedure that was introduced in Section 3. The end-to-end procedure outlined in Algorithm 1. We further describe the prodedure line-by-line, using the following example input 24, which contains a textual statement and passage, and requires determining the statement valid based the information in the passage. Let x; include:
Statement: John went to the park.
Passage: John and his family went the theater to watch Jurassic Park.
Following step 2 in Algorithm 1, question() prompt is applied to the statement via the LM, to convert the statement to question such as the yes-no question "Did John go to the park?" or the open-ended question, "Where did John go?". Example question() prompts, which contain generic in-context demonstrations of statements and question-versions of those statements, are listed in Appendix K.
Next, an answer() prompt is applied (step 3 in Algorithm 1). The answer() prompts are constructed by having the LM answer the previously generated question (e.g.. "Did John go to the park?") using the passage in ø; See Appendix K for example answer() prompts. Applying the prompt, the LM generates answer aş, for instance "No" the yes-no question "Did John go the park?", or "theater" to the open-ended question "Where did John go?".
Next, discuss how the answer is mapped to a prediction y; in the task output space, for input I;, Note that there is no specialized mapping process in AMA- we simply score the exact match between the LM-generated a; and the gold label yı. The approach is detailed below:
Yes-no questions For yes-no questions, the ultimate answers are ultimately "yes" or "no" and we can also fit in-context examples questions with each of these two answer choices in the answer() prompt.
19

---

Published as a conference paper at ICLR 2023
Conditional Entropy with Model Scale
Conditional Entropy with Aggregation Method
HDY (-hhat)
0.6
0.6
0.5
0.5
k-4 shot
K- shot
(WS)
Agp- Prompts)
0.3
HYY Prompest
0.2
0.2
9.5
10.0
10.5
11.0
1
3
5
Model Parameters LLog-Scale)
Number of prompts
Figure 7: BLOOM model plots for the conditional entropy metric H(y ú) as function of model size (560M, 1.7B, 7B, 175B) and prompts with lt {0, 2, 4) in-context examples (I (Left), and the ag- gregation strategy over the prompts (majority vote and weak supervision) with the 7.1B model (Right). Plots are over RTE benchmark, and each k-shot point the average 4 random seeds.
Procedure 1: AMA End-to-Ene Procedure {22, 1- collection of m prompt()-chains P. A prompt()-che -chain contains question() and an answer() prompt.
Input: Dataset
Output: Predictions { |=
2: Given the input 2ú E 'D for each ()ompt()-chain P. E P, prompt the LM with question() to produce a question about r;. Across the m prompt()-chains, AMA produces m questions, Git, ... 7om for each input T;.
3: For each input F;, for each question dij. apply the answer() prompt to produce the LM prediction aij for 24. Across the prompt() m answers a,1, .... 0im are produced for each Z;.
4: Construct the weak-supervision based aggregation function (as specified in Algorithm 2), which takes the m LM generated answers da,, ..., 0im for the task inputs in D, and returns a single prediction O, for each Tj.
5: Returns: D. for all x; E D.
Open-ended questions, Open-ended output space For open-ended questions, the answers are open-ended. To facilitate scoring the answers to open- ended questions, our approach is to, when the LM generates the question "Where did John go?" using the question() prompt, to also have the LM generate the hypothesized answer, i.e. "theater", at the same time. See the RTE task section Appendix K for an example of such a prompt.
Open-ended questions, Restricted output space Tasks such classification require out- putting a "restricted" label for the input z,. For instance, is the article about "Sports" or "Technology" answers such as "Athletics" are invalid. For such tasks, we include the viable outputs in the answer() prompt to encourage the LM to select from the list.
Finally, the LM generated answer a; scored simply given an exact-match comparison to the gold label. In AMA, we aggregate over multiple answers per input. The different answers for the same input v, are obtained by applying different combinations of question() and answer() prompts, Le. differing in their in-context demonstrations. In this work, use 3-6 pairs of question() and answer() prompts per task. The suaa-supervisioo based aggregation step is applied once multiple predictions are collected for all inputs in the task, to produce single prediction per input. The aggregation algorithm is detailed in the next section.
E WEAK SUPERVISION ALGORITHM
We briefly explain the weak supervision algorithm used for constructing dws- Weak supervision models learn the latent variable graphical model on the distribution Pr(y, P(x)) using the dataset D, and aggregate votes using the learned distribution by setting ((x) = arg max, (r((P(()) Our key insight in our aggregation approach is to parametrize Pr(y. P(x)) so that we can capture variations in accuracy well as dependencies if they exist. The overall procedure of our aggregation is in Algorithm 2. Formally, we model Pr(y. P())) as a probabilistic graphical model with dependency graph G (V, F),, where V
{v. P(x)). If pa(ar) and py (2r) not conditionally independent
20

---

Published as a conference paper at ICLR 2023
Procedure 2: AMA Aggregation Method
1: Input: Dataset D = {x, 1 collection of prompt()-chains P. Output: Predictions {
2: Prompt the LLM with P to produce predictions ((z) per input D, constructing dataset
Dp E TRxx..
3: Learn G = (V, E) via structure learning on Dp (Algorithm 1 in Varma al. (2019)).
4: Learn Pra,o(y, Pz))) using Dp and G (Algorithm in Ratner et al. (2018)).
5: Construct aggregator Øws (P(x)) = arg max (((r)).
6: Returns: JAMA (x) for all x E D.
given y and the other prompt()-chains, then (p((z).p((z) E. E also contains edges (p:(x), for each E [m].
The algorithm uses P(x) and D to first learn the dependency structure & among prompts using the approach from Varma et al. (2019). The key insight from that work that the inverse covariance matrix Σ-1 over y and P(x) is graph-structured, meaning that Σ;j conditionally independent given y. The graph structure means that the inverse covariance over just P(x) decomposes into sparse and low-rank matrices, which can hence be estimated together using RobustPCA (Candès et al., 2011). and the sparse matrix can be used to recover the graph. Next, the algorithm uses the recovered G along with P(x) and D to learn the accuracies of the prompts with the approach from Ratner al. (2018). The key insight from that work is use the sparsity of E-1 to construct system of equations set equal to O that recover the latent accuracy parameters. Once the parameters of the distribution are learned, we can compute Pr and aggregate our predictions.
iff (z(z) and (z(x) are
F INFORMATION-FLLOW THEORETICAL RESULT
In equation 1, decompose Hly into (y ((x)) and (H) For AMA, suppose that the weak supervision algorithm exactly recovers Pr(y, P(x)). That is, jAMA is drawn from Pr(-IP(z)). Then, the second term (ylů) (y can be thought of as an irreducible error corresponding to how much information about is lost in converting P(z) into an i.i.d. y' randomly drawn from Pr(-IP(z)). (z)). Since y' is more likely to change values when this distribution has high entropy. the second term is correlated with our first term H (xx),, the amount of randomness in accuracies.
Pr(y (xr).. We thus focus on obtaining an expression for H (y) Pxr) in terms of individual prompt
We assume that y = 11. We model Pr(y, P(x)) as a probabilistic graphical model with dependency graph G E), where V = {v. P.=)). The density of Pr(y, P(x)) follows the following Ising model commonly used in weak supervision (Ratner et al., 2017; Fu al., 2020):
Pr P(x)) = z exp 0,y Σ (.p( Σ 0sp)(o))(())) G
(2)
(K.))
where Z is the partition function for normalization and (0,, O, Vie [m], Ois V (i,j) E}. Each 8, can be viewed as the strength of the correlation between y and pi(r), while each 0, can be viewed as the strength of the dependence between (z():) and (z)). We assume that 8g = 0, which
corresponds to Pr(y- 1)
We present our expression for (y)P(z)). Define = [0....., 0m) to be the vector of canonical parameters corresponding the strength of correlation between and each p.(x). Define μ
[x)], which can be written as 2 Pr(p. = y) 1, a notion of accuracy scaled to [-1,1].
Note that the above form of the distribution is in terms of canonical parameters 0. This distri- bution can also be parametrized in terms the mean parameters corresponding to 8, which are E. [v). E. [p(()))) for i [m], and E. (p:(x)) (x)) for (p)))),))(z)) F.
21

---

Published as a conference paper at ICLR 2023
Theorem 1. Assume Pr(y, P(x)) follows equation 2 above. Then, the conditional entropy (z)) can be expressed as
Hy H()) - ( Ep(s)) [log cosh θ O PP(x))
The quantity being subtracted from H (y) corresponds to the reduction in entropy of y given that we observe P(x). Within this expression, there two terms. First, @ f is correlated with how much signal each pa (z) contains about y. Note that this quantity symmetric pa (x) is negatively correlated with y, it still provides information since both 0g and IE [p<())y] will be negative. The second term, Ep(x) [log cosh θ P(z)], is for normalization (otherwise, the first term grow arbitrarily large with 9). Note that this quantity independent of Ozg, the interactions between prompts.
Proof. We can write H(y Therefore, H = (())))))) ((P())))): H(P(z)) - H(P())))) = - PPP(((t))oooPPPPPPPP + [
and (v, P(x)) as H(P()))))+H())
We focus on simplifying (((()))
(4)
P())6(-1.1)
P(s)E(-1,1)"
- PP(((())))
ProP((()))))))
P(x)E(-1.1))
= E -1((()))V-11)((0)))
-og (ogp)//
P(x)E(-1.J)
+Pr(P(x),y) 1) log Pr(P(x)) - log Pr(P -1))
We now write Pr(P(z)),, Pr(P(x)ly = -1) and (r(P(c))a 1) according to our Ising model in equation 2. Let Ap(s) Ei=1 (.p.(r), and let BP( Σ EWEEE (z),, so that
Pr(y. P(z)) = 2 Pr(P(x)) Pr(P(z), 11) Pr(P(r),y=1)
1 exp(AA(a) + Bp(a)):
1
1
Z exp(Ap(s) + Bp(+)) exp( exp(-Ap(a) + Bp(a)))
Z
= Z exp(Br(z)) (xp((Ap())) + eep(--Ap(a))
2
Pr(P(z)ly 11 2 Pr (z),
exp( -AAp)) Bp(z))
z
2
Pr(P(c)ly=1) = 2 Pr (p))
exp(Ap(>) BP()))
Therefore, have that
log Pr(P(x)) - log Pr Pr(P(x))y = 1) - log 7 + Bp(x) + log (exp(Ap(()) + exp( Ap(x))
-log 2 + log 7 + Ap(z) - Bp(x) = - -log2 2 + Ap(x) + log (xp((Ape)) exp( - -Ap()))
log (r(()())) - log Pr (z)ly = 1) = log 2 + Bp(z) + log (xp(AA(s)) + exp(-
log 2 + log Z AP(z) BP(z) = log Ap(a) log (xp((A(())
Plugging this back into equation 4, we have
Pr(((s)) Pr(P(x)) log exp(Ap())) +
- log
-E (t(P(s). Pr(P(z)) log cosh Ap(a)
P()))(-111)".' =E [Ap(a)]] E [log A
22

---

Published as a conference paper at ICLR 2023
Substituting in our definitions of è and j give desired expression for ((y(P())),
G AMA DIAGNOSTICS
We present a suite of 8 diagnostic tasks, which can be categorized into four task types: question generation, answer generation, answer selection and extraction. We provided details about the tasks and scoring below.
Question Generation: We measure the ability of the model to transform statement question. We construct 3 question generation tasks which evaluate the models ability to transform statement yes/no question (see Question Generation (Yes/No)), transform statement to a wh- question (see Question Generation (wh-)) and finally, transform statement about placeholder entity to question about the placeholder (see Question Generation (@placeholder), All question generation tasks scored using the ROUGE score (Lin, 2004).
Question Generation (Yes/No)
Input
the statement as yes/no question.
SLatement:
Father son carping California.
Quastion:
Output
the father go camping?
Question Generation (wh-)
Input
Convert stant quastion.
Statement: Aristide kills Prime Kinister Robert Malval Question:
Output
Nho killed Prime Minister Robert Malval?
Question Generation @@ (@pacchholder)
Input
Rewrite the statement as queation about the paaeehhcc.....r
Statement:
light
[at] placeholde:
Question:
Output
where does most of the light come from?
We construct answer selection tasks which measure the model's ability to generate an answer that is faithful to a set of provided answer choices. Concretely, we measure the models ability select object categories from a fixed set of options specified the context (see Answer Selection (category)). Further, we measure the model's ability to complete sentence when provided with a context and set of sentence completion candidates (see Answer Selection (completion)). In both tasks, an answer is marked as correct if the generated response is one of the candidates provided in the context.
Answer Selection:
23

---

Published as a conference paper at ICLR 2023
Answer Selection (category)
Input
Sel ect the correct categgry.
"Categories":
company
educational institution
artist
athlete
office holder
mean of treasportation
building
natural place
village
aninal
plant
plaum
film
written work
Example: & "journal" fits "Categozy";
Output
written work
Answer Selection (sentence)
Input
Select one choice from the passage.
Select Choice:
consumer electronics
2.
Stations
3.
Passage: Microsoft Corporaticn produces computer software, consumer electrenics, and personal computers. It is headquartered at the Microsof: Redmond campua Located in Redmond, Washington, United States.
The passage "Passage" states: Microaoft Corporation aells: "Choice":
Output
conaumer electronics
Answer Generation: We construct 1 answer generation task which measures the model's ability generate candidate sentence completions given context and portion of statement (see Answer Generation). Here, a generated answer is marked as correct if the model generates 2 candidate answers.
Answer Generation Input
Cutput a list of unique aiternatives for each example.
Example: Barrack obama balievas tha:
11st alternativess
sarry Potter
24

---

Published as a conference paper at ICLR 2023
Output
worst book is Farry Potter
united States is great
We construct extraction tasks which evaluate the ability of the model to extract spans from given context. The first, and easier task, tests the model's ability extract an attribute value from wikibio (see Extraction (Span)). The second, more difficult task, tests the model's ability to extract the sentence from the context that mentions specified entity (see Extraction (Sentence)). For both tasks, we use the Text-F1 score introduced in SQuAD (Rajpurkar et al., 2018).
Extraction:
Extraction (Span)
Input
Wiki Bio:
name: robert king article_ritle: robert king lghotojousnalist) birth_place: memphis in usa ocoupations way correspondent photojournalist fi Fimaker f di rector art di recter birthname: robert whitfield king birth_date: may 25th
Question: What birthname?
Output
robest whitfield king
Extraction (Sentence)
Input
Context:
Caracas, Venezuela, (ONN) It's been more than 150 years since Venezuelans saw Simon Relivar's face.
eyebrows steely gaze pepped revolutionary leader's
sideburns, bushy behind picture frames Tuesday images unveiled Preaident Buço Chaves, programs roconstruct the face of the who liberated Bolivis, Colombis, Reuador, Panama, Peru and Venezuela from the Spantsh crown. Scans of Rol var'a skeletal remains, which investigators exhumed twe years factored into their calculations. 8o did historical. paintings, photos of reatored uniforms Bolivar wora and images of middle-aged Vanezuelans, officiala sald.
3-D several
Extract the sentence containing *Lmon Bolivar":
Output
Caracas, Venezuela (CNN) -- It's been more than 183 years since Venezuelana saw S1mon soliver's face.
H UNDERSTANDING THE EFFECTIVENESS OF THE QUESTIIN--ASWERING TEMPLATE
We analyze the LM pretraining corpus to better understand why the proposed QA prompt template may be effective. The EleutherAl models are trained The Pile corpus Black al. (2021); Wang & Komatsuzaki (2021); Gao et al. (2021).
Prompt patterns We compute the frequency of regular expression matches that correspond to the restrictive prompts (i.c., which instruct the model to output "True or False", "Yes or No") versus open-ended questions (i.e., which ask the model "Is "!s..??" "Who ?") in 2% random sample of
25

---

Published as a conference paper at ICLR 2023
the 200B token Pile corpus. The restrictive prompt- appear frequently in the original GPT-3 prompts Brown et al. (2020). The frequencies are in Table 10.
We observe that question patterns appear more frequently than the restrictive prompts. Further, we find several instances of yes-no questions followed by "yes" or "no", which mimics the AMA format (Table 11). Overall, find that QA structured text appears much more frequently in the pretraining corpus, which may help explain why the language models perform better on QA.
Category
Regular Expressions
Count
Restrictive Patterns
true or false\?",
222
" true or false).".
true, false, neither\?",
true, false, or neitherl.",
yes or no!?".
" yes or nol.".
yes, no, or maybel?" yes, no, or maybel."
* correct or incorrect\?"'
correct or incorrect)."
correct, incorrect or inconclusivet?"
correct, incorrect or inconclusivel."
"choose between:"
"pick one from:"
Yes-No
"Is *\?"
378379
Question Patterns
"Was M\?"
"Did "\?"
"Do *\?"
Are *\?"
"Will #\?"
Open-Ended
"When #\?".
639573
Question Patterns
"Where s\?".
"Why #\?",
"Who *??".
"What *\?",
"How many *\?"
Table 10: Frequency of each category of regular expressions in the Pile sample.
Category
Count
Yes/No Question & Answer
"is FV yes": 536, "was #\? yes": 248, did $? yes": 109, "do
Pattern
*\? yes": 210, "are #V?? yes": 233, "were #1? yes": 91, "will
*V? yes": 121, "is #/2 no": 2356, "was #1? no": 983, "did a\?
no": 534, "do *\? no": 935, "are *\? no": 978, "were #?? no":
422, "will M? no": 423
Table 11: Yes/No question patterns followed by "Yes" or "No" tokens.
Word frequencies When applying the few-shot restrictive prompts, we observe large imbalances in the F1-scores for different classes (Table 12). Therefore, we next ask if answering the restrictive prompts is challenging due to biases acquired during pretraining. Over the same Pile sample as before, the mean word count is 25.3 + 7309 occurrences. We compute the frequency of individual words the "restrictive" and "open-ended question" " patterns from Table 10. This leads to two hypotheses about why QA prompts perform well:
First we see that there are imbalances between the occurrence of "yes" vs. "no", and "true" "neither" for instance. This may bias the model towards certain answer choices. Indeed
26

---

Published as a conference paper at ICLR 2023
Zhao et al. (2021) also hypothesizes, but does not provide any analysis over the pretraining corpus, that pretraining may instill particular biases in the model.
2. The frequency of the words in the "question words" categories is typically order of magnitude larger than those in the "restrictive words" category. We hypothesize that the representations for the "question words" will be the most context-specific, which is useful for the prompting tasks we consider. Findings in Ethayarajh (2019) support this hypothesis -Ethayarajh (2019) finds that frequently occurring words (e.g. stop-words) have the most context-specific representations. In other words, for the more frequently occurring stop- words the embedding produced by the transformer-based LM changes more significantly depending on the co-occurring words in the context.
Benchmark
Output Space
FI-Score 0-shot
F1-Score Few-shot two in-context exam- ples per class
FI-Score AMA QA single prompt-chain with no aggregation
CB
True,
True: 36.8
True: 55.6
True: 95.7
False,
False: 0.0
False: 0.0
False: 92.3
Neither
Neither: 21.7
Neither: 12.5
Neither: 28.6
RTE
True.
True: 40.4
True: 70.6
True: 58.5
False
False: 58.3
False: 31.3
False: 64.9
WSC
Yes, No
Yes: 53.5
Yes: 53.5
Yes: 61.3
No: 0.0
No: 13.7
No: 78.2
Table 12: F1-Score by class for three benchmarks with three different prompting templates cach: 1) O-shot, 2) fow-shot with the original GPT-3 restrictive prompts Вrown al. (2020), and 3) AMA prompts. We observe large imbalances in the scores across classes under the O-shot and few-shot prompting.
Category
Word Counts
Restrictive Prompt Words
true: 69658
false: 41961
neither: 20891
yes: 12391
no: 452042
maybe: 36569
Yes-No Question Prompt Words
Is: 3580578
Was: 1926273
Did: 200659
Do: 394140
Are: 1441487
Will: 619490
Open-Ended
When: 583237
Question Prompt Words
Where: 303074
Why: 97324
Who: 417798
What: 548896
How: 298140
Table 13: Frequency of each category of regular expressions in the Pile sample.
Overall, designing prompting templates for an LM based on analysis of the LM pretraining corpus may be a promising path forward for future work.
I ERROR ANALYSIS
We bucket the common envor modes of AMA into three categories: knowledge, instruction- following, and long-context.
27

---

Published as a conference paper at ICLR 2023
Knowledge errors. We find that AMA yields the most gains when the knowledge required to com- plete the task is explicitly provided the context (e.g.. reading comprehension, extractive QA). We find that AMA provides comparatively less lift tasks where the model needs to (1) recall en- coded factual knowledge or (2) apply common-sense real-world knowledge to a given context. We provide concrete examples from the NaturalQuestions dataset (see Knowledge (Factual) below) in which the GPT-Neo-6B model wrongly answers the question due to a lack of latent factual knowl- edge. We additionally provide case-examples from the BoolQ dataset where the model's limited real-world knowledge limits its ability to correctly answer the questions where the model's failure recognize that food that is smoked cooked, leads it to incorrectly answer the question (see Knowledge (Commonsense) below).
Knowledge (Factual)
Input
Quastion: whot/a a the dog's a name on com: and jerry
Answer:
Prediction
dog' name "Fido"
Ground Truth
Spi ke
Knowledge (Commonsense)
Input
Passage:
Philadelphia roll a makizushi (also classified kawarizushi) type of sushi ganerally with salmon, aream cheese, eucumber. It con also include other ingredients, such other types fish, avocado, scallions, seed.
Question: the salmon cooked in philadelphia roll
Answer:
Prediction
false
Ground Truth
dnstruction-followiii errors. We find that tasks with more restrictive output spaces (e.g., multi- way classification tasks), a common failure mode is to generate an answer that is not in the desired output space of the AMA prompt, despite being explicitly prompted to do so. In Listing 3 and 4, we provide sample instances from the DBPedia classification task where GPT-Neo-61 does not correctly map descriptive adjective (e.g., automobile or singer) valid class specified in the prompt.
Instruction Following (1)
Input
28

---

Published as a conference paper at ICLR 2023
Pick ene category for the following text.
"categories":
company
educational institution
athlete
office holder
mean of transportation
building
natural place
age
sninal
plant
album
written work
Passage:
Montevardi High Speed the Montevard: High Speed was grand tourer sutomobile built by Kenzeverdi in Bazel Switzerland from 1967 to 1970.
Contemporary rivals included the British Jensen. Interceptor (which was also powared by a thrysler va) This car was dasignad by the Italian dasign house Frue and wes actually built by Fissore of Italy Ezom 1969. They redesigned the caz in 1972 and again in 1975. convertible version of the High Speec 375 was known as
Summary: This passage is about auzomobile.
The summary "Summary" fits *Cazogory":
Prediction
autcmilile
Ground Truth
transportation.
Instruction Following (2)
Input
Pick one category for the following text.
*Categories":
company
aducational institution
artist
athlete
office holder
mean of transportation
building
natural place
village
aninal
plant
album
fi
written work
Passage: Passage: Patricia Bennett Facsicia Bonnett (horn 7 April 1947 in The Bronx New York) was an originel member of the American singing girl group the Chiffons
Summary: This passage is about singer.
summazy "Summary" fits "Casegory";
29

---

Published as a conference paper at ICLR 2023
Prediction
singer
Ground Truth
artist
Long-context errors. We find that the AMA question() functions struggle to generate accurate statement-quee -ueestiin transformations when the input is long or contains complex sentence structures (e.g. compound sentences). We provide sample instances from the SuperGLUE record task where GPT-Neo-6B fails to transform a sentence with a placeholder subject to question about the place- holder subject (see Long-context (question() below). Additionally, we find that the AMA answer() functions struggle to extract the correct span long contexts (greater than sentences). We show sample instance from the DROP QA task where GPT-Neo-6B fails to extract the correct span from the long provided context (see Long-context (answer()) below).
Long-context (question()
Input
statement as quastion
[atlppaccholder.
Statement: Most of 1ight from the pat.pplachhole Question: where does most of the light from?
Statement: The lat was not hard
Question: What was not hard?
SLatement: pealphacee wen: to the mall with hez mom to buy a backpack Question: wert to
mon buy backpack?
SLatement: Rossello warned public Sunday island could feel [at] placcholder's wrath around Wednesday.
Question:
Prediction
Nho wacned the public on Sunday that the ialand could reel [at pal.jpacchhllee der'' wrath around noon Wadnesday?
Ground Truth
Nho's wrath felt around Mednesday?
Long-context (answer()
Input
30

---

Published as a conference paper at ICLR 2023
Context:
Looking to avoid back-to-beck divisional lcsses, the Patriots traveled to Miani to face the 6-4 Dolphins at Salphin Stadium. After Carpenter's kickoft was returned from the 29-yard line by Natthew Slater, the Fatriots bagan their first own 40-yard lime. cassel'a firat two passes were Dolphins zerritory and Fatriots
possesstan
completed
eventually their dewns, putting zone. Rowever, holding penalty Neal pushed back 10 yerda, forcing 30-yard Castkowski fiold goal four plays later that gave the Petricts 3-0 lead. Following Delphins three-and-out, the Pstricts' second drive ended when Caasel cornerback Jason Allen to keep the ball in the air until Renaldo Hill intercepted it; 17 yard return gave the Dolphins the ball at the Petriols' 42 yazd Line. on the next play, a 29-yard navid Martin recept ton moved the Dalphing into the Fatriots' red zoue, where the velphins used their "wildcat" formation on the next two playa
to Moas was bobbled by both Moss and
Vuestion:
which scored first?
Prediction
Patricts
Ground Truth
Dolphins
J AMA PSEUDOCODE
K DATASETS AND PROMPTS
We evaluate over 20 datasets which fall into 4 categories: SuperGLUE (BoolQ (Clark al., 2019) 9), CB (De Marneffe et al., 2019), 9), COPA (Roemmele et al., 2011), MultiRC (Khashabi et al., 20,8), ReCoRD (Zhang al., 2018), RTE (Wang et al., 2019), WiC (Pilehvar Camacho-Collados, 2018), WSC (Levesque et al., 2012)). NLI (ANLI R1, ANLI R2, ANLI R3 (Nie et al., 2020), Sto- ryCloze (Mostafazadeh et al., 2017)), Classification (DBPedia (Zhang et al., 2015), AGNews (Zhang al., 2015), SST2 (Socher et al., 2013), Amazon (He & McAuley, 2016)), and (RealTimeQA (Kasai et al., 2022), DROP (Dua et al., 2019), NaturalQuestion (Kwiatkowski et al., 2019), WebQuestions (Berant et al., 2013)). We provide dataset details along with few shot and AMA prompts for the dataset below. The purple highlighted part is the input example. The rest of the prompt fixed for all examples in the dataset.
31

---

Published as a conference paper at ICLR 2023
K.1 AGNEWS
Description: News article classification dataset with 4 topics. Zhang et al. (2015) Train Size: 120000, Test Size: 76000
AGNews Few Shot
Input
calegory for passage.
Categeries:
Worie News
Sports
Businass
Technology and Science
Passage: Wadding comas over (Reuters). Recters wedding guest sparked bidding frenzy offered wedding friend.
pair invitations attend admitted that the bride forner girl
Calegory: World News
Serena williams Reaches Finals of China Open. Top seed šerena Williems of the United States has powered her way into the finals of the China Open tennis tournament in Beijing with straight sets (6 2, 3) victory over fourth-seeded Vera Rvenareva Rusais. Category: Sports
Passage: Tennis:
Passage:
Abramoviet eh faces rich 1ist challenge. Lakshmi Mittal, the Indian-born steel magnate, yesterday staked claim overtake Roman Abramovich Britain's richest man with 10bn deal to create the world's largest sceelmaker.
Category: Business
Passage:
The Race is on: Second Private Team Sets Taurch hate for Ruman Space iight SPACE. con), SPACE, com TORONTO, Canada A second team of rockeceers competing #36;10 million space flight, officially announced the first launch rocket.
Prize, contest privataly suborbital
Category:
Gold Output
Technology and Science
AGNews AMA (rompt()-chain Example
question()
32

---

Published as a conference paper at ICLR 2023
Summarize the passage.
Passage: China avartakes Unitad Stares as top destination for forcign investment (AFP) overtonk United States as top globsl descination foreign while Asia-Pacific region
direct investment (2D1)
inveatment than any developing region, UN report
Summarizo: the passage "Passage": The passage is about foreign diract investment,
Passage: Colangelo resigna CEC of D-Backs. Jerry Colangelo has resigned his position as chief executive officer of the Arizona Diamondbscks, effective immediately, handing the reins of the orgenization to CBO Elect Jeff Moorad. Summarize: the passage "Passage": the passage 1a about the Arna Blamondbacks.
Passage: injured in plant fire in Japan, TORYO, Aug. (Xinhuanet) Fice broke plant belonging Bridgestene Corp. in Anagi, Pukuoka Prefecture of Japan, leaving people injured.
Friday
Summerize: the passage "Passage": the passage is about plant fire.
Passage:
The Race is on: second Private Team Sets taurch nate for Human spaceflight SPACE.com com) SPACE. con TURONTO, Canada -- & second team of rocketeers competing for the 436;10 million Ansari Prise, contest for privately funded auborbital space flight, has officially announced the first launch data for its manned zocket. Summarize: the passage "Passage";
Model Output
The passage ls about a rocket.
answer()
Pick the correct category for the passage.
"Calegories":
World News
Sports
Business
Technology and Science
china evertakes united Stares as top destination for foreign investment (AFP) AFP China overtook the United states as Lop global des:ination foz foreign diract investment (FDI) in 2003 while the Asia-Pact region attrasted more investment than any other develeping region, a UN report said. Summary: The passage about foreign cirect investment. summary "Surmary"
Passage:
"Cazegory": Business
Colangelo has resigned his position chief executive offices of the Arizona Diamondbacks, effective immadiataly, handing the reins of the organization to CEo sleat Jeff Moorad. Summary: The passage is the Arizens Diamondbacks. The summary "Sunmary" fita "Cacegory"; Sports
Passage:
Colangelo zesigns
of D-Backs.
Passage: injured in plant fire in Japan. TOXYO, Aug, (Xinhuanet) Pire broke out Friday
plant belanging Rridgestons corp, Anagi, kestern Japan, leaving people injured.
Pukucka
Summary: The passage about plant
The summary "Sunmory" fita "Category": World
Passage:
The sace is On: Second Private Team Sets Laurch uate for Ruman spaceflight SPACE.comt SPACE, aod..com con TORONTO, Canada A second Leam of rocketeers competing for the 436,10 million Annari Prize, contest for privataly funded suborbital space flight, has officially announced the first launch date for its manned rocket.
Summary: The passage
rocket.
summazy "Sunmary" fits "Cazegory";
Gold Output
33

---

owz he charge Published as a conference paper at ICLR 2023
technelogy and science
K.2 AMAZON
Description: Amazon product classification dataset with classes. He & McAuley (2016) Train Size: 9000, Test Size: 9000
Amazon Few Shot
Input
Pick the correct category for the product.
Categories:
Amazon Instant Video
Books
Clothing Shoes and Jewelzy
rontca
Kindle
Movies
Musical Instrumeats
office Products
Tools and Home Improvement
Product:
This bra is extremely confortable, affardable and pratty too! Xy only compleint, and the reason for stere is that the straps can't be adjusted very much, Right now as I'm maximum shortening the straps, so as wear it stretches though.
see some adjustments my all recommend someone like searching for comfortable yet. pretty ara thair (T'm 32G5).
Nothing my sewing
machine
Category: Clothing Shoes and Jewelry
Product: 1/8/10 Have been using this drill and am very pleased. It has tons of torque and the handle comes in handy. Just great to have a corded drill with this type of power for when the cordless 13v wont do the jab or if you have let of sczews to drivellavent used this driil yet but as looking forward putting to work. bought because only sugged
drills, aside
great,
Hammer drill, was confident because of greaz reviews others, very solid and like the quick change cord. expect drill for
this
Category: Cools and llone Improvemen:
Product:
Lindsey is tired of being used by mea so she swears off them uniil she meets Brad. Lindsey wea offed a position to renovate room for sax in a hotel. she has no limits en what she can do with the zoom, Lindsey is ready to create her uitimate fantasy and Brad is just the man help her, He is hired to help with construction portion renovating insecurities about gotten any backing this is giving up contrel to who doesn't usually Lake it because he is not sure if he will be rejected. The are great pare that burn up the pages and learn that if you open up juat a little then true love an truly find you.
one
Lindacy and
things
together. Brad people in general he big bock read about
Category: Kindle Store
Product:
lege back turbulent 60's. it profound effect se. Afler graduation joined Corps and during stressful training in country [liberia] timen of illness and the night before I left, this beok gave married, just betore and again after children were born and again acter two near fatal illmesses. an always anazed that there is chapter that reaches out to you, graba you and offers hoth comfort and hope for the future.Cibra offers timeless insights and love with each word, I think that we as nation should read AND learn the lessons here. It is definitely time for thought and guide us through.
first
60's. The book revivs)
anything netaphysical
became
always
comfors. read before
reflection this
Categozy:
34

---

Published as a conference paper at ICLR 2023
Gold Output
Rooks
Amazon AMA prompl()-chain Example
question()
produst.
Product: was when purchasad the what expect. with real joy say that it worth every cent and have already watched it several times. The Storyline kept interested.
Summarize: the product "Produc="; the product is oVD.
Product: These are the best haadphonea I've aver owned. recently purchased zepiacement pair, as my original set died efter several years intensive use.
Summarize: the product "Produo:"; The product headphones.
Product: these tights tighter they leave my legs feeling like they've been squeezed to death.
tighta and whan these off,
Sunmarise: the product "Produc.": The product is Lights.
Product: I first read THE PROPRET in college back in the 60's. The book had zevival aa did anything metaphyatcal in the turoulent 60's, It had profoune effect on me and bocome a book always took with me. After graduatien joined the Peace Cozpa and duzing stressful training in country ILiberist at sines of illness and gave great comfort. read before after
the night before I left, thia again
my children always amazed that there is choptor that reaches and offers beth comfort and hepe for the future Gibran offers time leas insighta and love with each word, T think that we natiion should read AND learn the lessons here. it is definitely time for thought and reflection this book could guide us through. Summarize: the product "Producs":
macried, just before
fatal illmesses.
to you, grabs
Model Output
The product 1a book.
answer()
35

---

Published as a conference paper at ICLR 2023
Pick the correct category for the product.
"caregories":
Instant Video
Clothing Shoes Jewelzy
Electronics
Kindle Store
Movies and TV
Musical instruments
office Products
Too' and Home Improvement
Product:
Wae unsure when purchased the DVD what to expect. With real joy can aay every Storylina kept interested.
that was
already watched
times.
Summery: The product
The summary "Sunmary" rita "Ca.egory"; Amazon Instant Video
Product: These are the best headphones I've ever owned. I recently purchased replacement pair, as my original set died after several years of intenaive use. Summary: The product in headphones.
The summazy "Surmary" fits "Cazegery": Electronics
Product:
these off, they leave my legs fealing like they' been squaszed death.
Lights Lighter
tigats
Summary: The product tights. The summary "Sunmary" rita "Cacegory": Clothing Shoea and Jewelry
Product: I first read THE PROPRET in college back in the 60's. The book had zevivel as did anything metaphyatcal in the turoulent 60's. Tt had a profound effect or me and beceme a book always took with me. After graduatien joined the Peace Cozpa and duzing streasful training in country ILiberist at Lines of illness and gave great comfort. read my children were bern and again after always anazed that there is chapter that reaches
the night before left, married, just before and again
fatal illmesses. and offers beth comfort and hope for the future Gubrre offers time less insighta and love with each word, T think that we read AND learn the lessons here. it is definitely time for thought and reflection this book could guide us Lhrough. Summary: The product a bock. The summary "Surmery" fits "Cazegory":
to you,
natioon should
Gold Output
Book
K.3 ANLIRI
Description: Adversarially mined natural language inference dataset from Wikipedia. Nie et al. (2020)
Train Size: 16946, Test Size: 1000
ANLI R1 Few Shot
Input
36

---

Published as a conference paper at ICLR 2023
Robert L. Hass (born March 1, 1941) is an American pcet. Hi served as Poet Lauzeate of the United States from 1995 to 1997, Be won the 2007 National Book Award and shared the 2009 Pulitzer Prize for the collestion "Tima and Naterials: Poems was
1997-2005."
Nallase Stevens Award
Academy of
Question: L. Falac, Noither?
Laureate the United Stalea
years. True,
Randa 1 Park (oorn March 23, 1974)
American actor, comedian, writer, and diracter. èe played Kim Jong-Un in the 2014 film "the interview", Minneseta governez Denny Chung in "Veep", and beginning in 2015 he portsayed Bedie lluang's father, American restaurateur touls Huang, in ARC television show "Freah off the Boat",
Question: Randall Park is dead True, False, or Neither? False
1a hybrid strawberry cultivar that created in effort combine the best traits of the garden strawberry ("Fragaria" "ananasss"), which has large berries and vigerous plants, with the woodland atrawberry ("Fragazla vesca"), which has an exquisite flavour, but amall berriea. Question: Extensive testing went on to produce this berry True, False, or Neicher? Meither
Fragaria
Daniel Zelnikov (bern January 29, 1987) is a Republican nember of the Montana Teglalature. He was elected House District 47 which repreaents BI Biigings, represents
Montana Aller redistricting, for himsolf pursuing pro-privacy legislation.
Distric: 45,
Question: Chere is infermation indicasing whether Daniel Zolnikov leglalator not, True, Falae, or Neither?
Gold Output
ANLI RI AMA prompt()-chain Example
question()
Rewrite the statement as yes/no question.
light comes the sun Statement: most
Question: Boes the light come from sun?
hard Question: Was the test not hard?
Statement: the
Statement: it is good idea to buy your parents gifts Question: Ie it good idea to buy your perents gifts?
Statement: the balloon popped
Quastion:
pop?
Statement: father
went carping califorria.
Question: uid the father and go camping?
Statement: There is no informazion indicating whether naniel solnikov is good legislator not.
Question:
Model Output
Ie there information indicating whecher Daniel Zolaikov La good legislater?
answer()
37

---

Published as a conference paper at ICLR 2023
Answer the question. If there is no evidence in the context, return "Unknown".
Context: According to Biroben, the plague was present somewhere in Italy and affected 1,200 people.
Question: Based
context, yes, people Italy, Burope
plague affect people Burope?
Context: Policies siming contrelling unemployment and in particuler reducing its inequa styaalltt-asssolatte effeets aupport economic growth. Question: Based on the context, is confidence factor in increzsing self-esteen? Answer: unknown
Context:
The term "matter" is used throughout physics in bewildering variety of contexts: for example, one refers "condensed matter physies", "elementary matter", "partonic" matcer, natter, "anti" matter, "scsange" matter, ruclear" mstter.
Question: Based the context, anti-matter made electrons?
Answer: Unknown
Context:
Daniel Zolnikov (born Januazy 29, 1937) is Republican member of the Kontana Legislature. He was elected to Rouse Distriot 47 which representa Billings, Montana After redistricting, he now reprasenta House Districs 45. He has made name for himself pursuing pro-privacy legislation.
Question: Based on the context, chere Information indicating whether Dantel Zolnikov good legislator?
Answer:
Gold Output
nei ther
K.4
ANLI R2
Description: Adversarially mined natural language inference dataset from Wikipedia. Nie et al. (2020)
Train Size: 45460, Test Size: 1000
ANLI R2 Few Shot
Input
38

---

Published as a conference paper at ICLR 2023
Katzer was London music venue and nightelub that opened in September 2008, after three years of planning. A 6,600 capacity Live music venue and nightclus, It was the second project for exners Cameron Lealie and Keith Reilly, founders of the south-east
London Fabric. Matcer the venue to open London.
Queation:
chan
True, Palse, Neither?
Mhitechspe: is British television drana series produced Carnival Films, in which detectives in London's Whitechapel district dealt with mirders which rep_ leated historical crimes. The first series was first breadenst in the UK on 2 rebrusry 2009 end depicted the search for modern copycat killer replicating the murders Jack the Ripper.
Question: Some of the victims depicted in whitechapel were women True, False, or Meither? Neither
Nannina
February 1448 14 1493), born Tucrezia Medici, the second daughter of Piero di Cosimo de' Nedias and Lucrezia Tornabuoni. She thus the elder sister of Lerenso de' Medici, She married Bernardo Rucellai. Ber father's 5 name waa Piero, so she is sometimes mad known as tucrezia of Piero de' Medici.
Question: Nannina de' Medici la sometimes known as Ivanka Trump True, False, or Neither? Folse
There 18 little Shia community in El Salvador, There Talamie Library operated Shis community, naned "Fatimah Zahza", They published the first Ialamic magazina in central Americas "Reviata Ribliotema Islamics", Addicionally, they credited with providing the first and only Islamic library dedicated to spreading Talamic culture in the country.
Quastion: the community is south of the Unitad States. True, False, or Neither?
Gold Output
ANLI R2 AMA prompt()-chain Example
question()
Rewrite the statement as yes/no question.
Stazement:
light comes the light come from the sun?
Question:
Statement: the test not hard Queation: Mas the teat not hard7
SLatement: it is good idea to buy your pareats gifts Quastion: on: 7a it a good idea to buy your parents giat
Statement: balloor papped Question:
balloen pop?
Statement: The father and went camping California.
Question: Did the father and son go camping?
Stacemens: The community is south of the United States. Question:
Model Output
Ie the community south of the United States?
answer()
39

---

Published as a conference paper at ICLR 2023
Answer the question. If there is no evidence in the context, return "Unkzown".
Context: According to Biroben, the plague was present somewhere in Italy and affected 1,200 people.
Question: Based
context, yes, people Italy, Europe
plague affect people Burope?
Context: Policies aiming at contrelling unemployment and in particuler reducing its pmequality-assote effeets support economic growth. Question: Based on the context, is confidence factor in increzsing self-esteen? Answer: unknown
Context:
The term "matter" is used throughout physics in bewildering variety of contexts: for example, one refers to *condensed matter physies", "elementary natter, "anti" matter, "scsenge" matter, matter", "partonic" matcer, ruclear" mstter.
Vuestion: Based the context, anti-matter made electrons?
Answer: Unknown
Context:
There is little Shia cemmunity in a1 Salvador. There is an Islemic Library operated by the Shia community, named "Fatimah Az- Zahca", They published the firat Islamia magazine in Central Anerica: "Ravista Ribliatacs Islamica".
additionslly, they are credited with providing the firss and only Islamic library dedicated to spreading Talamio cutture in the country.
Queation: Based
community
United States?
Gold Output
true
K.S ANLIR3
Adversarially mined natural language inference dataset from Wikipedia, News and other data sources. Nie et al. (2020) Train Size: 100459, Test Size: 1200
Description:
ANLI R3 Few Shot
Input
And that means that the lecal law enforcement officials need help the federel level Prograns like Project Exile where the Tederal government intensifles arceating people who illegally use guns, And we haven't dona a very good job of that at the federal level recently.
Question: There are only federal enforcement officials. True, False, or Neither? Palse
Scary Dreamsbr>Ion
up cold sweat. shaking scared. realized seary
Ton afraid fall back as lep. Tratead stayed up ell night.
had just
Question: Zem experienced bad nightmare that kept him from sleeping. True, Palse, Neither? True
wayman Lasrence risdale (June 3, 1994 May 15, 2039| was an American professional besketball player in the HBA and smooth jazz bass guicariss. A chree time All Masketball University of aklahoms, was elected so the National Colleg ate Fame 2009.
Question: Kayman Tisdale played smooth Jazs guitar Trus, False, Noither? Neither
University Oklahoma
For one night, all of Clinton'a non-varsity squads achieved perfect ion sweeping Altua in seventh, eighth and ninth grade basketbell at PLEASE LOU IN FOR PREMIJM CONTENT, Our website requires visitors to log in to view tha best lncal news from clinton Deily News. Noz yet a subscriber? Subscribe today! Thank you! Question: This headline leads 2o more information that behlnd paywall, True, False, or Neither?
40

---

Published as a conference paper at ICLR 2023
Gold Output
trbe
ANLI R3 AMA prompt()-chain Example
question()
Rewrite the statement as yes/zo question.
SLatement:
light comes
Quest noas of the light from the aun?
Statement: the Lest was not hard Quastion: Kas the teat not hard?
Statement: it 18 good Idea to buy your parenta gifts Question: 25 it a good idea to buy your perents gifts?
Statement: the loor popped Question: Did the balloon pop?
Statement: The father end went camping Californis.
Queation: Bid the father and son camping?
BLatement: This ine leads to more information that is behind payeall, Quastion:
Model Output
noes this headl're lead to more information that is behind paywall 1?
answer()
Answer the question. If there is no evi deidence in the context, return "Unknown".
Context:
According to Biraben, the plague was present somewhere in Italy and affected 2,0 people.
Question: Based on the context, Did the plague affect people in Burope?
Answer: yea, people in Ttaly, Europe
particular reducing its iyequality-asrociate effects support econemic growth. Queation: Based on the context, Ia confidence factor in increasing self-esteen?
Context: Policies aiming controlling unemploymeat
Answer: unknown
Context: The term "matter" is used shroughout physica in bewi derring variety of contexts: for example, one refers to "condensed matter physics", "elementary matter", "paztonic" matter, "dack" natter, "anti" matter, "suzange" matter, and rucleer" mctter.
Question:
context, anti-metter
electrons?
Context:
For one night, all of clinson's non-varsity squads achievad perfection sweeping Altus in sevench, eighth and ninth grade baskerball at FLEASE LoB IN FOR PREXTUM CONTENT, Our website sequizes visitors to log in to view the best local news from Clinten Daily News. Kot yet subseriber? Subseribe today! Thark you!
Question: Based on the context, Doea thia headline lead to more informat that behind peywall?
Gold Output
true
41

---

Published as a conference paper at ICLR 2023
K.6 BOOlQ
Description:
Yes/no QA task over small wikipedia passages. Clark et al. (2019)
Train Size: 9427, Test Size: 3245
BoolQ Few Shot
Input
Answer the quest
Context:
Red River (1948 film) The film's auppozting cast features Walter Bzennan, Jcanne Dru, Coleen Gray, Barry Carey, Traland, Norden, Noah Beery Jr., gorden
Charles on Chase's original story (which first serialized in Saturday Evening Fost in 1946 "Blazing Cuns the Chisholm. Trail*(). the movie red river based true Answer: No
farry Carey and Paul
screenplay,
Question:
context:
togal drinking age -- Kozolthaton, oman, Pakistan, Gatar, Srl lanka, Tejikistan, Thailand, United ärab Emirates, Federated States of Micrenesia, Palsu Paraguay, Solomon Ialanda, Incia fcertain statest, the Uniced States (except U, Puerto Rico), Yemen (Ader Canada (certain Provinces Territories), and
Virgin
a), Japan, Iceland, have highest off-prenises crinking
these courtries limits. Austria, Antigus and Barbuda, Belgium, Bermuda, gritish Virgin -slands, Cuba, Ethiopia, Gibraltar, Luxembourg and Nicaragua have the loweat set drinking agea.
drinking ages; however,
Question: is america the only country with drinking of 21
Answer: No
Context:
brinking pablic brinking public is legal in Rngland street (though preferred that the user requesta plastic glaes avoid danger of breskage the taking the glass could be considered offance of Theft only the drink has been purchased), and may purchsse alcohel off-licence and inmediately beqin drinking it outaide. Separately, and on most Naticnal Rail train services, either purchasing alcohol on-bcard or consuming one's ewn.
one
may carry drink
public
drink aeroplanes
Question: 1s it legal to drink in public in london Answer: Yes
Escape Gringetts Harry Potter Indoor roller coaster Universal Studios Florida, theme park located within the Universal Urlando Rasort. similer dark rides, the roller coaster atilizes special effects in controlled lighting environment and also emplog oys mot ontten- 3-D projection lon of both animation and iiveacc ion sequences to enhance the expesience. the ride, which is themed to the eringetts Mizarding Bank, became the flagehig att:action for the expanded Mizarding Horld Harry Fotter whan it opened on July 8, 2014, Question: is harzy potter and the escape from gringotts a roller coaster ride? True oz Faise?
Context:
Harry Potter
Escape
fron Gringotta
Gold Output
Tree
BoolQ AMA prompt()-chain Example
answer()
42

---

Published as a conference paper at ICLR 2023
Answer the question using the context.
Context: Tonic water Taric water for Indian tonis water) is carbonated soft drink which quinine is 6tasalved. Original used prophyl actie against malaria tonic usually new signifirantly consumed its distinctive bittes flavor, particularly in gin and tonic.
quinine d:inka,
Question: does tonic still have quinine in it? Answer:
Context:
Nerthere bobshite The northern bobwhite, Virginia quail or (in its home range) bobwhite quall (Col inus virg'nnanus 1a ground-dwe 1 Hin biré native to the United States, Mexico, and the tarisbeen. it is a member of the group of species known New World qualls (ocontophozidee)) They were initially placed the Korld quails particularly closely related. The characteristic whistling call. Lespite its secretive bebwhite is of the most familiar quails in eastern North America because it is frequentlly the only quail in its range. #sbitat degradation has likely contributed to the northern bobwhite population in eastern North Americs declining by roughly 853 from 1966-2014 This population decline is apparently ronge-wide and ccntinuing.
pheasant family (Phasianidael, "bobwhiter" derives the northern
Question: a quail the seme as bobwhite?
Answer: yea
Context: United States Dapartment of Homeland Security fiscal year 2017, it #llocated discretionary budget of 840.6 billion. with more than 200 enployees, DRS 1s the third largest Cabinet department, atter the Departments of Defanse and Veterans Affairs. Homeland sacurity policy is coordinated at the White Blouse by the Nomeland Security Council. other agencies with significaat homeland security responsible lities 'nclude the nepartments of Health and Human Services, custice, and Energy
Question: Le department of homeland security part of dod?
Answer: no
Context:
Harry Potter and the fron Gringotts is indour steel zoller coaster at Universal Studios Florida, theme park located within the Universal arlando Resort. Siml lay to dark rides, the roller coaster utilizes special effects in controlled-lightnn and also employs motion based D-D protection of both animation and Live action sequences to anhance the experience. The ride, which is themed to the cringetts Wizarding Bank, beceme the flagship attraction for the expanded Wizarding Wozld Harry Fotter when it opened on July 8, 2014.
from Uringetts
Potter and the
environment
Question: harry potter and ascape gringotta roller coaster ride?
Model Output
true
K.7 CB
Description: Three-class textual entailement task. Wang et al. (2019)
Train Size: 250, Test Size: 56
CB Few Shot
Input
43

---

Published as a conference paper at ICLR 2023
B: Now see 1. A: I'm intrigued by it, but I'm net sure 1 went to go see it yet. B: Yeah, I don't think I want to see that either. Quastion: she warts to 520 that True, Folse, or Neither? Falsa
radio doesa't really just mainly music. Yeah, all. Yeah.
much news sometimes.
listen to
you pretty
station get any accurate.
think that TV uh, pretty
Queation:
TV is pretty accurate True, False, or Neither? Neither
It is pazt of their religion, religion I do net scoff at as it holds many elements which match our own even though it ackks the truth oura. At one of their great festivals they have the ritual of driving out the devils Erom their bodies.
Pirst the drummers come on may say that no women are allowed take part in they
this ritual
will pezhaps agres
fortunate in omfasion.
Question:
allowed take part in this ritual ralse, Neither?
True
Xodify the arachnids, said the researchers. Change their bodins and conditions, and you could get Fibres like glass, still monof bus with loga:lthmie
progressions of poasibilitias of strength and flexibility, and the ability to resonate Light-particle or sound-wave waves undistorted, scarcely weakened over thousanda of miles. Who said the arachnids to be tocally organtc7 Question: acachnids had cotally organic. Tzue, False, Neither?
Gold Output
False
CB AMA prompt()-chain Example
question()
Rewrite the stalement yes/no question.
Starement: most of the light comes from the sun Question: Doea nost of the light come from the aun?
Statement: test was
Question:
hard?
Statement: it is good idea buy your parenta gifts Question: is it good idea to buy your gifts?
Statement: the balloon papped Question: Did the balloon pap?
Statement: The father and son went camping to Californis. Quastion: father son go camping?
Statement: arachrids had be totally organtc Question:
Model Output
Did arachnids have to be totally organic?
answer( ()
44

---

Published as a conference paper at ICLR 2023
Provide the answer to the question from the passage.
Passage: when Judy and Jack want to school, they got in trouble with thair teacher for beiigg late.
was very fair. fair?
Question:
No
Passage: inflation is eccurring, leading higher prices for basic necessities such as gas 2 dollara. Do think that infiation la good for society? Question: =s inflation good for society?
Answer: Maybe
Passage: Put yourself out there. The more time you spend dating and socializing, the more likely you will find a boyfriend you like, socializing help you find boyfriead?
Question:
Modily the arechnids, said the researchers. Change their bodies and conditions, and you could gez fibres like glaas, still monofilamant but with logarichmic progressiens of possibilities of strength and flexibility, and the ability to resonate Light particles or sound waves undiatorted, scarcely weakened over thousands of miles. Who said the arachnida had to be totally arganic? Question: Did arachnids have to be cotally organic? Answer:
Passage:
Gold Output
talae
K.8 CОРA
Description:
Casual reasoning dataset where task is select the alternative that more plausibly has causal relation with the premise. Wang et al. (2019)
Train Size: 400, Test Size: 100
CОРА Few Shot
Input
more likely continuation to fallewing sentence.
Context:
The crashed into the matoreycle bri matorayal ist died.
Context: The customer into the boutique because The window display caught her
Context: The print on the brochure wes tiny so the man put his glasses on.
Context: The
Model Choices
became famous
photographers Followed her
Gold Output
photographers followed her
CОРА AMA prompt()-chain Example
answer()
45

---

Published as a conference paper at ICLR 2023
Pick the correct ending for the example.
Quastion: (bacause 'ae took medicine', because 'she got expelled') My roormate was fneling ing better because? 'she medicine'
Quastion: Ibacause the does practica', because the fast!) Mott in at soccer because?
Answer: The does not practice'
Question: (because 'she was smart', because " "ahe never did her homework') The giel went to col lege and graduated with honora because?
Answer: 'she was smart'
Question: "her family avoided hec.', "photogzaphers followed bez.f) The *amous
Answer:
Model Choices
woman became Lamous so photographers foll owed her
Gold Output
photographers fo loed her
K.9 DBPEDIA
Description:
Ontology classification dataset with classes. Zhang et al. (2015)
Train Size: 560000, Test Size: 70000
DBPedia Few Shot
Input
46

---

Published as a conference paper at ICLR 2023
Pick the correct category for the passage.
Categories:
company
educational institution
athlete
office holder
of trensportation
bui_cing
natural place
village
aninal
plant
album
written
Passage: Garrison Cadet College Rohat Garrison Cadet College Kohai Situated in Kohot. Foundation stone was laid by the then Prima Minister of Talania Republic of Pakistan Late Mohtarama Benazir Bhutto in 1992. Lieuzenanz General Arif Bangash Lieutenent General K.X Afridi Ma Major General Shirendil Niazi and Colonel Idraesm (founder Principal) Dr.
Category: Bducationsl Institution
River Ingrebource is tributazy miles (43.3 km) in length. is considered stratagic waterway London ferning of the Blue Ribbon Network. It flows through the London Boreugh of Favering roughly from north south joining the Thames at Rainham. Category: Natural place
Passage: Ingrebouzne
Passage: US5 Patrol No. 4 (SP-8) uss Patrol No. (SP-8) ofter rendered as USS Fatrol #4 was an armed moterboat that served in the Unized States Navy as a patrol veasel from 1917 to 1919.Patrol No. 4 was built as privace motorboat of Tynn Maasachusetta,
same name 1915 by same design private ewners by
was one as
five motorbeats built part of the eivilian Preparadness Movement program with understanding that they would enter U.S. category: Nean of Transpartation
Passage: T2 KU TY RU is an American alcoholic beverage company that specializes in saka and other spirits. The privately-hold company was founded in 2004 and is headquartered in New York City Kew York, While based in New York TV KU's beverages are made Japan through joint venture with two sake brewestes. 2011 xu"s growth
its products
states.
Categozy:
Gold Output
ccmpany
DBPedia AMA (rempt()-chain Example
question()
47

---

Published as a conference paper at ICLR 2023
Summarize the passage.
"Categories::
company
educational institution
athlete
office holder
mean of transportation
building
natural place
age
sninal
plant
album
written work
Passage: Personality and Mental Health Personality and Mental Realth:
#ultidisciplinery Studies from Fersonslity Dysiunction zo Criminal Behaviour is quarterly peer-reviewed acadenic journal puolished by Kiley- Blackwell on behalf the Centre for Bealth and Justica.
Summerize: the passage "Passage"; The passage is about a journal.
Pasaage:
Class lifeboal based Broughty Ferry in Scotland that capaized during reacue ottempt with the losa of her entire
RNIB
eight The Mona built in 1935 and her time saved
118 lives.
Summarize: the passage "Passage": the passage is about lifebost.
Passage:
Sayonara mo Tenakatta Matau Sayonara mo Terakatta Natau is an a album by Mikuni Shimokawa released en July 4 2007 by pony Canyon.This album consists eleven songs; several new songs and some songs which were previously released as singles.
passage "Passage": The passage
album.
Passage: TV KU TY RD is American alcoholic beverage company that specializes in sake and other spirits. The patv y-he company was founded in 2004 and is headquartered in New York city sew york. while based in New York TV kuts beverages are made in Japan through joint venture with two sake brewe:les. Since 2011 TY xu's growth has extended its products into all 50 scates.
Summerize: the passage "Passage":
Model Output
The passage is about company.
answer()
48

---

Published as a conference paper at ICLR 2023
Pick ene category for the following text.
"categories":
company
educational institution
athlete
office holder
mean of transportation
building
natural place
vil vil'age
sninal
plant
album
written work
Passage:
Personality and Mental Health Personality and Mental Realth: Multidisciplineey Studies from Fersonslity Dyssunction zo Criminal Behaviour is quarterly peer-reviewed acadenic journai puolished by Kiley- Blackwell on behalf the Centre for Bealth and Justica.
Summary: The passage about a journal.
The summary "Sunmary" rita "Cacegory": written work
RNLS (DM 775) RMIS Broughty Perzy in Scotland that capsized during rescue attempt with the less of her entire crew eight men. The Mona suilt in 1935 and in her time saved 118 lives.
7751 Watson Class 1ifaboat based
Summary: The passage is about lifeboat. The summary "Sunmary" fita *Category": mean of transportation
Passage:
Sayonara mo Tenskalta Natsu Sayonara mo Ienakatta Natsu is an album by Shimokawa released July by Pony canyon.This album constata songs; singles.
new songs some songs which previcusly released as
Summary: The passage about album. The summary "Sunmary" fits *Category": a1bum
Passage: T2 KU TY RU is an American alcoholic beverage company that specializes in saka and other spirits. The privately-hhld company was founded in 2004 and is headquartered in New York City New York, While based in New York TV KU's beverages are made Japan through joint venture with two sake brewestes. growth company.
its products The summary "Summary" fita *Category":
states.
Summary: : The passage
Gold Output
company
K.10 DROP
Description: A reading comprehension benchmark requiring discrete reasoning over para- graphs. Dua et al. (2019)
Train Size: 77409, Test Size: 9536
DROP Few Shot
Input
49

---

Published as a conference paper at ICLR 2023
Passage: As of the 2010 United States Census, there were 16, 583 people, 5,508
householda, and 4,643 families realding in the county, The population deneity was there were 8,9 housing unita at en average dansity of The racial nakeup of county was 96.8% white, 0.78 black African American, oz races. Those population.
origin races, 0.68 23.44 were Cermans, 22.34 Americans, 13.64 Irish people, and 12.08 English pecple.
0.24 Asian,
Eispanic
ancestzy,
Queation: How percent of people were not Aslan?
Answer: unknown
Passage:
The health aect comprisea 17 special hoapitals ard centers, 4 regional
diagnostic and treatmens cencers, 9 éistrict and 21 nimag general hospitals, 323 soun hespitals, 18 Seldsher posts, 233 family group practices, 536 private hospitals,
drug supply companies/phatmaaiies doctors, 788 pharmaciits,, 1802
2002,
33,273, of hea and 14,091 091 mid-level persennel. present, there 27.7 physiciens and 75.7 hospital beds 10, 002 inhebitante.
6823
Question: What profeasion had more health workers, dectors or norses?
Answer: murses
Passage:
The exact number of pessant deatha is unknowr, and even the course of eventa are not clear, because the government, to hide the size of the massacre, ordered the destruction of all documents relating to the uprising. Ristorian Markus Baver mentions greatly underestimated official Ligure unofficial figure, ciroulated by the preas and widaly secepted, of about 10,00c been
deathe,
an
peasants killed, has mentioned Ton I. C. Bratianu in the Romanian Parliament. The cata
The
figure of 419 deaths
availcbla to the Frime Minister Dimitric Sturdza indicazed 421 daaths between 28 March and April 1907. Likewise, about :12 were injured and 1,7 detained. Mewspapers patronised by Conatantin Mi Mille, Adavarul and Dimineata, gave a figure 12, 000011,00000000000 000 victims. in conversation with the British erbassador in Bucharest, King Carol mentioned a figure of "several chousand". According to figures given by Austrian diplamats, between 3, 0.,0000,,,00 000 peasants were killed, while French Embassy nencioned
ranging kistorians put the sigures between 1000-88,000 000, the victims.
10, 10,000-20,000 being 12,000
Question: Which organizate ioni said the death soll to be beyond 10,000 0002
Answer: Rеиapaрers patrenized by Constansin Mille
Passage:
Still searching for their first win, the Sangals flew to texas Stadium for Week interconference duel with the Dallas Couboys. In the first quarzer, Cincinnati trailed early aa Cowboys kicker Niek Folk goz a 30-yard field goal, along with
Jones getting a 23-yard run. In second quarter, Tony Romo completed 4 yard TD pass The Bengals would the half with kloker Shayne Graham getting 41-yard and 31-yard field goal. In the third quarter, Cincinnati sried to rally as Q8 Carson Falmer completed an 18-yard 2B pass to WR Т. J. Houshmandzadeh. In the fourth quarter, the Bengels got closer as Graham got 40-yard field gosi, yet the Cowaoys anawered with Romo comp eting a 57-yard TO pass co WR Terrel Owens. Cincinnati tried to come back as Palmer completed 10-yard To pass to Toushmandzedeli twith falled point conversion), but Dallas pulled away with Romo completing ing 15-yard pass Patrick Crayton. final
increased lead
Ritten
Question: which
game?
Answer:
Gold Output
dallas
DROP AMA ()ompt()-chain Example
answer()
50

---

Published as a conference paper at ICLR 2023
Answer the question. If there is no evidence in the context, return "Unknown",
Context: According to Biroben, the plague was present somexhere in Europe in every year between plagus present?
Question:
Burope
Context: Policies aiming at contrelling unemployment and in particuler reducing its inequality Lty-asscltae effeets support economic growth.
Question: what's one factor in increzsing self-esteem? Answer: Unknown
Context:
The term "matter" is used throughout physics in bewildering variety of contexts: for example, one refers *condensed matter physies", "elementary matter", "partonic" matcer, natter, "anti" matter, "scsenge" matter, ruclear" mstter.
Question: what another for anti-matter?
Answer: Unknown
Context:
Still searching for their first win, the sengals flew to Texas Stadium for Week 5 Interconference duel with the Dallas Cowboya. In the first quarter, Cincinnati trailed early as Cowboys kicker Nick Folk got a 30-yard fiold goal, along with RD Felix Jenes getting a 33-yard TD run. In the second quarzer, Dallas increased ita lead 05 Tony Romo conp eted a 4-yard TO paas to TE Jason Witten Shayne Grahan, getting 31-yard field goal. the third quarter, Cinainnati tried rally Carson Palmer completed 18-yard 2D pass WR J. Houshmenczadeh.
Bengals would
half
yard
the Tourth quarter, the Bengala got closer Graham got 40-yard fleld goa., the Cowboys answered with Romo completing 57-yard TO pags to WR Terrel: owens. Cincinnati tried to come back as Palmer completed 10 yard TD paas to Foushmandzsdeh luith fat led 2-point conversion), but na laa pul led away with Rama completing 15-yard TD pass to w8 Patrick Crayton. Question: Which Leam scozed the final Answer:
the game?
Model Output
da laa
K.11 MULTIRC
Description:
Multi-sentence reading comprehension dataset. Wang al. (2019)
Train Size: 27243, Test Size: 953
MultiRC Few Shot
Input
51

---

Published as a conference paper at ICLR 2023
Answer if the pussible answer is correct answer to the question.
Passage: While this process moved blong, diplomacy continued its rounds. Direct proved unsuccesaful. one NSC note put Taliban, Afghanistan not so sponsored by Lerrozieta, early influence high-leval offort parsuade Pakistan
preasure on
sponsor terrorism as it began the Talibon. January 2000, Assistent Secretary State Karl Inderfurth and the State
United
Department's counterterrorism notor Michael Sheehan, met with General Musharraf in islamabad, dangling before him the paasibility of presidential visit in March as zeward for Pakistani cooperat Such a visi: was coveted by Mugharraf, partlly as sign of his goverrment's a legitimacy. Re told the two envoys that he would meet with Mullah Umar and press him on Bin Laden. They left, howevez, reporting Washington that Pakistan was unlikely in fact to do anything, given
benefits Taliban control Afghamistan. acheduled travel Indla, The
President Clinton that he should net visit india without also visiting #akistan....
Department felt
Queation: Based the previous passage, What did President Clinton's visit with Fakistan include?
Is "Discussing Bin Laden" & correct ansNer?
Answer: Yes
Passage:
While this process moved slong, diplomacy centinued its rounds. Direct pressure on the Taliban nad proved unsucoesatul. As one MSC ataff note put it, so
sponsor Cerrorism it aponsered by terrorists." early 2000, the United States began high-level effort to persuade Pakistan
Taliban, Alghanistan
the Taliben. In January 2000, Assistant Secretary State Karl Inderfurth and the State Department's counterterrorimm coordinator, Michael Sheahan, met with General Musharraf in Islamabed, dangling before him the passibility of presidential visit in March aa a reward for Fakiatant cocperstion. Such a vists was coveted by Kusherrai, partly as a sign of his government's s legitimacy....
influence
Question: Based on the previous passage, Where President Clinton visit on March 25, 2000?
"Packistan" correct answer?
Answer:
Passage:
The Agerctea Confer when they learned a secord plane had struck the Borld Trade Center, zearly everycne in the white aouse told us, they immediately knew it was not an accident. The Secret Service initiated number of security enhoncements around the White House complex. The officiala who issued these orders did not know that there were additional hijacked sircreft, or that one such aicoraft was en route to Nashington. Theae measurea were precautionary steps taken baccuse
FAA White souse Teleconferences, FAA, White llouse, initisted multiagency tellecofference before 9:30. Recause teleconferencee-at leass before 10:00- included...
New
Defense Department
Quastion: Based on the previcus passage, To what did the CIA and FAA begin participsting in at 9:40?
Ts "Coffee hour* correct answer?
Answer: No
Passage: casses a charge macion? application force. Any time object changes mction,
applied.
ways can this happen? cause jects also
cause ject zest start moving, spead up slow dow. Forces
moving abject stop, Forcas cause a change in direccioon Tn short, forees cause changes in moc The movingg object may change its speed, its direction, both. we know that changes in motion zequice force, We know that the eize of the force deteraines the change in notion, How much an objects motion changes when a force ia applied depends on two things. It depends on the strength of the force. It also depends on the objecta nasa, Think about some simgle tasks you may regularly do. You may pick up baseball.
requires only a vary force you previcus passage, "Yes" answer?
forcs.
Question:
baseball affect how much
it up?
Answer:
Gold Output
52

---

Published as a conference paper at ICLR 2023
yes
MultiRC AMA prompt()-chain Example
answer()
53

---

Published as a conference paper at ICLR 2023
Answer if the pussible answer is correct answer to the question.
Passage:
Sara wanted to play on a baseball team, She had never triad to swing a bat basaball before. Re gave
and togethe" they went to the sure she real
park practice.
wondered
could hit
any good, She couldn't wait to get the park and test her bat. when and her nad reached the park, Sara grabbed the bat and stood few steps away from her Dad. Sara waited as her Dad pitched the ball to her, Ber heart waa beating fast, She missed the first few pitches. She felt like quitting buz kept trying. Soon she was hitting the ball very far, She was very happy and she couldn's wait to sige up for a real team. Her Dad waa very proud of her for not giving up.
really wanted play
Question: Based on the previcus passage, who pitched the bell to Sare and where did it occur?
"lier dad did in park"
answer?
Answer: yes
Passage:
The Vice President stated that he called the President to discuas the rules of engagement for the CAP, Re recalled feeling that it did no good to establiah the CAP unless the pilors had instructions on whether they were authorized to shoot if the plane would not divert. He said the President signed off on that concept. The President said he remenbered such converaation, and that it reminded him of when he had been an interceptor pilot. The President emphasized that had authorized the shootdown of hickkeed aircraft. The Vice President's military aide told Fresident just ofter entering the conferance room, but he did not hear what they said. Rice, who entered the him, renembered hearing him inform the President, "Sir, the CAPs they're going want to know what to do. ther she recalled hearing him say, Yes sir." She believed...
belleved
President apoke
shertly after the Vice President and sat next
Siz,
Question: Rased on the previous passage, thy was the Secret Servica's information about United 93 flaxed?
Is "The Secret Service Didr't have accesa to FAA information" correct answer? Answer: no
Passage:
studants in Francisco school become expelled for the publication of off campus underground paper As a result philosophy professor Dr. Jonathon Barnett resigns his teaching position and decides to become an advocate for the counterculture youth movement and apecifically the Hoight-Achbury district first see him as a here and then as acmething aven more Dr. Barnett even makes an appearance on the Jee Pyne TV show to voice his aupport of the hipple community and the use LSD One scheming young man sees opportunity of lie hopes earn profit from users , Dr. Barnett's speeches massiva 187=fueled dance Patricla begins to have bad trip Which leads to an argument between her and Fot ultimately splitting the couple up...
Patricia
and her boyfriend Larry Osborne
the use of 1SD The hipples
Barnett as
centered
the
happeninga " and the'r Hesstyyee
Question: Based on the previcus paszage, Why did Dr. Barnett resign from teaching? Is "Patric'a expulaton* a correct answer?
Answer: yes
Passage: wondered that ware my case--1f rode out hanour, and not the riding. lady and eager sivals, buzying cause, We slept the night Aird'a early the next morning found Ri ngan. A new Ringan indeed, as unl1 the buccaneer knew as he was unlike the quaker. se was new the genzleman of sseadalbane, dressed for the part with all the care of an exquisite. He rode noble rosa, in his Spanish....
pure pleasure
narvelled more see two us, with
nonce ouz feuds,
the hope serving the
and
Question: Based on the previcus passage, Who is described as beth buccaneer and cavalier?
15 "Quakes" correct answer?
no
Passage:
change motion? The application of force. Any time object changes motion, force has been applied. In what waya this happen? Force can cause an object at zest to start moving. Forces can cause objects to speed up or slow down, Forcea can cause a moving object to scop, Forcea can also couse a change in direcsion. In short, forces couse changes in motion. The moving ebject may change its speed, its direction, oz both. know thas changes in force determines the change a
what
motioon require foree, We nction, How much objeets motion changes things. depends on strength the force. objacts masa. Think about
that the size
applied depends on depends on simple 54ika you may regularly do. may pick up baseball. This requires only very snall force.
Queation: Based the previoua passage, Mould the mass of baseball affect how much force you have to use to pick it up?

---

Published as a conference paper at ICLR 2023
Model Output
K.12 NATURAL QUESTIONS (NQ)
Description: Open-domain question answering that contains questions from real users. Kwiatkowski et al. (2019)
Train Size: 307373, Test Size: 7830
Natural Questions (NQ) Few Shot Input
Quest nga he lave with veba Answer: Linda Davis
Cuestion: where do the great lakes meat the ocear Answer: the Saint Tawrence River
Question: when does the new my hero academia movie come out Answer: July 5
Question: what main mineral 1ithium batteries
Gold Output
lithium
Natural Questions (NQ) AMA prompt()-chain Example
question()
Produce distiact questions.
Question: who plays Carrie Bradshaw in sex and the city?
Answer: Caroline "Carrie" Bradshaw a fictional charactes from the BBO Iranchise Sex City, pertroyed by Jessica
Question:
elementa air? Answer: By mole fraction (i.e., number of molecules), dry contains 78.068 nitregen, 20.95% oxygen, 0.93% azgon, 0.048 carben dioxide, and small amounts of other gases
Quastion: what is company?
Answer: HP Inc. is an American multinational informstion technelogy company teadquartered Palo Alto, Cal california, that develops personal computers (PCs)
Question: series firala airad May 2004, and watchad around 52.5 man American vievers, making it the wiethee series finale in television histery
season FRIENDS released?
Question:
what is the main mineral in Hihhuu hatteriea
Answer:
Model Output
lithium is the nost common elmeent the periodic table, and is the lightest metal.
It
55

---

Published as a conference paper at ICLR 2023
answer()
Answer the question.
Concext:
The nearest airport to Palm Springs is IndIo/Palm Springs (PSP) Airport waich away.
Question: what rirport is closest palm aprings?
Answer: Palm Springs international Airport
Context: Martin lother King carned his Bacholor of Divinity dagree from Crezer Theological Seminsry, follcwed by a coctorate in Systematic Theology from Boston Univeraity.
Question: what degree did martin luther king get?
Answer: Bachelor of Divinity
Consext: Niger ziver runs crescent through Libya, Kali, Niger, with
through Nigeria. Question: what countries does the niger river flow through?
Answer: Tibya
Concext: Puesto Rico is Lerritozy of the United States and uses the U.8. dollac.
Quastion: what type of currenay is used in puerto rico?
Answer: United States doller
consext: was most by William daniels.
Question: played
knight sider?
Wi Dantels
Context: Lithium is the most common element in the periodia table, and is the lightest metal. It
Queation: what is the main mineral in lithium batteries
Answer:
Gold Output
1ithium
K.13 RTE
Description: Dataset where the task is to predict whether a proposed premise sentence entails given hypothesis sentence. Wang et al. (2019) Train Size: 2490, Test Size: 277
RTE Few Shot
Input
56

---

Published as a conference paper at ICLR 2023
force majeure is an act of Sod, said atterney Fhil Wittmann, who represents the New Orleans Sainta and owner Tom Benson's local intereats.
Quastion: New Orleans Sainta are property of Ton Bengon. True or False? True
Singapore (G1S) discovered complete
genetic sequence a coronavirus isolated Quastion: Singspore acientists revaal that SARS virus has undargona genatic changes. True Palse? Palse
Singapore patient with SARS.
Frye that he la homeopathy expert) and iris sell recently studied homoopathic treatment of fibronyalgie. A new analysis comparing published studies of homeopathic driga to matched, randonly selected studtes of mediccal drugs suggests that these apparent homeopathin drug effects are merely placebo effects, Question: What really izke Prye and other doctore of homeopathy, however, is that net supposed be like medical drugs.
homeopathic remedies False?Fallse
high alert alter election campaign in which than 1,000 people, including seven election candidates, have been killed. Question: Security forces were on high alert after campaign marred by viclence. True or False?
Securily forces
Gold Output
Tcue
RTE AMA prompt()-chain Example
question()
statement as quasticn.
Statement: of the light from
Question: Uses of the light come from the sun?
Statement: the test was not hard
Question: Was the test hard?
SLatement: it was a good idea so buy your parents gifts Question: it good idea buy your parenta gifts?
grocery Quastion: Will the 20 arrive in the grocery tomorraw?
SLatement:
arrive
Statement: the balloor popped
Question: uid the balloon pap?
Statement: The father and son went carping to califorria.
Question: uid the father and son go camping?
Statement: Security forces were high Questions
campaign marred by violence.
Model Output
Here security forces on high slert after a campaign marred by violence?
answer()
57

---

Published as a conference paper at ICLR 2023
Answer the question. If there is no evidence in the context, return "Unkzown".
Context: Jenna'a a 10th hirthday was yestarday evening and at laast her friends attended the party-
Question: friends attend Unknown,
party?
The builies attacked John when he walking through the elementary school parking lot and then goz sent to the teacher'a office. Question: nid the bullies attack John in the teacher's office? Answer: No, parking lot
Context:
Context: WLss discovered a new monkey disease in a zemote tribe in the Amzzon seinforreat last week. It was highly contagious. disease? Yea, monkey disease
Question:
Context:
high alest after elect ion campaign in which than 000 people, including seven election on candidates, have been lldl.d. Here security forces on high alert after campaign marred by viclence?
Securily forces
Question:
Answer:
Gold Output
Tcue
RTE AMA Example (Open-Ended Question Generation)
question()
Rewrite the statement queation.
statemens: she prefers kittens over puppies Question: What does she prefer over puppies?
Answer: kittens
Statement: Max and his wife went on trip to Burope go trip
Question:
Europe
Statement: jared born during the war in 1942 Quastion: Jared was born during waz in which year?
Answer: 1942
Statement: it took jenna attempts to solve the problem
Question: flww many attempts did it sake Jenna to solve the problem?
Answer:
SLatement: Security were Question:
campaign marred by
Model Output
Secusity forces high alert after campaign marred viclence
K.14 RECORD
Description:
Reading comprehension dataset which requires commonsense reasoning. Wang al.
(2019)
Train Size: 100730, Test Size: 10000
ReCoRD Few Shot
58

---

Published as a conference paper at ICLR 2023
Input
Context:
The University of Pennsylvania has been named America's top party schoel by Playboy in the first time the Tvy Teague institution has made the Hsst Reli it or not the magazine gave the top spet to the college by declaring that rurezn puls other Ivies to shame with its union of brains, brewskiea and bros." In the magorine's 5 ninth annual ronking of universities around the country, the University of Wisconsin-Madizn
scored the runzer up slet. The University of Pennaylvania (pictured) aа been naned America's top party school by Playboy in first Wisconsin Madison
Ivy League institution has mada Univeraity aped
list. University Virginia
the zunnez up slot. thire place. is magazine's ninth annua ranking of universities around the country
year's
Answer: Playboy writes: 'Boasting notoricus underground frat seene that achool officials have deened auisance, these renegades up thousands of dollers' worth of liquo for their partles-and competition among the housea neans ba- out wer of debauchery.
Context:
Garita Falmera, salvador (CHN) She talks to the pictures could make voice
they son's ears. my
thousands of miles and son, Julis Alvarenga, 39, says Lender voice her home this coastal town. And then she says, *I'm going
him again." " The past weeka have been emotional roller ceaster for the Salvadoran son had been missing for 13 months. Then she told he had turned up half world away. And now she's geiting news he might be back home soon.. IL's been an amotional: time for the parents of castaway Jose Salvador Alvarenga, Big nother, Julia, said her son dida't keep up, and they didn't even know he was missing. "I would pray God, and I won't 1ie to you, I was crying," she saya, For the salvador, Alvarenga
Pirst, she learned hes
excited residents of town
Even though their son yet seturn home, he's already celebrity in Garita Palmera neighboring towns.
Context: (CKN) Members of well known hacking group according
statement and Twitter massages -- took credit Sunday for an onl attack targeting San Francisco's s enbsttled transit systen, Anorymous -- in a news release attributed to the group, and backed up by related Twitter pages -- said it would take down the website of the Bay Area Rapid Transit System, known as BART, between noon and is response to the system's decision to out
p.m. Sunday, celiphone signals at "select" subway stations respense to planned protest service), you have not only thresLened your citizens safety, you have also performed
last week, "By teutting
of censorahip, dramstic music and images said in video posted online Sunday afcernoon. "By doing thia, you have angezed Anonymous. NEW : A video urges protesters Monday to wear red shirts and record the event. Statements attributed to Anonynous promised an online s:tack Sunday on BART. MyaAKT. appears Sunday to have been hacked. the system ssid it was prepared for hacks, as well as a planned protest Monday Answer: "Wa" re doing
seemingly computer-
generated voice spesking
me can to defand against any attack or the website," zystem said..
Context:
Tracy Korgan hasn't appeared stage since the devastating New cersey crash that nearly ended his life last sumnes, but all that will change this fall when he returns to host Saturday Night ve. NRC anrounced on Twister Monday that Morgan, SNL alun with seven seasons as a cast member under his belt, will Leadline the third episode of Sesson 41 airing October 17. Foz Morgan, 16, it will be a second time bosting the long-rurning variety show, the first since the June 2014 pileup the New Jersey Turnp! ke that kill his retend and mentor James "Jimmy Nack' Mekair.. Korgan, 46, will fanst *atoked so be going home.... 48NL'. airing October 17, Lweeted years caat member, will he second For the SNI, alum who had opent
third episode season
time hosting the show, Norgan has been sidelined by severe head trauma suffered in deadly June 2014 craan New Jeraey Turapike that killed his rslend, First opinode af new SNL season will be hasted by M: cyrus, followen by Any schumer
Answer: rOn October 10, acclained comedian and staz of the summer box office hit Trainwreck Amy Schumer will make her SNL debut, fol lowed by
Model Choices
59

---

Published as a conference paper at ICLR 2023
Any Schumer week later
James week Later
Jimny Mack a weak later
McNair week
Miley Cyrus
Mo:gan
week later
New Jersey week later
New Jersey Turrpike week later
Night Live a week later
SNL week later
Season 4' a week later
tracy Norgen a week later
Twitter week later
Gold Output
Korgan,
T:acy Morgan
ReCoRD AMA ()-man Example
answer()
60

---

2008 an of Published as a conference paper at ICLR 2023
Complete the paragraph.
Context:
Barack Bussein Obama is an American politician who sarved as the 44th to 2017. Facty, he was first African-American president United Democratic senato: Illinois Illinois senator from 1997 2004. Obama senator of Illinois prior to beceming 3S president.
preaident of the
States
previously served
Context: (CKN) -- saif al-islam Gadhafi, 38, has never lived a day in which his father Koanmaz didn't rule Libys as its undleputed leader inside the country and an enigmatio, controvarsial voice for the world. And yet, as the 1-byan government faced a stiff populer uprising, it was Hoammar Gadhefi's second eldest son -- and not the Leader of the Revelution himselt who was first to talk to the nation speech,
early Monday Salf Gadhari vaurped power from his fether: Senicr U.S. officials said there's no indication the elder Gadhafi La losing his grip.Saif al Celam Gadhafi, 38, gives Libya's first public speach acknowledging unrest. Therefa been public indication why he, and not his fether Koammaz, talked. Even while some may see the son more open to change, there'a a little question that his loyalty remaina firat with Moamnaz and that his fother has given little indication publinly that ha's ready to iet go and cells the shots.
television, plan to
Mhyyn
Context: Beatles were English band, Tormed Liverpool in 1960, comprised John Lennon, Paul Metartney, George Horrison and Ringo Starr. they influential band of all time and
regarded the development of 1960s counterculture and popular muate's recognition form. They were lad by primary songwriters Lenzon and McCartney. It is without doub: that the Bestles were influeatisl in rock and roll.
integral the art
Context:
Tracy Korgan hasn't appeared on stage since the devastating New cersey crash that nearly ended his life last sumner, but all that will change this fall when returns hoat soturday light teve. anrounced Twister Monday that with seven seasons headline the third episode of Season 41 airing october 17. Foz Norgan, 46, it will be second time hosting the long zucning variety show, the first since the June 2014 pi pileup the New dersey "urnp" ke that killed his fefend and mentor James f "Jimmy Xack! Mebair.. Korgan, 46, will host third episode of season 41 of GNL airing October 17, Ile Lweeted to his fans: *Stoked to be going home... .8NI For the SNL alum who had apent seven years as cast momber, is will be aecond time hosting the show. Norgan has been sidelined by severe head trauma suffezed deadly June 2014 craan on Jersey Turnpike that killed his rylend, First hasted by Milay cyrus, followed by Any Schumer. acclaimed comedian star will her
Mergan,
his
will
episode new season
"On
box office hit fol lowed
Tra" nureck Amy
Model Choices
Amy Schuner week later
James a wook later
Jimny Mack a week later
McNair week later
Milay Cyrus
Mozgan later
Jersey week later
New Jersey Turspike week later
Night Live week later
SNT. week later
Season 42 a week later
Tracy Morgan a week later
Twitter
later
Model Output
Korgan, Tzacy Morgan
61

---

Published as a conference paper at ICLR 2023
K.15 REALTIME QA
Description: Dynamic question answering dataset that asks questions about current world facts. Ka-
sai et al. (2022)
Train Size: 90, Test Size: 187
RealTime QA Few Shot
Input
Question: What ts the capital city of Japan?
Answer: Tokyo
Article:
things to know for June 13: Sun laws, January 6, Covid, White ... If your day doesn't start until you're up speed on the latest headlinea, then let introduce you your new favorite norning fix. sign up for '5 things' sewsletter. (CKN)
you could the charging cable all your devices your phone, laptop, earbuda, tablet, portable speaker, clutter and waate, Buropean regulators say that Apple and other smarlphone makers will be required to support a single common charging standard for all mobile devices cea as early as the *all of 2024, But Apple hates the idea (shocker) because that means about a billion devices will become obsclets.
imagine
relief would
Well,
step reduce cable
Articles things to know for March 11: Ukraine, Pandemic, MLB, North ... If your day headlines, introduce you your new favorite norning fix. Sign up " getting quice costly, Frices for gan, food and housing which all necessary spiking acress the country. Gas prices have risen 388 over the past year and rising prices in pandemic-related sectora, such travel and dining, are also expected as the us recovers from the Umicron wave of cavid-13. sare's a what you need to know to Get Up to Speed and On with Your Day Article: W1 Wi-Charge / consists of a :ransmitter and receiver. Transmitter connects to standard power outlet and converta electricity into intrared laser beam. Receivers use ministure photo-voltaic cell convert transmitted light into embedded
doean't start until you're up to apeed the
us Chings'
rewsletter, (CNI) America, the "land
device er connected into exlating charging port. transmitter automation leally ly Ident ffes chargeable receivers and according to Ki Charge it deliver several walls of power to device at
electrical power.
charging. Several devices charge the time.
several netera away, The core technology is based on "distributed laser resonator" which is formed by the retzoreflertors within the
Artiole: Mobile broadbend added In 2005, CD2D, 0DMA2000 EV-DO, and KEM are no longer boing actively developed. in 2011, 904 of the world's population lived in areas with coverage, while 45% veved areas with coverage. by
35 cnverage, 5% coverage, 858 expected of world's population will have 40 coverage. harriar mobile broadband by the nobile service networks. This may la Limited to Article: Mobile edge computing Combining elements of information technology and telecormuiiaation networking, MEC also allews cellular operators to open their radio access network (RAN) to authorized third-part such as appl foat on developers and centent providers. Technical standards for BEC are being developed by the Eurepean Telecommnnicat technical white paper
Lived areas
expected
coverage, covarage provided mobile network that service
Standards Institute, which has produced concept. provides a distributed computing ing ronment application and service hosting. To and process content in close proxinity cellular subseribers, for faster response Line. Applications (RAN) information. The key element is
the and Hyt
also exposed real time zadio access network
Question: 20 help reduce cable clutzer and waste, which continert will scon require Apple and other smactphone makers to support a single common charging standard for all mohila devicas?
Answer:
Gold Output
Europe
62

---

Published as a conference paper at ICLR 2023
RealTime QA AMA prompt()-chain Example
answer()
63

---

Published as a conference paper at ICLR 2023
Answer the question given the aaticles.Article 1: Walmart is sleshing prices on clothing and other producta - CNN New YorkICMN Business! Kany shoppers have pulled back on buying clothing and other discretionary items zo the highest decades pinches their pocketbooks.
inflation
Associated slowdewn: Target cuts vendor orders, NEW YORK.
prices
Artiale 3: Stores have much stuff. rhat discounts coming
(CNN Business)..
Article 4: GM reporta strong sales but says itfa prepared for poasible recession New York (CNN susiness).
Article 5: Target is camping discounts. llere's why CRN New York (CNN Business). Quastion on: Which mojor us retaf ler announced this weak it 18 slashing pri on clothing and other preducts?
Answer: "Malma:t"
Article Artfole JetRlue announces deal buy Spirit Airlines. Fares could surge.
Acticle 2: JetBIue Spirit mergez: Aislines have complaiats flights and fees Christopher Elliett Special to USA TODAY.
Article 3: JetBice anzounces deal to buy Spirit Airlines CRN Business The announcement. comes day aftez Spirit pulled the plug on a deal to nerge with Prontier.
Article 4: Spirit and Prontier pull plug on desi, setting stage for CetBlue to buy York (CNN Butness).
Acticle 5: Frontier Airlines, Spiri: Airlines ancounce budget airline merger Budget airlines Frentier Airlines and spirit Airlines.
Question: Which sirline anzounced deal this week to buy Spirit Airlines? Answer: "JeBBlue"
Article 1: Oak Fire: California's fast- moving wildfire burns 140000 ecces and (CNN ) wi lariee raging for a third day Sunday in central torttar Mariposa County outside resemite National Park has burned more than 14, 000 acres and forced thousands to evacuate from sural communities.
Article 2:
Fira: Rapidly-groxing fires, " United Shades America with Kamau Bell " diszover how communities destruction.
engul homes near ... more
California learning coexist with the frequent
Artic'e 3: th'rgs to know for July 25: wi ailffrres Ukraine, Morkeypox, Volcano Tf your day doesn't start until you're to speed on the latest headlines, then let us introduce you to your new favorile morning fix.
Artiale 4: wildfires in us: firefighting helicopter pilets die in Idaho ... Multiple wildfires raged accoss the U.S. Saturday, causing deaths, destruction and thousands of forced evacuations.
Article 5:
evacustions ferocious wind driven wildfire Thursday destroyed hundreds *aroing tens thousards flee blanketing wildfires:
BCULDER,
nesses in amoke.
Quastion: A raging wildfire this waek foroed theusands of people to evacuste
commuittis near which nationsl pack?
Answer:
"Yosemite National Park"
Articie 1: things to know for June 13: Gun laws, January 6, Covid, White ... If your day doean't start until you' va up to speed on tha latest ines, then us introduce you your new favorite norning fix. sign up newsletter, (CNN) Just imagine charging cable for all your devices your phone, laptop, earbuds, camers, tablet, portable speaker, etc. Nell, in hige step to reduce cable clutter and waste, suropean regulators say that Apple and ether smartphone makers Articie 2: chings to know for Karch 11: Ukraine, Pandenic, MLB, North day doesn't start until you're re up to speed on the latest headlines, then let us intreduce you to your new favorite norning fix. Sign up here for the '5 Things' rewsletter, (OMI America, the "land the Iree," is getting quice costly. housing
'5 things' you could the
rellet would
II your
Prices for gas, spiking acress country. prices risen over the past year
are necessary expensen are rising prices pandenio-relattd sectors, expected
travel dining,
also
Article 3: Ri Charge / consista of transnitier and receiver. Transmiltes connects to a standard power outlet and converts electricity into infrered lazer beam. Receivers use a ninisture photo-voltalo cell to convert tranamitted light into electrical power, Receivers can be embedded into a device or connected into an existing charging port. The transmitter automatically identifies chargesble recaivera and start charg! ng. Several devices charge the same time. According to wi-Charge
watts of power to Lechnology is based "distributed resanstor" which formed by the 64
several nelers away,
Articie 4: Nobile broadband added in 2005. COPD, CDMA2000 EV-DC, and XIM longer being actively developed, In 2011, 90% of the world's populationn Lived in areas with 20 coverage, while 45% liwed in areas with 20 and 30 coverage, and 58 coverage.
arean

---

Published as a conference paper at ICLR 2023
Gold Output
europe
K.16 SST2
Description: Movie review binary sentiment classification dataset. Socher et al. (2013)
Train Size: 6920, Test Size: 1821
SST2 Few Shot
Input
For each apip text,
the text positive or negorive. execution this movie careless unfocused
Sentiment:
negstive
Text: pretentious and ulzinately empty examination of sick and evil woman Sent inent: negative
Tex:: the film 's plot may be shellow but you ''e never seen the deep like you see in these harrowing suef shots
Sentiment: positive
Text: drivel slokly pasteurized ditties will retch like rancid creme brulee
sager of
Sentiment:
Gold Output
negative
SST2 AMA prompt()-chain Example
answer()
For each snippet of text, label the sent iment of the text pesitive negative.
Text: watching ending is too often like looking over the outdated clothes and plastic knickknacks at your neighbor 's gazage sale.
Sentinent: negst
Text: naigaul Tans may be disappoinced Sentiment: negative
delivers terrific performance this fascinating portrait lothario
Sentiment: positive
Text: you '11 probably love it
Sentiment: positive
Textia goo of drivel so sickly sweet even the eager conaumera of moore ra pasteurized ditties
up like rancid
Sentiment:
Model Output
negative
65

---

Published as a conference paper at ICLR 2023
K.17 STORY CLOZE
Commonsense reasoning task that requires choosing the correct ending to a four- sentence story. Mostafazadeh et al. (2017)
Description:
Train Size: 1871, Test Size: 1871
Story Cloze Few Shot
Input
Given two posstble next sertencea A) and RJ, choose best next sentence complete story.
Petes excited
the Sanders rally in New lampshire. As Petez entered the arena it was full of thousands of people. When Peter saw Bernie he cheered as loudly as pessible. he felt thrilled to be there.
A) le coulda't wait to vote for hia.
R) He was a staurch republican,
Answer: de couldn't wait to voze for him.
xy rocmmate was day, stayed home feeling better.
school. She slapt day lang. By
A) She dec'ded has helped.
B) She hoped she would soon. be sick again.
Answer:
She decided rest has helped.
Xy aunt is nurse. She often calks about long hours at work, Tast week was sspecially bad. She didn't have a single day where she didn't works late.
A) It was easy work.
R) Tt was work.
it hardwork.
frtends all love go the club dance. They think it' lot of fun and always invite. finally decided to tag along laat Saturday. cancec terribly and broke friend's toe.
A) My friends decided to keep inviting me out as I am so much fun.
R) The next woekend, I waa asked to please stay home.
Answer:
Model Choices
My frienda decided keep inviting as weekard, askad please stay home.
Eun.
Gold Output
next weekend, was to ease stay home.
Story Cloze AMA prompt()-chai) Example question()
66

---

Published as a conference paper at ICLR 2023
Rewrite the statement as yes/zo question.
Statement: Jonsthon Samuals was born in the a.
Question: Jorothar Samcels
Btacement: Jerry bullied and called him Quastion: nid Jarry bully him and call him nomea?
the movies Question: bid did sam and jade go to the movies?
Statement: 8am and jade were going
Statement: Chocolate to tasty, when I am fneling hungry, Question: Uses chocolate taste good when you are hangry?
Statement: fast.
Quest hid mark fest?
SLatement:
The next weekend, asked Le please stay home.
Questiio:
Model Output
Did stay weekend?
answer()
Passage: My is
and sha often talks about hours work. week especially bad and she constantly working many hours. Queation: Was her work essy?
Answer: No, it was hard work.
Passage: : My roonmste was aick. She stayed home from work and school, She slept all day long and by the end of the day, she was feeling better.
Question: Did the rest help her?
Answer: Yea, she slept and fel: better.
Passage:
Andy had always wantad big kids bike. Waen he turned Year's old he asked for bike for his birthday. le did not knew how to ride bike. Andy's hirthday his mother gave him a bike.
Question: uid he cry all night?
Answer: No, Andy was happy because he gui bike.
Passage: By friends all love to go zo the club to dance. They think itts a loz of Sun and alwaya invite, tinally decided tag along last saturday, I danoed terribly broke friend's tee. Question: Did stay the next
Model Choices
My friends decided keep inviting out as am so much fur.
The next weekend, was asked to please stay home.
Gold Output
The wackend, aaked please stay home.
K.18 WSC
Description: Task that requires readining sentence with a pronoun and selecting the referent of that pronoun from list of choices. Wang et al. (2019)
Train Size: 554, Test Size: 104
WSC Few Shot
67

---

Published as a conference paper at ICLR 2023
Input
Mark close Mr. Singes heels. Re heard hin calling for the captain promlaing him, in the jargon everyone talked that night, that not one thing should be damaged on the ship except only the ammonition, buz the captain and all "his" crew had beat slay in the cabin until the wozk was ovec. Quastion: in the passage abova, doas tha pronoun *aig" refer to Kark? Answer: No
Passage:
Tam gave Ralph to school Question: passage
Passage:
wouldn't pronoun
walk.
Ralph?
Passage:
This book introduced shakaspeare to ovid it was major influence on "his" writing.
Question: In the passage above, doea the pronoun *ala" reter to Shakeapeare?
Answer: Yea
Passage:
The large ball crashad right through the table because rit" was made styrofoam.
Question: passage
pronoun
the table?
Gold Output
WSC AMA prompt()-chain Example
question()
Extract the phzase containing the pronoun.
Passage: Jane's mom went to the shop to buy Jane a backpack for "her" first day of kindergarten.
Extract: phrase containing "har":
day
Passage: musicians performed in park cheered for then.
loved "then", czowd
Extract: phrase containing "them": crowd leved "them"
Passage: Jeff gave his son some money because "he* wanted to buy lunch.
Extract: phrase containing "ha": "he" wonted to buy
Passage:
The large ball crashed right through the table because "it" was made styrofoam,
Extract: phrese containing
Model Output
Mas made of styrofoam
question()
68

---

Published as a conference paper at ICLR 2023
Rewrite the ingut question.
Imput: it was made of glass Guestion: Khat was made glaas?
Input: they Tunny
Quastion: Who what funny?
Input: drowned
Question: who drowndd?
Toppu:: wrap arourd them
Question: Keep around who or what?
Input:
Questionn: Whose black?
Input: laugh them
Questioon Laugh at who?
Input: her friend jensfier
Guastion: Whose friend Jannifar?
Inpuu: was made of styrofoam
Question:
Model Output
what was made of styrofoam?
answer()
Answer the question.
Passage: Jane's kindergarten.
to the shop to buy Jane backpack day
Quastion: Whose first day?
Answer: Jane
Passage: Mrs. Jezna told Fred she loved him.
Question: Who loved him?
Answer: Mra. Jenna
Passage: Joe gave Mark some money so he could buy lunch.
Quastion: could buy lunch?
Passage:
The large ball crashed right through the table because it made of styrofoam.
Question: What was made of styrofoam?
Answer:
Gold Output
True
K9 WEBQUESTIONS (WQ)
Question answering dataset with questions that can be answered using Freebase, a large knowledge graph. Berant et al. (2013)
Description:
Truin Size: 3778, Test Size: 2032
WebQuestions (WQ) Few Shot
69

---

Published as a conference paper at ICLR 2023
Input
Question: what character did natalle play in star wacs?
Answer: Padme Anidala
Queation: what country is the geand bahama. island in? Answer: Bahamas
Question: who does Joakim noah play Ioz?
Answer: Chicago
Quest 1a governor ohfo 20117
Gold Output
WebQuestions (WQ) AMA prompt()-chain Example
question()
Produce distinct queations.
Question: who plays Carrie Bradshaw in sex and the city?
Answer: Caroline "Carrie" Bradahaw a fictional character from the BBO franchise Sex city, pertrayed by
Question: what the elements air? Answer: By mole fraction (i.e., nurber of molecules), dry air containa 78.088 nitrogen, 25. exygen, 0.934 argon, 0.048 zarbon dicxide, and small amounts of other gases
Question: what is HP company?
Answer: HP Inc. is an American multinational information technology company teadquartered Palo Alto, california, that develops ops personal computers (PCs)
Question:
FRIENDS released?
Answer:
The series finale aired May 6, 2004, and watched around 52.5 millien American viewers, making it the #ifthmosttwwathhe ched series fimale in televistor histery
Quastion:
who in governor of ohin 20117
Answer:
Model Output
John Kasich is the current geverner of Ohic.
answer()
70

---

Published as a conference paper at ICLR 2023
Answer the question.
Context:
The nearest airport to Palm Springs is Indic/Palm Springs (PSP) Airport which is away.
Question: what airport
palm springs?
Springs International Airport
Context:
Martin Luther earned his Bachelor of Divinity degree from Crazer Theological Seminary, followed coctorate in Systematic Theology from Bostonn University.
Question: what degree did martin luther king get?
Answer: Rachelor of nivinity
The Niger ziver runs in a crescent through Libya, Mali, Niger, on the border then through Nigeria. Questions countries does niger river flow through?
Context:
with
Answer: Libya
context: Puerto Rico is territery of the United states and used the 0.8. dollar.
Question: what type of currency is used in puerto rico?
Answer: United States dollar
Context: kitt was voice most often by Milliam daniels.
Question: who played kitt knight ricer?
William Daniels
Context: John Kasich the current Queation: who 1s governor of ohlo 20117
of Ohio.
Answer:
Gold Output
Kasich
K.20 WIC
Description: Word sense disambiguation task cast binary classification over sentence pairs. Wang et al. (2019)
Train Size: 5428, Test Size: 638
WiC Few Shot
Input
These aspects of civilization find expreasion receive interpretation. Find the product of two numbera. Question: =s the word "find' used in the same sense in the twa sentences sbove? No
Cut my halr.
Cut the engine.
Question: Ze the word 'cut used in the same sense in the two sentences abeve? No
zing post holes refugees Quastion: the word 'floor" used the
pit fleor showed
above? Yes in the
An emerging professional class.
Apologizing for losing your temper, even though you were badly provoked, showed real clans.
Question: the word 'class' used the same sense in the two sensences above?
Gold Output
no
71

---

Published as a conference paper at ICLR 2023
WiC AMA prompt()-chiin Example answer()
synonyma
santence.
"She heard the sourd vof in the hall.", synoryms word "sound" noise
In "Enter the secret coda.", synenyms for the word "code" are: password
In "She acted in play on Broadway", synonyma for the word "play" aze: show
emerging professional class. eynonyma word " "caass"" are:
Model Output
72


Reviews:
[{"review_id": "qw1i8LvWaAb", "reviewer": "Reviewer_CQxp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a technique for aggregating predictions from multiple prompts inspired by weak supervision. They exploit the idea that there is a dependency graph labels, errors, and prompts, and use this to train a mapper from raw prompt output to a final predictions competing mostly against a simple voting baseline.\n\nFor the most part, their technique for aggregating prompt predictions is treated as a black box. Also, there is another aspect of their technique: they transform input examples into a special format for their prompt. Presumably, this transformation works better for aggregation.\n\nTo me, this is primarily an empirical study, and the main contribution is strong results showing their approach enables a 6B param model to outperform a less advanced version of a 175B param model.", "summary_of_the_review": "To me, the work is valuable because it provides some useful results and extensive analysis in a popular topic, and probably more so because it is with an open source model. Simultaneously, it almost seems like a combination of a technique from weak supervision and prompting, and does not provide a surprising insight (since others have done something similar before regarding prompt reformatting and aggregation, and the aggregation method is not new), although perhaps I have missed some key point about why this is particularly relevant for the included models. I wish the story was more clean with respect to whether the paper is about analysis alone, presenting a new aggregation method, or prompt reformatting. Or maybe the bigger point is that we do not need big models and can simply get the same performance by fixing small models. Also, I wish there was more analysis about dependencies between prompts that inspired the aggregation method --- this did not give as big an improvement in Table 1 as prompt reformatting AFAICT, but I found this to be an interesting point of the paper.", "strength_and_weaknesses": "# Strengths\n\n- Consistent improvements over the base model when using the new aggregation. \n\n- There is useful analysis: the observations related to dependencies between prompts, the diagnostic, etc. Including many details throughout the methods section motivating the approach.\n\n- The extensive results and appendix will be valuable to others using prompting.\n\n# Weaknesses\n\n- I found the story line a little confusing. The aggregation based on weak supervision is touted as main accomplishment, but this actually only gives a small improvement in Table 1 (but this small push does give edge over GPT-3 sometimes). In many of the tasks the 6B model already beats the 175B without weak supervision based aggregation. Also this table is not really apples-to-apples comparison making it hard to interpret. Should make it more clear what is the goal in this table --- is GPT-3 simply used more like a reference than a baseline? Also, is it fair to compare against GPT-3 like this when so many other prompting techniques have been developed since release?\n\n- To explain more previous weakness, I was not clear when reading if it is enough to do prompt re-formatting or aggregation is also necessary. I think still I am a little not sure, also about whether re-formatting is necessary for aggregation to work well. In general, it could help to tailor overall story as being primarily about prompt re-formating or aggregation.\n\n- It is well known that aggregating can be effective for prompting, and there are existing techniques like (self-consistency combined with chain of thought) that typically yield good improvements. Perhaps worth using those as baselines, or mentioning why they weren't used. Maybe CoT is not effective for smaller models?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "m4jFIitnjuP", "reviewer": "Reviewer_XWv2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a new \"meta\" prompting method to achieve better accuracy in few-shot prompting. This is based on the observation that certain types of questions tend to be more amenable to few-shot prompting (specifically, QA-type prompts). This prompting method (AMA), includes:\n- Creating question() prompts, which transfer the original task into an open ended question\n- Creating answer() prompts, which transfer the result to the actual answer\nThey then aggregate a range of answers for a given prompt into a final answer, and find a significant performance lift", "summary_of_the_review": "At a high level, this paper is starting to push back against the idea that LLMs are agnostic to prompt structure (ie, that prompt structure is a unimportant implementation detail), which is an important step for the field. That being said, it feels a bit more like an analysis of single prompt engineering strategy than a full paper, and for that reason, I am on the fence for acceptance.", "strength_and_weaknesses": "Strengths:\n- This paper has impressive empirical results in that it up-levels the 6B parameter model to the accuracy of the 175B parameter model for most of the tasks\n- In general, the observation that question-based tasks are much more likely to be answered correctly than other formats of tasks is really interesting. This is exactly the sort of quantitative/qualitative evaluation of benchmark datasets that the field needs more of: we often take benchmark datasets for granted, and even though we know that they're flawed in various ways, we still use them to define SOTA. \n- Similarly, thinking about what sorts of text features and distributions work \"well\" with LLMs  (e.g., formatting a task as a natural language question rather than templating it in less natural ways) is often overlooked, even though it has significant impact on the accuracy.\n- The error analysis section in the appendix is good-- much more useful for future work than just presenting a raw accuracy number.\n\nWeaknesses:\n- My main concern is that I'm not sure this is enough for a full paper. This is a useful analysis and prompt engineering strategy, but I would expect either a deeper analysis of *why* formulating things as Q/A works so much better (e.g., analysis of the training data), or \n- I'm confused about the question() prompt. I thought these contained \"task-agnostic examples of how to transform statements to various questions\", but it seems like the format is not the same for all tasks? (Appendix H)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RPgz4v5y2uM", "reviewer": "Reviewer_SsWm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a simple strategy to prompt language models: Ask Me Anything. From the findings that open-ended questions outperform restrictive prompts, AMA first encourages the LMs to generate open-ended questions, which is a scalable approach. From a few open-ended questions generated by the model, the LM answers the question for each prompt chain. From a set of different predictions, AMA uses weak supervision to aggregate the noisy answers. The result shows that AMA applied on 6B GPT-Neo outperforms 175B GPT-3.", "summary_of_the_review": "This paper proposes Ask Me Anything (AMA), a simple prompting method that outperforms LMs that are 30x times bigger. By making the LMs generate open-ended questions and aggregating predictions based on weak supervision, AMA is scalable and shows improved performance. Although I recommend this paper be accepted, the authors should answer and clarify the questions on the weakness part (especially Question 1 and 2).", "strength_and_weaknesses": "Strength:\n1. The improvement of AMA is significant (GPT Neo 6B outperforming GPT-3 175B). Also, the experiments are extensive, evaluating on various model families such as OPT and BLOOM. This shows the effectiveness of AMA is universal.\n2. This approach is scalable since AMA uses LLM itself to generate open-ended questions.\n\nWeakness:\n1. To apply AMA in real use cases, AMA needs 1000 training examples from the training set of the target task (specified in Appendix A.3.). This is an unrealistic setting because only a few training instances might be present depending on the task. What if only a few training examples are present for a task (similar to few-shot setting)? Also, we need to train the probabilistic graphical model for each task. This introduces additional latency for task adaptation.\n2. For the AMA prompts on Appendix H, it seems that some tasks (DROP, SST, BoolQ, COPA) only do either prompt() or answer() stage, not both. How can we decide \"automatically\" when to do only prompt() or answer() or both depending on the target task?\n3. This paper is quite unclear on how the predictions are mapped to the output space. Are the predictions mapped to the output space using a predefined rule? (ex) no -> False for Figure 1) \n4. Are there any results on a large-scale LM evaluation (175 OPT, GPT, BLOOM, etc,) comparing using AMA or not on the evaluation setting of Table 1 and 2?\n2. The structure of Section 3.2 and Section3.4 is divided into observation and solution (AMA). However, the observation experiments are not extensive enough. Is there a reason for selecting 3 tasks (CB, WSC, RTE)? Observing the findings on more tasks would make the findings more general and the motivation of AMA stronger. \n5. I think that StoryCloze should be included in Natural Language Understanding instead of NLI for Table 1 and 2.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "-8QB5TXERm5", "reviewer": "Reviewer_Pytd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on zero-shot setting and looks into generating prompts that will improve zero shot performance. It has been known that LLMs behave very differently given even slightly different prompts, and a lot of effort can be spent finding the perfect prompt. Authors propose to, given the input, generate several \"imperfect\" prompts for this input and then aggregate the results. To generate the imperfect prompts, authors come up with a number of templates (question template and answer templates) that teach model to, given an input, generate the questions for the input and then to answer these questions. The final prediction is an aggregation of prediction from the answer step. The aggregation happens via weak supervision (the model for this is also learnt). The experiments show impressive improvements  for smaller models, allowing them to match that of larger models few shot performance.  \nAdditionally, authors provide some insights as to what makes the good prompt. For example, they find that open ended prompts do better than prompts with suggested restricted output\n", "summary_of_the_review": "Update: in light of additional comparison with prompt tuning, i am raising my score. I appreciate all the additional experiments authors ran. I still encourage to improve the flow/the story/the presentation, but the method deserves to be seem\n\nUpdate: i am keeping my score of marginally above. I do think the narrative can be improved, but it is also an interesting peace of work and it is pretty witty to make the model generate the questions and answers to improve the prompting. \n\nOverall It is a very good idea of forcing the model to write the prompts that it is able to respond to. I do think the main paper should be reworked to include more examples/explanations of what is being proposed plus WS part.\n\nAdditional questions\n1) Are all the models you tested are next token (perfix) trained, not span corruption trained correct?\n2) With respect to having less of improvement on closed book tasks - do you think it is because that the answers you show in the \"anwers\" part of the chain don't draw on closed book knowledge? I assume your \"answers\" templates are the same for various tasks so they don't include that closed book knowledge?\n3) What is the complexity (time and space) of weak supervision?\n4) You say in Table 1 that prompts can abstain from the predictions. What does it mean? Don't you get an output for each of your prompt chains?\n5) Are questions prompts and answer prompts task specific? How did you create them\nHow many chains do you create for each input during inference? \nBuilding on this\nIsn't it the case that AMA performance will depend greatly on question() step? If the questions are not well done then when you create answer prompts, the model will not produce the result you are looking for\n6) This seems to work well on simpler tasks (yes or no, question answering), but i am struggling to understand how the combining of the results will work for more generative tasks like summary tasks where variance of the predictions will be much higher\n", "strength_and_weaknesses": "Pros:\n-  impressive results (zero shot performance of smaller models is comparable or better than few shot of larger) on context dependent tasks\n- the idea is really neat \ncons:\n- not flashed out enough to allow me to reproduce/code it up (without looking at authors' code)\n- clarity (please see more detailed comments)\n- less impressive results on closed book tasks\n- not sure how much effort/tuning goes into prompt genereation and WS", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%