Bidirectional Language Models Are Also Few-shot Learners
Authors: Ajay Patel
Bryan Li
Mohammad Sadegh Rasooli
Noah Constant
Colin Raffel
Chris Callison-Burch
Venue: ICLR 2023 poster (2023)
Keywords: prompting
prompt-based learning
mt5
t5
machine translation
llm
large language models

Abstract: Large language models such as GPT-3 (Brown et al., 2020) can perform arbitrary tasks without undergoing fine-tuning after being prompted with only a few labeled examples. An arbitrary task can be reformulated as a natural language prompt, and a language model can be asked to generate the completion, indirectly performing the task in a paradigm known as prompt-based learning. To date, emergent prompt-based learning capabilities have mainly been demonstrated for unidirectional language models. However, bidirectional language models pre-trained on denoising objectives such as masked language modeling produce stronger learned representations for transfer learning. This motivates the possibility of prompting bidirectional models, but their pre-training objectives have made them largely incompatible with the existing prompting paradigm. We present SAP (Sequential Autoregressive Prompting), a technique that enables the prompting of bidirectional models. Utilizing the machine translation task as a case study, we prompt the bidirectional mT5 model (Xue et al., 2021) with SAP and demonstrate its few-shot and zero-shot translations outperform the few-shot translations of unidirectional models like GPT-3 and XGLM (Lin et al., 2021), despite mT5's approximately 50% fewer parameters. We further show SAP is effective on question answering and summarization. For the first time, our results demonstrate prompt-based learning is an emergent property of a broader class of language models, rather than only unidirectional models.

Published as a conference paper at ICLR 2023
BIDIRECTIONAL LANGUAGE MODELS ARE ALSO FHOT LEARNERS
Ajay Patel
Bryan Li University of Pennsylvania
Mohammad Sadegh Rasooli Microsoft
University of Pennsylvania
Noah Constant Google Research
Colin Raffel
Chris Callison-Buri
UNC Chapel Hill
University of Pennsylvania
ABSTRACT
Large language models such as GPT-3 (Brown et al., 2020) can perform arbitrary tasks without undergoing fine-tuning after being prompted with only a few labeled examples. An arbitrary task can be reformulated a natural language prompt, and a language model can be asked to generate the completion, indirectly perform- ing the task a paradigm known as prompt-based learning. To date, emergent prompt-based learning capabilities have mainly been demonstrated for unidirec- tional language models. However, bidirectional language models pre-trained on denoising objectives such as masked language modeling produce stronger learned representations for transfer learning. This motivates the possibility of prompting bidirectional models, but their pre-training objectives have made them largely in- compatible with the existing prompting paradigm. We present SAP (Sequential Autoregressive Prompting), a technique that enables the prompting of bidirec- tional models. Utilizing the machine translation task as a case study, we prompt the bidirectional mT5 model (Xue et al., 2021) with SAP and demonstrate its few- shot and zero-shot translations outperform the few-shot translations of unidirec- tional models like GPT-3 and XGLM (Lin al., 2021). despite mT5's approx- imately 50% fewer parameters. We further show SAP is effective on question answering and summarization. For the first time, our results demonstrate prompt- based learning is an emergent property of broader class of language models, rather than only unidirectional models.
INTRODUCTION
Recent work on GPT-2 (Radford et al., 2019) and GPT-3 (Brown et al., 2020) have shown that large language models possess few-shot learning capabilities and zero-shot instruction following capa- bilities, despite only being pre-trained with self-supervised causal language modeling objective (which is to predict the next token).
An arbitrary task can be converted into natural language task specification, often called a prompt. Prompting a task in this way makes its format similar to the language modeling objective used to pre-train large language models. In the zero-shot setting, this prompt contains just the task with instructions, whereas in the few-shot setting, the prompt contains both the task and several example demonstrations. When language model is tasked generate text complete this prompt, it can perform the task in the process. The broader paradigm of reframing all tasks as text generation is known as prompt-base based learning. In the few-shot setting, the learning that occurs from examples provided in a given prompt (the context) is known as in-context learning (Liu et al., 2021). In the zero-shot setting, models perform instruction following (Ouyang al., 2022), with their perfor- mance guided through natural language instructions provided in the prompt.
Emergent prompt-based learning capabilities have mainly been demonstrated for unidirectional lan- guage models. Bidirectional language models have stronger learned representations (Devlin et al., 2019; Conneau et al., 2020; Raffel et al., 2020); however, they have not been able to broadly
"Correspondence jjayp@upenn.er edu

---

Published as a conference paper at ICLR 2023
Prompt:
Prompt
Tronslate Spanish English.
Transiste Spanish English.
Translate clima
Sponish: es
Spantsht soleado.
English The weather sunny.
English: The weather sunny.
English The weather is sunny.
Spanish: MI perto cachorto.
Spanisl:: Mi parro de un cachomo
Spanish: Mi parro un cacharno.
Englisit puppy
English: My dog puppy.
English puppy.
Spanish: Los erboke son importantee.
Spanial: Los arboles son importantes.
Spanish: Los érboles importantes.
Figliay <d>
Fnglish: Trees <X>>
Figlish Trees ane <X>
Generation:
Generation:
Generation:
*x Trans son importantes.
<Xs importantes.
<X- important
Trees
Trees are
Trees are important
Figure 1: A visualization of our SAP technique extracting high-quality translations from mT5, In the zero-shot setting, the examples used in the prompt are synthetic examples retrieved in fully unsupervised manner.
demonstrate the same few-shot in-context learning capabilities zero-shot instruction following capabilities due to the incompatibility bidirectional denoising pre-training objectives have with the prompting paradigm. Instead, they typically require fine-tuning. Bidirectional models are not able to generate long, fluent completions to prompts since they are usually only trained to output sin- gle tokens or short spans of text to in-fill masked tokens during pre-training. We discuss this more in-depth in Section 2.1.
Today, language model architects are faced with difficult choice between unidirectional bidirec- tional models. The authors of GPT-3 lay out this design dilemma in Brown et al. (2020):
"GPT-3 has several structural and algorithmic limitations as result our experiments do not include any bidirectional architectures other training objectives such denoising our design decision comes at the cost of potentially worse performance on tasks which empirically henefit from bidirectionality
making bidirectional model at the scale GPT-3, and/or trying to make bidirectional models work with few- or zero-shot learning, is promising direction for future research, and could help achieve the 'hest of both worlds"."
In this paper, we directly address this dilemma. We contribute new technique, SAP (Sequential Autoregressive Prompting), that enables bidirectional language models to take advantage of prompt- ing and allows them to perform at the level of unidirectional models in few- zero-shot learning without fine-tuning. SAP iteratively prompts bidirectional models, concatenating previous genera- tions back into the prompt, to produce longer generations from models that were only pre-trained to output short, mask-infill spans. We acknowledge efficiency concerns in Section and we discuss the importance and impact of SAP and its results to the field regardless of those concerns.
Using the machine translation task as in-depth case study, we empirically demonstrate mT55 (Xue al., 2021), a bidirectional language model, used with AP outperforms its unidirectional counter- parts, GPT-3 and XGLM (Brown et al., 2020; Lin et al., 2021) in both the few-shot and zero-shot settings, while utilizing approximately 50% fewer parameters. We then examine SAP's effective- other tasks such question answering and summarization, demonstrating that bidirectional models can be prompted for tasks beyond machine translation.
Our work hints at the possibility of more efficient and performant few-shot learners through pre- trained language models that incorporate bidirectionality. We discuss this impact and outline future research directions to this end in Section 6. In summary, our key contributions are:
2

---

Published as a conference paper at ICLR 2023
1. We introduce SAP, technique that enables bidirectional language models to work with few-shot and zero-shot prompt-based learning at a level that exceeds unidirectional mod- els. Our results demonstrate in-context learning and instruction following emergent properties of a broader class of language models, rather than only unidirectional models, addressing long-standing challenge in language model design and use.
2.
We perform an in-depth study of the effectiveness of a bidirectional language model, mT5, with SAP on the machine translation task. Evaluating over 14 language pairs, despite using approximately 50% fewer parameters than GPT-3 and XGLM, we find SAP with mT5 has improved average few-shot and zero-shot performance over all language pairs, and especially has improved performance individual low-resource language pairs.
3. We propose a range of improvements -ilttennnnttiiiiinnnn... prompt ensembling, and mnglish-centric bootstrapping to the unsupervised machine translation procedure outlined by Han et al. (2021) to better adapt the bootstrapping process for unsupervised low-resource machine translation.
4. We assess SAP'S performance the tasks of question answering and summarization, and we find the technique enables few-shot in-context learning and zero-shot instruction fol- lowing capabilities of bidirectional models in tasks beyond machine translation.
RELATED WORK
2.1 UNIDIRECTIONAL AND BIDIRECTIONAL LANGUAGE MODELS
Transformer-basee language models (Vaswani et al., 2017) be broadly categorized into bidirec- tional and unidirectional models. Bidirectional models are models that use a denoising pre-training objective (such as masked language modeling), allowing them to utilize bidirectional context when learning language representations. Unidirectional language models are models with causal-or left-to-right language modeling objective (such as next token prediction), restricting them to be unidirectional when learning representations (Liu et al., 2021).
The T5 family of models, such as T5 v1.1 and mT5, and BART-style models (Lewis et al., 2019) are bidirectional, while GPT-style models, such as GPT-2, GPT-3, and XGLM are unidirectional. Usually, but not always, bidirectional models are paired with an encoder-decoder architecture, while unidirectional models are paired with decoder-only architecture (Devlin et al., 2019; Raffel et al., 2020; Xue et al., 2021: Radford et al., 2019; Brown et al., 2020; Lin et al., 2021; Wang et al., 2022). BERT-style models are an example of an exception. BERT-style models are bidirectional, but they cannot be easily utilized for prompting and text generation since they are encoder-oni only (Wang & Cho, 2019). Of the available bidirectional models, T5 models are the only models with long enough sequence length (unlimited with their relative position embeddings) to support many in-context prompt examples and with large enough number of parameters to be effective zero- shot and few-shot performers (Radford et al., 2019; Brown et al., 2020; Kaplan et al.. 2020). See Appendix J for survey of popular open source language models. Aside from sequence length and model size, BART is not purely trained on the span denoising objective SAP exploits, but is also trained on many other corruption objectives like "Sentence Permutation." " For this reason, we utilize the T5 models for experiments and leave the exploration of the generalization of SAP to other models, that could become available later, as future work.
Devlin et al. (2019) and Raffel et al. (2020) have both shown that after transfer learning, bidi- rectional denoising pre-training objectives such as BERT's masked language modeling and T5's random span corruption outperform causal language modeling downstream tasks. Brown et al. (2020) concedes this to be a potential source of weakness for the GPT-3 model on certain tasks where bidirectionality is important.
Despite the advantages of denoising objectives, prompting and in-context learning capabilities have not been broadly demonstrated for bidirectional language models like T5, disqualifying them when few-shot in-context learning and zero-shot instruction following is desired. Lester et al. (2021) explains this may be because:
3

---

Published as a conference paper at ICLR 2023
"...a TS model pre-trained exclusively on span corruption, such T5.1.1, has seen truly natural input text (free of sentinel tokens), nor has ever been asked predict truly natural targets"
In other words: bidirectionality are only conditioned to output single token short spans of tokens (the in-fill of the mask) rather than full and complete sentences; this inhibits their ability to generate arbitrary- variety of prompts.
when pre- mre-trained on their denoising objectives, language models like T5 that utilize
length natural responses
Despite the stronger learned representations of bidirectional models, their shortcomings in prompt- based learning motivate Brown al. (2020) and Lin al. (2021) to explicitly choose unidirectional models over bidirectional models for GPT-3 and XGLM.
2.2 PROMPTING BIDIRECTIONAL LANGUAGE MODELS
Unlike prior approaches to incorporate prompt-based learning capabilities into bidirectional models, our technique, SAP, neither requires fine-tuning, weight updates, nor supervised instruction-tuning datasets. demonstrates that bidirectional language models develop innate few-shot learning capa- bilities with in-context learning and zero-shot instruction following capabilities.
Cloze-style prompts Schick & Schiitze (2021a) and Schick Schütze (2021b) find that bidirec- tional models such as RoBERTa and ALBERT (Liu et al., 2019; Lan et al., 2019) can be "prompted" with cloze-style phrases. They propose few-shot training paradigm called PET where the model's predicted mask in-fill, called "verbalizer," is used to label fine-tuning examples for the model. These verbalizers are only a single word or a few words, e.g. "yes". "no", "amazing", "worse". Ni & Kao (2022) follow similar technique, but with the ELECTRA model (Clark et al., 2020). These works primarily demonstrate zero-shot effectiveness on classification tasks such as sentiment question
analysis, rather than more challenging generation tasks such as machine translation answering. Furthermore, they still require fine-tuning for effective few-shot learning, a major lim- itation that does not achieve the prompt-based in-context learning or instruction following abilities of unidirectional models such as GPT-3.
Lester et al. (2021) finds some success with prompting the T5 v1.1 models after continued pre-training on the unidirectional prefix-LM objective described in Raffel et al. (2020). The resulting model, T5 v1.1 LM-adapted (T5+LM), is described a late-stage adaptation to uni- directional objective. Adaptation requires performing weight updates, and given that representations learned by the original denoising objective have been shown be superior (Raffel et al., 2020), we hypothesize that such an adaptation could degrade the quality of the learned representations.
LM-adaptation
Prompt-tuning Lester al. (2021) and Li & Liang (2021) find by fine-tuning only portion of the parameters in an otherwise frozen pre-trained bidirectional language model, a "soft prompt" can be discovered through backpropagation. Soft prompts are prompts discovered in the embedding space of the model and are not grounded in natural language. As a form of parameter-efficien fine- tuning (Liu et al., 2022), this approach requires training the prompt embeddings and benefits from initialization from LM-adaptation, both of which require performing weight updates. The nature of soft prompts lacking grounding in natural language makes their use and flexibility limited, stark difference from the instruction following capabilities of unidirectional models (Liu et al., 2021).
Instruction-tuning Language models can be fine-tuned on supervised dataset consisting of nat- ural language prompts and their respective target completions (Wei et al., 2021; Sanh et al., 2022; Ouyang et al., 2022; Min et al., 2021). This instructioontuning technique allows these models to improve performance instruction following and therefore exhibit few-shot and zero-shot capabil- ities through prompting. The TO model in particular an instruction-tuned version of the T5+LM model (Lester et al., 2021), augmenting it with prompting capabilities. While instruction-tuning likely bolsters the instruction following performance of a model, we hypothesize that by instruction- tuning, the TO model is to some degree surfacing the innate prompting ability that the bidirectional model already has. We provide evidence towards this hypothesis by demonstrating that bidirectional models can be prompted without instruction-tuning.
4

---

Published as a conference paper at ICLR 2023
English- Russian Russian- Russian-English
English-Russian
Prompting (mT53,77)
geseration the time step
1.9
5.6
Sequential Prompting (m733..7 SP)
Concatenztnng the full geseration at esch time step CONCATCC C.
9.3
17.9
Sequential Auloregressive Prompting (m1T33777 7B SAP)
Cuncolonnsing word genention
CONCAT(F
20.1
26.9
Table I: Few-shot (2-shot) machine translation results on FLORES- 101 devtest (spBLEU) using m733. as described in Section 3. In this experiment, over two language pairs, English-Russia and Russian- Engllih,, we compare a) simply prompting the model once and taking the full generation Go b) concatenating the full generation at each time step G; to the prompt in the next time step c) concatenating just the first word the generation at each time step F; to the prompt in the next time step.
2.3 UNSUPERVISED MACHINE TRANSLATION THROUGH PROMPTING
GPT-2 (Radford al., 2019) and GPT-3 (Brown et al., 2020) have shown it is possible to perform few-shot machine translation and unsupervised zero-shot machine translation with large language models using prompting and in-context learning. The XGLM model (Lin et al., 2021) trains similar architecture to GPT-3 on a diverse multilingual corpus, resulting in improvements on few-shot, low-resource machine translation. Han et al. (2021) introduce bootstrapping and ceff-impliiication techniques to further improve unsupervised zero-shot performance on machine translation.
3 FEW-SHOT MACHINE TRANSLATION
To motivate our method for enabling few-shot in-context learning in bidirectional language models, we first focus on applying mT537B (mT5-XL) (Xue et al., 2021) to the machine translation task as an in-depth case study since this task benefits greatly from bidirectionality (Conneau et al., 2020; Lin et al., 2021). We largely follow the procedure of Lin et al. (2021), except with mT5 and SAP. mT5 is a massively multilingual bidirectional model trained on random span corruption, a variant of masked language modeling. We demonstrate that with SAP, m'T5 can perform few-shot machine translation using prompting and in-context examples with no fine-tuning. We first formulate a prompt format that utilizes its random span masking scheme to complete the translation task, such as:
Translate Spanish to English. Spanish: El clima es soleado.. 

English: The weather is sunny.</s 

Spanish: Mi perro es un eachorro. </>>
English: My dog is a puppy.

Spanish: Los árboles son importantes. 

English: <X>
3.1 SEQUENTIAL AUTOREGRESSIN PROMPTING (SAP) TECHNIQUE
By requiring mT5 to in-fill <X>1 we are effectively asking it translate the Spanish sentence. However, due to the limitations of the denoising pre-training objective on prompting (described in Section 2)1), observe mT5 often outputs partial translation of the beginning of the source sentence, rather than the full translation. To overcome this, we prompt mT5 T times until the model generates stop token 
2 resulting in longer translation. At each time step of iteration, keep the first word generated (using the space character as delimiter) and concatenate it into the last line of the prompt to use in the next time step. This iterative prompting enables us extract longer generations. Formally, we denote the generation at each time step as G,. We denote the first word generated at each time step as F,, where F, SPLIT(G,, " ") [O]. We update the prompt each time step P. to include the cumulative generation from all previous time steps concatenated in the
We use the first sentinel token from the mT55 vocabulary as our mask token.
We repurpose the 100th sentinel token from the mT5 vocabulary as our stop token.
5

---

Published as a conference paper at ICLR 2023
Thensbee firina in
vide Speath if rightt
Rule Bade-Based Filters
Pegiics
Pogaie
The weather
1. Sampling
2. Filtering
3. Deff-Amplification
Unsuparvised zero-shet translation on source language monolingual corpus
Unsupervised fittering high-quality transiations for use as synthetic few-shot examples
Unsupervised translation using high quality synthetic examples
Figure 2: A visualization of the bootstrapping process described in Section 4.
last line of the prompt. The prompt used at each time step Pt is as follows:
Translate Spanish to English.
Spanish: El clima es soleado. 

English: The weather is sunny. 

Spanish: Mi perro un eachorro. 

English: My dog is a puppy.

Spanish: Los árboles son importantes. 

English:
CONCAT( P:-1) <X>
In Table 1, we also consider sequential prompting concatenating the entire generation G, instead of just the first word of the generation F,-bu find that it produces significantly inferior results as low-quality tokens are generated after the first word. By conditioning the model to generate the next word in the translation based on previous words generated, this technique resembles autoregression. mT5 is already autoregressive, but it is autoregressive only at the decoder level. Adding previously generated words back into the prompt allows them to pass through the encoder layers as well. For this reason, we call this technique SAP (Sequential Autoregressive Prompting). To provide signal to stop generation, we add our stop token at the end of each example in the prompt. We stop prompting after the model generates stop token. The overall process is graphically depicted, with stop tokens omitted, in Figure 1.
3.2 RESULTS
Following Lin et al. (2021), we evaluate our technique on 14 languages from the FLORES- dataset (Goyal et al., 2021) that span high-resource and low-resource languages*. We evaluate Sen- tencePiece BLEU (spBLEU) (Goyal et al., 2021) in every direction, leading to an evaluation over 182 language pairs in total. Abbreviated results be found in Table and the matrix of full results can be found in Appendix A. Examples generations can be found Appendix K.
On average spBLEU score all 182 pairs, our model matches the performance of the uni- directional XGLM and GPT-3 models-with approximately 50% fewer parameters and 16x fewer examples. Notably, our technique has significant improved performance on language pairs with least one low-resource language, while trailing only slightly on high-resource pairs.
4 UNSUPERVISED ZERO-SHOT MACHINE TRANSLATION
To extend our in-depth case study on the machine translation task, we now perform fully unsu- pervised zero-shot machine translation with SAP and mT55 following the procedure of Han et al. (2021), which uses self-amplification technique to boost performance. A comparison of zero-shot
We also implement a basic post-pracessing step strip any generated text after repeated sequence of three or more tokens following settings available common decoding implementations (Wolf et al., 2019). 'HR: English (en), German (de), French (fr), Catalan (ca), Finish (fi), Russian (nu), Bulgarian (bg), Chinese (zh), Korean (ko), Arabic (ar), Swahili (sw); LR: Hindi (hi), Malayalam (my), Tamil (ta)

---

Published as a conference paper at ICLR 2023
performance without self-ampliiee coff-ampliiication can be found in Appendix G. We ultimately will replace the examples in the few-shot prompt with synthetic parallel examples. These synthetic parallel exam- ples are bootstrapped in completely unsupervised fashion using zero-shot translation prompt with no examples. The zero-shot prompt format looks like:
Translate Spanish to English.
Spanish: Los árboles son importantes,420 

English: <X>
We adapt the bootstrap process of Han et al. (2021) to retrieve these synthetic parallel examples. The process, depicted in Figure 2, consists of three steps:
Step (sampling): Generate synthetic parallel examples using a zero-shot translation prompt
(with no examples) to translate sentences from monolingual source language corpus.
Step 2 (filtering): Filter out low-quality synthetic examples to keep only high-quality synthetic examples using an unsupervised scoring technique (discussed in Section 4.1).
Step 3 (self-ampiification)) Translate any source language sentence desired using these synthetic parallel examples in the few-shot prompt.
We iteratively run multiple rounds of this bootstrap by repeating step 2 and step 3 to form a better few-shot prompt. The few-shot prompt after self-amplification is used translate more source language sentences. These are then filtered using the scoring technique used in step 2 and so on. In our experiments, we run four bootstrapping rounds and sample 100 source language sentences from the training dataset in each round. Note that none of the target language parallel sentences from the training dataset are used in this sero-shot setting; following Han al. (2021), only the source language sentences are used.
4.1 FILTERING DOWN TO HIGH-QUALITY TRANSLATIONS
The filtering step of the bootstrap requires unsupervised scoring method for assessing the quality of translations. We first use langdetect? a language identifier, as a simple rule-based filter to ensure the generated text is in the desired target language. We then score the remaining generated translations against their corresponding original sentence in the source language. For this unsu- pervised multilingual similarity metric, we utilize the BERTScore (Zhang et al., 2019) algorithm with mT5000 (mT5- dubbing it "mT5Score". We ablate the use of mT5Score as filter in Appendix
We take the top two synthetic parallel examples with the highest m5Sccore in the filtering step and use those as synthetic few-shot examples in the prompt the self-amplification step.
4.2 TRANSLATING WITH AN ENSEMBLE OF PROMPTS
Because the two examples used in the prompt can greatly affect the quality the generated trans- lations, some prompts containing low-quality synthetic examples may cause poor translations for certain sentences. To combat this and reduce variation in performance, we keep the top N synthetic examples instead of two synthetic examples. We use these to form N different few-shot prompts with two synthetic parallel examples each. Each sentence in the test set is then translated with these N. different prompts produce N translations. The best translation of the A translations chosen 2 in fully unsupervised manner with mT5Score, as done in the filtering step of the bootstrap.
We find this ensembling technique helps make unsupervised zero-shot performance competitive with few-shot performance. Experiments varying the number of prompts in the ensemble can be found in Appendix D. Unless otherwise stated, we use prompt ensemble this paper:
In sum, we sample and zero-shot translate 100 sentences from monolingual corpus, keep the top eight synthetic parallel examples scored by mTSScore, and use them to form four few-shot prompts, each of which has two synthetic examples.
'https: / http://pppp..ppppppp.. www.orrgg/rrr.ooooo / angdetect// 'The BERTScore Python lihrary (Zhang et al., 2019) directly supports using mT5 instead of BERT.
7

---

Published as a conference paper at ICLR 2023
HR
HR
LR
HR
HR
LR
LR
LR
All
Number of Language Pairs
110
33
33
6
182
Supervised
21.5
10.3
8.6
4.3
16.6
GPT-36
(32-shol)
8.1
0.4
0.1
0.1
5.0
XGLM275B
(32-shot)
15.3
3.8
mT73.. + SAP
(2-shot)
14.5
9.8
8.2
7.1
12.3
mT53.7 + SAP
(zero-shot) 15.5
10.7
9.1
8.2
13.2
Table Abbreviated few-shot and unsupervised zero-shot machine translation results on FLORES- 101 devtest (spBLEU). The matrix of full results can be found in Appendix A. Results are average spBLEU scores over subsets of the 182 language pairs (src + tgt) where "LR" is low-resource language and "HR" is high-resource language. "All" represents the average spBLEU score over all 182 language pairs. Supervised results correspond to the M2M-124 615M model from Goyal et al. (2021). XGLM2 results correspond to the model from Lin al. (2021), Bold denotes best of GPT-3, XGLM, and mT5. spBLEU computed using the implementation from Goyal (2021).
4.3 ENGLISH-CENTEN BOOTSTRAPPING
While Han et al. (2021) only performed a bootstrap on English-French and French-English pairs, perform bootstrapping some language pairs which may contain least low-resource language or non-English language.
It has been found that multilingual language models perform best in English, due to imbalance of languages in the pre-training corpus where English has the highest amount of data (Lin et al., 2021). Therefore, when running the bootstrap various language pairs, modify the bootstrap favor generating English, or pivot through English when neither the source nor target language is English. Ablation experiments can be found Appendix E. We outline examples of our modified English- centric bootstrapping process for various language pairs in Appendix F.
4.4 RESULTS
We report results with the same method used for our few-shot evaluation. Abbreviated results can be found in Table and the matrix of full results can be found in Appendix A.
In this unsupervised setting, we find our zero-shot results exceed our 2-shot results; furthermore, they significantly exceed the performance of the XGLM and GPT-3 results reported in Lin et al. (2021) on an average spBLEU score over all 182 pairs (+1.0 spBLEU). Again, we note strong performance on language pairs that contain one or more low-resource languages.
Intuitively, we can explain the zero-shot performance surpassing the few-shot performance through of prompt ensembling in the zero-shot setting. As prompt ensembling utilizes four prompts with two synthetic parallel examples each, it essentially uses eight synthetic examples, instead of just two real examples in the few-shot setting. Our synthetic examples are nearly as high-quality as real examples (similar to the findings of Han et al. (2021)) as demonstrated by Appendix D. Prompt ensembling not only reduces performance variation if low-quality synthetic examples are selected during the bootstrap, but it also boosts performance beyond the few-shot setting as demonstrated by Table and the Appendix experiments (Russian-English 26.9 + 27.9 spBLEU).
In Appendix B, we also evaluate on WMT14 (Bojar et al., 2014) to compare with the results reported Han et (2021) using GPT-31758-
OTHER LANGUAGE GENERATION TASKS
We next demonstrate that bidirectional models have generalized ability, beyond machine transla- tion, to be prompted for arbitrary tasks. We evaluate their performance on question answering and summarization language generation tasks. Example generations be found in Appendix K.
8

---

Published as a conference paper at ICLR 2023
5GLM
shno-shoo) 19,5/319
12.2/25.3
7,2/2822
12,5/21.2
13,2/203
25.0/48.8
17,429944
1944430
9.7410
15044.1
6.6321
16.13990
2.8/7.44
15.037.0
18.241.9
15.29.01
14.637.3
Table Zero-shot multilingual question answering results (EM/F1) on the XQuAD set (Artetxe et al., 2020).
ROUGE-1 ROUGE-2 ROUGE-L
Zero-shot
Zeno-shot
M3+ mT53.7 mT53.u
(zcro-shot)
5.3
0.6
4.9
T5+LMsB
(zero-shot)
23.5
48.4
(zero-shot)
4.6
mT527u + SAP (zero-shot)
30.2
54.0
SAP (zero-shot)
22.0
7.4
20.1
Few-shor
Few-shor
(16-shot)
23.0
54.5
TS+LMau
(2-shol)
mT53.7 + SAP (16-shot)
35.4
60.0
mT53mB
(2-shot)
15.9
4.5
15.0
mT537 SAP
(2-shol)
22.0
6.8
20.3
Table 4: Zero-shot and few-shot question answering results on the SQuAD v1.1 dev set (Rajpurkar et al., 2016).
Table 5: Zero-shot and few-shot summarization results on the CNN / Daily Mail v3.0.0 test set evaluated with ROUGE (Nallapati al., 2016; See et al., 2017; Hermann et al., 2015; Lin, 2004).
5.1 QUESTION ANSWERING
We compare the zero-shot question answering performance of mT5 against XGLM on the XQuAD dataset (Artetxe et al., 2020), multilingual question answering dataset, in Table We find mT5 with SAP outperforms XGLM significantly (+1.7 EM, +12.3 FI).
In Table we also compare against T5+LM (Lester et al., 2021). As T5+LM English-only, we compare using the English-only SQuAD v1.1 dataset (Rajpurkar et al., 2016). We still utilize the multilingual mT5 with SAP due to observations that the English-only T5 v1.1 model does not per- form as well mT5 in prompt-based learning". SAP achieves +6.7 EM and +5.6 Fl over T5+LM.
SAP, as an iterative technique, is useful for producing long generations from bidirectional model for tasks such as machine translation. We find, however, it still has utility tasks like question answering where answer generations are shorter spans of text. We ablate utilizing SAP with mT5 against the simple approach of prompting mT5 once and using the mask in-fill generated on SQuAD v1.1. In the few-shot (16-shot) setting, we find that utilizing SAP still markedly improves perfor- mance (+12.5 EM, +5.5 F1) even short-form generation tasks like question answering.
5.2 SUMMARIZATION
We next perform summarization on the CNN/Daily Mail dataset (Nallapati et al., 2016; See al., 2017; Hermann et al., 2015) as another long-form text generation task. We compare mT5 with T5+LM and ablate the usage of SAP once again in Table 5. In the few-shot setting, we find a significant lead against T5+LM (+7.1 ROUGE-L). Of that +7.1 ROUGE-L boost, the ablation of our usage of SAP finds the technique itself is responsible for large component of the boost (+5.3).
CONCLUSION AND FUTURE DIRECTIONS
We demonstrate SAP with the bidirectional mT5 model enables few-shot and zero-shot machine translation and zero-shot multilingual question answering, outperforming unidirectional models de- spite using far fewer parameters and examples. Our results suggest that the bidirectional represen- tations learned by models such as mT5 contribute to this improved performance. Still, we concede that our results do not conclusively prove bidirectionality explains the difference in performance. Beyond bidirectionality and pre-training objectives, mT5, XGLM, and GPT-3 further differ in ar- chitecture, pre-training corpus, and hyperparameters. complete ablation experiment would be
We discuss this observation in detail in Appendix H.

---

Published as a conference paper at ICLR 2023
computationally expensive, and we leave this as future work. The main limitation of SAP lies in its computational efficiency, discussed further in Appendix along with potential mitigations.
Importantly. these results demonstrate bidirectional models possess few-shot in-context learning and zero-shot instruction following capabilitiess innately, without the post-hoc modifications required by prior work. Our results finally contribute strong evidence towards the strength and efficiency of bidirectional pre-training objectives and motivate further research into bidirectional architectures, pre-training objectives, and language models designed and optimized for prompting and few-shot learning. We hypothesize these future bidirectional training schemes could yield an approach that overcomes the efficiency limitations of SAP, while maintaining the performance and parameter size reduction benefits. Concurrent recent work that compares or mixes unidirectional and bidirectional pre-training objectives (Wang al., 2022; Tay al., 2022; Soltan et al., 2022) already provide some early evidence towards this hypothesis.
ACKNOWLEDGMENTS
We thank Daphne Ippolito for reviewing versions of this draft and Victor Sanh for answering queries related to earlier directions of this work. This research is based upon work supported in part by the DARPA KAIROS Program (contract FA8750- 19-22- 1004), the DARPA LwLL Program (contract 19-2-0000,- the IARPA BETTER Program (contract 2019- 19--19051 1955160000 600004), the IARPA HIA- TUS Program (contract 2022-22072200005) and the NSF (Award 1928631), Approved for Public Release, Distribution Unlimited. The views and conclusions contained herein are those of the au- thors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, DARPA, IARPA, NSF, or the U.S. Government.
REFERENCES
Alex Andonian, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hallahan, Josh Levy- Kramer, Connor Leahy, Lucas Nestler, Kip Parker, Michael Pieler, Shivanshu Purohit, Tri Songz, Wang Phil, and Samuel Weinbach. GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch, 8 2021. URL https: www. github. com/ wwwetheraiggggttt.. -nexx.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross-lingual transferability of mono- lingual representations. In Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics, pp. 4623-4637, 2020.
BigScience. BigScience Language Open-science Open-access Multilingual (BLOOM) Language http:///ginnacccooooeeeeeeeeeeee
Model.
hbleee/blooomooeoeeeee 7b1, May 2022.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Au- toregressive Language Modeling with
March 2021. URL https://doi.
org/10.52811 2eed///eeodo.oooooo 5297715.
Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. Findings of the 2014 workshop statistical machine trans- lation. Proceedings of the Ninth Workshop on Statistical Machine Translation, pp. 12- 58, Baltimore, Maryland, USA, June 2014. Association for Computational Linguistics. URL http::/ http::///www... web. wwwgg///nttaaooooee ogy //WW1//W/// 4 4/W1 4-302.
Tom Brown, Benjamin Mann. Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Merbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL https: httpttppppp/roceeiigg..o neurips.co cc/ wtt.cc/pppppr.e http:/pppprr2222 filee//iee///// http:////0../......o....aaaaa... er.
10

---

Published as a conference paper at ICLR 2023
Kevin Clark, Minh-Thang Luong, Quoe V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003. 2020.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning scale. In ACI., 2020.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186, 2019.
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, San- jana Krishnan, Marc' Aurelio Ranzato, Francisco Guzman, and Angela Fan. The flores-101 evaluation benchmark for low-resource and multilingual machine translation. arXiv preprint arXiv:2106.03199, 2021.
Jesse Michael Han, Igor Babuschkin, Harrison Edwards, Arvind Neelakantan, Tao Xu, Stanislas Polu, Alex Ray, Pranav Shyam, Aditya Ramesh, Alec Radford, al. Unsupervised neural ma- chine translation with generative language models only. arXiv preprint arXiv:21 2021.
Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In NIPS, pp. 1693- 1701, 2015. URL http: http//ppprss.iii nips cccccc/pppa.. wc/pppper/99//// teaehiittaciinggg machines-t machhnen--achheese- o o-read- arread-and-eee anddcooprehendd
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXIv:2001.0036)1, 2020.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. ALBERT: A lite BERT for self-supervised learning of language representations. CoRR, abs/1909.11942, 2019, URL http: org/ abs/ 1909. 11942.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuming. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Pro- cessing, pp. 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. As- sociation for Computational Linguistics. doi: ht https: https////acaanhhloy.ooorooeo org/oog2..222/. cmmnpp-main22.
http3///1112111m//p//. 2433. URL
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising scquence-to-sequence pre- training for natural language generation, translation, and comprehension. arXiv preprint arXiv:/910.1 134611, 2019.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Saško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Can- wen Xu, Nicolas Patry, Angelina McMillan-Major Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Gochringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 175-184, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/221 1t663////2222..mmppddmmm..../// emnlp-demo.21. URL https: nlp-demo 21.
h http://aaaaaaaaaaaeaaeae org org/2021-111 201..021..00 em
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers),
pp. 4592,4997 2021.
11

---

Published as a conference paper at ICLR 2023
Chin-Yes Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out. pp. 74-81, Barcelona, Spain, July 2004, Association for Computational Linguis- tics. URL https: /alaaaaaaaattooooooooooaoooooolt wwwprhhooooooooooo-eo--e.e-
org//04-1013
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasumuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O'Horo, Jeff Wang. Luke Zettlemoyer, Zornitsa Kozareva, Mona T. Diab, Veselin Stoyanov, and Xian Li. Few-shot learning with multilingual language models. CoRR, abs/2112. 10668, 2021. URL https::/p//... /tt//rriiiv. wwx.v. org/ abs 2212. 10668.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learn- ing. arXiv preprint arXiv:2205.0500 2022.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. arXiv preprint arXiv:2 13586, 2021.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint
2019.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. Trans- actions of the Association for Computational Linguistics, 8:726-742, 2020.
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. Metaicl: Learning to learn in context. arXiv preprint arXiv:21 15943, 2021.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gülgehre, and Bing Xiang. Abstractive text summarization using peneence-to-sequenee RNNs and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning. pp. 280-290, Berlin, Ger- many, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/K16 1028. URL http::tt//// pestts::/aclle /aclaaaaaaaaatoyooaaaaaaaa org/ 6 1028.
Shiwen Ni and Hung- Yu Kao. Electra a zero-shot learner, too. arXiv preprint arXiv:2207.0814) 2022.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint 2231v:2003002155, 2022.
Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Con- ference on Machine Translation: Research Papers, pp. 186-191, Brussels, Belgium, Octo- ber 2018. Association for Computational Linguistics. doi: 10. N0.18633/v1/W 1- 6319. URL https: httpss///claatthhooooooo org//lrg/// 63199.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67, 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, 2016.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika. Zaid Alyafeai, Antoine Chaffin, Arnau Stiegler, Teven Le Scao, Arun Raja, al. Multitask prompted training enables zero-shot task generalization. In The Tenth International Conference on Learning Representa- fions, 2022.
12

---

Published as a conference paper at ICLR 2023
Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In EACL, 2021a.
Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2339-2352, 2021b.
Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pp. 1073-1083, Vancouver, Canada, July URL https:
2017. Association for Computational Linguistics. doi: 10. / //www. aclweb. org/ anthology/P11-22
Saleh Soltan, Shankar Ananthakrishnan, Jack FitzGerald, Rahul Gupta, Wael Hamza, Haidar Khan, Charith Peris, Stephen Rawls, Andy Rosenbaum, Anna Rumshisky, et al. Alexatm 20b: Few-shot learning using large-scale multilingual seq2seq model. arXiv preprint arXiv:2208.01444 2022.
Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. Unifying language learning paradigms. arXiv preprint arXiv:2205.05131, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Alex Wang and Kyunghyun Cho. Bert has mouth, and must speak: Bert as a markov random field language model. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pp. 10-36,222 2019.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https: //ttt::://ithbb.../ com/ wwtpooooogooooooooooooooooooooll z/ tresshhhttasssormmee jaax May 2021.
Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. What language model architecture and pretraining objective 2022..
work best for zero-shot generalization?, 2022.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint (XXiv:200900160) 2021.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's 's transformers: State-of-the-art natural language processing. CoRR, abs/1910.03371, 2019. URL
http:///rxi: www./rgg//////ooooo/ooo///..eese /1 03771.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 483-498, Online, June 2021. Association for Computational Liniitatiiioaannnniiiiiiiiii Linguisties. doi: 10.18653/v1/2022 www.aa--mai....... URL https: / //aclanthology org 202122221122
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer 2022.
language models. arXiv preprint arXiv:2205.01068 2022.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluat- ing text generation with bert. In International Conference on Learning Representations, 2019,
13

---

Published as a conference paper at ICLR 2023
A FLORES-101 FEW-SHOT AND UNSUPERVISED ZERO-SHOT MACHINE
TRANSLATION
un
de
Fr
6
bg
2h
ko
ar
SW
16
my
ta
avg
Supervised
27-1
37.4
19.3
26.9
28.1
3.5
24.0
(32-shor) (32-shot)
36.1
5.9
0.3
15.6
11.0
=175 20153.78
SAP SAP
54.2
20.1
9.5
10.4
11.0
17.9
(2cro-shot)
26.0
33.2
27.1
11.3
19.1
13.2
19.0
Supervised
35.8
35.5
25.8
22.6
24.6
31.5
17.2
16.6
14.8
21.0
23.4
2.3
2.3
21.0
GPT--3 7n
(30-shut)
40.4
26-2
17.2
8.1
9.3
4.8
9.0
1.0
0.9
0.5
0.3
0.1
0.1
0.1
4e
XGI M7
(52-shot)
38.5
27.9
19.1
20.5
19.7
25.8
12.3
3.4
6.6
11.7
14.3
9.9
4.8
16.5
m0133,78 m1131,7
SAP SAP
(2-shot)
33.0
24.4
17.8
14.1
15.7
20.2
8.2
9.1
7.7
11.0
10.0
9.8
9.6
14.7
(acro shot)
35.9
25.9
22.5
14.3
17.4
21.0
8.2
8.4
8.7
13.4
10.4
9.01
10.8
15.8
Supervised
28.5
17.6
16.7
22.0
(32-shot)
20.9
4.6
9G
(32 shot)
20.4
32.1
19.4
19.8
26.3
10.6
2.4
14.5
13.7
9.2
17.1
mT3
SAP
(2-shul)
38.0
19.3
26.
13.7
18.3
25.5
8.6
9.2
15.0
12.1
10.3
9.7
16.5
2075 SAP
(2cr--sho))
18.1
21.1
30.1
12.9
18.1
22.3
8.7
9.2
11.1
15.7
11.0
'E.
TI.1
16.8
Supervised
33.4
24.8
35.1
19.0
21.1
28.6
15.1
13.9
13.4
18.7
20.5
2.1
2.6
19.1
shot)
40.2
18.6
31.4
7.0
9.3
4.3
3.0
0.9
0.5
0.3
0.4
0.1
0.1
9.3
68
XGLMM 36 SAP
(32-shut)
11.1
18.9
33.8
11.3
3.3
23.9
10.8
1.3
0.8
13.8
5.1
7.9
3.1
13.6
2015
(2-shot)
29.5
15.6
6.5
7.0
7.3
677
20155.2p SAP
(2cro-shot)
19.3
32.4
16.7
19.1
7.9
7.4
9.4
30.3
15.6
Supervised
27.2
23.0
29.3
21.6
20.6
26.4
16.0
14.8
12.4
14.2
19.8
1.7
0.9
17.5
(32-shot)
25.3
13.5
17.1
10.0
5.4
2.6
5.7
0.3
0.3
0.1
0.0
XGLMM;
(52-shor)
20.2
17.4
22.2
17.0
16.5
17.5
12.4
7.5
7.6
K.0
10.1
6.2
2.0
13.4
26133.78 SAP
(2-shor)
24.1
16.1
19.8
14.9
14.2
17.0
7.0
5.8
7.1
8.3
5.6
NLS
3.1
11.7
mIT5 SAP
(ere-shot)
23.2
16.1
20.5
16.3
14.5
16.3
8.0
5.9
6.3
10.0
7.5
37.0
8C2
12.2
Supersised
27.5
23.5
30.1
22.0
19.4
31.0
16.5
15.3
13.5
18.1
20.9
2.2
18.6
GPF1-3677
(52-shor)
28.1
14.8
20.4
13.1
7.4
1.2
01.2
XGGLM--4 1n
(52-shot)
17.9
14.6
26.3
11.6
9.1
7.3
SAP
23.4
13.5
22.5
19.4
25.4
8.3
Supervised
33.0
26.1
33.7
24.9
20.8
26.5
17.5
16.4
14.5
20.9
25..1
2.3
2.4
20.22
GPT 7n
(32-shot)
21.5
11.4
16.0
9.7
4.3
6.5
1.2
0.2
0.2
0.1
0.2
0.1
0.1
5.5
bg
XGLMM
(52-shol)
35.5
19.2
26.3
12.9
14.3
22.9
11.9
6.8
9.2
9.4
7.5
3.2
1.0
13.0
SAP
(2-shot)
31.0
17.0
23.8
18.3
10.9
22.9
7.2
8.3
8.1
11:7
7.4
9.5
6.6
14.1
ws151
SAP
(zer-shot))
32.5
17.3
24.5
21.7
10.6
23.2
8.7
7.5
9.0
13,0
8.6
7.9
10.1
15.0
Supervised
24.3
17.4
22.1
15.5
(32-shot)
0.3
2h
XGLM
(32-shot)
20.7
8.3
8.5
10.5
14.8
12.0
8.5
m135.78
SAP
(2-shot)
19.0
10,9
14.9
11.9
8.0
10.6
11.9
8.9
9.1
8.0
10.0
7.6
10.5
mB SAP
(zcro-shol)
18.5
10.9
14.8
12.8
10.7
11,8
9.2
9.0
8.9
8.2
8,9
10.7
Supervised
20.9
16.7
22.1
16.5
14.9
15.5
21.1
15.7
10.6
15.1
18.7
1.9
4.0
14:9
GPP1-5-78 78
(32-shot)
8.3
4.6
6.4
4.4
2.1
1.7
0.8
2.5
0.1
0.1
0.1
0.1
2-4
k0
XGLMM; 18
(32-shut)
19.9
10.3
13.7
5.3
1.4
1.2
10.9
11.9
2.7
3.2
1.0
2.2
L.4
6.5
mT5, m
SAP
(2-sho))
18.3
10.1
13.7
11.3
10.1
12.6
7.8
7.2
6.6
3.6
9.2
SAP
Gero-sho))
18.1
10.1
13.8
12.8
9.9
11.4
7.6
8.0
6.7
W.1
Supervised
25.5
18.9
25.8
13.1
13.3
19.4
1.8
GPT-33,m
(32-sho))
10.5
5,3
9.6
6.0
2.2
2.2
0.9
0.9
0.1
0.1
0.2
0.0
2.9
XGLMs
(32-hho)
27.7
12.2
17.9
8.8
8.5
9.1
18.4
8.9
0.8
7.7
7.8
3.4
3.7
10.4
25133,7
SAP
(2-shor)
23.7
10.8
17.5
11.0
8.0
12.2
13.8
3.9
7.1
10.3
8.0
8.0
11.1
m=133.3
SAP
(2cro shot)
26.9
11.5
19.8
15.9
7.5
14.5
13.6
6.3
7.6
11.0
8.0
N.S
9.3
12.4
Supervised
30.4
19.4
26.7
20.1
15.6
17.6
23.8
13.2
12.2
12:0
19.2
2.1
16.6
GPT-3477
(30-sho))
2.9
3.9
2.8
1.7
1.3
1.3
0.4
0.1
XGLMM158
(32-shot)
21.8
13.1
7.6
m1133.78
12 shot))
12.6
19:0
15.8
5.9
6.5
(scro shot)
20.0
18.0
15.8
6.9
Supervised
27.9
19.4
25.9
18.9
15.7
16.9
23.9
13.5
13.9
12.3
16
2.5
3.8
16.2
GPT-33 38
(32 (32-shot)
1.2
0.9
1.4
0.8
0.4
0.4
0.3
0,2
0.1
0.1
0.1
0.1
0.2
0.5
hi
XGLMM158
(32 shot)
25.2
12.3
15.4
8.8
9.5
11.5
11.3
10.8
8,5
6.1
4.7
L.5
1.9
9.8
m115.m SAP
(2-shot)
25.7
12.1
13.0
13.0
8.0
12.2
15.4
7.2
4.1
7.4
8.9
9.6
9.0
11.6
a115
SAP
(zero-shot)
27.1
12.6
17.3
14.3
9.0
12.4
14.5
8.0
6.7
8.1
N.9
10C2
12:8
12.5
Supervised
10.4
9.5
5.7
9.1
3/773:778
(32-shot)
8.9
8.9
SAP
(2-shot)
11.9
9.2
5.2
230
7.0
a0135:10
SAP
(zcro-shot)
9.0
11.9
7.1
Supervised GPI15522
8.3
4.9
6.8
5.8
5.0
4.7
7.0
2.5
2.3
1.1
5.2
6.9
1.2
5.8
(52-shot)
1.0
0.5
0.8
0.5
0.2
0.3
0.5
0.1
0.2
0.1
0.1
(1.2
0100
0.5
12.
XGGLMM
(52-shot)
16.3
8.4
10.3
5.1
5.2
8.1
7.6
8.1
6.2
5.4
2.8
7.2
0..9
2.1
m013 m
SAP
(2-shot)
18.7
10.4
137
10.9
0.3
9X
11.6
3.2
0.5
5.0
L
R5
SAP
(ero-shot)
21.4
10.5
14.7
T29
13.2
6.8
MT
2.N
Supperised
26.0
20.2
26.7
20.0
24.5
14.1
13.5
19.3
GPP1-357
(52-shot)
9.9
14.2
2.7
4.0
0.3
XGLM:
(32-shot)
28.5
14.9
20.6
14.4
10.9
18.5
10.9
9.7
5.8
12.2
mm
SAP
(2-bhot)
35%
14.1
20.2
15.6
10.0
13.7
16.9
6.9
6.8
74
10.5
8.5
8CT
12.3
SAP
(zero-shot)
27.1
15.0
20,9
18.2
10.5
14.8
17.1
7.9
7.5
11.5
9.2
8.0
9.7
13.2
Table 6: Few-shot and unsupervised zero-shot machine translation results on FLORES-101 devtest (spBLEU). Source language in rows, target language in columns. GPT-3&>B and XGLM,,5B use 32 examples from the dev set for few-shot learning. mTT5.778 uses 2 examples from the dev set for few-shot learning. Supervised results correspond to the M2M- 124 615M model from Goyal et al. (2021). XGLM results correspond to the model from Lin et al. (2021). Underline denotes better than supervised, bold denotes best of GPT-3, XGLM, and mT5. spBLEU computed using the implementation from Goyal et al. (2021).
14

---

Published as a conference paper at ICLR 2023
B
WMT14 UNSUPERVISED ZERO-SHOT MACHINE TRANSLATION
English- 30.0
English-French French
French-Engihn
GPT-3175h
(self-amplified)
31.8
mT53.7 + SAP
(self-ampliiied)
29.8
31.4
Table 7: Unsupervised zero-shot machine translation results on WMT14 English- Frenihhh test set (SacreBLEU) (Bojar al., 2014; Post, 2018). GPT-31758 (self-amplified) results correspond the unsupervised zero-shot "GPT-3 (self-amplified)" results from Han et al. (2021) prior to performing distillation, initial backtranslation, and iterative backtranslation which involved unsupervised weight updates. mT53.7B (self-amplified) is fully unsupervised zero-shot approach outlined in Section with a 16 prompt ensemble. The BLcre signature used also follows Han et al. (2021):
BLEU+case mi xedtnumrefs. smooth. expttok. intl+version 1.2.20...
C FILTERING AND SELECTION ABLATION
English- Russian
Russian- Russian-Fnglish
Random Selection
0.0
25.5
mT5Sccre Filtering and Selection
20.0
26.3
Table 8: Unsupervised zero-shot machine translation results on FLORES- 101 devtest (spBLEU) using mT53.78 as described in Section 4. In this experiment, we ablate utilizing mT5Score to fil- ter and select the high-quality synthetic examples during bootstrapping over two language pairs, English- and Russian-English. When using random selection, the synthetic parallel exam- ples choosen may be extremely low-quality or non-sensical leading to 0.0 spBLEU score after self-amplification as shown for the English-Russian language pair.
D PROMPT ENSEMBLE SIZE
English-Rusi Russian
Russian-Englhe
Single Prompt
20.0
26.3
4 Prompt Ensemble
20.9
27.9
8 Prompt Ensemble
20.7
28.6
16 Prompt Ensemble
20.9
28.6
Table 9: Unsupervised zero-shot machine translation results on FLORES-101 devtest (spBLEU) using mT53,7p 7B as described in Section 4. In this experiment, we compare utilizing a single few- shot prompt with two synthetic parallel examples to perform the final translation with utilizing an ensemble of 4, 8, and 16 distinct few-shot prompts each with two synthetic parallel examples that generate 8, and 16 translations respectively from which the best translation (by mT5Score) selected as the final translation over two language pairs, Inglish-Russian and Russian-English.
E ENGLISH-CENTRIC BOOTSTRAP ABLATION
English-Russian Russian
English-
Standard bootstrap
20.9
5.8
English-centee bootstrap
21.2
8.3
Table 10: Unsupervised zero-shot machine translation results FLORES- 101 devtest (spBLEU) using mT5,77 as described in Section In this experiment, we ablate utilizing the English-centric bootstrap described in Section 4.3 over two language pairs, English-Russe Russian and
15

---

Published as a conference paper at ICLR 2023
F ENGLISH-CENTRIC BOOTSTRAP EXAMPLES
We outline examples our modified English-centric bootstrapping process for various language pairs below:
Example 1 (Russian-English): No change.
Example 2 (English-Russian)) In step 1, generate Russian-English synthetic examples using a Russian monolingual corpus. Then, reverse the examples obtain English-Russian Russian synthetic examples.
Example (Russian-Chinese): In step 1, for the first three rounds of the bootstrap, generate Russian-English synthetic examples and Chinese-English synthetic examples using Russian and Chinese monolingual corpora. On the fourth and final round, use English monolingual corpus along with the reversed previous synthetic examples to produce English-Russial Russian and English- Chinese synthetic examples. Since the same English sentences are used to produce both sets, we can align these to form synthetic Muasian-Chhness examples. In step we filter examples using the harmonic mean of the two m TSSocess..
G ZERO-SHOT PERFORMANCE WITHOUT SELF-AMPLIIICATIO
English English-Russian Russian
Russian -agginhh
Standard
0.4
4.6
Bootstrapping and celf-amplification
21.2
27.9
Table 11: Unsupervised zero-shot machine translation results FLORES-IP -01 devtest (spBLEU) using mT5,, 7 as described in Section 4. this experiment, we compare the standard zero-shot performance of mT5 with SAP against the zero-shot performance achievable implementing the boot- strapping and self-amplification techniques from Han et al. (2021) with the adaptations described in Section 4.
H PROMPTING T5 v1.1 WITH SAP
Careful readers may ask why use a multilingual model, mT5, obtain results for the English- only tasks of QA (SQuAD) and summarization (CNN/DailyMail), While a suitable English- version of T5 could in theory improve performance, we found issues with T5 v1.1's performance. We choose to run SAP with mT5 due to the observation that T5 v1.1 cannot be prompted as easily as m'5, and thus underperforms.
The inputs seen by T5 v1.1 and mT5 during pre-training are of sequence length 512 tokens where multiple spans in the sequence are dropped (Raffel al., 2020). Therefore, the prompt template we describe in Section 3, would be out-of-distribttion from the pre-training inputs since it may have a sequence length shorter or longer than 512 tokens and only contains single mask instead of multiple masks.
We find that the mT5 model has generalized to sequences shorter and longer than 512 tokens and to sequences that only contain single mask. while the T5 v1.1 model has not. It is still possible to prompt the T5 v1.1 model with SAP, but requires formulating a prompt constrained to the same in-distribution length of 512.
Due to this complication, we forgo prompting T5 v1.1 this paper. Since mTS and T5 v1.1 were trained identically (the same model architecture and hyperparameters), apart from mT5 being pre- trained on the multilingual mC4 dataset instead of the primarily English C4 dataset, we further hypothesize that this difference between T5 v1.1 and mT5 may be artifact of which checkpoint is selected after pre- pre-traine training or the length of pre-training (Xue et al., 2021; Raffel et al., 2020).
16

---

Published as a conference paper at ICLR 2023
I LIMITATIONS
SAP requires 7 total forward passes to produce generation instead of single forward pass, where I' equals the number of words in the generation before reaching stop token. For example, to produce a translation that has 14 words, SAP requires 14 inferences of the bidirectional model. For tasks with shorter generations with only few words, such multilingual question answering, SAP is more practical, especially since it uses fewer parameters. Depending on the size of inference data, SAP as an inference-only prompting technique may be faster and easier implement than methods that require fine-tuning. While these inferences must be performed sequentially due to the autoregressive nature of the technique, utilizing batching over a test set can still ensure maximum GPU utilization, which is how our experiments were performed. For longer generation tasks, believe SAP is prohibitively computationally expensive and it likely would not be suitable for use by practitioners directly despite some evidence of improvements in performance. Nevertheless, SAP uncovers an important result: prompt-based learning is an emergent property of bidirectional models. We hypothesize that further research into pre-training objectives and language model design following Wang et al. (2022), Tay et al. (2022), and Soltan et al. (2022) could yield bidirectional pre-training objective better optimized for few-shot prompting. lifting the requirement to perform multiple forward passes sequentially to generate longer completions.
17

---

Published as a conference paper at ICLR 2023
J
SURVEY OF OPEN SOURCE LANGUAGE MODELS
Model
Architecture
Large
Max Sequence
Pre-training
[>1B params?)
Length during
Objective
Pre-training
Unidirectional Pre-training Objectives
GPT-family models
Decoder-only
1024-2048
Next Token Prediction
(GPT-2, GPT-3)
(Radfoed at. 2019; Brown
et al., 2020)
Eleuther EleutherAT-famil fa-mmlly
Decoder-only
2048
Next Token Prediction
models
(GPT-Nco.
GPT-J, GPT-NeoX)
(Black al, 2021; Wang
& Komassuzaki. 2021;
Andonion et al., 2021)
XGLM
Decoder-only
2048
Next Token Prediction
in al., 2021)
OPT
Decoder-only
2048
Next Token Prediction
(7hing
BLOOM
Decoder-only
2048
Next Token Prediction
(BigScience, 2002)
Bidirectional Pre-training Objectives
BERT-style models
Encoder-only
512
Masked Language Modeling:
Next Sentence Prediction
(BERT, RoBERTa, ALBERT, elc.)
2019;
2019)
BART-style models
Sncoder-Decodee
512/512
Token Masking:
Token Deletion:
(BART,
Sentence Permutation;
mBART, elc.)
Document Rotation;
(Lewis et al., 2019: Liu
Text Infilling
at al., 2020)
T5.style models
Encoder-Decodee
1024/512
Random Span Corruption
Relative
(T5, mT5, etc.) (Matfel al.
Position
Embeddings
at al., 2021)
18

---

Published as a conference paper at ICLR 2023
K
SELECTED EXAMPLE GENERATIONS
Task: Few-shot Machine Translation (Example #1)
Dataset: FLORES-101 (Arabic - English)
Prompt Template:
Translate Arabic English. [[examples]]
Arabic: ((scurce_text))
English:
Ground Truth:
The 102.11n standard operates both the 2. 4Gh and 5.0Ghz frequencies. Generation (mT53,78 + SAP):
The wireless standard 802. 11n operates at the frequency of 2.4 GHz and
5 GHz
Generation (mT33.78):
The 802.11n wireless standard operates at 2.4 and 5.0
Commentary:
SAP generates a lengthier and more fluent translation and correctly translates the units of the frequencies. Both generations add the word "wireless" which is used correctly and is likely to appear next to the words 1802.1In" and "standard" but does not exist in the ground truth translation.
19

---

Published as a conference paper at ICLR 2023
Task: Few-shot Machine Translation (Example #2)
Dataset: FLORES-101 101 (Russian - English)
Prompt Template:
Translate Russian English.
[[xamples]]
Russian: ((source_text))
English:
Ground Truth:
In 1956 Stania moved to Sweden, where three years later he began work for the Swedish Post Office and becane their chief engraver.
Generation (mT33.78 + SAP):
In 1956, Slania noved to Sweden, where he worked for three years for the Swedish Post Office and became its chief engraved worker.
Generation (mT33,78):
In 1956, slanya moved to sweden and became...
Commentary:
SAP generates a full length translation and more correctly translates "Slania" to "Slania" in- stead of "Slanya". While the translation without SAP only generates a partial translation, the word "became" indicates the direction translation would be less close to the ground truth translation than the direction of translation taken by SAP. Notably, SAP produces relatively high-quality translation, but a common failure mode is displayed in this example. SAP translates "chief engraver" "chief engraved worker" which an imperfect paraphrase likely due to an imperfect multilingual alignment of the word "engraver" in the embedding space of the model.
20

---

Published as a conference paper at ICLR 2023
Task: Few-shot Question Answering (Example #1)
Dataset: SQuAD v1.1
Prompt Template:
Answer the question based on the fellowing passage.
[examples)]
Passage: ((passage)
Question: ((question))
Answer:
Passage:
In 1874, Tesla evaded being drafted into the Austro-Hungarian Army in Smiljan by running away to Tomingaj, near Gračac. There, he explored the mountains in hunter's s garo. Tesla said that this contact with nature made him strenger, both physically and mentally. He read many books while in Tomingaj, and later said that Mark Twain's s works had heiped him to mi miraculousi recover from his earlier illness.
Question:
Why did Tesla avoid by fleeing Smiljan?
Ground Truth:
being drafted into the Honngaaaan Army
Generation (mT53,78 + SAP):
because he was ill and wanted to avoid being drafted into the Austro-Hungarian Army
Generation (mT53,78):
because he was ill and could not leave the country
Commentary:
In this example, the grammaticality of the question itself ("Why did Tesla avoid by fleeing Smiljan?" vs. "What did Tesla avoid by fleeing Smiljan"") has issues. This seems cause both generations to attempt to answer a "why" style question with "because" instead of "what" style question. Notably, the answer generated by SAP does eventually reach correct answer where as the the answer generated without SAP hallucinates fact: "he [TTslaa] could not leave the country".
21

---

Published as a conference paper at ICLR 2023
Task: Few-shot Question Answering (Example #2)
Dataset: SQuAD v1.1
Prompt Template:
Answer the question based on the fellowing passage.
[examples)
Passage: ((passage)
Question: ((question))
Answer:
Passage:
The Bronces took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five solo tackles, 24 sacks, and two forced fumbles.
Question:
Who won the MVP for the Super Bowl?
Ground Truth:
Mi ler
Generation (mT53,7B + SAP):
Von Miller
Generation (mT53,7)):
Von Mi Miller, Denver, NFL, NELPA
Commentary:
Without using SAP, the answer generated is correct, but contains hallucinated tokens towards the end of the generation.
22

---

Published as a conference paper at ICLR 2023
Task: Few-shot Summarization (Example #1)
Dataset: CNN / Daily Mail v3.0.0
Prompt Template:
Summarize the article.
[examples)]
Article: ((article))
Summary:
Article:
London (CNN) The Hatton Garden heist, it will surely was every safe deposit box holder's nightmare, every movie director's dream. Thieves using heavy cutting equipment and rappelling gear broke into the vault of esteemed 60-year-old safe deposit сопраnу in the heart Lendon over the past heliday weekend, possibly taking advantage of many as four days to rifle through uncounted number of safe deposit boxes. And they reportedly got away with hundreds of thousands of pounds worth of gems and cash even, in the educated guess of one former police official, as much as 200 million pounds, or... [truncated]
known,
Summary:
Robbers may have taken advantage of a four-day holiday weekend. Estimates the value of the items taken rage from hundreds of thousands of pounds to 200 million pounds. The heist took place in a historic heart London's s jewelry business.
Generation (mT53,78 + SAP):
The Hatton Garden heist was a nightmare for jewelers and their customers. They were 'treated like criminals', while officers said they would double bail for anyone who was arrested before. They were arrested while on the sidewalk
Generation (mT33,70):
The latest Jewel heist in London tock place over the weekend.
Commentary:
SAP generates lengthier generation with multiple sentences summarizing the article, Some of the facts presented in the generation, however, are not coherent with the article.
23

---

Published as a conference paper at ICLR 2023
Task: Few-shot Summarization (Example #2)
Dataset: CNN / Daily Mail v3.0.0
Prompt Template:
Summarize the article.
[examples)]
Article: ((article))
Summary:
Article:
Sanaa, Yemen (CNN) Al Qaeda fighters attacked prison in the coastal Yemeni city of Al Mukallah early Thursday, freeing at least 270 prisoners, third of whom have Qaeda links, senior Defense Ministry official has told CNN. Khaled Batarfi, senior al Qaeda figure, was among the escapees, officials said. Dozens attackers took control of government buildings, including the city's Central Prison, Centra- Bank and radio station during the assault early Thursday, according to officials. Government troops arrived early Thursday and clashed with the al Qaeda fighters, and most of the militants fled,... med,....trunee, (tr.ncatedd
Summary:
Al Qaeda fighters attack a prison and other government buildings, freeing many priscners. Government troops clash with the fighters, most of whom flee. Yemen is descending into chacs as Shia-Sunni conflict draws in regional rivals Saudi Arabia Iran.
Generation (mT53,7 + SAP):
Yemen's s al Qaeda fighters escape from prison after clash with government forces.
Generation (mT53.78):
Yemen's priscners escape from al Qaeda fighting in Yemen
Commentary:
SAP correctly characterizes the major theme of the article in its summary. Without SAP, the generation devolves in to an incoherent fact ("prisoners escape from al Qaeda fighting"") and only partial summary generated. The ground truth summary, however, notably longer and contains multiple sentences, while the summary generated by SAP in this instance is only single sentence.
24

---

Published as a conference paper at ICLR 2023
L RESOURCES
We provide links and citations to resources used this paper which provide license information, documentation, and their intended use. Our usage follows the intended usage of resources.
We utilize the following models:
mT5 (Xue et al., 2021):
https://gii /tttt/gtttttbtt com/ http:o/peeeseeer////uitiinnaaa-a
T5 v1.1 (Raffel et al., 2020; Lester et al., 2021):
https://github. er/
T5+LM (Raffel et al., 2020; Lester et al., 2021):
https://gitee com/ resear reesearcc/aee or er/
We utilize the following datasets:
FLORES-101 (Goyal et al., 2021):
http://ww........oo.a/....ooooa. https://ai. facebook. com/
tion
WMT14 (Bojar et al., 2014):
hctps://ww..tte stw.tttttttttttmmmmmmmtmmmmmm /
XQuAD (Artetxe et al., 2020):
SQuAD v1.1 (Rajuurar al., 2016):
https://raajirrkarggtthhbb. 1 wtt.squD--eexplrrrr/////
CNN Daily Mail v3.0.0 (Nallapati et al., 2016; See al., 2017; Hermann et al., 2015):
cwav/eaa//eaeeeeeeeeeeeeeeeeeeee sevv/enn dai Lymail
We utilize the following software:
Transformers (Wolf al., 2019):
Datasets (Lhoest al., 2021):
com http:/ttwwb..omm/muugggggggaaaed
SacreBLEU (Post, 2018; Goyal et al., 2021):
ROUGH 2004):
http:///ttttuu.ccmm//pr./r//
BERTScore (Zhang et al., 2019):
langdetect: http:://ppppl.oorrrrooorcc/////s
We estimate the total compute budget and detail computing infrastructure used to run the computa- tional experiments found this paper below:
Ix NVIDIA RTX A 87GB RAM/ / 4x CPU 686 hours
25


Reviews:
[{"review_id": "VNStyQygkso", "reviewer": "Reviewer_iGmv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors explore the use of bidirectional models, particularly T5-style models, for in-context learning. The pre-training objectives of bidirectional models do not make it directly amenable to prompt-based learning, but representations from bidirectional models might be better. This paper introduces an iterative procedure (Sequential Autoregressive Prompting) to adapt mT5 for incontext learning at inference time. The results show that with this simple procedure, bidirectional models can also perform prompt-based learning. The results are also competitive with larger decoder-only language models. The major downside is the need to run multiple iterations of generations to complete the generation, thus making the process inefficient. The paper also discusses the strategy applied to the tasks of summarization and question answering. ", "summary_of_the_review": "The paper proposes a quick inference-time fix for adapting seq2seq models trained on denoising objectives for prompt-based learning. This just that seq2seq models can perform incontext learning, but the particular methodology proposed in the paper has a lot of inference-time overhead and alternative proposals of LM adaptation of seq2seq models in concurrent work will eventually be adopted.  Hence, the impact of this work might be limited.", "strength_and_weaknesses": "Strengths: \n\n- The paper shows that bidirectional models can perform prompt-based learning with the right prompting strategy without the need for any model adaptation. This is an interesting fix for models trained for denoising objectives.\n- The paper also proposes an interesting idea for English-centric bootstrapping for unsupervised, though this is somewhat orthogonal to the central contribution of the paper. \n\nWeakness\n\n- The method is applicable to mT5 style models, but the long-term solution is probably to adapt mT5 models with causal LM or prefix LM objectives. This will make the inference more efficient as well. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QumyN7IQMhn", "reviewer": "Reviewer_77VL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper motivates the interest for making bidirectional language models better few-shot/zero-shot reasoners from natural language prompts/instructions. The paper argues that bidirectional models can potentially have richer representations but the unidirectional training in the causal decoder-only models typically happens to be more amenable to few-shot/zero-shot instruction following. The paper resolves the dilemma by introducing a new prompting technique, SAP, to get bidirectional models like mT5 to do few-shot/zero-shot tasks on par with (or better than) other large causal decoder-only models like GPT3.  \n\n\nIn each time step, SAP generates a token using mT5 from a given prompt but then adds the first generated token to the prompt for the next step (in the next time step the whole encoder-decoder is run again with the new prompt). \n\n\nThe paper also adopts several engineering techniques on top of that like bootstrapping with synthetic few-shot samples (sampled and then filtered with unsupervised scorers) and self-amplication (using those synthetic samples as few shots); also prompt ensembling. \n\n\nPaper shows better or on par results of mT5 with unidirectional models on zero-shot and few-shot translation/QA/summarization. ", "summary_of_the_review": "SAP, a technique to effectively zero-shot/few-shot prompt bidirectional language models, is introduced. Simple method; results look decent. Bidirectional models are shown to have the potential to be on par with or better than unidirectional model on instruction following and in-context learning. However, some experimental comparisons may not be completely fair and inference speed may take a big hit. ", "strength_and_weaknesses": "Strength:\n\n1. The core technique is quite simple and seems effective. \n\n2. The insight that bidirectional models can be utilized effectively for few-shot prompting without weight updates similar to GPT-like models can motivate more research into this direction and towards building of better larger bidirectional models. \n\nWeakness:\n\n1. I am unclear of some of the experimental settings and how much we can conclude from them. Particularly, when is bootstrapping+self-amplification+ensembling are used? Is it used only during translation? \n\nIt also seems odd to use those techniques only for zero-shot and not adapt them (or explore possible adaptions) for 2-shot (given the explanation in 4.4) \n\nMy main concern is, however, that if XGLM is not using bootstrapping+self-amplification etc. in Table 2 I am not sure how fair of a comparison it is. Perhaps with those techniques XGLM would  exceed mT5 by a high margin. \n\n(In appendix G the additional techniques over SAP seems to make a huge difference. Appendix B does show that mT5 performs on par with a much larger GPT3 on translation when both are self-amplified but still would be curious about XGLM given its multilingual emphasis similar to mT5. However, I acknolwedge the effectiveness of SAP compared to other variants of prompts to mT5) \n\n\n2. As the paper acknowledge, SAP requires running the whole encoder-decoder architecture in an autoregressive loop. This can be slow. Particularly critical is that caching isn't anymore possible (wherein caching is generally usable to speed up autoregressive decoding in causal decoders), because the encoder is bidirectional and in the autoregressive loop. \n\n-------------------------------\n\nI am increasing the score to 8. While the weaknesses are not completely resolved: re. 1 there are still enough evidence that mT5 can be made to perform on par with undirectional decoders as the authors have clarified. Regarding 2, authors are upfront about it. \n\nThere are still some limitations. The conclusions rely only on mT5 but authors are clear about their justification and there are not many other models at a similar large enough scale. In terms of impact, I am not entirely sure how meaningful the result will be - given other works on improving Seq2Seq pre-training is already underway independent of this work and SAP, as it is now, may not be as practical for adoption. However,  I think the paper still brings a neat empirical insight, and sets up a bar with SAP that we can attempt to approximate or overcome. It can also open questions and investigation of why this sort of autoregression over the whole seq2seq setup works so well and if there can be some ways to make to computation easier. There is also a decent amount of experiments in the paper. \n\nOverall, given these factors, I decided on raising the score. \n\nSome (optional) discussion topic the paper may (or may not) add:\n\nNot related to prompt tuning but - [1] and [2] also uses a similar strategy of puting the bidirectional encoder within an autoregressive loop and shows better compositional generalization. This could be worth some discussion and encourage future directions in more closely studying what this form of technique is bringing to the table. \n\n[1] Disentangled Sequence to Sequence Learning for Compositional Generalization - Zheng et al ACL 2022\n[2] Recursive Decoding: A Situated Cognition Approach to Compositional Generation in Grounded Language Understanding -Setzler et al. Arxiv 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QPyHj5xWBc", "reviewer": "Reviewer_CyM9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the effectiveness of bidirectional language models (specifically, mT5 model) on low-resource multilingual generative tasks with the proposed Sequential Autoregressive Prompting(SAP) method. The paper presents a representative case study on the low-resource MT task utilizing the proposed SAP and a self-amplification technique. Further, the generalized ability of the proposed SAP method on bidirectional LMs is also validated on few-shot / zero-shot QA and Summarization task.", "summary_of_the_review": "This paper proposes an inference-only SAP prompting technique for mT5 model to achieve remarkable results on low-resource multilingual generative tasks, including MT, QA and Summarization. The proposed method outperforms unidirectional models (GPT-3, XGLM) within fewer parameters and in-context training examples. Even though the paper presents sufficient experiments and analysis, it does not give strong evidence to support the point that the low-resource performance boost comes from bidirectional pre-training objectives. What’s more, the technical novelty of the paper is kind of limited.", "strength_and_weaknesses": "**Strength:**\n1. The proposed inference-only SAP approach is an interesting way to use the bidirectional LMs’ low-resource transfer ability on generative tasks such as MT, QA and summarization;\n2. Combines multiple techniques (e.g. filtering , prompt ensembling, and English-centric bootstrapping) and applies them to perform better unsupervised low-resource MT results;\n3. Sufficient experiments and analysis.\n\n**Weakness:**\n1. The technical novelty of the paper is somewhat limited, since the SAP prompt technique is just feeding the generated token back the prompt again so as to get the new representation of the encoder.\n2. The paper only verifies the effectiveness of the proposed SAP method on the mT5 models, however, claims that the performance improvements come from the Bidirectional LMs, which is not reliable. This part of performance improvement may only come from SAP's effect on mT5, or SAP's effect on the encoder-decoder architecture model. Not only that, the underperforming attempt of running SAP on the T5 1.1 proves the unreliability of this conclusion.\n3. Lack of detail analysis on the computational efficiency.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ai_-ZqkZV0", "reviewer": "Reviewer_ZTjg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on prompt-based learning applied to bidirectional language models. This class of models doesn't lend itself well to standard prompt-based approaches due to their objectives only training them to fill in short spans of text at most. The approach proposed in this paper is to iteratively prompt such models, feeding back the original prompts concatenated with the first words of previous generations, to enable them to generate much longer sequences in an \"autoregressive\" manner.\n\nThe authors evaluate the method on a translation task, obtaining better performance with mT5 compared to the much larger causal language models GPT-3 and XGLM, and are also not too far off supervised techniques even on low(ish)-resource languages. They further evaluate on QA and summarisation tasks, showing again strong performance against XGLM.", "summary_of_the_review": "A strong paper proposing a conceptually simple yet effective prompt-based learning approach for models like mT5. Some questions around computational efficiency I believe should be given a bit more prominence.", "strength_and_weaknesses": "## Strengths\n\n* A clear, well structured paper with a well thought out set of experiments. Most of the questions I could come up with while reading it I found answered in one of the appendices, including an extensive set of ablation studies.\n* Translation/generation performance is impressive compared to traditional prompt-based methods on GPT-like models, in spite of the much smaller model size.\n* The evaluation is generally strong, with 14 languages being chosen for translation, and the model being also compared to supervised approaches.\n\n## Weaknesses\n\n* The main question I have is: why? I don't think this is spelled out clearly enough in the main paper. Do the authors expect SAP-like approaches to eventually overtake supervised approaches? If so, the questions around computational complexity are central and should be discussed in much more detail in the main section of the paper, including perhaps some FLOPS statistics for the proposed approach and the current SOTA.\n* The supervised translation baseline include in Table 2 should perhaps be replaced with a more recent and stronger multilingual model.\n* I would definitely challenge the classification of languages such as `ko`, `ar`, `sw` etc. as being low-resource. These are all languages for which at least 1M training sentences can be found (see e.g. Table 1 in <https://arxiv.org/abs/2207.04672>)\n\nMinor: I believe the meaning of \"Supervised\" in Table 2 is only explained in the Appendix. I would recommend explaining it in the table caption.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster