Venue: ACL 2023

# Crosslingual Generalization through Multitask Finetuning

Niklas Muennighoff\({}^{1}\)   Thomas Wang\({}^{1}\)  Lintang Sutawika\({}^{2,3}\)    Adam Roberts\({}^{4}\)

Stella Biderman\({}^{3,5}\)   Teven Le Scao\({}^{1}\)   M Saiful Bari\({}^{6}\)   Sheng Shen\({}^{7}\)

Zheng-Xin Yong\({}^{8}\)   Hailey Schoelkopf \({}^{3,9}\)   Xiangru Tang \({}^{9}\)   Dragomir Radev\({}^{9}\)

Alham Fikri Aji\({}^{10}\)   Khalid Almubarak\({}^{11}\)   Samuel Albanie\({}^{12}\)   Zaid Alyafeai\({}^{13}\)

 Albert Webson\({}^{8}\)   Edward Raff\({}^{5}\)   Colin Raffel\({}^{1}\)

\({}^{1}\)Hugging Face \({}^{2}\)Datasaur.ai \({}^{3}\)EleutherAI \({}^{4}\)Google Research, Brain Team

\({}^{5}\)Booz Allen Hamilton \({}^{6}\)NTU \({}^{7}\) UC Berkeley \({}^{8}\) Brown University

\({}^{9}\) Yale University \({}^{10}\) MBZUAI \({}^{11}\) PSAU \({}^{12}\) University of Cambridge \({}^{13}\) KFUPM niklas@hf.co

###### Abstract

Multitask prompted finetuning (MTF) has been shown to help large language models generalize to new tasks in a zero-shot setting, but so far explorations of MTF have focused on English data and models. We apply MTF to the pretrained multilingual DLOOM and mT5 model families to produce finetuned variants called BLOOMZ and mT0. We find finetuning large multilingual language models on English tasks with English prompts allows for task generalization to non-English languages that appear only in the pretraining corpus. Finentuning on multilingual tasks with English prompts further improves performance on English and non-English tasks leading to various state-of-the-art zero-shot results. We also investigate finetuning on multilingual tasks with prompts that have been machine-translated from English to match the language of each dataset. We find training on these machine-translated prompts leads to better performance on human-written prompts in the respective languages. Surprisingly, we find models are capable of zero-shot generalization to tasks in languages they have never intentionally seen. We conjecture that the models are learning higher-level capabilities that are both task- and language-agnostic. In addition, we introduce xP3, a composite of supervised datasets in 46 languages with English and machine-translated prompts. Our code, datasets and models are freely available at [https://github.com/bigscience-workshop/xmtf](https://github.com/bigscience-workshop/xmtf).

+
Footnote †: July 9-14, 2023 ©2023 Association for Computational Linguistics

_Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers_, pages 15991–16111

July 9-14, 2023 ©2023 Association for Computational Linguistics

## 1 Introduction

Large language models pretrained on vast amounts of text show some capability of solving tasks expressed in natural language, even without explicit training on these tasks [Brown et al., 2020]. Fine-tuning on groups of language tasks has been shown to significantly boost this zero-shot task generalization of language models [Wei et al., 2021; Sanh et al., 2022; Min et al., 2021]. For example, Sanh et al. (2022) finetune on tasks like summarization and question answering leading to better performance on unseen tasks like natural language inference. Previous work has focused on multitask finetuning in the context of large English language models and tasks.

Multilingual large language models show the same zero-shot learning capabilities for both monolingual and crosslingual tasks [Goyal et al., 2021a; Lin et al., 2021; Patel et al., 2022; Soltan et al., 2022]. However, zero-shot performance tends to be significantly lower than finetuned performance. Thus, task-specific or language-specific transfer learning via finetuning remains the predominant practice [Devlin et al., 2018; Conneau et al., 2019; Aribandi et al., 2021]. This is particularly challenging for low-resource languages or tasks with limited data available, such as writing a table that teaches a specified moral. In the spirit of multitask finetuning, it would be desirable to improve the zero-shot task generalization of multilingual models to make them usable on tasks from low-resource languages without requiring further finetuning.

To address this goal, we focus on crosslingual multitask finetuning. Due to the difficulty of collecting supervised task data in low-resource languages, previous work typically aims to transfer capabilities learned from finetuning on English data, which can improve performance on non-English language tasks [Wu and Dredze, 2019; Phang et al., 2020; Chalkidis et al., 2021; Vu et al., 2022]. We investigate whether English-only multitask finetuning also improves performance on non-English held-out tasks using the multilingual BLOOM [Scao et al., 2022a] and mT5 [Xue et al., 2020] models. We find that after finetuning on the English-only multitask mixture used for T0 [Sanh et al., 2022] (P3), performance on a diverse set of

non-English held-out tasks increases.

To investigate whether multilingual task data can further improve performance, we extend P3 to xP3 by adding datasets from 46 different languages that cover tasks previously not present in P3 (such as translation and program synthesis). Finetuning on xP3 leads to even better zero-shot task generalization in both English and non-English compared to the P3-trained baseline. Models finetuned on xP3 perform best on English prompts, even for non-English samples. Hypothesizing that better performance could be attained by training on non-English prompts, we construct a variant of xP3 with machine-translated prompts called xP3mt. We find that finetuning on machine-translated prompts is enough to significantly increase performance on held-out tasks with non-English human-written prompts. However, reducing the number of English prompts in the finetuning also worsens English prompt performance on multilingual tasks.

Notably, we also find that models finetuned on xP3 generalize to held-out tasks in languages never intentionally seen during pretraining nor finetuning. We conduct a contamination analysis and find that only small amounts of these languages were included in the pretraining corpus. Thus, we hypothesize the models learn some language- and task-agnostic capabilities.

We publicly release all our datasets and models (URLs in Appendix SSC).

## 2 Related work

### Multitask learning

Multitask finetuning Sanh et al. (2022) (or instruction tuning Wei et al. (2021)) has emerged as a

Figure 1: An overview of datasets in xP3. Datasets added to P3 in this work are marked **bold**. Yellow datasets are trained on. Green datasets are held out for evaluation.

Figure 2: Language composition of xP3, ROOTS, and the corpus of mT5. All ROOTS and xP3 languages are depicted. The mT5 corpus covers additional languages that are not included in the graph.

 

recipe for improving the zero-shot task generalization of large language models. Typically, these works define a task as a collection of datasets that require a certain set of skills. To inform large language models which task to perform given an input, a prompt is used to add natural language instructions to dataset instances (Schick and Schutze, 2020; Scao and Rush, 2021). In this line of work, zero-shot task generalization refers to the ability to perform a held-out task based on prompted instructions alone. Our work builds on T0 (Sanh et al., 2022), a variant of T5 (Raffel et al., 2020) that underwent MTF and was subsequently shown to have strong zero-shot task generalization capabilities.

Increasing the number and diversity of finetuning tasks and datasets has been shown to increase model performance (Min et al., 2021; Fries et al., 2022; Wang et al., 2022d; Scialom et al., 2022; Chung et al., 2022; Mishra et al., 2021b). PromptSource (Bach et al., 2022) is a software application that provides a framework for developing and applying prompts. PromptSource was used to construct P3, the training dataset of T0. While most prior work has focused on using English prompts on English datasets, Wang et al. (2022c) trained both English and multilingual models on prompted datasets. Their multilingual model, called mTk-Instruct, attains strong crosslingual performance. In contrast with Wang et al. (2022c), our sole focus is crosslingual zero-shot generalization. Therefore, we consider a wider variety of prompting settings and perform a more detailed evaluation of multilingual capabilities. Separately, Radford et al. (2019) find that accidental inclusion of non-English text gave the GPT-2 model a limited ability to process and generate non-English text. We similarly discover that our finetuned models can process text in languages not intentionally trained on.

### Multilingual models

Many language models are pretrained on English data only. Multilingual pretrained language models (Lample and Conneau, 2019; Conneau et al., 2019; Fan et al., 2021) aim to enable processing a wide variety of non-English languages. Unlike monolingual models, multilingual models can also be used for crosslinking tasks, such as translation. For language generation, recent efforts have focused on two different model architectures based on the Transformer (Vaswani et al., 2017). On the one hand, encoder-decoder transformers trained with a denoising objective such as mBART (Liu et al., 2020) and mT5 (Xue et al., 2020) learn to predict tokens masked out in the input sequence. Predicting masked tokens is only a pretraining task and these models are generally finetuned on downstream datasets before being used. On the other hand, decoder-only models pretrained on next token prediction such as mGPT (Shliazhko et al., 2022), XGLM (Lin et al., 2021) and BLOOM (Scao et al., 2022a) can be used to solve tasks expressed in natural language directly in a zero-shot or few-shot setting (Brown et al., 2020). XGLM demonstrated competitive few-shot performance even when the model was prompted in a language different than the sample being processed. In particular, using English prompts for multilingual datasets provides better performance with XGLM than human-translating the English prompt to the dataset language.

In this work, we use the BLOOM models (Scao et al., 2022a, b), which were pretrained on the

Figure 3: Comparison of dataset variants P3, xP3, and xP3mt on a sample from PAWS for P3 (Zhang et al., 2019) and PAWS-X (Yang et al., 2019) for xP3 and xP3mt. P3 pairs English datasets with English prompts, xP3 pairs multilingual datasets with English prompts and xP3mt pairs multilingual datasets with prompts machine-translated from English to match the dataset language. Expressions in curly brackets are replaced, e.g. for xP3mt the target shown as {(Choices[label]})) becomes Si.

 

ROOTS corpus Laurencon et al. (2022) in 46 natural languages and 13 programming languages. We also finetune mT5 Xue et al. (2020) to compare encoder-decoder and decoder-only performance. mT5 is pretrained on a corpus sampled from mC4 covering 101 languages.

## 3 Finetuning data and models

To study crosslingual multitask prompted finentuning, we create xP3 by extending the P3 dataset collection with additional non-English tasks. We finetune both BLOOM and mT5 models on xP3. We refer to Appendix SSC for public links to released models and datasets.

### Finetuning data

We build on the P3 Sanh et al. (2022) task taxonomy and add 30 new multilingual datasets illustrated in Figure 1. We define four task clusters previously not present in P3: translation, simplification, program synthesis, and miscellaneous code datasets. As 11% of BLOOM's pretraining data is code, we add code datasets classified as program synthesis (text-to-code) or miscellaneous. The latter includes tasks such as estimating the computational complexity of a provided code snippet and generating a name for a given function. We extend the XWinograd dataset Tikhonov and Ryabinin (2021) with winograd schemas from CLUE Xu et al. (2020) to increase its Chinese samples from 16 to 504. Similar to P3, a fraction of our prompts invert the task at hand. For example, a prompt may invert a closed-book QA sample by asking the model to generate a question given an answer.

With xP3 we aim to replicate the language distribution of the ROOS corpus Laurencon et al. (2022) used to pretrain BLOOM. Thus, xP3 consists of the same 46 natural languages and code as ROOTS. ROOTS, xP3 and the mT5 corpus Xue et al. (2020) language distributions are visualized in Figure 2. 39% of xP3 data is English, slightly more than the 30% of English data in ROOSTs. Various African languages such as Twi (tw) and Bambara (bm) form the tail of xP3's language distribution. Many of them are not included in the mT5 pretraining corpus. In xP3, Twi and others are represented solely as a translation task using data from Flores-200 NLLB Team et al. (2022).

To study the importance of non-English prompts, we construct a machine-translated variant of xP3, xP3mt. We translate prompts of monolingual datasets into the respective dataset language. For example, for the Chinese dataset C3 Sun et al. (2020) prompts in xP3mt are in Chinese instead of English in xP3. For crosslinkingal datasets prompts remain in English in xP3mt (such as Wiki-Lingua, which involves producing a summary in one language based on text in another language). We use the Google Cloud API for machine translation1. Figure 3 compares the dataset variants we train on.

Footnote 1: [https://cloud.google.com/translate](https://cloud.google.com/translate)

### Models

We use publicly available pretrained BLOOM models ranging from 560 million to 176 billion parameters. BLOOM models are large decoder-only language models pretrained for around 350 billion tokens with an architecture similar to GPT-3 Brown et al. (2020). We finetune the models for an additional 13 billion tokens with loss only being computed on target tokens. For example, given the input "Translate to English: Je t'aime." and a space-separated target "I love you.", the model is trained to predict only the targets. As targets vary in length from just one to hundreds of tokens, we downscale the loss of each token by the length of the target it belongs to. This ensures short targets (e.g. for multiple-choice QA) get the same weight as long targets (e.g. for translation). We skip samples longer than 2048 tokens and use packing to train efficiently on multiple samples at a time Kosec et al. (2021). We select the final checkpoint based on validation performance.

For mT5 models, we finetune using the T5X Roberts et al. (2022) framework on TPUs. mT5 uses the same encoder-decoder architecture, pretraining objective (masked language modeling), and pretraining length (1 trillion tokens) as T5 Raffel et al. (2020). For finetuning mT5, we follow the same procedure as described above for BLOOM, except that inputs are fed into the encoder and thus are not space-separated from targets.

We produce three core model variants available in different sizes:

* **BLOOMZ-P3 / mT0-P3:** Models finetuned on the English-only P3.
* **BLOOMZ / mT0:** Models finetuned on xP3, which consists of multilingual datasets with English prompts.
* **BLOOMZ-MT / mT0-MT:** Models fine-tuned on xP3mt, which consists of multi 

lingual datasets with English and machine-translated prompts.

We evaluate on three held-out tasks: coreference resolution, sentence completion and natural language inference (NLI) as depicted in Figure 1. We also evaluate on HumanEval due to its popularity for code evaluations using the pass@k metric Chen et al. (2021). For datasets that involve choosing the correct completion from several options, we follow prior work Sanh et al. (2022); Brown et al. (2020) and use rank classification: We compute the log-likelihood of each possible completion and select the highest scoring option. For each evaluation dataset, we select 5 prompts at random from PromptSource and use them for all language splits of the dataset. We report the median of the 5 prompts for results per language split. Thus, in constrast to XGLM Lin et al. (2021), we do not tune prompts based on performance on validation data. A selection of prompts can be found in Appendix SSM. For evaluation on generative tasks, such as translation, we use lm-evaluation-harness Gao et al. (2021) and report BLEU scores Papineni et al. (2002).

## 4 Results

We first examine generalization to new tasks in languages included in finetuning in SS4.1. Then, in SS4.2, we look at language generalization: Can models generalize to tasks in languages that **(a)** they have only seen during pretraining and **(b)** they have never seen intentionally? In SS4.3, we investigate performance on multilingual prompts and finetuning on xP3mt. Scaling laws are analyzed in SS4.4. Finally, SS4.5 looks at performance on generative tasks and SS4.6 at the effect of language proportions on performance.

### Task generalization

Previous work has shown that large language models finetuned on prompted multitask mixtures generalize to unseen tasks Zhong et al. (2021); Wei et al. (2021); Mishra et al. (2021, 2021); Wang et al. (2022). In Figure 4, we show that the same applies to multilingual models: Finetuned BLOOMZ and BLOOMZ-P3 models significantly improve over BLOOM and XGLM on held-out tasks. Despite an order of magnitude fewer parameters, mT0 (13 billion parameters) is ahead of BLOOMZ (176 billion parameters). We attribute this to the encoder-decoder architecture paired with a masked language modeling pretraining objective Wang et al. (2022); Tay et al. (2022) as well as the longer pretraining of mT5 Hoffmann et al. (2022); Su et al. (2022) (1 trillion tokens for mT5 vs. 366 billion for BLOOM). Despite also having gone through crosslingual multitask finetuning, mtK-Instruct performs significantly worse than the same-sized mT0. We attribute this to our prompting style, which aims to replicate natural human communication. mTk-Instruct is finetuned on more structured prompts with specific "Definition", "Input" and "Output" fields. Similarly, Wang et al. (2022) find that T0 performs worse than Tx-Instruct on their prompts. We also find models finetuned on the 39% English xP3 (BLOOMZ, mT0-13B) outperform models finetuned on the 100% English P3 (BLOOMZ-P3, mT0-13B-P3) on _English tasks_ (Appendix SSB). Even the fully English T0-11B model Sanh et al. (2022) is outperformed by our mTO-13B model on entirely _English tasks_. Ignoring embedding parameters T0-11B and mTO-13B have about the same size. This is likely due to xP3 adding additional

Figure 4: Zero-shot multilingual task generalization with English prompts. BLOOM models have 176 billion parameters. Scores are the language average for each task. Appendix §B breaks down performance by language.

 

tasks and prompts, which has been shown to help generalization Chung et al. (2022); Iyer et al. (2022). mT0-13B beating T0-11B indicates that the benefit of scaling tasks is larger than the benefit of pretraining and finetuning on relatively more English tokens.

### Language generalization

Here we add another layer of generalization: languages. Figure 4 already shows that finetuning on English data only (P3) leads to better performance on non-English data: For example, BLOOMZ-P3 improves by over 50% on multilingual sentence completion compared to BLOOM. Thus, zero-shot task performance in languages only seen during pretraining improves after finetuning on English. This has major practical benefits as it can be more difficult to collect data for low-resource languages.

Next, we investigate performance on languages the model has _never intentionally seen_. Due to the scale of large language model pretraining, it is difficult to label tasks or languages as strictly unseen. It is likely that the training data unintentionally includes small fractions of these languages (just as many tasks might appear "implicitly" in the pretraining corpus Sanh et al. (2022)). In Figure 5 we show that after multitask finetuning on xP3, the models can perform unseen tasks in languages that were not intentionally trained on. After probing the pretraining corpus of BLOOM, we do find small amounts of these languages that were unintentionally included (Appendix SSD). However, for XNLI, performance increases across all languages, many of which only show up in tiny fractions in our language contamination analysis, such as Thai with 0.006%. If we extrapolate this proportion to the entire ROOTS corpus, the BLOOM models would have seen a mere 20 million tokens of Thai during pretraining. One possibility is that better-than-random XNLI performance can be attained with little or no language understanding. In Appendix SSB, we investigate edit distances of XNLI samples and find that there are differences across labels, however, likely not significant enough to enable this kind of generalization.

### Multilingual prompting

Since all prompts in xP3 ar

\begin{table}
\begin{tabular}{l l|c c|c c} \hline Task & Prompt & \multicolumn{5}{c}{Average accuracy} \\  & & BloomZ & BloomZ-MT & mT0-13B & mT0-13B-MT \\ \hline XNLI & EN & **52.99** & 49.01 & 48.24 & **51.29** \\  & MT & 37.56 & **41.16** & 39.31 & **41.66** \\  & HT & 40.4 & **43.88** & 44.95 & **46.87** \\ \hline SCOPA & EN & 72.52 & **73.24** & **81.4** & 80.36 \\  & MT & 70.04 & **71.84** & **81.16** & 79.64 \\ \hline XStoryCloze & EN & **81.73** & 81.39 & 81.99 & **82.3** \\  & MT & 80.89 & **81.76** & **83.37** & 82.86 \\ \hline XWinograd & EN & **60.07** & 59.15 & 70.49 & **73.24** \\  & MT & 58.48 & **60.14** & 66.89 & **72.33** \\ \hline \end{tabular}
\end{table}
Table 1: Comparison between EN (English), MT (machine-translated) and HT (human-translated) prompts for 176B BLOOMZ and 13B mT0 models finetuned on either only English or English and machine-translated multilingual prompts (-MT).

Figure 5: Zero-shot task and language generalization using English prompts on tasks and languages not intentionally seen during pretraining nor finetuning. Language codes are ISO 639-1, except for JP (Japanese).

 

multilingual datasets), we created xP3mt, an extension with machine-translated prompts. To investigate performance on non-English prompts, we additionally human- and machine-translated the English evaluation prompts from Figure 4. In Table 1, we report performance on these. Results on machine-translated prompts in languages that are not part of the finetuning corpus, such as those in Figure 5, are in Appendix SSI. Table 1 shows that BLOOMZ performs much better on English than on non-English prompts. BLOOMZ-MT, which is fine-tuned on xP3mt, significantly improves on multilingual prompts. On XNLI, BLOOMZ-MT raises the average performance on human-translated prompts from 41.13 to 45.55. This comes at the cost of a reduction in its performance on English prompts, from 53.58 to 49.74. For mT0, the MT version provides similar performance gains on XNLI and XWinograd non-English prompts, while results on XCOPA and XStoryCloze are mixed. Similar to Lin et al. (2021), we also find that models perform better on human-translated prompts than machine-translated ones for XNLI.

### Scaling

In Figure 4, the average performance of BLOOM is near the random baselines of 0.50 for Sentence Completion and Coreference Resolution and 0.33 for NLI. We think this is due to all of our experiments being zero-shot and using untuned prompts Perez et al. (2021). We find in Figure 6 that even at 560M parameters, multitask finetuning improves zero-shot generalization. The gap between pretrained and multitask finetuned models grows significantly as parameters increase. Scaling up parameters benefits all languages evaluated.

### Generation tasks

In this section, we investigate the impact of multitask finetuning on generative tasks. In Figure 7, we plot validation performance throughout the training process. We find that while performance on natural language understanding tasks continues to increase, generative performance jumps initially and then decreases. Relatedly, in Table 2, we find that multitask finetuning does not improve performance on HumanEval Chen et al. (2021). Only for small models, such as BLOOM-560M vs. BLOOMZ-560M, there are meaningful performance gains. When no code data is included in finetuning (BLOOMZ-P3) performance decreases significantly. mT0 models, which have not been pretrained on code, fail to solve any HumanEval problems (see full results in Appendix SSK). Given a Python docstring, HumanEval requires models to complete a function. Inspecting generations reveals that the multitask finetuned models are biased towards short generations. In Appendix SSE, we show example solutions from HumanEval and compute

Figure 6: Aggregate performance vs. size. Transparent lines correspond to individual languages, while thick lines are average accuracy scores.

Figure 7: Validation performance during training on natural language understanding (NLU) and natural language generation (NLG) tasks. The former are scored using accuracy and the latter using BLEU Papineni et al. (2002). The NLG tasks measured are translation and summarization. For BLOOMZ(-7.1B) the performance at 0 training tokens corresponds to the performance of BLOOM(-7.1B). For mT0 there is no data point at 0 tokens, as its base model, mT5, is not suitable for evaluation without finetuning. Performance on individual tasks is in Appendix §K.

 

average length statistics. BLOOMZ tries to solve problems with 70% fewer characters than BLOOM.

This bias towards short answers and the performance drop on generative tasks come from fine-tuning on short texts. Most tasks in our finetuning dataset, xP3, are single sentences. We show in Appendix SSG that finetuning on fewer short tasks via early stopping, adding long tasks or upweighting long tasks leads to longer generations and slightly better performance. We find it most effective, however, to force a minimum generation length at inference. This is done by ignoring any probability mass the model assigns to its end-of-sequence token for a desired number of tokens. Only after the generation has reached the desired length, can the model generate the end-of-sequence token, thus finishing the generation. Forcing a minimum generation length improves the BLEU score on a translation task by 9 points, see Appendix SSG for quantitative and Figure 15 for qualitative results.

### Effect of language proportions

In Figure 8, we find that finetuned BLOOM models perform better on languages seen extensively during pretraining. As the language distribution in the finetuning dataset, xP3, closely follows that of pretraining, these languages are also seen most frequently during finetuning. Specifically, XCOPA and XNLI show significantly better performance on these high-resource languages, such as English, Spanish or French, which all make up more than 10% of pretraining individually. The trend is less consistent for XWinograd. This may be caused by the fact that XWinograd language subsets are not translations of each other and have a significantly different number of samples. Thus, some language subsets of XWinograd may be inherently more difficult than others.

## 5 Conclusion

In this work we investigated crosslingual multitask finetuning. We developed xP3, a corpus consisting of tasks in 46 languages. Further, we have extended xP3 to xP3mt with machine-translated prompts. We have finetuned pretrained BLOOM and mT5 models on the newly created corpora as well as the English-only P3 corpus to produce BLOOMZ and mT0 models.

We found that English-only finetuning suffices for a multilingual pretrained large language model to generalize to tasks in other pretrained languages.

\begin{table}
\begin{tabular}{l c c c} \hline \hline  & \multicolumn{2}{c}{Pass@_k_} \\  & \(k=1\) & \(k=10\) & \(k=100\) \\ \hline GPT-Neo 1.3B & 4.79\% & 7.47\% & 16.30\% \\ GPT-Neo 2.7B & 6.41\% & 11.27\% & 21.37\% \\ GPT-J 6B & 11.62\% & 15.74\% & 27.74\% \\ GPT-NeoX 20B & 15.4\% & 25.6\% & 41.2\% \\ \hline Codex-300M & 13.17\% & 20.37\% & 36.27\% \\ Codex-679M & 16.22\% & 25.7\% & 40.95\% \\ Codex-2.5B & 21.36\% & 35.42\% & 59.5\% \\ Codex-12B & 28.81\% & 46.81\% & 72.31\% \\ \hline BLOOM-560M & 0.82\% & 3.02\% & 5.91\% \\ BLOOM-1.1B & 2.48\% & 5.93\% & 9.62\% \\ BLOOM-1.7B & 4.03\% & 7.45\% & 12.75\% \\ BLOOM-3B & 6.48\% & 11.35\% & 20.43\% \\ BLOOM-7.1B & 7.73\% & 17.38\% & 29.47\% \\ BLOOM & 15.52\% & 32.20\% & 55.45\% \\ \hline BLOOMZ-560M & 2.18 \% & 4.11\% & 9.00\% \\ BLOOMZ-1.1B & 2.63\% & 6.22\% & 11.68\% \\ BLOOMZ-1.7B & 4.38\% & 8.73\% & 16.09\% \\ BLOOMZ-3B & 6.29\% & 11.94\% & 19.06\% \\ BLOOMZ-7.1B & 8.06\% & 15.03\% & 27.49\% \\ BLOOMZ & 12.06\% & 26.53\% & 48.44\% \\ BLOOMZ-P3 & 6.13\% & 11.79\% & 18.73\% \\ \hline \hline \end{tabular}
\end{table}
Table 2: Code continuation on HumanEval. Non-BLOOM results come from prior work Chen et al. (2021); Fried et al. (2022). Codex is a language model finetuned on code, while the GPT models Black et al. (2021); Wang and Komatsuzaki (2021); Black et al. (2022) are trained on a mix of code and text like BLOOM. Following Chen et al. (2021) we generate 200 samples for each problem with top \(p\) = 0.95 and compute pass rates. We perform this evaluation three times for temperatures 0.2, 0.6 and 0.8 and pick the best pass rate.

Figure 8: Performance across languages by size in the BLOOM pretraining corpus, ROOTS.

 

However, finetuning on multiple languages using xP3 provided even better performance. We have further observed finetuned models to be capable of generalization to new tasks in languages they have never intentionally seen. We investigated multilingual prompting and found performance after finetuning on English prompts only to be poor. However, finetuning on a corpus with machine-translated prompts (xP3mt) lead to significantly better performance on human-written non-English prompts. Comparing models from 560 million up to 176 billion parameters revealed that the performance gap between only pretraining and finetuning widens as parameters increase. Lastly, we found multitask finetuning on billions of short targets biases models to produce short answers, which can hurt performance on generative tasks. We proposed a simple workaround by forcing a minimum generation length at inference.

To contribute to future progress on improving zero-shot generalization, we release all datasets and models introduced in this work.

## 6 Limitations

We highlight several limitations of our work:

Unnatural prompting formatThe choice to separate inputs and targets using a space character has proven effective to multitask finetune our decoder-only models. Nonetheless, poorly formatted prompts may result in undesirable behavior. For example, given the following prompt: "Translate to English: Je t'aime", the model may continue the input with additional French content before starting to solve the task, i.e. translating the input from French to English. This can be mitigated by improving the prompts with a trailing full stop or a newline symbol. Encoder-decoder models, such as our mT0, do not suffer from this problem, as inputs and targets are fed into different parts of the model.

Limited languages in xP3The pretraining corpus of mT0 contains more than 101 languages Xue et al. (2020), however, we finettune on only 46 languages. Likely, finetuning on the full 101 languages mT0 has seen during pretraining would lead to better performance. However, we decided to use only the languages of BLOOM in order to study language generalization (SS4.2). Similarly, one could likely attain better performance by enhancing xP3 with more datasets, such as via BIG-Bench Srivastava et al. (2022); Suzgun et al. (2022), or more prompts, such as via NL-Augmenter Dhole et al. (2021). We have released an extended version of xP3 dubbed xP3x that covers 277 languages and is around ten times larger than xP3, but are yet to finetune models on it.

PerformanceWhile our models show strong capabilities of performing tasks zero-shot, there remain numerous failure modes that are common in large language models Rae et al. (2021); Bomassani et al. (2021); Zhang et al. (2022); Smith et al. (2022); Ouyang et al. (2022); Taylor et al. (2022); Chowdhery et al. (2022); Biderman et al. (2023); Allal et al. (2023); Li et al. (2023). In Figure 16 of Appendix SSF, BLOOMZ fails to understand the moral of a fable resulting in an undesirable generation. Similarly, in Figure 15, mT0-13B is asked to provide an explanation, but answers with a question. We have made several modifications to the multitask finetuning recipe, such as loss weighting, mixing in long tasks, and various multilingual aspects, leading to the strong zero-shot performance of our models. However, there are many other changes to the multitask finetuning procedure that are worth exploring to get better models Honovich et al. (2022); Wang et al. (2022b); Longpre et al. (2023); Liu et al. (2023); Dettmers et al. (2023); Yin et al. (2023). Further, the pre-trained models we use, BLOOM and mT5, are suboptimal in many aspects such as compute allocation Hoffmann et al. (2022); Muennighoff et al. (2023), pre-training datasets Longpre et al. (2023); Touvron et al. (2023); Chung et al. (2023), pre-training objective Tay et al. (2022b) and possibly model architecture Komatsuzaki et al. (2022); Shen et al. (2023). Future work should investigate multitask finetuning better base models.

Learning new languages during finetuningWhile we have investigated generalization to languages only seen during pretraining, we did not investigate generalization to languages only seen during finetuning. Our mT0 models are finetuned on several new languages not seen in pretraining (see Figure 2). Out of those, we only evaluated on code HumanEval), where mT0 performed at the random baseline (0.00 in Table 10). We point to follow-up work that has investigated the question of teaching BLOOMZ new languages Yong et al. (2022); Cahyawijaya et al. (2023) and work investigating adaptation of BLOOM Ennen et al. (2023); Yong and Nikoulina (2022) 



## Acknowledgments

This work was granted access to the HPC resources of Institut du developpement et des ressources en informatique scientifique (IDRIS) du Centre national de la recherche scientifique (CNRS) under the allocation 2021-A0101012475 made by Grand equipement national de calcul intensif (GENCI). In particular, all the evaluations and data processing ran on the Jean Zay cluster of IDRIS, and we want to thank the IDRIS team for responsive support throughout the project, in particular Remi Lacroix.

We thank the XGLM team for providing access to XStoryCloze. We thank volunteers who human-translated XXLI prompts. We thank Noah Constant and Douwe Kiela for feedback on drafts of this paper. We thank Victor Sanh, Stephen Bach, Sasha Rush and Jordan Clive for support throughout the project.

## References

* [1]D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. B. Bernstein, J. Bohg, A. Bosselut, C. Brunskill, et al. (2021) On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258. Cited by: SS3.
* [2]J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. (2021) Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Cited by: SS1.
* [3]S. H. Bach, V. Sanh, Z. Yong, A. Webson, C. Raffel, N. V. Nayak, A. Sharma, T. Kim, M. Saiful Bari, T. Fevry, Z. Alyafeai, M. Dey, A. Santilli, Z. Sun, S. Ben-David, C. Xu, G. Chhablani, H. Wang, J. Alan Fries, M. S. Al-shaibani, S. Sharma, U. Thakker, K. Almubarak, X. Tang, Y. Tang, M. Tian-Jian Jiang, and A. M. Rush (2022) Promptsource: an integrated development environment and repository for natural language prompts. External Links: 2206.09455 Cited by: SS1.
* [4]T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litvin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei (2020) Language models are few-shot learners. In Advances in Neural Information Processing Systems, Vol. 33, pp. 1877-1901. Cited by: SS1.
* [5]K. Cho, L. Zhang, W. Li, and X. Liu (2020) An open source database for web learning. arXiv preprint arXiv:2009.03038. Cited by: SS1.
* [6]L. Deng, L. Dong, and J. Jia (2019) Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1-10. Cited by: SS1.
* [7]Y. Chung, H. Lu, and X. Liu (2020) Learning multiple representations from a single document corpus. arXiv preprint arXiv:2009.03038. Cited by: SS1.
* [8]H. Dai, Y. Zhou, and J. Shi (2020) Multi-lingual multi-lingual multi-label classification dataset for zero-shot cross-lingual transfer. arXiv preprint arXiv:2009.00946. Cited by: SS1.
* [9]X. Cheng, J. Tworek, H. Jun, Q. Yuan, H. Poonde de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. (2021) Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Cited by: SS1.



Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. _arXiv preprint arXiv:2204.02311_.
* Chung et al. (2023)Hyung Won Chung, Noah Constant, Xavier Garcia, Adam Roberts, Yi Tay, Sharan Narang, and Orhan Firat. 2023. Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining. _arXiv preprint arXiv:2304.09151_.
* Chung et al. (2022)Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyu Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scaling instruction-finetuned language models. _arXiv preprint arXiv:2210.11416_.
* Conneau et al. (2029)Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. _arXiv preprint arXiv:1911.02116_.
* Conneau et al. (2018)Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. xnli: Evaluating cross-lingual sentence representations. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
* Cui et al. (2018)Yiming Cui, Ting Liu, Li Xiao, Zhipeng Chen, Wentao Ma, Wanxiang Che, Shijin Wang, and Guoping Hu. 2018. A span-extraction dataset for chinese machine reading comprehension. _arXiv preprint arXiv:1810.07366_.
* Dettmers et al. (2023)Tim Dettmers, Artidotro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. _arXiv preprint arXiv:2305.14314_.
* Devlin et al. (2018)Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. _arXiv preprint arXiv:1810.04805_.
* Dhole et al. (2021)Kaustubh D Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Mahamood, Abianya Mahendiran, Simon Mille, Ashish Srivastava, Samson Tan, et al. 2021. NI-augmenter: A framework for task-sensitive natural language augmentation. _arXiv preprint arXiv:2112.02721_.
* Ding et al. (2022) Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zhonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weizee Chen, et al. 2022. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. _arXiv preprint arXiv:2203.06904_.
* Ennon et al. (2023)Philipp Ennon, Po-Chun Hsu, Chan-Jan Hsu, Chang-Le Liu, Yen-Chen Wu, Yin-Hsiang Liao, Chin-Tung Lin, Da-Shan Shiu, and Wei-Yun Ma. 2023. Extending the pre-training of bloom for improved support of traditional chinese: Models, methods and results. _arXiv preprint arXiv:2303.04715_.
* Fan et al. (2021)Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. 2021. Beyond english-centric multilingual machine translation. _J. Mach. Learn. Res._, 22(107):1-48.
* Fried et al. (2022)Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022. Incoder: A generative model for code infilling and synthesis. _arXiv preprint arXiv:2204.05999_.
* Fries et al. (2022)Jason Alan Fries, Leon Weber, Natasha Seelam, Gabriel Altay, Debajyoti Datta, Samuele Garda, Myungsun Kang, Rusii Su, Wojciech Kusa, Samuel Cahyahwijaya, et al. 2022. Bigbio: A framework for data-centric biomedical natural language processing. _arXiv preprint arXiv:2206.15076_.
* Gao et al. (2021)Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evaluation.
* Goyal et al. (2021)Naman Goyal, Jingfei Du, Myle Ott, Giri Anantharaman, and Alexis Conneau. 2021a. Larger-scale transformers for multilingual masked language modeling. _arXiv preprint arXiv:2105.00572_.
* Goyal et al. (2021b)Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc'Aurelio Ranzato, Francisco Guzm'an, and Angela Fan. 2021b. The flores-101 evaluation benchmark for low-resource and multilingual machine translation.
* Guzm'an et al. (2019)Francisco Guzm'an, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc'Aurelio Ranzato. 2019. Two new evaluation datasets for low-resource machine translation: Nepali-english and sinhala-english.
* Hasan et al. (2021)Thamid Hasan, Abhik Bhattacharjee, Md. Safiful Islam, Kazi Mubassshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahrijar. 2021. XLsum: Large-scale multilingual abstractive summarization for 44 languages. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 4693-4703, Online. Association for Computational Linguistics 

Vincent J. Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, and David Bieber. 2020. Global relational models of source code. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
* Hendrycks et al. (2021) Dan Hendrycks, Steven Basart, Sauva Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring coding challenge competence with apps. _NeurIPS_.
* Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. _arXiv preprint arXiv:2203.15556_.
* Honovich et al. (2022) Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2022. Unnatural instructions: Tuning language models with (almost) no human labor. _arXiv preprint arXiv:2212.09689_.
* Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. _arXiv preprint arXiv:2106.09685_.
* Iyer et al. (2022) Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihalyov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. 2022. Opt-impl: Scaling language model instruction meta learning through the lens of generalization. _arXiv preprint arXiv:2212.12017_.
* Kim et al. (2021) Joongwon Kim, Mouncia Maddela, Reno Kriz, Wei Xu, and Chris Callison-Burch. 2021. BiSECT: Learning to split and rephrase sentences with bitexts. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 6193-6209, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Komatsuzaki et al. (2022) Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. 2022. Sparse upcycling: Training mixture-of-experts from dense checkpoints. _arXiv preprint arXiv:2212.05055_.
* Kosec et al. (2021) Matej Kosec, Sheng Fu, and Mario Michael Krell. 2021. Packing: Towards 2x nlp bert acceleration. _arXiv preprint arXiv:2107.02027_.
* Lample and Conneau (2019) Guillaume Lample and Alexis Conneau. 2019. Crosslingual language model pretraining.
* Laurencion et al. (2022) Hugo Laurencion, Lucile Saulinier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo Gonzalez Ponferrada, Huu Nguyen, et al. 2022. The bigscience roots corpus: A 1.6 tb composite multilingual dataset. In _Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.
* Levenshtein et al. (1966) Vladimir I Levenshtein et al. 1966. Binary codes capable of correcting deletions, insertions, and reversals. In _Soviet physics doklady_, volume 10, pages 707-710. Soviet Union.
* Lewis et al. (2019) Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2019. MIqa: Evaluating cross-lingual extractive question answering. _arXiv preprint arXiv:1910.07475_.
* Li et al. (2023) Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Koceetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you! _arXiv preprint arXiv:2305.06161_.
* Lin et al. (2021) Xi Victoria Lin, Todor Mihalyov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, et al. 2021. Few-shot learning with multilingual language models. _arXiv preprint arXiv:2112.10668_.
* Liu et al. (2022) Haokun Liu, Derek Tam, Mohammed Munqeth, Jay Mohtha, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. _arXiv preprint arXiv:2205.05638_.
* Liu et al. (2023) Qian Liu, Fan Zhou, Zhengbao Jiang, Longxu Dou, and Min Lin. 2023. From zero to hero: Examining the power of symbolic tasks in instruction tuning. _arXiv preprint arXiv:2304.07995_.
* Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marijan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pretraining for neural machine translation. _Transactions of the Association for Computational Linguistics_, 8:726-742.
* Logan et al. (2021) Robert L Logan, Ivana Balazevic, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. 2021. Cutting down on prompts and parameters: Simple few-shot learning with language models. _arXiv preprint arXiv:2106.13353_.
* Longpre et al. (2023a)Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023a. The final collection: Designing data and methods for effective instruction tuning. _arXiv preprint arXiv:2301.13688_.
* Longpre et al. (2023b) Shayne Longpre, Gregory Yaunye, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al. 2023b. A pretrained's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. _arXiv preprint arXiv:2305.13169_.
* Min et al. (2021) Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirizi. 2021. Metaicl: Learning to learn in context. _arXiv preprint arXiv:2110.15943_ 

Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2021a. Cross-task generalization via natural language crowdsourcing instructions. _arXiv preprint arXiv:2104.08773_.
* Mishra et al. (2021b) Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2021b. Natural instructions: Benchmarking generalization to new tasks from natural language instructions. _CoRR_, abs/2104.08773.
* Muennighoff (2022) Niklas Muennighoff. 2022. Sgpt: Gpt sentence embeddings for semantic search. _arXiv preprint arXiv:2202.08904_.
* Muennighoff et al. (2023) Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. 2023. Scaling data-constrained language models.
* Muennighoff et al. (2022) Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2022. Meteb: Massive text embedding benchmark. _arXiv preprint arXiv:2210.07316_.
* Team et al. (2021) NLLB Team, Marta R. Costa-jussas, James Cross, Onur Celebi, Maha Elbayad, Kenneth Hefaield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prongthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angelen Fan, Cynthia Gao, Vedani Goswami, Francisco Guzman, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation. _arXiv preprint 2207.04672_.
* Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wanwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. _arXiv preprint arXiv:2203.02155_.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_, pages 311-318.
* Patel et al. (2022) Ajay Patel, Bryan Li, Mohammad Sadegh Rasooli, Noah Constant, Colin Raffel, and Chris Callison-Burch. 2022. Bidirectional language models are also few-shot learners. _arXiv preprint arXiv:2209.14500_.
* Perez et al. (2021a) Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021a. True few-shot learning with language models. _Advances in Neural Information Processing Systems_, 34:11054-11070.
* Perez et al. (2021b) Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021b. True few-shot learning with language models. _CoRR_, abs/2105.11447.
* Phang et al. (2020) Jason Phang, Iacer Calixto, Phu Mon Hutut, Yada Pruksanchukun, Haokun Liu, Clara Vania, Katharina Kann, and Samuel R Bowman. 2020. English intermediate-task training improves zero-shot cross-lingual transfer too. _arXiv preprint arXiv:2005.13013_.
* Ponti et al. (2020) Edoardo M. Ponti, Goran Glavas, Olga Majewska, Qianchu Liu, Ivan Vuli'c, and Anna Kornonen. 2020. XCOPA: A multilingual dataset for causal commonsense reasoning. _arXiv preprint_.
* Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. _OpenAI blog_, 1(8):9.
* Rae et al. (2021) Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. _arXiv preprint arXiv:2112.11446_.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _J. Mach. Learn. Res._, 21(140):1-67.
* Raganato et al. (2020) Alessandro Raganato, Tommaso Pasini, Jose Camacho-Collados, and Mohammad Taher Pilehvar. 2020. Xlwic: A multilingual benchmark for evaluating semantic contextualization. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7193-7206.
* Roberts et al. (2020) Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Attor Lewkowycz, Alex Salcianu, Marc van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitng Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasminja Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp, Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin-Shepard, Noah Fieldet, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newman, and Andrea Gesmundo. 2022. Scaling up models and data with t5x and seqio. _arXiv preprint arXiv:2203.17189_.
* Roemmele et al. (2011) Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In _2011 AAAI Spring Symposium Series_.
* Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Stauwika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2022. Multitask prompted training enables zero-shot task generalization. In _The Tenth International Conference on Learning Representations_ 

* Le Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagne, Alexandros Sashu Luccioni, Francois Yvon, Matthias Galle, et al. 2022a. Bloom: A 176b-parameter open-access multilingual language model. _arXiv preprint arXiv:2211.05100_.
* Le Scao and Rush (2021) Teven Le Scao and Alexander M Rush. 2021. How many data points is a prompt worth? _arXiv preprint arXiv:2103.08493_.
* Le Scao et al. (2022b) Teven Le Scao, Thomas Wang, Daniel Hesslow, Lucile Saulnier, Stas Bekman, M Saiful Bari, Stella Bideman, Hady Elsahar, Niklas Muennighoff, Jason Phang, et al. 2022b. What language model to train if you have one million gpu hours? _arXiv preprint arXiv:2210.15424_.
* Schick and Schutze (2020) Timo Schick and Hinrich Schutze. 2020. Exploiting cloze questions for few shot text classification and natural language inference. _arXiv preprint arXiv:2001.07676_.
* Schick and Schutze (2020) Timo Schick and Hinrich Schutze. 2020. Exploiting cloze questions for few-shot text classification and natural language inference. _CoRR_, abs/2001.07676.
* Sciolam et al. (2022) Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. 2022. Continual-tD: Progressively instructing 50+ tasks to language models without forgetting. _arXiv preprint arXiv:2205.12393_.
* Shen et al. (2023) Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, et al. 2023. Flan-moe: Scaling instruction-finetuned language models with sparse mixture of experts. _arXiv preprint arXiv:2305.14705_.
* Shliazhko et al. (2022) Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlovz, and Tatiana Sharvina. 2022. mgpt: Few-shot learners go multilingual. _arXiv preprint arXiv:2204.07580_.
* Smith et al. (2022) Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajibhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. 2022. Using deep-speed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. _arXiv preprint arXiv:2201.11990_.
* Soltan et al. (2022) Saleh Soltan, Shankar Ananthakrishnan, Jack FitzGerald, Rahul Gupta, Wael Hamza, Haidar Khan, Charith Peris, Stephen Rawls, Andy Rosenbaum, Anna Rumshisky, Chandana Satya Prakash, Mukund Sridhar, Fabian Triefenbach, Apurv Verma, Gokhan Tur, and Prem Natarajan. 2022. Alexatm 20b: Few-shot learning using a large-scale multilingual seq2seq model.
* Srivastava et al. (2022) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abuwal Md Shoeb, Abbubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adria Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. _arXiv preprint arXiv:2206.04615_.
* Su et al. (2022) Hui Su, Xiao Zhou, Houjing Yu, Yuwen Chen, Zilin Zhu, Yang Yu, and Jie Zhou. 2022. Welm: A well-read pre-trained language model for chinese. _arXiv preprint arXiv:2209.10372_.
* Sun et al. (2020) Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. 2020. Investigating prior knowledge for challenging chinese machine reading comprehension. _Trans. Assoc. Comput. Linguistics_, 8:141-155.
* Suzgun et al. (2022) Mirac Suzgun, Nathan Scales, Nathanael Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoco V Le, Ed H Chi, Denny Zhou, et al. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. _arXiv preprint arXiv:2210.09261_.
* Tay et al. (2022a) Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huiaziu Steven Zheng, Neil Houlsby, and Donald Metzler. 2022a. Unifying language learning paradigms. _arXiv preprint arXiv:2205.05131_.
* Tay et al. (2022b) Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, et al. 2022b. Ul2: Unifying language learning paradigms. In _The Eleventh International Conference on Learning Representations_.
* Tay et al. (2022c) Yi Tay, Jason Wei, Hyung Won Chung, Vinh Q Tran, David R So, Siamak Shakeri, Xavier Garcia, Huaixiu Steven Zheng, Jinfeng Rao, Aakanksha Chowdhery, et al. 2022c. Transcending scaling laws with 0.1% extra compute. _arXiv preprint arXiv:2210.11399_.
* Taylor et al. (2022) Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science. _arXiv preprint arXiv:2211.09085_.
* Realistic data sets for low resource and multilingual MT. In _Proceedings of the Fifth Conference on Machine Translation_, pages 1174-1182. Association for Computational Linguistics.
* Tikhonov and Ryabinin (2021) Alexey Tikhonov and Max Ryabinin. 2021. It's all in the heads: Using attention heads as a baseline for cross-lingual transfer in commonsense reasoning.
* Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gutier Izaccard, Xavier Martinet, Marie-Anne Lacthaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. _arXiv preprint arXiv:2302.13971_.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all 

you need. _Advances in neural information processing systems_, 30.
* Vu et al. (2022) Tu Vu, Aditya Barua, Brian Lester, Daniel Cer, Mohit Iyyer, and Noah Constant. 2022. Overcoming catastrophic forgetting in zero-shot cross-lingual generation. _arXiv preprint arXiv:2205.12647_.
* Wang and Komatsuzaki (2021) Ben Wang and Aran Komatsuzaki. 2021. Gpt-j-6b: A 6 billion parameter autoregressive language model.
* Wang et al. (2022a) Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. 2022a. What language model architecture and pretraining objective work best for zero-shot generalization? _arXiv preprint arXiv:2204.05832_.
* Wang et al. (2022b) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaeh Hajishirzi. 2022b. Self-instruct: Aligning language model with self generated instructions. _arXiv preprint arXiv:2212.10560_.
* Wang et al. (2021) Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arijun Ashok, Arut Selvan Dhanasekaran, Athara Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Hiazhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Maitreya Patel, Kuntal Kumar Pal, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehab Singh Puri, Rushang Karia, Shailaja Keyrur Sampat, Savann Joshi, Siddhartha Mishra, Sujan Reddy, Sumanta Patro, Tanay Dist, Xudong Shen, Chitta Baral, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi, and Daniel Khashabi. 2022c. Super-naturalistructors: Generalization via declarative instructions on 1600+ nlp tasks. _arXiv preprint arXiv:2204.07705_.
* Wang et al. (2022d) Zhenhailong Wang, Xiaoman Pan, Dian Yu, Dong Yu, Jianshu Chen, and Heng Ji. 2022d. Zemi: Learning zero-shot semi-parametric language models from multiple tasks. _arXiv preprint arXiv:2210.00185_.
* Webson and Pavlick (2021) Albert Webson and Ellie Pavlick. 2021. Do prompt-based models really understand the meaning of their prompts?
* Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. _arXiv preprint arXiv:2109.01652_.
* Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1112-1122. Association for Computational Linguistics.
* Wu and Dredze (2019) Shijile Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 833-844, Hong Kong, China. Association for Computational Linguistics.
* Xu et al. (2020) Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yadong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, et al. 2020. Clue: A chinese language understanding evaluation benchmark. _arXiv preprint arXiv:2004.05986_.
* Xue et al. (2020) Liting Xue, Noah Constant, Adam Roberts, Mihir Kale, Ramil Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. _arXiv preprint arXiv:2010.11934_.
* Yang et al. (2019) Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification. In _Proc. of EMNLP_.
* Yin et al. (2023) Da Yin, Xiao Liu, Fan Yin, Ming Zhong, Heritik Bansal, Jiawei Han, and Kai-Wei Chang. 2023. Dynosaur: A dynamic growth paradigm for instruction-tuning data curation. _arXiv preprint arXiv:2305.14327_.
* Yong and Nikolaina (2022) Zheng-Xin Yong and Cassilina Nikoulina. 2022. Adapting bgiscience multilingual model to unseen languages. _arXiv preprint arXiv:2204.04873_.
* Yong et al. (2022) Zheng-Xin Yong, Hailey Schoelkopf, Niklas Muennighoff, Ahlam Fikri Aji, David Ifeoluwa Adelani, Khalid Almubark, Mai Saiful Bari, Lintang Sutawika, Jungo Kasai, Ahmed Baruvus, et al. 2022. Bloom+ 1: Adding language support to bloom for zero-shot prompting. _arXiv preprint arXiv:2212.09535_.
* Zaken et al. (2021) Elad Ben Zaken, Shauli Rayfogel, and Yoav Goldberg. 2021. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. _arXiv preprint arXiv:2106.10199_.
* Zeng et al. (2022) Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. Glm-130b: An open bilingual pre-trained model. _arXiv preprint arXiv:2210.02414_.
* Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artereko, Myga Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. _arXiv preprint arXiv:2205.01068_.
* Zhang et al. (2019) Yuan Zhang, Jason Baldridge, and Luheng He. 2019. Paws: Paraphrase adversaries from word scrambling. _arXiv preprint arXiv:1904.01130_.
* Zhong et al. (2021) Ruiqi Zhong, Kristy Lee, Zheng Zhang, and Dan Klein. 2021. Meta-tuning language models to answer prompts better. _CoRR_, abs/2104.04670 

Ming Zhu, Aneesh Jain, Karthik Suresh, Roshan Ravindran, Sindhu Tipirneni, and Chandan K. Reddy. 2022. Xlcost: A benchmark dataset for cross-lingual code intelligence.

 



###### Contents

* 1 Introduction
* 2 Related work
	* 2.1 Multitask learning
	* 2.2 Multilingual models
* 3 Finetuning data and models
	* 3.1 Finetuning data
	* 3.2 Models
* 4 Results
	* 4.1 Task generalization
	* 4.2 Language generalization
	* 4.3 Multilingual prompting
	* 4.4 Scaling
	* 4.5 Generation tasks
	* 4.6 Effect of language proportions
* 5 Conclusion
* 6 Limitations
* A Contributions
* B Task generalization breakdown
* C Artifacts
* D ROOTS language contamination
* E Code generations
* F Qualitative examples
* G Increasing generation length
* H XNLI edit distances
* I Multilingual prompting in unseen languages
* J Ideas that did not work
* K Full results
* L Version control
* M Prompts used

Contributions

This research was conducted under the BigScience project for open research, a year-long initiative targeting the study of large models and datasets. The goal of the project is to research language models in a public environment. The project has hundreds of researchers from more than 50 countries and over 250 institutions. The BigScience project was initiated by Thomas Wolf at Hugging Face, and this collaboration would not have been possible without his effort. In the following, we list contributions made to this work.

**Niklas Muennighoff** evaluated all models, created xP3 and wrote most of the paper.

**Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts and Hailey Schoelkopf** wrote the training and evaluation code.

**Niklas Muennighoff and Adam Roberts** trained the models.

**Niklas Muennighoff, Teven Le Scao, Hailey Schoelkopf, Zheng-Xin Yong, Thomas Wang, Khalid Almubarak, Alham Fikri Aji, M Saiful Bari and Zaid Alyafeai contributed prompts or datasets.

**Lintang Sutawika, Stella Biderman, Zheng-Xin Yong, Khalid Almubarak, M Saiful Bari and Albert Webson** initiated the project.

**Sheng Shen** conducted the contamination analysis.

**Samuel Albanie** wrote the prompt appendix.

**Thomas Wang and Zheng-Xin Yong** converted checkpoints.

**Colin Raffel, Thomas Wang, Teven Le Scao, M Saiful Bari, Edward Raff and Dragomir Radev advised the project.

**Niklas Muennighoff, Lintang Sutawika, Teven Le Scao, Colin Raffel, Stella Biderman, Alham Fikri Aji, Adam Roberts, Samuel Albanie, Sheng Shen, M Saiful Bari, Albert Webson, Xiangru Tang, Dragomir Radev and Edward Raff** contributed to the paper.

## Appendix B Task generalization breakdown

In Figure 9, we compare performance on English held-out tasks. We find that **(a)** finetuning on xP3 outperforms P3 **(b)** multilingual mT0 is better than monolingual T0 on _English tasks_. We think both improvements come from xP3 having more prompts and datasets than P3 Chung et al. (2022).

Figure 9: Zero-shot English task generalization. Each dot represents performance on one English evaluation prompt.

 

In Figure 10, we visualize task generalization to multilingual datasets. The same data is aggregated in Figure 4. Performance by prompt varies substantially highlighting that prompt engineering may still be necessary after MTF. We also find that mT0 consistently outperforms BLOOMZ on Swahili (SW), possibly due to it being a larger part of its pretraining corpus (see Figure 2 and SS4.6).

Figure 10: Zero-shot multilingual task generalization on languages seen during pretraining and finetuning. Each dot represents performance on one English evaluation prompt.

 

Artifacts

Table 3 lists all artifacts used or released in this work. We make all our work accessible under the most permissive licenses available to us.

## Appendix D ROOTS language contamination

While the BLOOM ROOTS corpus Laurencon et al. (2022) was collected from 46 natural languages and 13 programming languages, we find that sentences from the same document do not always belong to the collected (meta) language. Some sentences use languages like Russian or Japanese that were not the intentionally collected parts. This "language contamination" may stem from "code-mixing" or different languages being used in code comments. To investigate the extent of contamination, we randomly sample 1% of the documents from ROOTS for a total of 51M documents. For each document, we use cld\(3^{2}\)Xue et al. (2020) to identify the languages used in each sentence and compare them with the meta language of the document. We summarize our results in Figure 11. It shows that ROOTS contains unintentionally collected languages, such as Burmese (my: 0.00003%), Thai (th: 0.006%), Turkish (tr: 0.03%), Greek (el: 0.03%), Russian (ru: 0.03%), Bulgarian (bg: 0.05%), Estonian (et: 0.06%), Haitian (ht: 0.12%), German (de: 0.21%), Italian (it: 0.28%) and Japanese (ja: 0.54%). These "unseen" languages only have

\begin{table}
\begin{tabular}{l|c|l} \hline \hline
**Artifact** & \multicolumn{1}{c|}{**Explanation**} & \multicolumn{1}{c}{**Public link**} \\ \hline ROOTS & Multilingual pretraining corpus of BLOOM & [https://huggingface.co/bigscience-data](https://huggingface.co/bigscience-data) \\ pC4 & Multilingual pretraining corpus used for mtS & [https://huggingface.co/datasets/mt](https://huggingface.co/datasets/mt) \\ pJ3 & Multitask finenturing dataset with English data & English prompts & [https://huggingface.co/datasets/bigscience/pJ3](https://huggingface.co/datasets/bigscience/pJ3) \\ pJ51 & Multitask finenturing dataset with multilingual data & English prompts & [https://huggingface.co/datasets/bigscience/pJ3](https://huggingface.co/datasets/bigscience/pJ3) \\ pJ3all & Same as rJ9N with held-out evaluation sets & [https://huggingface.co/datasets/bigscience/rJ9all](https://huggingface.co/datasets/bigscience/rJ9all) \\ pJ5mt & Same as xP3H with English \& multilingual machine-simulated prompts & [https://huggingface.co/datasets/bigscience/rJ9ml](https://huggingface.co/datasets/bigscience/rJ9ml) \\ pJ7gnegds & Processed version of xP3 for easy using Meta-DeepSpeed & [https://huggingface.co/datasets/bigscience/rJ9nggds](https://huggingface.co/datasets/bigscience/rJ9nggds) \\ pJ8s & Extension of xP3 to 277 languages & [https://huggingface.co/datasets/Memightfor/xP3nk](https://huggingface.co/datasets/Memightfor/xP3nk) \\ \hline XGLM-7.5B & 7.5B parameter pretrained multilingual transformer & [https://huggingface.co/facebook/xglm-7.5B](https://huggingface.co/facebook/xglm-7.5B) \\ T0-11B & 11B parameter model fintured on PJ3 & [https://huggingface.co/bigscience/t0](https://huggingface.co/bigscience/t0) \\ nTR-Instruct-3.7B & 3.7B parameter multitask finentured multilingual transformer & [https://huggingface.co/allen/ntk-instruct-lb-def-pos](https://huggingface.co/allen/ntk-instruct-lb-def-pos) \\ nTR-Instruct-13B & 13B parameter multitask finentined multilingual transformer & [https://huggingface.co/allen/ntk-instruct-ltb-def-pos](https://huggingface.co/allen/ntk-instruct-ltb-def-pos) \\ \hline BLOOAI-560M & 560M parameter model pretrained on ROOTS & [https://huggingface.co/bigscience/blomo-560m](https://huggingface.co/bigscience/blomo-560m) \\ BLOOAI-1.1B & 1.1B parameter model pretrained on ROOTS & [https://huggingface.co/bigscience/blomo-1b1](https://huggingface.co/bigscience/blomo-1b1) \\ BLOOAI-1.7B & 1.7B parameter model pretrained pretrained on ROOTS & [https://huggingface.co/bigscience/blomo-1b1](https://huggingface.co/bigscience/blomo-1b1) \\ BLOOAI-3B & 3B parameter model pretrained pretrained on ROOTS & [https://huggingface.co/bigscience/blomo-1b1](https://huggingface.co/bigscience/blomo-1b1) \\ BLOOAI & 17GB parameter model pretrained on ROOTS & [https://huggingface.co/bigscience/blomo-1b1](https://huggingface.co/bigscience/blomo-1b1) \\ \hline BLOOAIZ-560M & 560M parameter model finentuned on xP3 & [https://huggingface.co/bigscience/blomz-560a](https://huggingface.co/bigscience/blomz-560a) \\ BLOOAIZ-1.1B & 1.1B parameter model finented on xP3 & [https://huggingface.co/bigscience/bloozm-1b1](https://huggingface.co/bigscience/bloozm-1b1) \\ BLOOAIZ-1.7B & 1.7B parameter model finentent on xP3 & [https://huggingface.co/bigscience/bloozm-1b2](https://huggingface.co/bigscience/bloozm-1b2) \\ BLOOAIZ-3B & 3B parameter model finented on xP3 & [https://huggingface.co/bigscience/bloozm-3b](https://huggingface.co/bigscience/bloozm-3b) \\ BLOOAIZ-1.7B & 7.1B parameter model finented on xP3 & [https://huggingface.co/bigscience/bloozm-7b1-art](https://huggingface.co/bigscience/bloozm-7b1-art) \\ BLOOAIZ-7.1B-RY & 7.1B parameter model finented on xP3T & [https://huggingface.co/bigscience/bloozm-7b1-g3](https://huggingface.co/bigscience/bloozm-7b1-g3) \\ BLOOAIZ-7.1B-RY & 7.1B parameter model finented on xP3T & [https://huggingface.co/bigscience/bloozm-7b1-g3](https://huggingface.co/bigscience/bloozm-7b1-g3) \\ BLOOAIZ-7.1B-RY & 7.1B parameter model finented on xP3T & [https://huggingface.co/bigscience/bloozm-7b1-g3](https://huggingface.co/bigscience/bloozm-7b1-g3) \\ BLOOAIZ-7.1B-RY & 7.1B parameter model finented on xP3T & [https://huggingface.co/bigscience/bloozm-7b1-g3](https://huggingface.co/bigscience/bloozm-7b1-g3) \\ BLOOAIZ-7.1B-RY & 7.1B parameter model finented on xP3T & [https://huggingface.co/bigscience/homezi-7.1B-RY](https://huggingface.co/bigscience/homezi-7.1B-RY) \\ BLOOAIZ-7.1B-RY & 7.1B parameter model finented on xP3T & [https://huggingface.co/bigscience/bloozm-7b1-g3](https://huggingface.co/bigscience/bloozm-7b1-g3) \\ BLOOAIZ-7.1B-RY & 7.1B parameter model finented on xP3T & [https://huggingface.co/bigscience/bt0-gsw1](https://huggingface.co/bigscience/bt0-gsw1) \\ BLOOAIZ-7.1B-RY & 7.1B parameter model finented on xP3T & [https://huggingface.co/bigscience/bt0-gsw1](https://huggingface.co/bigscience/bt0-gsw1) \\ BLOOAIZ-7.1B-RY & 7.1B parameter model finented on xP3 & [https://huggingface.co/bigscience/bt0-gsw1](https://huggingface.co/bigscience/bt0-gsw1) \\ \hline nTS-300M & 300M parameter model pretrained on sampled version of mC4 & [https://huggingface.co/google/nt5-small](https://huggingface.co/google/nt5-small) \\ nTS-55.30M & 580M parameter model pretrained on sampled version of mC4 & [https://huggingface.co/google/nt5-base](https://huggingface.co/google/nt5-base) \\ nTS-12B & 1.2B parameter model pretrained on a sampled version of mC4 & [https://huggingface.co/google/nt5-large](https://huggingface.co/google/nt5-large) \\ nTS-3.7B & 3.7B parameter model pretrained on a sampled version of mC4 & [https://huggingface.co/google/nt5-xl](https://huggingface.co/google/nt5-xl) \\ nTS-13B & 13B parameter model pretrained on a sampled version of mC4 & [https://huggingface.co/google/nt5-xl](https://huggingface.co/google/nt5-xl) \\ nTS-13B & 13B parameter model pretrained on a sampled version of mC4 & [https://huggingface.co/google/nt5-xl](https://huggingface.co/google/nt5-xl) \\ \hline nTM-300M & 300M parameter model finentuned on xP3 & [https://huggingface.co/bigscience/nt0-gsw1](https://huggingface.co/bigscience/nt0-gsw1) \\ nTO-580M & 580M parameter model finentuned on xP3 & [https://huggingface.co/bigscience/nt0-bsw2](https://huggingface.co/bigscience/nt0-bsw2) \\ nTO-1.2B & 1.2B parameter model finentent on xP3 & [https://huggingface.co/bigscience/nt0-lsw2](https://huggingface.co/bigscience/nt0-lsw2) \\ nTO-3.7B & 3.7B parameter model finentent on xP3 & [https://huggingface.co/bigscience/net0-gsw1](https://huggingface.co/bigscience/net0-gsw1) \\ nTO-13B & 13B parameter model finentuned on xP3 & [https://huggingface.co/bigscience/nt0-gsw1](https://huggingface.co/bigscience/nt0-gsw1) \\ nTO-13B-MT & 13B parameter model finentuned on xP3T & [https://huggingface.co/bigscience/nt0-skrl-mt](https://huggingface.co/bigscience/nt0-skrl-mt) \\ nTO-13B-P3 & 13B parameter model finentuned on xP3 & [https://huggingface.co/bigscience/nt0-skrl-pl](https://huggingface.co/bigscience/nt0-skrl-pl) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Links to all models & datasets used as part of this work. BLOOMZ models have an additional repository containing the final optimizer states for training with Megatron-Deepspeech that can be found by appending “-optimizer-states” to the respective URL. BLOOM(Z) models are released under the RAIL license, while mT5 / mT0 models are licensed under Apache 2.0 

small sentence proportions in our subsample compared to English (en: 46.23%), French (fr: 15.73%) and Spanish (es: 13.38%). Yet, they may help the language generalization of BLOOMZ models described in SS4.2. Japanese is mostly mixed in the meta English documents (47%), meta Code documents (8%) and meta Chinese documents (5%). Meanwhile, Russian is mostly mixed in the meta English documents (52%), meta Code documents (19%) and meta French documents (11%).

## Appendix E Code generations

Table 4 provides statistics on code generations and code data. We find that BLOOM generates on average 70% more characters and 17x more comments than BLOOMZ for a given problem from HumanEval. Figure 12 compares an example solution from BLOOM and BLOOMZ. While both solutions are correct, BLOOMZ is biased towards short and concise answers.

## Appendix F Qualitative examples

\begin{table}
\begin{tabular}{l|c|c} \hline Data (\(\rightarrow\)) & HumanEval generations & Fine-tuning data \\  & BLOOM & BLOOMZ & in rS?s (code data) \\ \hline Average characters & 247 & 144 & 531 \\ Average Python comments (**r**) & 0.69 & 0.04 & 0.85 \\ \hline \end{tabular}
\end{table}
Table 4: Number of characters and comments for generations and fine-tuning data. For finetuning data, the statistics are computed for the targets that the model is tasked to generate, not the input.

Figure 11: Language composition of ROOTS-IDENTIFY-1%, ROOTS-1% and the mT5 corpus. All mT5 languages are depicted. ROOTS-1% is a random 1% sample of ROOTS with its assigned meta-languages. ROOTS-IDENTIFY-1% are the actual languages in ROOTS-1% re-identified using c1d3.

Figure 12: Code generations of BLOOM and BLOOMZ on HumanEval. The model is prompted to generate after the final ” ” ””. The generation is stopped after an end-of-sequence token or a return statement followed by a newline.

 

Figure 14: Greedy generations for zero-shot query expansion, a task not trained on. The models sometimes fail to output at least five terms as requested in the prompt.

Figure 13: Greedy generations for sentiment analysis, a task trained on. BLOOMZ and mT0-13B have not been trained on non-English prompts, but are still able to handle them. BLOOMZ, however, answers in English. The review is a five star review of Star Wars Episode IV.

 

Figure 16: Non-greedy fable generations given a moral, a task not trained on. The generations are cherry-picked from 16 outputs with no minimum length, a temperature of 0.9 and top \(k\) of 40. **Left:** BLOOMZ generates an interesting fable with the desired moral. mT0 is significantly worse at writing stories likely due to its different pretraining objective. **Right:** BLOOMZ does not seem to understand the moral correctly.

Figure 15: Greedy generations on question answering, a task trained on. **Left:** Specifying the language in the prompt is an effective way to force the output language. **Right:** Setting a minimum token length as a generation hyperparameter is an effective way to force long generations. The output of BLOOM is shortened (marked with ...).

 

Increasing generation length

In SS4.5, we found performance on generative tasks to worsen in later stages of training. To investigate this problem further, we study a 7.1 billion parameter BLOOM model that is finetuned for 13 billion tokens, which results in a low BLEU score of 0 and very short generations as shown in Table 5 (Default). We can solve this problem with two high-level strategies: **(a)** Reducing short tasks during finetuning and **(b)** Forcing a minimum generation length.

For **(a)**, we do so by either early stopping, upweighting long tasks or adding new long tasks. As the majority of our finetuning data are single sentences, early stopping has the effect of finetuning on fewer short sentences. Upweighting long tasks is done by removing the loss normalization explained in SS3.2. This has the effect of each token getting equal weight regardless of the task, which upweights long tasks, as they have more tokens. Finally, for adding long tasks, we add tasks that require multi-sentence generations, such as generating an entire news article given a title. These long tasks collectively make up 10% of finetuning data for this ablation. All three solutions result in longer average generations as shown in Table 5 and slightly better BLEU scores, albeit effects are still small.

For **(b)**, we force the model to generate a minimum number of tokens at inference. Our benchmarking task, MultiEURLEX Chalkidis et al. (2021), requires multi-sentence generations with an average target length of 1965 characters (about 491 tokens). By forcing the model to generate at least 768 tokens, we ensure that the generation is at least as long as the target. This boosts the BLEU score significantly to 9.05. This approach is thus an effective strategy to maintain long generations of good quality.

For our final models, we employ early stopping, adding of long tasks and recommend forcing a minimum generation length at inference for long generations. We do not upweight longer tasks, as it worsens accuracy on our NLU validation tasks by 10%. The number of tokens our final models are fine-tuned for are displayed in Table 6.

## Appendix H XNLI edit distances

As models are surprisingly capable of solving XNLI in languages they were never intentionally trained on (SS4.2), we investigate whether XNLI can be solved without any language understanding. To do so, we compute edit distances using the Levenshtein methodology Levenshtein et al. (1966) between premise

\begin{table}
\begin{tabular}{l|c c c c c} \hline \hline Model & mT0-300M & mT0-560M & mT0-1.2B & mT0-3.7B & mT0-13B \\ \hline Tokens & 4.62 & 4.62 & 4.62 & 1.85 & 1.29 \\ \hline Model & BLOOMZ-560M & BLOOMZ-1.1B & BLOOMZ-1.7B & BLOOMZ-3B & BLOOMZ-7.1B & BLOOMZ \\ \hline Tokens & 3.67 & 0.502 & 8.39 & 8.39 & 4.19 & 2.09 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Tokens in billions that final models are finetuned for. We early-stop models based on validation performance. For -MT and -P3 variants we take the checkpoint after the same number of steps as for their default versions.

\begin{table}
\begin{tabular}{l|c c} \hline \hline Model & Finetuning tokens & BLEU Score & Average generation length (characters) \\ \hline Default & 13 billion & 0.00 & 122 \\ \hline Early stopping & 6 billion & 0.00 & 155 \\ Upweight longer tasks & 13 billion & 0.06 & 364 \\ Add more long tasks & 13 billion & 0.06 & 136 \\ \hline Forcing 768 tokens at inference & 13 billion & 9.05 & 3072 \\ \hline \hline \end{tabular}
\end{table}
Table 5: 7.1 billion parameter BLOOMZ models with various modifications benchmarked on MultiEURLEX English-French translation Chalkidis et al. (2021). We benchmark three prompts on both English to French and French to English translation. We then take the median performance across the three prompts for each translation direction and average the two scores to arrive at the BLEU score reported.

 

and hypothesis. Table 7 shows three samples from the English XNLI and their edit distances. Our hypothesis is that entailment pairs generally need to cover similar content, and thus have similar distance. Contradiction pairs still need to cover similar content but differ in at least one major way. Meanwhile for neutral pairs, hypothesis and premise may be about completely different topics, hence they should have the highest distance. In Table 8 we compute distances across all Thai, Turkish and Greek samples, three languages where we found language generalization to occur for BLOOMZ. Results confirm our hypothesis that distances are generally largest for neutral samples and smallest for entailment samples. However, the aggregate differences are very small with only a few edits difference. For example, Thai contradiction samples only have 2.5 edits more on average than entailment samples. Thus, comparing characters based on edit distance alone is likely not sufficient to fully explain the language generalization of models in SS4.2.

## Appendix I Multilingual prompting in unseen languages

Table 9 shows aggregate performances on languages not intentionally seen during pretraining nor fine-tuning for BLOOMZ and only seen during pretraining for mT0. For BLOOMZ, performance drops significantly when translating the prompts to the respective unseen languages. Unlike on translated prompts for seen languages (SS4.3), BLOOMZ-MT performs worse than BLOOMZ for machine-translated prompts in unseen languages. This is likely because BLOOMZ-MT has not been finetuned on prompts in these languages. For mT0 differences are less significant.

\begin{table}
\begin{tabular}{l c c c} \hline Premise & Hypothesis & Lev. distance & Label \\ \hline probably so probably so un-hum & probably yes so uh-huh & 13 & Entailment \\ equivalent to increasing national saving to 19 . & National savings are 18 now . & 34 & Neutral \\ The Inglethorps did not appear . & The Inglethorps were the first ones to turn up . & 26 & Contradiction \\ \hline \end{tabular}
\end{table}
Table 7: Three samples from the English XNLI split. To solve XNLI models need to classify whether the premise entails, is neutral to or contradicts the hypothesis. Samples are cherry-picked.

\begin{table}
\begin{tabular}{l l l l} \hline Label (\(\rightarrow\)) & Entailment & Neutral & Contradiction \\ Language (\(\downarrow\)) & & & \\ \hline Thai (th) & 79.08 & 82.64 & 81.52 \\ Turkish (tr) & 76.93 & 80.59 & 80.24 \\ Greek (el) & 90.90 & 95.10 & 93.93 \\ \hline \end{tabular}
\end{table}
Table 8: Levenshtein distances between hypothesis and premise averaged across samples from different XNLI labels. Each label has 830 samples per language subset.

 



## Appendix J Ideas that did not work

We list several experiments that did not improve over baseline results:

Non-causalIn a non-causal or prefix language model, the model attends bidirectionally over input tokens and only causally over target tokens. Given a pretrained causal decoder, other work found that multitask finetuning in a non-causal setup performed better than causal finetuning (Wang et al., 2022; Tay et al., 2022). However, in our experiments, non-causal finetuning did not improve over causal finetuning.

Special tokensInstead of separating inputs and targets with a space, we experimented with special tokens. Using the end-of-sequence token as a separator or a completely new token that the model would learn during finetuning significantly worsened results. The models may need to train on more tokens, possibly even during pretraining, to learn these new special tokens (Zeng et al., 2022).

Fixing promptsPromptSource has been written with encoder-decoder models in mind, where inputs and targets are fed into different models. As a consequence, human-written prompts in PromptSource often lack separators between input and target. For our decoder models, we decided to separate them with a space. We additionally experimented with leaving them as is or rewriting a significant amount of prompts, but neither improved significantly over space separation.

BitFitPrevious work has shown bias-only finetuning (Zaken et al., 2021) of large language models to be sufficient for strong downstream performance (Logan et al., 2021; Hu et al., 2021; Muennighoff, 2022; Liu et al., 2022; Ding et al., 2022; Muennighoff et al., 2022). We found multitask finetuning of only biases to perform 15 absolute percentage points worse on the average of held-out tasks for BLOOMZ-7.1B.

## Appendix K Full results

Table 10 shows all evaluation results on test datasets. Table 11 displays evaluation results on validation datasets which we use for checkpoint selection.

\begin{table}
\begin{tabular}{l l|r r|r r} \hline \hline Task & Prompt & \multicolumn{4}{c}{Average accuracy} \\  & & BLOOMZ & BLOOMZ-MT & mT0-13B & mT0-13B-MT \\ \hline XNLI & EN & **45.65** & 43.2 & 48.52 & **51.33** \\  & MT & **36.48** & 35.67 & **41.86** & 39.78 \\ \hline XCOPA & EN & **54.27** & 53.67 & **72.67** & 71.6 \\  & MT & **53.2** & 53.0 & **71.57** & 70.87 \\ \hline XYGlorCloze & EN & **61.59** & 61.36 & 79.31 & **80.13** \\  & MT & **60.5** & 59.91 & 80.21 & **80.28** \\ \hline XWinograd & EN & **55.98** & 54.54 & 70.81 & **72.0** \\  & MT & **53.11** & 52.46 & 67.86 & **70.45** \\ \hline \hline \end{tabular}
\end{table}
Table 9: Comparison between EN (English) and MT (machine-translated) prompts for 17GB BLOOMZ and 13B mT0 models finetuned on either only English or English and machine-translated multilingual prompts (-MT). For BLOOMZ the evaluation languages averaged are never intentionally seen, such as Japanese and Russian for XWinograd (see Figure 5). For mT0 the evaluation languages are only seen during pretraining.



* [14] M. C. Bader, A. D. Kessler, and R. J. Sorensen. The role of the social network in a social network. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 35-46, 2007.

[MISSING_PAGE_POST]

 



## Appendix L Version control

**V1 \(\rightarrow\) V2:**

* Added evaluation results for the validation datasets used for checkpoint selection (Appendix SSK)
* Added a section on the effect on generation length (Appendix SG) and rewrote parts of SS4.5
* Added a mention of xP3x, the extension of xP3 to 277 languages in Appendix SSC
* Added an example of XNLI to Appendix SSAH

## Appendix M Prompts used

This section describes the prompts used for training and evaluation.



In the following, dataset naming conventions follow those used in the Hugging Face datasets library. Since xP3 expands upon the P3 dataset employed by Sanh et al. (2022), we refer the reader to that work for example prompts from datasets that fall within P3. Here, we provide prompts curated for datasets that belong to xP3 but not to P3. The prompts provided are not exhaustive. Code will be released to provide a canonical reference. For each dataset considered, a dataset example is provided for additional context. Next, it is noted if the prompt does not match the original task formulation of the dataset. This is followed by a reference for the data, an input template and a target template. For prompts with predefined answer choices, these are also included. To provide examples of both human-translated and machine-translated prompts, samples of each kind are included for the xmli es dataset.

###### Contents

* 1 Prompts
	* 1.1 Simplification
		* 1.1.1 GEM/BiSECT en
		* 1.1.2 GEM/BiSECT es
		* 1.1.3 GEM/BiSECT fr
	* 1.2 Summarization
		* 1.2.1 GEM/wiki_lingua en
		* 1.2.2 GEM/wiki_lingua es
		* 1.2.3 GEM/xlsum bengali
		* 1.2.4 GEM/xlsum english
	* 1.3 Translation
		* 1.3.1 HelsinkiNLP/tatoeba_mt ben-eng
		* 1.3.2 HelsinkiNLP/tatoeba_mt eng-fra
		* 1.3.3 facebook/flores ben_Beng-eng_Latin
		* 1.3.4 facebook/flores ben_Beng-fra_Latin
	* 1.4 Program Synthesis
		* 1.4.1 Muennighoff/mbpp sanitized
		* 1.4.2 codeparrot/apps all
		* 1.4.3 codeparroft/github-jupyter-text-code-pairs
		* 1.4.4 codeparroft/xlcost-text-to-code C++program-level
		* 1.4.5 codeparroft/xlcost-text-to-code C-program-level
		* 1.4.6 codeparroft/xlcost-text-to-code Cs sharp-program-level
		* 1.4.7 codeparroft/xlcost-text-to-code Java-program-level
		* 1.4.8 codeparroft/xlcost-text-to-code Javascript-program-level
		* 1.4.9 codeparroft/xlcost-text-to-code PHP-program-level
* 1.4.10 codeparroft/xlcost-text-to-code Python-program-level
* 1.4.11 neural_code_search evaluation_dataset
* 1.4.12 teven/code_contests
	* 1.5 Coreference Resolution
 

1.5.1 Muennighoff/xwinograd en 1.5.2 Muennighoff/xwinograd fr
1.6 Question Answering Multiple Choice 1.6.1 clue c3
1.7 Question Answering Extractive 1.7.1 clue cmrc2018 1.7.2 clue drcd
1.7.3 mlqa mlqa.vi.vi
1.7.4 mlqa mlqa.zh.zh
1.7.5 xquad xquad.vi
1.7.6 xquad xquad.zh
1.8 Topic Classification 1.8.1 clue csl 1.8.2 clue tnews
1.9 Code Misc. 1.9.1 codeparrot/codeccomplex codeparrot-codecomposable 1.9.2 great_code 1.9.3 teven/code_docstring_corpus top_level
1.10 Word Sense Disambiguation 1.10.1 pasinit/xlwic xlwic_en_zh 1.10.2 pasinit/xlwic xlwic_fr_fr
1.11 Paraphrase Identification 1.11.1 paws-x en 1.11.2 paws-x es
1.12 Sentence Completion 1.12.1 xcopa vi 1.12.2 xcopa zh
1.13 Natural Language Inference 1.13.1 xml en 1.13.2 xml es 1.13.2.1 Human-translated prompts 1.13.2.2 Machine-translated prompts

## 1 Prompts

### Simplification

#### GEM/BiSECT en

Dataset from Kim et al. (2021). Used in training.

 



## Data Example

\begin{tabular}{l l} \hline Key & Value \\ \hline gem\_id & BiSECT\_en-train-1 \\ source & To view any of the video clips belo... \\ target & If you want to watch one of the vid... \\ references & If you want to watch one of the vid... \\ \hline \end{tabular}

### Prompts

Input Template:

Split and simplify the following sentence while retaining its full meaning:  (source)   Simplified version:

Target Template:

{target}

Input Template:

{target}

Input Template:

{resource}

The above sentence is very complicated. Please provide me a simplified synonymous version consisting of multiple sentences:

Target Template:

{target}

Input Template:

{resource}

```
{resource}

Input Template:

{resource}
``` {resource}

#### Gem/BiSECT es




\begin{tabular}{l l} \hline Key & Value \\ \hline gem\_id & BiSECT\_es-train-1 \\ source & Al final de la Santa Misa , mienetra... \\ target & Al finalizar la santa misa , mienert... \\ references & Al finalizar la santa misa , mientr... \\ \hline \end{tabular}

### Prompts

Input Template:

{(source)}. Estas frares es dificil de entender. Una version mas simple con significado equivalente es las siguiente:

Target Template:

{target}

Input Template:

Divids y simplifique la siguiente oracion conservando su significado completeo: {(source)} Version simplicada:

Target Template:

{target}

Input Template:

{(source)} La frase anterior es muy complicada. Por favor, proporcione una versión sinonima simplificada que consta de varias oraciones:

Target Template:

{(target)}

#### 1.1.3 Gem/BiSECT fr

**Data Example**

### Prompts

Input Template:

* ```
Listing 1: The first example of the program is a program that can be used to generate the program.

The program is called the program, and the program is called the program.
```
0:() {\(x_{i}=x_{j}+y_{j}\); \(x_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(y_{i}=x_{j}-y_{j}\); \(z=x_{j}-y_{j}\) 

Input Template:

{source}} === Write a summary of the text above in English : Target Template:

{target}}

Notes: xsum 'article_DOC_summary' template Input Template: Article in English: {resource} Summary in English:

Target Template:

{target}}

Notes: xsum 'DOC_how_would_you_rephrase_few_words' template Input Template:

{(source)} How would you rephrase that briefly in English?

Target Template:

{{target}}

Notes: xsum 'DOC_tidr' template Input Template: 

{(source)}
TL;DR in English: Target Template: {{target}}
**Notes:** xsum 'read_below_DOC_write_abstract' template Input Template: First, read the English article below. {{source}} Now, please write a short abstract for it in English. Target Template: {{target}}

Input Template: {{target}}

Input Template: {{target}}

Given the above abstract, write an English article for it. Target Template: {{source}}

Input Template: {{target}}

I'm interested in that, but I only have a few mins. Can you give me the first 500 characters of an article about that? Target Template: {{source[:500]}}

#### 1.2.2 Gem/wiki_lingua es
 



### Prompts

**Notes: xsum templates Input Template: {(source)} === Write a summary of the text above in Spanish: Target Template: {{target}} Notes: xsum templates Input Template: First, read the Spanish article below. {(source)} Now, please write a short abstract for it in Spanish. Target Template: {{target}} Notes: xsum templates Input Template: {{source}} TL;DR in Spanish: Target Template:

({target})

**Notes: xsum templates Input Template: Article in Spanish: {source} Summary in Spanish: Target Template: {{target}} Notes: xsum templates Input Template: {source} How would you rephrase that briefly in Spanish? Target Template: {{target}}

#### 1.2.3 GEM/xlsum Bengali

Dataset from Hasan et al. (2021). Used in training.

##### Data Example

\begin{tabular}{l l} \hline Key & Value \\ \hline gem\_id & xlsum\_bengali-train-2 \\ url & [https://www.bbc.com/bengali/news-50](https://www.bbc.com/bengali/news-50)... \\ title & program search so fast enough code.. \\ target & data file with partial data.. \\ references & data file with partial data.. \\ text & use a new file for different software.. \\ \hline \end{tabular}

##### Prompts

Input Template: 

* [{text[:500]}]

Input Template: {{{text[:7000]}}}

Target Template: {{target}}

Input Template: {{text[:8500]}}

Target Template: {{target}}

Input Template: {{text[:8500]}}

Target Template: {{target}}

Input Template: ...{{{text[3000:3500]}}

Target Template: {{text[5000:]}}

Input Template: {{title}}

Target Template: {{text[:7000]}}

Input Template: {{text[:7000]}}

Input Template: {{text[ text[:600]}}

Target Template: {{title}}

Input Template: {{text[ text[:7000]}}

Target Template: {{title}}

Input Template: 

* {title}

Target Template:  {text[:7000]}}

Input Template:  {title}\(\{\){text[:5000]}\(\}\)

Target Template:  {target}

Input Template:  {text[:1000]}\(\}\)

Target Template:  {text[1000:5000]}\(\}\)

#### gem/xlsum english

Data Example

\begin{tabular}{l l} \hline Key & Value \\ \hline gem_id & xlsum.engilish-train-2 \\ url & [https://www.bbc.com/news/uk-scotlan](https://www.bbc.com/news/uk-scotlan)... \\ title & hugetidal turbine installed at Ork... \\ target & The massive tidal turbine AK1000 ha.... \\ references & The massive tidal turbine AK1000 ha... \\ text & Atlantis Resources unveiled the mar... \\ \hline \end{tabular}

### Prompts

Input Template:

Doc to summarize: {(text[:8500])}nSummary in the same language as the doc:

Target Template:  {target}

Input Template: 

Content: {{text[:7000]}}\nThe previous content can be summarized as follows:

Target Template:  {{target}} Input Template:  {title}}\n({text[:5000]})\n|m|t|;dr: Target Template:  {{target}} Input Template:  {{text[x]}} \n|m|Oive me a good title for the article above. Target Template:  {title}} Input Template:  {{text[x]}} \n|m|Oive me a good title for the article above. Target Template:  {title}} Input Template:  Given the below title and summary of an article, generate a short article or the beginning of a long article to go along with them. Title: {title}\n|m|Oummy:  {{target}}\n|Article (Max S00 characters): Target Template:  {{text[x]}} Input Template:  {title: {title}}\nGiven the above title of an imaginary article, imagine the article.\n|Target Template:  {{text[x]}} Input Template:  {{text[x]}} Input Template: 

Title: {(title)}\nGiven the above title of an imaginary article, imagine the article.un

Target Template:

{{text[:7000]}}

Input Template:

{{text[:1000]}}... Continue the article for another 4000 characters max:

Target Template:

{{text[1000:5000]}}

Input Template:

...{{text[3000:3500]}}... Write the rest of the article:

Target Template:

{{text[5000:]}}

### Translation

#### 1.3.1 Helsinki-NLP/tatoeba_mt ben-eng

Dataset from Tiedemann (2020). Used in training.

##### Data Example

\begin{tabular}{l l} \hline Key & Value \\ \hline sourceLang & ben \\ targetlang & eng \\ sourceString & Tateoba*w@ ? \\ targetString & What does "Tatoeba" mean? \\ \hline \end{tabular}

##### Prompts

Input Template:

Translate the following text from English to Bengali {{targetString }}

 

Target Template: { sourceString }}

Input Template: Translate the following text from Bengali to English {( sourceString )}

Target Template: {{ targetString }}

#### Helsinki-NLP/tatoeba_mt eng-fra Data Example

\begin{tabular}{l l} \hline Key & Value \\ \hline sourceLang & eng \\ targetlang & fra \\ sourceString & Aah. Now I understand. \\ targetString & Ah! Maintenant, je comprends. \\ \hline \end{tabular}

### Prompts

Input Template:

Translate the following text from French to English {targetString }

Target Template: {{ sourceString }}

Input Template:

Translate the following text from English to French {sourceString }

Target Template:

{targetString }

 



#### 1.3.3 facebook/flores ben_Beng-eng_Latin

Dataset from NLLB (2022). Used in training.

##### Data Example

\begin{tabular}{l l} \hline Key & Value \\ \hline id & 2 \\ URL & [https://en.wikinews.org/wiki/Scient](https://en.wikinews.org/wiki/Scient)... \\ domain & wikinews \\ topic & health \\ has\_image & 0 \\ has\_hyperlink & 0 \\ sentence\_ben\_Beng & [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] \\ sentence\_eng\_Latn & Lead researchers say this may bring... \\ \hline \end{tabular}

### Prompts

Input Template:  {sentence_ben_Beng} Target Template:  {sentence_eng_Latn} Input Template:  A text in Bengali: {sentence_ben_Beng} Target Template:  {sentence_eng_Latn} Input Template:  {sentence_ben_Beng} Target Template:  {sentence_eng_Latn}

#### 1.3.4 facebook/flores ben_Beng-fra_Latin




### Prompts

Input Template: {sentence_ben_Beng} Target Template: {sentence_fra_Latn} Input Template: {sentence_ben_Beng} Target Template: {sentence_fra_Latn} Input Template: {sentence_ben_Beng} Target Template: {sentence_fra_Latn} Input Template: A text in Bengali: {sentence_ben_Beng} Target Template: {sentence_fra_Latn}

### Program Synthesis

#### 1.4.1 Muennighoff/mbpp sanitized

Dataset from Austin et al. (2021). Used in training.

##### Data Example

Prompts

Input Template:

```
``` {{ prompt }}HereisasolutioninPython: TargetTemplate: {( code )}
```

Note: the prompt does not correspond to the original task intended by the dataset authors. Input Template: {{ prompt }}ThiscanbesolvedinPythonwiththefollowingcode: TargetTemplate: {{ code }} ```

#### 1.4.2 codeparrot/appsall

Dataset from Hendrycks et al. (2021). Used in training.

##### Prompts

Input Template: 

Solve in Python: { question })

Target Template:

{solution }}

Input Template:

{ question }}

Can you solve the above problem using Python?

Target Template:

{ solution }}

Input Template:

I found an interesting problem on {{url}}: { question }}

I tried it in Python, but could not do it. Can you solve it?

Target Template:

{solution }}

#### 1.4.3 codeparrot/github-jupyter-text-code-pairs

Data Example

\begin{tabular}{l l} Key & Value \\ \hline markdown & Extract the dataset from the compre... \\ code & num_classes = 10 \\  & np.random.seed(133... \\ path & machine-learning/deep-learning/udac... \\ repo_name & pk-ai/training \\ license & mit \\ \end{tabular}

Prompts

Input Template: 

*{ markdown })" Please write code following the instructions in jupyter notebook style.

Target Template:

{( code }) Input Template:

 I am working on the file "{{ path }}". The first task is: {( markdown }) [ Can you write Python code for it?

Target Template:

{{ code }} Note: the prompt does not correspond to the original task intended by the dataset authors.

Input Template:

{{ markdown }}

Target Template:

{{ code }} Note: the prompt does not correspond to the original task intended by the dataset authors.

Input Template:

{{ code }} Given the above code, generate some markdown instructions for it.

Target Template:

{{ markdown }}

#### 1.4.4 codeparrot/xlcost-text-to-code C++-program-level

Dataset from Zhu et al. (2022). Used in training.

**Data Example 



#### Prompts

Input Template:

"{{ text }}" Solution in C++: Target Template:

{( code_clean )}

Input Template:

"{{ text }}" How can the above be solved in C++?

Target Template:

{( code_clean )}

#### codeparrot/xlcost-text-to-code C-program-level Data Example

\begin{tabular}{l l} \hline Key & Value \\ \hline text & Logarithm tricks for Competitive Pr... \\ code & **@include <stdio.b> NEW_LINE @includ...** \\ \hline \end{tabular}

### Prompts

Input Template:

"{{ text }}" Solution in C:

Target Template:

{( code_clean )}



Input Template: {{ text}} How can the above be solved in C7

Target Template: {{ code_clean }}

#### codeparrot/xlcost-text-to-code Csharp-program-level Data Example

\begin{tabular}{l l} Key & Value \\ \hline text & Check if a number can be represente... \\ code & using System ; class GFG { static b... \\ \end{tabular}

### Prompts

Input Template:

"{{ text }}" Solution in C#:

Target Template: {{ code_clean }}

Input Template:

"{{ text }}" How can the above be solved in C-Sharp?

Target Template:

{( code_clean })
 



\begin{tabular}{l l} \hline Key & Value \\ \hline text & Check if a number can be represente... \\ code & import java . io . * ; class GFG { ... \\ \hline \end{tabular}

### Prompts

Input Template:

"{{ text }}" Solution in Java:

Target Template:

{( code_clean )}

Input Template:

"{{ text }}"

Input Template:

"{{ text }}"

How can the above be solved in Java?

Target Template:

{( code_clean )}

#### 1.4.8 codeparrot/xlcost-text-to-code Javascript-program-level Data Example

\begin{tabular}{l l} \hline Key & Value \\ \hline text & Check if a number can be represente... \\ code & function sumOfTwoCubes ( n ) { var ... \\ \hline \end{tabular}

### Prompts

Input Template:

"{{ text }}" Solution in Javascript:

Target Template:

{( code_clean )}



Input Template:

"{ { text}}" How can the above be solved in JS?

Target Template:

{{ code_clean }}

#### codeparrot/xlcost-text-to-code PHP-program-level Data Example

\begin{tabular}{l l} Key & Value \\ \hline text & Rearrange the array to maximize the... \\ code & < ? php function solve ( $ a , $ n ... \\ \end{tabular}

### Prompts

Input Template:

"{{ text }}" Solution in pbp:

Target Template:

{{ code_clean }}

Input Template:

"{{ text }}" How can the above be solved in PHP?

Target Template:

{{ code_clean }}
 

* ```

### Prompts

Input Template:

"{{ text }}" Solution in Python:

Target Template:

{{ code_clean }}

 Input Template:

"{{ text }}" How can the above be solved in Python?

 Target Template:

"{{ text }}" How can the above be solved in Python?

Target Template:

{{ code_clean }}

#### 1.4.1 Neural\(\_\)code\(\_\)search evaluation\(\_\)dataset

Dataset from hug (2018). Used in training.

##### Data Example

\begin{tabular}{l l} \hline Key & Value \\ \hline stackoverflow\(\_\)id & 4616095 \\ question & How to get the build/version number... \\ question\(\_\)url & [https://stackoverflow.com/questions](https://stackoverflow.com/questions)... \\ question\(\_\)author & Fahad Ali Shaikh \\ question\(\_\)author\(\_\)url & [https://stackoverflow.com/users/565](https://stackoverflow.com/users/565)... \\ answer & try { \\   PackageInfo plinfo = this.ge... \\ answer\(\_\)url & [https://stackoverflow.com/a/6593822](https://stackoverflow.com/a/6593822) \\ answer\(\_\)author & plus- \\ answer\(\_\)author\(\_\)url & [https://stackoverflow.com/users/709](https://stackoverflow.com/users/709)... \\ examples & 4130029;3398176;2320640 \\ examples\(\_\)url & [https://github.com/altanizio/Concei](https://github.com/altanizio/Concei)... \\ \hline \end{tabular}
 

Note: the prompt does not correspond to the original task intended by the dataset authors.

Input Template:

Description: {( question )} Implementation:

Target Template:

{( answer )} Note: the prompt does not correspond to the original task intended by the dataset authors.

Input Template:

Given the following code: {(answer )} Describe it:

Target Template:

{{ question }}

#### 1.4.12 **T****even/code_contests Data Example**

\begin{tabular}{l l} Key & Value \\ name & 1575\_A. Another Sorting Problem \\ description & Andi and Budi were given an assignm... \\ source & 2 \\ difficulty & 7 \\ solution & \#include <bits/stdc++.h> \\  & using name... \\ language & CPP \\ \end{tabular}

### Prompts

Input Template:

{description} Target Template: 

Input Template:

Can you solve the below in {language}?  {description)} Target Template:

 {solution)} Input Template:

 {description)} The above is tricky. Write me a correct solution in {language}.

Target Template:

 {(solution)}

Input Template:

 Input Template:

 {description)} The above is tricky. Write me a correct solution in {language}.

Target Template:

 {description)} Solve the task in {language}.

Target Template:

 {(solution)}

Input Template:

 {description)} Using {[ language | lower]} can you solve the prior task?

Target Template:

 {solution)}

Input Template:

 {description)} {[solution[:5]]}

Target Template: 

({solution[5:]})

Input Template: {language} solution for "{description}": Target Template: {isolution}

### Coreference Resolution

#### 1.5.1 Muennighoff/xwinograd en

Dataset from Tikhonov and Ryabinin (2021). Used in evaluation.

##### Data Example

\begin{tabular}{l l} \hline Key & Value \\ \hline sentence & The city councilmen refused the dem... \\ option1 & The city councilmen \\ option2 & the demonstrators \\ answer & 2 \\ \hline \end{tabular}

##### Prompts

Input Template:

{(sentence)} Replace the _ in the above sentence with the correct option: - {option1} - {option2}

Target Template:

{% if answer == '1' %} {{option1}} {% else Y} {{ option2 }} (% endif %}

Answer Choices Template:

{{option1}} || {{option2}}

Input Template: 

Fill in the_inthebelowsentence: {(sentence)}

Choices: - ({ option1 }) - ({ option2 }) Answer:

Target Template:

{% if answer == '1' %} {{option1}} {% else X} {{ option2 }} {% endif %}

Answer Choices Template:

{{option1}} |||| {{option2}} Note: the prompt does not correspond to the original task intended by the dataset authors. Input Template:

The_inthesentencebelowrefersto{{option1}}. TrueorFalse? {{sentence}} Target Template:

{{answerchoices[answer|int - 1]}} Answer ChoicesTemplate:

True|||False

Input Template:

{{ sentence }} In the previous sentence,does_referto{{ option1 }} or {{ option2 }}??

Target Template:

{% if answer == '1' %} {{option1}} {% else X} {{ option2 }} {% endif %}

Answer Choices Template:

{{ option1 }} ||{{ option2 }}

 

{(sentence)} Whatdoesthe_intheabovesentencereferto?{{option1 }} or{{ option2 }}?

Target Template:

{% if answer=='1'%}{{option1}}{% elseX}{{ option2 }}{%endif X}

Answer Choices Template:

{{option1}}||||{{option2}}

Input Template:

In the sentence below, does the_stand for {{answer_choices[0]}} or {{answer_choices[1]}}? {{sentence}}

Target Template:

{{answer_choices[answer|int-1]}}

Answer Choices Template:

{{option1}}||||{{option2}}

#### Muennighoff/xwinogradFr

**Data Example**

\begin{tabular}{l l} \hline Key & Value \\ \hline sentence & La coupe n'entre pas dans la valise... \\ option1 & La coupe \\ option2 & la valise \\ answer & 2 \\ \hline \end{tabular}

### Prompts

Input Template:

{{ sentence }} Dans la phrase precedente,_fait-ilreference a{{ option1 }} ou {{ option2 }} ?

Target Template: 

(% if answer == '1' %) {{option1}} {% else X} {{ option2}} {% endif %}

Answer Choices Template:

 {{ option1}} |||| {{ option2}}

Input Template:

 Dans la phrase ci-dessous, le _ signifie-t-il {answerchoices[0]}} ou  {answerchoices[1]} ?  {sentence}

Target Template:

 {answerchoices[answer | int - i]}}

Answer Choices Template:

 {{option1}} |||| {{option2}}

Input Template:

 {{sentence}}  Remplacez le _ dans la phrase ci-dessus par l'option correcte :  - {{option1}} -  {{option2}}

Target Template:

 {% if answer == '1' %) {{option1}} {% else X} {{ option2}}} {% endif %}

Answer Choices Template:

 {{option1}} |||| {{option2}}

Input Template:

 {{sentence}}  A quoi le _ dans la phrase ci-dessus fait-il rdference ? {{ option1}} ou {  option2}} ?

Target Template:

 {% if answer == '1' %) {{option1}} {% else X} {{ option2}}} {% endif %}

 

Answer Choices Template:

{option1}} || {option2}}

Input Template:

Le_ dans la phrase ci-dessous fait reference a {(option1)}. Vrai ou faux? {(sentence)} Target Template:

{answer_choices[answer|int = l]} Answer Choices Template:

Vrai ||| Faux

### Question Answering Multiple Choice

#### c3

Dataset from Sun et al. (2020). Used in training.

##### Prompts

Input Template:



Input Template:

```
{\% for statementincontext%} {{statement}} (% endfor%) {\# z,\##[]\# x + {{answer}}} \# y {\# z,\##[]\# x + {{answer}}}
```

Target Template:

```
{{ question}}
```

Input Template:

```
{{\% forstatementincontext%} {{statement}} {{\% endfor%} Giventhedialogue/passageabove,usethefollowingoptionstoanswerthequestion'<{question}".Options: -{{answer_choices|join('\n-')}} }
```

Target Template:

```
{{ answer}}
```

Answer ChoicesTemplate:

```
{{choice|join(''||")}}
```

Input Template:

```
{\% forstatementincontext%} {{statement}} {\% endfor%} {\# z,\#[]\# x + {{question}}". ++ -{{answer_choices|join(' -')}} }
```

Target Template:

``` {{answer}} AnswerChoicesTemplate: 

Input Template:

 Passage: {% for statement in context %}  {{ statement }}  {% endfor %} Question: "{(question)}" Answer choices: {answer_choices[:-1] | join(', ') }}, or {{ answer_choices[-1] }}? Target Template:

 {{ answer }}

Answer Choices Template:

 {{ choice | join(" || ") }} Note: the prompt does not correspond to the original task intended by the dataset authors. Input Template:

Passage: {% for statement in context %}  {{ statement }}  {% endfor %} What kind of question would elicit an answer response of {{ answer }}?

Target Template:

 {{ question }}

Input Template:

 {% for statement in context %}  {{ statement }}  {% endfor %}  {\# if : {(answer_choices[:-1] | join(', ') }) } } } } } } } } } } } } } } } } } } } } } } } } } } } } ```

Target Template:

 {{{ answer }}

 

(% for statement in context %)

{{ statement}}  (% endfor %) Given the dialogue / passage above, what is the answer for the question "{question}" Answer choices: {answer_choices[:-1] | join(', ') }}, or {{ answer_choices[-1] }}? Target Template:

{{ answer}}

Answer Choices Template:

{{ choice|join('||')}} Input Template:





### Question Answering Extractive

#### 1.7.1 clue cmrc2018

Dataset from Cui et al. (2018). Used in training.

##### Data Example

\begin{tabular}{l l} \hline Key & Value \\ \hline id & TRAIN 186_QUERY.1 \\ context & [] (),

In an exam, you are asked ({ question }), and you are tasked to find the answer from the following passage. {( context }) What's the answer?

Target Template:  {{ answers['text'][0] }} Input Template:  {{ context }} The answer to {{ question }} is in the passage above. What is it?

Target Template:  {{ answers['text'][0] }} Input Template:   Answer the question using the given context. Question: {{ question }} Context: {{ context }} Answer:

Target Template:  {{ answers['text'][0] }} Input Template:   Q: {{ question }} Can you write some context to answer the question?

Target Template:  {{ context }} Input Template:  {{ context[:answers["answer_start"][0]-5] }}... How would you continue the prior text to answer "{{ question }=? 

{ ( context[answers["answer_start"][0]-5:] )}

Input Template:  {context[:ansvers["answer_start"][0]-5] }... %%%%%%%%%%%%%%%%* 



## Data Example

### Prompts

Input Template: {{context }} {{ question }} #{\#} #{\#} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} #} & } }} \\ }} \#} \# \#}\#}\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\##\#\##\##\

({ context })

Input Template:  {context } The answer to { question } is in the passage above. What is it?

Target Template:  { answers['text'][0] } Input Template:  { question }, %\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\##\#\##\# 

Target Template:

{( answers['text'][0] )} Note: the prompt does not correspond to the original task intended by the dataset authors. Input Template:

Given this context "{{context }}", generate a question that would return the answer of "{answ['text'][0] }". Target Template:

{{question }} Input Template:

In an exam, you are asked {{ question }}, and you are tasked to find the answer from the following passage. {{context }} What's the answer?

Target Template:

{{answer['text'][0] }}

Input Template:

In an exam, you are asked {{ question }}, and you are tasked to find a passage answering the question. Write such a passage:

Target Template:

{{context }}

#### mlqa mlqa.vi.vi

Dataset from Lewis et al. (2019). Used in training.

**Data Example 



### Prompts

Input Template: {{context[:answers.answer_start[0]-5]}}...Tiep tuc d tr\(\bar{\text{n}}\), sao cho n\(\bar{\text{o}}\) tr\(\bar{\text{l}}\) lsi "{question}": Target Template: {{context[answers.answer_start[0]-5:]}}

Input Template: {{context[answers.answer_start[0]-5:]}}

Input Template: {{context}} V6i sw tham chiu d\(\bar{\text{n}}\)ng@ c\(\bar{\text{n}}\)hn tre\(\bar{\text{n}}\), {{question}}

Target Template: {{answers.text[0]}}

Input Template: {{context} H: {Question} D$ c\(\bar{\text{p}}\)d\(\bar{\text{e}}\)d\(\bar{\text{e}}\)d\(\bar{\text{e}}\)on v\(\bar{\text{n}}\)tr\(\bar{\text{n}}\), caut tra loi d\(\bar{\text{m}}\)g cho c\(\bar{\text{a}}\)hbi d\(\bar{\text{s}}\)cho trong ng@ ng@ c\(\bar{\text{u}}\)da doan v\(\bar{\text{n}}\)la Target Template: {{answers["text"][0]}}

Input Template: {{Question} Ngt c\(\bar{\text{c}}\)t\(\bar{\text{c}



{(answers.text[0])}

Input Template:

Tham kho doan vsn duoi day va sau do tra losi cau hbi saus 46 bang ngon ngu tudong tu nht door wn:

Doan: {context}

Cau hbi: {question}

Target Template:

{{answers["text'][0]}}

Input Template:

Toi da tim thay mot van ban tra la1 *{question}= bang {(answers.text[0])}. No bdt dAu bhang *{{ context[:10]}}". Ban co thde tiep tuc na khlong?

Target Template:

{{ context[10:]}}

Input Template:

Doc doan vsn san u sau do tra losi cau hbi tip theo bang cach trich m@t phAn dung from ocan vsn:

{context}} {question}

Target Template:

{{answers.text[0]}}

Input Template:

D: {context}

H: {question}

A:

Target Template: 

{answers["text"][0]})

#### 1.7.4 mlqa mlqa.zh.zh

##### Data Example

\begin{tabular}{p{285.3pt} p{69.0pt}} \hline Key & Value \\ \hline context question answers id & \\ \hline \end{tabular}

### Prompts

Input Template:

```
{{context}} {question}
```

Target Template:

```
{answers.text[0]}
```

Input Template:

```
{{context[:answers.answer_start[0]-5]}...####,(%=%+"(Question)": TargetTemplate:

Target Template:
``` {{context[answers.answer_start[0]-5:]}} ```
Input Template:
``` D: {{context}} {question} ```
Target Template:
``` {{question}} ``` 

Input Template:

``` {answers.text[0]}\)"#"(!answer.text[0]})"#"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%"%""%"%"%"%"%""%"%"%"%"%""%"%"%"%""%"%"%""%"%"%""%"%""%"%"%""%"%""%"%"%""%"%""%"%""%"%""%"%""%"%""%"%""%"%""%"%""%"%""%"%""%"%""%"%""%""%"%""%""%"%""%""%"%""%""%"%""%""%""%""%""%""%""%""%"%" 

({context})
[#: {question})

Target Template:

{answers["text"][0]}}

#### xquad Xquad.vi

Dataset from Artetxe et al. (2019). Used in training.

##### Data Example

\begin{tabular}{l l} Key & Value \\ id & 56beb4343aeenaa14008c925c \\ context & D@i thu c aa Panthers chi thua 308 d... \\ question & Jared Allen cc bo haibu l an vat mg... \\ answers & {'text': ['136'], 'answer_start': [... \\ \end{tabular}

##### Prompts

Input Template:

{context}} V@i sw than chieu daen ngQ canh tren, {{question}} Target Template:

{answers.text[0]}}

Input Template:

Dun ra ca u tra la loi {{ansvers.text[0]}} cho {question}}, hhy vit mqt van bah giki \\ thi chai u nay. Cau tra la loi phai bdt dhu s so ky tv {{amswers.answer_start[0]}}. \\ Van ban: \\ Target Template:

{context}} Input Template: 

{(question)} R6 rang la {answers.text[0]}}. Ban c@ th& cung cdp cho t@i m$t s@ b@i cahn?

Target Template:  {(context)}

Input Template:  {{context}} H: {Question} D@ cbp den doan vAn trn, cau tra l@i chinh xac cho cau hbi dusc data ra la Target Template:  {answers["text"][0]}} Input Template:  {{context} H: {question} A: Target Template:  {answers["text"][0]}} Input Template:  {answers["text"][0]}} Input Template:  {answers["text"][0]}} Input Template:  {answers["text"][0]}} Output Template:  {answers["text"][0]}

 

Tham khao doan v\n dudi d\y va trl lsi cau hbi sau:

Doan: {context}

Cau hbi: {question}

Target Template:

{answers["text"][0]}}

Input Template:

{context}

Tu doan van tren, mtt cau hbi hop1 yoi *{answers["text"][0]}}= nhu cau tra lbi s$ la:

Target Template:

{{question}}

Input Template:

{context}

Tao cau hbi tt doan v\n t\en:

Target Template:

{question}

#### Xquad xquad.zh

##### Data Example

\begin{tabular}{p{284.5pt} p{396.7pt}} \hline Key & Value \\ \hline id context question answers & 56beb4343aeaa14008c925c \\ \end{tabular}

### Prompts

Input Template: 

[{context}] (question)

Target Template:

{{answers.text[0]}}}

Input Template:

{{context}}

[]: {{question}}

### Target Template:

{{answers["text"][0]}}

Input Template:

[{answers["text"][0]}]

Input Template:

[{answers["text"][0]}]

Input Template:

[{answers["text"][0]}]

Input Template:

[{context}]

[], ->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->+->\]+->->+->->+->->+->+->->+->+->->+->+->->+->+->->+->+->->+->+->->+->+->->+->+->->+->+->->+->+->->+->+->->+->+->->+->+->->+->+->->+->+->->+->+->->+->+->->+->+->->+->+->->+->+->->+->->+->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->-<+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->->+->+->->+->->+->+->->+->+->->+->+->->+->+->->+->+->->+->+->->+->+->->+->+->->+->+->->+->+->->+->+->->+->+->->+->+->+->->+->+->->+->+->+->->+->+->+->->+->+->+->->+->+->+->->+->+->+->+->->+->+->+->+->+->->+->+->+->+->->+->+->+->+->+->->+->+->+->+->+->+->+->+-> 

({context})

Target Template: {{question}}

Input Template: {{context}}

## 12.4 X, {Question}

Target Template: {{answers.text[0]}}

Input Template: {{context}}

Input Template: {{context}}

[]: {{question}}

## 13.1 Target Template: {{answer["text"][0]}}

### Topic Classification

#### 13.1 clue csl

Data Example

\begin{tabular}{l l} \hline Key & Value \\ \hline idx & **1** \\ corpus\_id & **2565** \\ abst & * 

Input Template:

After John wrote the abstract "{abst}", he wrote these keywords "{{ keyword | join('.')}". Do you think his choice of keywords was correct? Answer { answer_choices[i]} or {{answer_choices[0]}}. Target Template:

{{ answer_choices[label]}}

Answer Choices Template:

no || yes

Input Template:

Do these keywords "{{ keyword | join(',')}" represent key concepts in the abstract "{{ abst }}"?

Target Template:

{{ answer_choices[label]}}

Answer Choices Template:

no || yes

Note: the prompt does not correspond to the original task intended by the dataset authors. Input Template:

Given the abstract {{abst}}, list out {{ keyword | length }} keywords for it.

Target Template:

{% if label == 1 %} {{ keyword | join(',')}} {% endif %}

Input Template:

-- {\(\#\#\) ## ## ## ## ## ## ### ###  

{ (answer_choices[label])} Answer Choices Template:

Input Template:

Target Template:

```
{{abst},%|#|+({ keyword|length})}%%%%.
```

Target Template:

```
{%iflabel==1%} {{keyword|join(',')}}"\%%%%: {$endif}
```

Target Template:

``` {%iflabel==1%} {{keyword|join(',')}}"\% 

*{ keyword | join(', ')}" #{ abst }}" #{

Target Template:

{ answer_choices[label]}}

Answer Choices Template:

Input Template:

A scholar used "{{ keyword | join(', ') }}" as search terms. Do you think the search engine would return the abstract "{abst}"? Answer {{ answer_choices[1] }} or {{ answer_choices[0] }}.

Target Template:

{{ answer_choices[label]}}

Answer Choices Template:

no || yes

Input Template:

Write an abstract about "{{ keyword | join(', ') }}";

Target Template:

{% if label == 1 %} {{abst}} {% endif }

Answer Choices Template:

no || yes

[MISSING_PAGE_POST]

**The
 



[MISSING_PAGE_POST]

oin('\n-')})
* ({ answer_choices| join('\n-')})
* [#{ answer_choices| join('\n-')})
* ({ answer_choices| join('\n-')})
* [#{ answer_choices| join('\n-')})
* ({ answer_choices| join('\n-')})
* [#{ answer_choices| join('\n-')})
* ({ answer_choices| join('\n-')})
* [#{ answer_choices| join('\n-')})
* ({ answer_choices| join('\n-')})
* [#{ answer_choices| join('\n-')})
* ({ answer_choices| join('\n-')})
* [#{ answer_choices| join('\n-')})
* ({ answer_choices| join('\n-')})
* [#{ answer_choices| join('\n-')})
* ({ answer_choices| join('\n-')})
 

Answer Choices Template:

story ||| culture || entertainment ||| sports ||| finance ||| real estate ||| car  ||| education ||| tech ||| military ||| travel ||| world news ||| stock ||| agriculture ||| game

Input Template:

[] "{{sentence}}" \(\#\){{answer_choices[0] | capitalize }},  {{ answer_choices[1:-1] | join(', ')} \(\#\){{answer_choices[-1] }}?

Target Template:

{{ answer_choices[label] }}

Answer Choices Template:

[] "{{answer_choices[i-1] | join(', ')} \(\#\){{answer_choices[-1] }}, \(\#\){{answer 

Answer Choices Template:

story ||| culture || entertainment ||| sports ||| finance ||| real estate ||| car
1|| education ||| tech ||| military ||| travel ||| world news ||| stock ||| agriculture ||| game

### Code Misc.

#### codeparrot/codecomposable codeparrot-codeccomplex

##### Data Example

\begin{tabular}{l l} \hline Key & Value \\ \hline src & import java.util.Scanner; \\  & public ... \\ complexity & linear \\ problem & 1197_B. Pillars \\ from & CODEFORCES \\ \hline \end{tabular}

##### Prompts

Input Template:

{{ code }} What is the time complexity of the previous code?

Target Template:

{{ complexity }}

Note: the prompt does not correspond to the original task intended by the dataset authors.

Input Template:

Identify the time complexity of the following codes as constant, linear, quadratic, cubic, log(n), nlog(n) or NP-hard. {{ code }} Complexity:

Target Template:

{{ complexity }}

Note: the prompt does not correspond to the original task intended by the dataset authors.

Input Template:

{{ code }} Which one is the correct time complexity of the code snippet: constant, linear, quadratic, cubic, log(n), nlog(n) or NP-hard? 

Target Template: {complexity }}

#### 1.9.2 great_code

Dataset from Hellendoorn et al. (2020). Used in training.

##### Data Example

\begin{tabular}{l l} \hline Key & Value \\ id source\_tokens & **?NEULINE#;def test\_get\_params(;self... \\ has\_bug & True \\ error\_location & 76 \\ repair\_candidates & 2;76;4;11;18;22;30;40;48;58;66;80;8... \\ bug\_kind & 1 \\ bug\_kind\_name & VARIABLE\_MISUSE \\ repair\_targets & 4;11;18;22;30;40;48;58;66;80;88;103... \\ edges & [!~before\_index': 1, 'after\_index'.... \\ provenances & {‘datasetProvenance': {'datasetName...} \\ \hline \end{tabular}

##### Prompts

Note: the prompt does not correspond to the original task intended by the dataset authors.

Input Template:

``` {%setmask='def<FUNCNAME>('%sinthedent=''.%s"h".%s"inhes='namespace(indent_size=0,result=[],masked=false,target='')%k) {%fortokeninsource_tokens%} {%ifns_maskedisfalsefandtoken.startswith('def')%}  {%setns.target=token.split('def')[1][::-1]%k}  {%settoken=mask%} {%setns.masked=true%} {%endif%} {%id token=='#INDENT*%s"h".%s"inhentation_size=ms.indent_size+1%s"h".

 

Target Template:

{ (ns.target )} Note: the prompt does not correspond to the original task intended by the dataset authors.

Input Template:

```
{%setresult=""%} {%setindent=".%} {%setns=namespace(intend_size=0,line_number=0,buggy_line=0,bug_location=0,buq_len=0,result=[],result_lines=[])%} {%setfixed_token=source_tokens[repair_targets[0]]%} {%setbigsy_line_content=""%} {%setfixed_buggy_line_content=""%}
``` {%ifhas_bugand(repair_targets|length>0)%} {%fortoinosenisource_tokens%} {%ifloop.index0==error_location%} {%setns.bsuggy_line=ns.line_number%} {%setns.bug_location=(ns.result|join("")|length)%} {%setns.bug_len=(token|length)%} {%endif%} {%iftokens=='#INDENT*%} {%setns.indent_size=ns.indent_size+1%} {%setns.result=ms.result+[indentation*ns.indent_size]%} {%eliftoken=='#NEullNINE*%} {%setns.result_lines=ns.result_lines+[ns.result|join("")]%}  {%setns.result=[]%} {%setns.line_number=ns.line_number+1%} {%eliftoken=='#UNINDEN*''%} {%setns. indent_size=ns.ident_size-1%} {%else%} {%ifnotloop.firstandloop.previtem=='#NEULNINE#'%} {%setns.result=ns.result+[indent*ns.indent_size]%} {%endif%} {%setns.result=ns.result+[token|replace('\n','\n'),"~"] %} {%endif%} {%endfor%} {%setns.result_lines=ns.result_lines+[ns.result|join("")]%} {%setresult=ns.result_lines|join("")} {%setns.result_lines|joining("")} {%result|replace(".",".")|replace(",",",")|replace("(",")|"))|replace("[","["]|replace("|","]^))} {%setbutggy_line_content=ns.result_lines[ns.busuggy_line]|trim|replace(".",".")|replace(",".")|replace(",".")|replace("["]|replace("|"),")")| replace("["]|replace("|"),"]")} {%setfixed_buggy_line_content=ns.result_lines[ns.buggy_line][:ns.bug_location]+fixed_token+ ns.result_lines[ns.bugy_linel]ins.bug_location+ns.bug_len:]|trim|replace(".",".")|replace(",",",")|replace(",",")")|replace("["]|replace("],",")")} ```

Fixthebuggyline:{{buggy_line_content}

 

({fixed_buggy_line_content}) (%endif%)

Input Template:

 [% set mask = '<MASK>' %\]  (% set indent = '' ' %\]  (% set ns= namespace(indent_size=0, result=[]) %\]

 [% if has_bug %]  {% for token in source_tokens %\]  {% if loop.indexO == error_location X}  (% set token = mask %\]  {% endif}  {% if token== '#INDENT#' %  {% set ns.inendent_size = ns.inendent_size + 1 %} % set ns.result == ns.result + [indent * ns.inendent_size] %\]  {% elif token == '#NEMININE" %}  (% set ns.result == ns.result + ["\n"] %\]  {% elif token == '#UNINDIENT#' %  {% set ns.inendent_size = ns.inendent_size - 1 %}  {% else %}  (% if not loop.first and loop.previtem == '#NEMININE#' %\]  (% set ns.result = ms.result + [indent * ns.inendent_size] %)  {% endif %}  {% set ns.result = ns.result + [token | replace('\n', '\n'), " "]  %)  {% endif %}  {% endfor %}  {ms.result | join("") | replace(" . ", ".") | replace(" , ", ", ") | replace("( ", "(") | replace(" )", "]"))}

 Given the code above, what is a proper replacement for {mask}?

Target Template:

 {(source_tokens[repair_targets[0]])}  {% endif %}

Note: the prompt does not correspond to the original task intended by the dataset authors.

Input Template:

 [% set indent = ' ' %]  {% sets n = namespace(indent_size=0, result=[]) %}  (% for token in source_tokens %]  {% it tokens = '#INDENT#' %  {% set ns.inendent_size = ns.inendent_size + 1 %}  {% set ns.result = ns.result + [indent * ns.inendent_size] %}  {% elif token == '#NEMININE#' %  {% set ns.result = ns.result + ["\n"] %}  {% elif token == '#UNDENT#' %  {% set ns.inendent_size = ns.inendent_size - 1 %}  {% else %}  {% if not loop.first and loop.previtem == '#NEMININE#' %  {% set ns.result = ns.result + [indent * ns.inendent_size] %}

 

(%endif%)  {%setns.result=ns.result+[token|replace('\n','\n'),"="]%}  {%endif%} (%endfor%)  {{Ins.result|join("")|replace(".",".")|replace(",",",")| replace("(",")|replace(")",")")}
*Isthereabuginthecodeabove? TargetTemplate:

{{True:"Yes",False:"No^}[has_bug]}

AnswerChoicesTemplate:

Yes||No Note: the prompt does not correspond to the original task intended by the dataset authors.

Input Template:

{%setmask='<MASK>'%}  {%setindent=''%}  {%setns=namespace(indent_size=0,result=[])%}

{%iflas_bug%}  {%foretokeninsourcetokens%}  {%ifloop.index0==error_location%}  {%settoken=mask%}  {%endif%}  {%ifokemem='#INDENT#'%}  {%setns.indent_size=ns.intend_size+1%}  {%setns.result=ns.result+[indent*ns.indent_size]%}  {%eliftoken==''#NELINES#'%}  {%setns.result=ns.result+["\n"]%}  {%filtertoken=="#UNINDIENT#'%}  {%setns.indent_size=ns.indent_size-1%}  {%else%}  {%ifnotloop.firstandloop.previterm=='#NELINES#'%}  {%setns.result=ms.result+[indent*ns.indent_size]%}  {%endif%}  {%endfor%}  {%endfor%}  {{Ins.result|join("")|replace(".",".")|replace(",",",")| replace("(","(")|replace(")",")")|replace("[","["]|replace("]","]")})}

 Giventhecodeabove,whatisaproperrolacementfor{{mask}}?Chooseamong:{answerchoices|join(",")})

TargetTemplate:

{{source_tokens[repair_targets[0]]}}  {%endif%}

 

(% if has_bug %) {% set nss = namespace(choices=[]) %} {% for i in repair_candidates %} {% set nss.choices = nss.choices + [source_tokens[(i | int)]] %} {% endfor %} {nss.choices | unique | join("|| *)}} {%endif %}

#### 1.9.3 **teven/code_docstring_corpus top_level Data Example**

\begin{tabular}{l l} \hline Key & Value \\ \hline desc & ’XXX22: This has to be present' \\ decl & def XXXI1(): \\ bodies & pass \\ \hline \end{tabular}

### Prompts

Input Template:

Complete the below  {decl}  '''{desc | replace('  '', '  ')}""

Target Template:

{(bodies)}

Input Template:

 I wrote the below code  {boolties} What's a good function header?

Target Template:

{{decl}}

Input Template:

{{decl}}

 

Target Template:

"""*{desc | replace('  ',  ') | replace("", '')}"""

{bodies}}

### Word Sense Disambiguation

#### 1.10.1 pasinit/xlikic xliwc_en_zh

Dataset from Raganato et al. (2020). Used in training.

##### Data Example

\begin{tabular}{l l} \hline Key & Value \\ \hline id & EN\_1 \\ context\_1 & We like to summer in the Mediterranean... \\ context\_2 & We summered in Kashmir. \\ target\_word & summer \\ pos & V \\ target\_word\_location\_1 & {'char\_start': 11, 'char\_end': 17} \\ target\_word\_location\_2 & {'char\_start': 3, 'char\_end': 11} \\ language & EN \\ label & 1 \\ \hline \end{tabular}

##### Prompts

Input Template:

```
#1%:{context_1}
#2:{context_2}
#3%:"{target_word}"\#4%:"{target_word}"\#5%:"\#6%:"\#7%:"\#8%:"\#9%:"\#10%:"\#11%:"\#12%:"\#13%:"\#14%:"\#15%:"\#16%:"\#17%:"\#18%:"\#19%:"\#19%:"\#11%:"\#12%:"\#13%:"\#14%:"\#15%:"\#16%:"\#17%:"\#18%:"\#19%:"\#



* # # +(target_word)>" {{context_1}}

Target Template:

{% if label != -1%}

{answer_choices[label]}}

{% endif %} Answer Choices Template:

###### Abstract

In this paper, we present a new method for constructing the class of objects in a given set. We show that the class of objects is not necessarily unique and can be used to construct an object from a set. This approach allows us to use the class of objects with respect to the class of objects. The class of objects is defined as follows:

* # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # ## # # # # # # # # 

Input Template:

{(context_1)} {context_2}} Target Template:

{% if label != -1\%} {answer_choices[label]} {% endif %} Answer Choices Template:

```
1111&&#> InputTemplate:

{{context_1}} {{context_2}} [],H]: "{target_word}" {% if label != -1\%} {{answer_choices[label]}} {%endif %} Answer Choices Template:
```
1111&&#> InputTemplate:

```
113478690 

Input Template:

"({target_word})" {{context_1}}  {context_2}}

Target Template:

{% if label != -1%}  {answer_choices[label]}}  {% endif %}

Answer Choices Template:

\(\times\) || #}

Input Template:

\(\ll\) A: {{context_1}}  \(\ll\) B: {{context_2}}  \(``\){{target_word}}" \(\ll\) A \(\ll\) B \(\ll\)A \(\ll\) X, \(\ll\) A \(\ll\) A ?

Target Template:

{% if label != -1%}  {answer_choices[label]}}  {% endif %}

Answer Choices Template:

\(\ll\) || &&}

Input Template:

{{context_1}}  {Context_2}  []/R: "{{target_word}}" \(\ll\) A \(\ll 



#### 1.10.2 pasinit/xlwic xlwic_fr_gr

**Data Example**

\begin{tabular}{l l} \hline Key & Value \\ \hline id & FR\_1 \\ context\_1 & L'e$clarice est g~en$ralement une co... \\ context\_2 & Am$liorations utiles. \\ target\_word & am@lioration \\ pos & N \\ target\_word\_location\_1 & \{'char\_start': 41, 'char\_end': 53\} \\ target\_word\_location\_2 & \{'char\_start': 0, 'char\_end': 13\} \\ language & FR \\ label & 1 \\ \hline \end{tabular}

### Prompts

Input Template:

{{context_1}} {(context_2)} Sens similaire de {{target_word}} ?

Target Template:

{% if label != -1%} {(answer\_choices[label])} {% endif %}

Answer Choices Template:

Non ||Uui

Input Template:

Devoirs

Decidezsi le mot "{{target_word}}" est utilis$avec le m~emes sans les deux phrases suivantes. R~epondez par oui ou non. {{context_1}} {{context_2}}

Target Template:

{% if label != -1%}  {answer_choices[label])}  (%endif %}

Answer Choices Template: Non ||I| Oui Input Template: Le mot "{target_word}" a-t-il le m\(\widetilde{\text{ names sans ces deux phrases ? Oui Non? {context_1}} {{context_2}} Target Template: {% if label != -1%}  {answer_choices[label]}}  (%endif %}

Answer Choices Template: Non ||I| Oui Input Template: {{context_1}}  {{context_2}} Question : Le mot '{{target_word}}' est-il utilise dans le m\(\widetilde{\text{ names sans less deux phrases ci-dessus ? Oui Non? Target Template: {% if label != -1%}  {answer_choices[label]}}  (%endif %}

Answer Choices Template: Non ||I| Oui Input Template: Le mot "{{target_word}}" a plusieurs significications. A-t-il le m\(\widetilde{\text{ names sans dans les}}\) phrases i et 2 ? Oui ou non? Phrase 1 : {{context_1}} Phrase 2 : {{context_2}}

 

Target Template:

{% if label != -1%}  {answer_choices[label]}} (% endif %) Answer Choices Template: Non||Uui Input Template: Phrase 1 : {{context_i}} Phrase 2 : {{context_2}} Determinez si le mot "{target_word}}" est utilise dans le memes sans des deux phrases. Oui ou non? Target Template:

 {% if label != -1%}  {answer_choices[label]}} (% endif %) Answer Choices Template: Non||Oui Input Template: Le mot "{target_word}}" a-t-il leme sens dans ces deux phrases ? {{context_1}} (% context_2) Target Template:

 {% if label != -1%}  {answer_choices[label]}} (% endif %) Answer Choices Template: Non||Uui 

Phrase A : {{context_1}} Phrase B : {{context_2}}

"{target_word}" a une signification similaire dans les phrases A et B. Vrai ou faux ?

Target Template:

{% if label != -1%}  {answer_choices[label]}}  (% endif %) Answer Choices Template:

Faux ||Vrai

Input Template:

Determinez si le mot '{target_word}' est utilise de la meme maniere dans les deux phrases ci-dessous. {{context_1}}  {context_2}

Target Template:

{% if label != -1%}  {answer_choices[label]}}  (% endif %) Answer Choices Template:

Non |||Oui

Input Template:

{{context_1}}  ({context_2})   Question : Le mot '{{target_word}}' est-il utilise dans le memes sans dans les deux phrases ci-dessus ?

Target Template:

{% if label != -1%}  {answer_choices[label]}}  (% endif %) Answer Choices Template: 



### Paraphrase Identification

#### 1.11.1 paws-x en

Dataset from Yang et al. (2019). Used in training.

##### Data Example

\begin{tabular}{l l} \hline Key & Value \\ \hline id & 2 \\ sentence1 & The NBA season of 1975 -- 76 was th... \\ sentence2 & The 1975 -- 76 season of the Nation... \\ label & 1 \\ \hline \end{tabular}

##### Prompts

**Notes:** Generalized prompt format, task_description-input.

Input Template:

Determine if the following two sentences paraphrase each other or not. Sent 1: {sentence1}

Sent 2: {sentence2}

Target Template:

{answer choices[label]}}

Answer Choices Template:

No || Yes

##### Notes:** Natural question.

Input Template:

Setence 1: {sentence1}

Sentence 2: {sentence2}

Question: Do Sentence 1 and Sentence 2 express the same meaning? Yes or No?

Target Template:

{answerchoices[label]}}

Answer Choices Template:

No || Yes

**Notes:** Generalized prompt format, context-question without any label.

Input Template:

{(sentence1)} Is that a paraphrase of the following sentence?

{(sentence2)}?

Target Template:

{(answer_choices[label])}

Answer Choices Template:

No || Yes

**Notes:** Natural Question without label.

Input Template:

Sentence 1: {{sentenced}} Sentence 2: {{sentence2}} Question: Can we rewrite Sentence 1 to Sentence 2?

Target Template:

{(answer_choices[label])}

Answer Choices Template:

No || Yes

**Notes:** Generalized prompt format, context-question.

Input Template:

{(sentence1)} Is that a paraphrase of the following sentence?

{(sentence2)}? Yes or No.

Target Template:

{(answer_choices[label])}

Answer Choices Template: 

No || Yes

**Notes:** Concatenation of sentence 1 and sentence 2.

Input Template:

 Sentence 1: {sentence]} Sentence 2: {sentence2} Question: Does Sentence 1 paraphrase Sentence 2? Yes or No?

Target Template:

{answer_choices[label]}} Answer Choices Template:

No || Yes

Note: the prompt does not correspond to the original task intended by the dataset authors.

**Notes:** Create a generative paraphrase task.

 Input Template:

 {% if label == 1 %} Paraphrase the sentence: {sentence1}} Target Template:

 {(sentence2)}  {% endif %}

**Notes:** Concatenation of sentence 1 and sentence 2 without any label.

Input Template:

 Sentence 1: {sentence1} Sentence 2: {sentence2} Question: Does Sentence 1 paraphrase Sentence 2?

Target Template:

 {answer_choices[label]}} Answer Choices Template:

No || Yes 



**Notes:** Natural question without label.

Input Template:

Sentence 1: {sentence]} Sentence 2: {sentence} Question: Do Sentence 1 and Sentence 2 express the same meaning? Target Template:

{answer_choices[label]}} Answer Choices Template:

No || Yes Prompt from Brown et al. (2020) **Notes:** ANLI prompt format from Table G7 in the GPT3 paper Brown et al. (2020) Input Template:

{sentence1} Question: {sentence2} True or False? Target Template:

{answer_choices[label]}} Answer Choices Template:

False || True

**Notes:** Natural Question.

Input Template:

Sentence 1: {sentence1} Sentence 2: {sentence2} Question: Can we rewrite Sentence 1 to Sentence 2? Yes or No? Target Template:

{answer_choices[label]}} Answer Choices Template:

No || Yes

Prompt from Brown et al. (2020)**Notes:** ANLI prompt format from Table G7 in the GPT3 paper Brown et al. (2020). Additionally added task information without any label.

Input Template:

{sentence1} Question: {(sentence2}) Paraphrase or not?

Target Template:

{answer_choices[label]}}

Answer Choices Template:

No || Yes

#### paws-x es

##### Data Example

\begin{tabular}{l l} \hline Key & Value \\ \hline id & 2 \\ sentence1 & La temporada de la NBA de 1975: 76 ... \\ sentence2 & La temporada 1975 - 76 de la Asocia... \\ label & 1 \\ \hline \end{tabular}

### Prompts

Input Template:

Oracion 1: {(sentence1}) Oracion 2: {(sentence2}) Pregunta: ?La oracion 1 parafrasea la oracleion 2? ?Si o no?

Target Template:

{answer_choices[label]}}

Answer Choices Template:

No || Si

Input Template:

{isentence1} Pregunta: {(sentence2}) ?Parafrasear o no? 

Target Template: {answer_choices[label]}}

Answer Choices Template: No ||Si Input Template: {{sentence1}} _Esuna parafrasis de la siguiente oracion? {{sentence2}}?? Si ono. Target Template: {{answer_choices[label]}} Answer Choices Template: No ||Si Input Template: Oracion 1: {sentence2} Oracion 2: {sentence2} Pregunta: _iLa Oracion 1 y la Oracion 2 expressan el mismo significado? Target Template: {{answer_choices[label]}} Answer Choices Template: No ||Si Input Template: {% if label==1% Parafrasea la oracion: {sentence1}

 

{(sentence2)} (%endif%)

Input Template:  {{sentence1}}) Pregunta: {{sentence2}} _|Verddero o falso? Target Template:  {answer_choices[label]}} Answer Choices Template: Falso ||! Verdadero Input Template:  Oracion 1: {{sentence1}}  Oracleon 2: {{sentence2}}  Pregunta: _|a_oracion 1 parafrasea la oracion 2? Target Template:  {{answer_choices[label]}} Answer Choices Template: No ||! Si Input Template:  Determine si las siguientes dos oraciones se parafrasean entresi o no. Enviado 1: {sentence1}}  Enviado 2: {{sentence2}} Target Template:  {{answer_choices[label]}} Answer Choices Template: No ||! Si Input Template: 

Oracion 1: {sentence]} Oracion 2: {sentence2} Pregunta: _|La Oracion 1 y la Oracion 2 expressan el mismos significicado? _|Si o no? Target Template:

{answerchoices[label]}} Answer Choices Template: No ||I Si Input Template: Oracion 1: {sentence1} Oracion 2: {sentence2} Prepunta: _|Podemos reescribir la Oracion 1 a la Oracion 2? _|Sio ono? Target Template: {(answerchoices[label])} Answer Choices Template: No ||I Si Input Template: {(sentence1)}  LE S una parafrasis de la siguiente oracion? {(sentence2)} Target Template: {(answerchoices[label])} Answer Choices Template: No ||I Si Input Template: Oracion 1: {(sentence1)} Oracion 2: {sentence2} Prepunta: _|Podemos reescribir la Oracion 1 a la Oracion 2? 

Target Template: {answer_choices[label]}} Answer Choices Template: No || Si

### Sentence Completion

#### xcopa vi

Dataset from Ponti et al. (2020). Used in evaluation.

##### Data Example

\begin{tabular}{l l} Key & Value \\ premise & C6 gai tim th day con bo trong ngd c6... \\ choice1 & C8 d s@n vdo bat. \\ choice2 & C8 mht c4m giac nogn mi@ng. \\ question & effect \\ label & 1 \\ idx & 1 \\ changed & False \\ \end{tabular}

##### Prompts

Input Template:

{{ premise }} Toi dang lu@ng l ygita hai lu chon. Giup toi chonn nguyen nnhan (% if question == "cause" %) co kha nng ray ra cao hon: (% else %) effect: (% endif %) - ({choice1}) - ({choice2})

- ({choice2})

Target Template:

{% if label != -1 %} {{ answer_choices[label] }} {%endif %} Answer Choices Template: {{choice1}} || {{choice2}}

Input Template: 

{ ( premise )} Lua chon t\t nh4 l a gi?  - ({choice1})  - ({choice2})

Ching toi dang tim kiem (% if question == "cause" %) m0t nguyen nhan (% else %) m0t anh huOng (% endif %) Target Template:

{% if label != -1 %} {answer_choices[label]} (%endif %) Answer Choices Template:

{{choose1}}|||{{choose2}} Input Template:

{{ permise}} (% if question == "cause" %) Di@un nay xay ra vi ... (% else %) Do do ... (% endif %) Giop t@ch onty chon hdp ly hon:  - ({choice1})  - ({choice2})

Target Template:

{% if label != -1 %} {{ answer_choices[label]}} (%endif %) Answer Choices Template:

{{choice1}}|||{{choose2}} Input Template:

*{{ amsver_choices[0]}' hay ^{{ answer_choices[1]}}+? {{ premise }}} (% if question == "cause" %) bbi vi (% else %) nen (% endif %) Target Template:

{% if label != -1 %} {{ answer_choices[label]}} (%endif %) Answer Choices Template:

{{choose1}}|||{{choose2}}

 

{ { premise }} Chon nguyen nnhan (% if question == "cause" %) hop ly nhhat: (% else %) effect: {% endif %} - ({choice1}) - ({choice2})

Target Template:

{% if label != -1 X} {{ answer_choices[label] }} {%endif %} Answer Choices Template:

{{choice1}} || {{choice2}}

#### 1.12.2 xcopa zh

##### Data Example

\begin{tabular}{l l} \hline Key & Value \\ \hline premise & \\ choice1 & \\ choice2 & \\ question & \\ label & 1 \\ idx & 1 \\ changed & False \\ \hline \end{tabular}

### Prompts

Input Template:

{{ premise }} {% if question == "cause" %) % 

{ { premise }}

[X if question=="cause"%] X[X]: {% else X} X(X. endif X)  - {choice1}}

- {{choose2}}

Target Template:

{% if label!=-1 X}{answer_choices[label] }}{Vendif}

Answer Choices Template:

{{choice1}} || {choice2}}

Input Template:

 "{{ answer_choices[0] }}" x[X] ^* {{ answer_choices[1] }}"? {{ premise }} {X if question=="cause"%] X[X] ^* { {\% else X} X[X] ^* endl }}

Target Template:

{% if label!=-1 X}{{ answer_choices[label] }}{X endif X}

Answer Choices Template:

{{choice1}} || {{choice2}}

Input Template:

 {{ premise }}

[X] ^* { { choose1}} - {{choose2}}

Output Template:

 {{ premise }}

[X] ^* { { choose1}}

[X] ^* { { choose2}}

[X] ^* { { choose2}}

[X] ^* { { choose3}} - {X} # {X endif X}

Target Template:

{% if label!=-1 X}{answer_choices[label]}}{Vendif}

Answer Choices Template:

 {{choice1}} || {{choice2}}

Input Template: 

{ ( premise )}

#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\\#\#\#\\#\#\\#\#\\#\\#\\#\\#\\#\\#\\\#\\\#\\\#\\\\#\



({choice1}) || [{choice2}]

**Notes:** Adapted from Perez et al. (2021) and Schick and Schutze (2020).

Input Template:

"{{ answer_choices[0]}" %EUR={answer_choices[1]}*? {{ premise }} {% if question=="cause"%}\#\%{@ else%}\#\%{@ endif X}

Target Template:

{% if label!=-1%}{{ answer_choices[label]}}{%endif %}

Answer Choices Template:

{{choice1}} || {{choice2}}

Input Template:

{{ premise }}  -- {{choice1}}  -- {{choice2}}  -- {{choice2}}  -- {{choice2}}  -- {{choice1}}  -- {{choice2}}  -- {{choice2}}

Target Template:

{% if label!=-1%}{{answer_choices[label]}}{%endif}

Answer Choices Template:

{{choice1}} || {{choice2}}

Input Template:

{{ premise }}   

({choice1}) || [{choose2}]

### Natural Language Inference

#### 1.13.1 xnli en

Dataset from Conneau et al. (2018). Used in evaluation.

##### Data Example

\begin{tabular}{l l} \hline Key & Value \\ \hline premise & you know during the season and i gu... \\ hypothesis & You lose the things to the followin... \\ label & 0 \\ \hline \end{tabular}

##### Prompts

**Notes:** Sanh et al. (2022)

Input Template:

Take the following as truth: {{premise}} Then the following statement: "{hypothesis}" is {["true"}, {["false"}, or {[" inconclusive'}]?

Target Template:

{{ answer_choices[label })}

Answer Choices Template:

True ||| Inconclusive ||| False

**Notes:** Sanh et al. (2022)

Input Template:

{{premise}} Question: Does this imply that "{hypothesis}"? Yes, no, or maybe?

Target Template:

{answer_choices[label]}

 

Yes ||I Maybe ||N No

**Notes:** Same as reported in Figure G7 of Brown et al. (2020), except that there is no task identifying tokens like "anli R1: ".

Input Template:

{{premise}} Question: {(hypothesis)} True, False, or Neither?

Target Template:

{{ answer_choices[label] }} Answer Choices Template:

True |||Neither |||False

**Notes:** Sanh et al. (2022)

Input Template:

Given that {{premise}} Does it follow that {{hypothesis}} Yes, no, or maybe?

Target Template:

{{ answer_choices[label] }} Answer Choices Template:

Yes ||Maybe ||No

**Notes:** Adapted from the BoolQ prompts in Schick and Schutze (2020).

Input Template:

{{premise}} Based on the previous passage, is it true that "{(hypothesis)}"? Yes, no, or maybe?

Target Template:

{{ answer_choices[label] }} Answer Choices Template:

Yes ||Maybe ||N 



**Notes:**: Webson and Pavlick (2021)

Input Template:

Given {{premise}} Is it guaranteed true that "{(hypothesis)}=? Yes, no, or maybe?

Target Template:

{{ answer_choices[label] }}

Answer Choices Template:

Yes ||I Maybe ||| No

**Notes:**: Webson and Pavlick (2021)

Input Template:

Given {{premise}} Should we assume that "{(hypothesis)}" is true? Yes, no, or maybe?

Target Template:

{{ answer_choices[label] }}

Answer Choices Template:

Yes ||Maybe |||No

**Notes:**: Sanh et al. (2022)

Input Template:

Given that {{premise}} Therefore, it must be true that "{(hypothesis)}"? Yes, no, or maybe?

Target Template:

{{ answer_choices[label] }}

Answer Choices Template:

Yes ||I Maybe ||| No

**Notes:**: Webson and Pavlick (2021)

Input Template:

Suppose {(premise)} Can we infer that "{hypothesis}"? Yes, no, or maybe?

Target Template:  {{ answer_choices[label] }} Answer Choices Template:  Yes ||| Maybe ||| No**Notes:** Webson and Pavlick (2021) Input Template:  {{premise}} Are we justified in saying that "{hypothesis}"? Yes, no, or maybe? Target Template:  {{ answer_choices[label] }} Answer Choices Template:  Yes ||| Maybe ||| No**Notes:** Sanh et al. (2022) Input Template:  {{premise}} Based on that information, is the claim: "{(hypothesis)}" {("true")},  {{"false"}}, or {("inconclusive")}? Target Template:  {{answer_choices[label] }} Answer Choices Template: True ||| Inconclusive ||| False**Notes:** Sanh et al. (2022) Input Template:  {{ premise}} Keeping in mind the above text, consider: {{hypothesis}} Is this {("always")},  {{"sometimes"}}, or {{"never"} correct? 

Target Template: {{ answer_choices[label]}} Answer Choices Template: Always||Sometimes||Never Notes:Sanh et al. (2022) Input Template: Suppose it's true that {{premise} Then, is "{hypothesis}" {"always"}, {{"sometimes"}}, or {("never"})true? Target Template:

{{ answer_choices[label]}} Answer Choices Template: Always|||S Sometimes||Never Notes:Sanh et al. (2022) Input Template: Assume it is true that {{premise} Therefore, "{hypothesis}" is {"guaranteed"}, {{"possible"}}, or {{"impossible"}}?? Target Template: {{answer_choices[label]}} Answer Choices Template: Guaranteed||Possible||Impossible Notes: Adapted from Williams et al. (2018) instructions to crowdsourcing workers. Input Template: {{premise}} Using only the above description and what you know about the world, "{hypothesis}" is definitely correct, incorrect, or inconclusive? Target Template: 

({ answer_choices[label])}

Answer Choices Template: Correct ||Inconclusive ||Incorrect

#### xnli es Data Example

\begin{tabular}{l l} \hline Key & Value \\ \hline premise hypothesis label & Used sabe durante la temporad a y s... Pierdes las cosas al siguiente nive... \\ \hline \end{tabular}

### Prompts

#### Human-translated prompts

**Notes:** Same as reported in Figure G7 of Brown et al. (2020), except that there is no task identifying tokens like "anli R1: ".

Input Template:

{{premise}}

Pregunta: {(hypothesis)} Verddero, Falso, o Ninguno?

Target Template:

{{answer_choices[label]}}

Answer Choices Template:

Verddero |||Ninguno |||Falso

**Notes:** Webson and Pavlick (2021)

Input Template: Supongamos {{premise}} Podemos inferir que "{{hypothesis}"? Si, no, o tal vez? Target Template: 

* [1]

Answer Choices Template: Si|||Tal vez||No

**Notes:** Webson and Pavlick (2021) Input Template: {{premise}} Estamos justificadosen decir que "{{hypothesis}}"? Si, no, otal vez7 Target Template: {{ answer_choices[label }}} Answer Choices Template: Si|||Tal vez||No

**Notes:** Sanh et al. (2022) Input Template: Supongamos ques ciertoque {{premise}} por lo tanto, "{{hypothesis}}" es {{"garantizedo"}}, {{"possible"}}, o {{"imposible"}}?

Target Template: {{answer_choices[label }}} Answer Choices Template: Garantizado|||Possible|||Imposible

**Notes:** Adapted from Williams et al. (2018) instructions to crowdsourcing workers. Input Template: {{premise}} Usando solo la descripci on anterior y lo que sabe sobre el mundo, "{hypothesis}" es definitivamente correcto, incorrecto o noconcluvente? Target Template: {{ answer_choices[label }}} Answer Choices Template: 

Correcto ||INoconcluyente|II|Incorrecto

##### Machine-translated prompts

Input Template:  {(premise)}_iEstamosjustificadosaldecirque&quot;{(hypothesis)}@quot;?_iSi,no otalve27 TargetTemplate:  {(answer_choices[label] }) AnswerChoicesTemplate: S1||Quizas||No InputTemplate:  {(premise)}Pregunta:{(hypothesis)}_iVerdadero,falsooingmuno? TargetTemplate:  {( answer_choices[label] }) AnswerChoicesTemplate: Verdadero|||Ningunodelosdoss||Falso InputTemplate:  {(premise)}Usandosoloalescricpion anterioryloquesabesobreelmundo, &quot;{(hypothesis)}@quot;esdefinitivamentecorrecta,incorrectaono concluyente. TargetTemplate:  {(answer_choices[label] }) AnswerChoicesTemplate: Correcto||Prooconcluyente||Incorrecto 

Supongamos {(premise)}_iPodemos inferir que &quot;{(hypothesis)}&quot;?_iSi, no o tal vez?

Target Template: {{ answer_choices[label]}} Answer Choices Template: Si||Quizas|||No Input Template: Supongamos ques escietque {{premise}} For lontanto, &quot;{(hypothesis)}&quot; es {{"guaranteed"}}, {{"possible"}} o {{"impossible"}}. Target Template: {{answer_choices[label]}} Answer Choices Template: garantizado||Posible||Imposible 



**ACL 2023 Responsible NLP Checklist+**

Footnote †: margin: The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

A For every submission: A1. Did you describe the limitations of your work? _Abstract & Section 1_. A2. Did you discuss any potential risks of your work? _We did not find our work to have major potential risks._ A3. Do the abstract and introduction summarize the paper's main claims? _Abstract & Section 1_ A4. Have you used AI writing assistants when working on this paper? _Left blank._ B **Did you use or create scientific artifacts?**_Section 3 & Appendix D_ B1. Did you cite the creators of artifacts you used? _Section 3 & Appendix D_ B2. Did you discuss the license or terms for use and / or distribution of any artifacts? _Appendix D_ B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? _The artifacts used did not detail their intended use except for license compliance, which we adhere to. The intended use of our artifacts is clear from their accompanying licenses._ B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? _As we reuse existing datasets (ROOTS, XCOPA, etc), we refer to the original works for such information._ B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? _We explain the languages covered by our artifacts in Section 3._ B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. _Dataset sizes are reported in Figures 1 and 2, as well as the accompanying text._ C **Did you run computational experiments?**_ Sections 3 & 4._ C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? _Model parameters and infrastructure details are reported in Section 3.



C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? _Hyperparamters are reported in Section 3._ C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? _Results are reported in Figures 4 - 8 and Tables 1 - 2 as well as in the Appendix._ C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? _We report parameter settings in Section 3 and will provide code for reproduction of all our experiments._ D **D****M****Did you use human annotators (e.g., crowdworkers) or research with human participants?**

_Left blank._

* D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? _Not applicable. Left blank._

* D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants' demographic (e.g., country of residence)? _Not applicable. Left blank._

* D3. Did you discuss whether and how consent was obtained from people whose data you're using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? _Not applicable. Left blank._

* D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? _Not applicable. Left blank._

* D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? _Not applicable. Left blank._