Crosslingual Generalization through Multitask Finetuning
Year: 2023

Abstract:
Multitask prompted finetuning (MTF) has been shown to help large language models generalize to new tasks in a zero-shot setting, but so far explorations of MTF have focused on English data and models. We apply MTF to the pretrained multilingual BLOOM and mT5 model families to produce finetuned variants called BLOOMZ and mT0. We find finetuning large multilingual language models on English tasks with English prompts allows for task generalization to non-English languages that appear only in the pretraining corpus. Finetuning on multilingual tasks with English prompts further improves performance on English and non-English tasks leading to various state-ofthe-art zero-shot results. We also investigate finetuning on multilingual tasks with prompts that have been machine-translated from English to match the language of each dataset. We find training on these machine-translated prompts leads to better performance on humanwritten prompts in the respective languages. Surprisingly, we find models are capable of zero-shot generalization to tasks in languages they have never intentionally seen. We conjecture that the models are learning higher-level capabilities that are both taskand languageagnostic. In addition, we introduce xP3, a composite of supervised datasets in 46 languages with English and machine-translated prompts. Our code, datasets and models are freely available at https://github.com/ bigscience-workshop/xmtf.

heading: 1 Introduction
text: Large language models pretrained on vast amounts of text show some capability of solving tasks expressed in natural language, even without explicit training on these tasks (Brown et al., 2020). Finetuning on groups of language tasks has been shown to significantly boost this zero-shot task generalization of language models (Wei et al., 2021; Sanh
et al., 2022; Min et al., 2021). For example, Sanh et al. (2022) finetune on tasks like summarization and question answering leading to better performance on unseen tasks like natural language inference. Previous work has focused on multitask finetuning in the context of large English language models and tasks.
Multilingual large language models show the same zero-shot learning capabilities for both monolingual and crosslingual tasks (Goyal et al., 2021a; Lin et al., 2021; Patel et al., 2022; Soltan et al., 2022). However, zero-shot performance tends to be significantly lower than finetuned performance. Thus, task-specific or language-specific transfer learning via finetuning remains the predominant practice (Devlin et al., 2018; Conneau et al., 2019; Aribandi et al., 2021). This is particularly challenging for low-resource languages or tasks with limited data available, such as writing a fable that teaches a specified moral. In the spirit of multitask finetuning, it would be desirable to improve the zero-shot task generalization of multilingual models to make them usable on tasks from low-resource languages without requiring further finetuning.
To address this goal, we focus on crosslingual multitask finetuning. Due to the difficulty of collecting supervised task data in low-resource languages, previous work typically aims to transfer capabilities learned from finetuning on English data, which can improve performance on nonEnglish language tasks (Wu and Dredze, 2019; Phang et al., 2020; Chalkidis et al., 2021; Vu et al., 2022). We investigate whether English-only multitask finetuning also improves performance on non-English held-out tasks using the multilingual BLOOM (Scao et al., 2022a) and mT5 (Xue et al., 2020) models. We find that after finetuning on the English-only multitask mixture used for T0 (Sanh et al., 2022) (P3), performance on a diverse set of
15991
non-English held-out tasks increases.
To investigate whether multilingual task data can further improve performance, we extend P3 to xP3 by adding datasets from 46 different languages that cover tasks previously not present in P3 (such as translation and program synthesis). Finetuning on xP3 leads to even better zero-shot task generalization in both English and non-English compared to the P3-trained baseline. Models finetuned on xP3 perform best on English prompts, even for non-English samples. Hypothesizing that better performance could be attained by training on nonEnglish prompts, we construct a variant of xP3 with machine-translated prompts called xP3mt. We find that finetuning on machine-translated prompts is enough to significantly increase performance on held-out tasks with non-English human-written prompts. However, reducing the number of En-
glish prompts in the finetuning also worsens English prompt performance on multilingual tasks.
Notably, we also find that models finetuned on xP3 generalize to held-out tasks in languages never intentionally seen during pretraining nor finetuning. We conduct a contamination analysis and find that only small amounts of these languages were included in the pretraining corpus. Thus, we hypothesize the models learn some language- and task-agnostic capabilities.
We publicly release all our datasets and models (URLs in Appendix §C).
2 Related work
2.1 Multitask learning
Multitask finetuning (Sanh et al., 2022) (or instruction tuning (Wei et al., 2021)) has emerged as a
Sentence1 He was a scholar in Metaphysical Literature, Theology and Classical sciences. Sentence2 He was a scholar in metaphysical literature, theology, and classical science. Label 1 Sentence1 He was a scholar in Metaphysical Literature, Theology and Classical sciences. Sentence2 He was a scholar in metaphysical literature, theology, and classical science.
Label 1
Sentence1 He was a scholar in Metaphysical Literature, Theology and Classical sciences. Sentence2 He was a scholar in metaphysical literature, theology, and classical science. Label 1
P3
sentence1 He was a scholar in Metaphysical Literature, Theology and Classical sciences. sentence2 He was a scholar in metaphysical literature, theology, and classical science. label 1
Oración 1: {{sentence1}} Oración 2: {{sentence2}}
Pregunta: ¿La oración 1 parafrasea la oración 2? ¿Si o no? {{Choices[label]}}
Sentence 1: {{sentence1}} Sentence 2: {{sentence2}}
Question: Can we rewrite Sentence 1 to Sentence 2? Yes or No? {{Choices[label]}}
Choices=[No,Sí]
xP3
Sentence1 He was a scholar in Metaphysical Literature, Theology and Classical sciences. Sentence2 He was a scholar in metaphysical literature, theology, and classical science. Label 1 Sentence1 He was a scholar in Metaphysical Literature, Theology and Classical sciences. Sentence2 He was a scholar in metaphysical literature, theology, and classical science.
Label 1
Sentence1 He was a scholar in Metaphysical Literature, Theology and Classical sciences. Sentence2 He was a scholar in metaphysical literature, theology, and classical science. Label 1 sentence1 Fue académico en literatura metafísica, teología y ciencias clásicas." sentence2 Fue académic en literatura metafísica, teología y ciencia clásica.
label 1
xP3mt
Sentence1 He was a scholar in Metaphysical Literature, Theology and Classical sciences. Sentence2 He was a scholar in metaphysical literature, theology, and classical science. Label 1 Sentence1 He was a scholar in Metaphysical Literature, Theology and Classical sciences. Sentence2 He was a scholar in metaphysical literature, theology, and classical science.
Label 1
Sentence1 He was a scholar in Metaphysical Literature, Theology and Classical sciences. Sentence2 He was a scholar in metaphysical literature, theology, and classical science. Label 1 sentence1 Fue académico en literatura metafísica, teología y ciencias clásicas." sentence2 Fue académic en literatura metafísica, teología y ciencia clásica.
label 1
Choices=[No,Yes]
Sentence 1: {{sentence1}} Sentence 2: {{sentence2}}
Question: Can we rewrite Sentence 1 to Sentence 2? Yes or No? {{Choices[label]}}
Choices=[No,Yes]
ROOTS corpus (Laurençon et al., 2022) in 46 natural languages and 13 programming languages. We also finetune mT5 (Xue et al., 2020) to compare encoder-decoder and decoder-only performance. mT5 is pretrained on a corpus sampled from mC4 covering 101 languages.

heading: 3 Finetuning data and models
text: To study crosslingual multitask prompted finetuning, we create xP3 by extending the P3 dataset collection with additional non-English tasks. We finetune both BLOOM and mT5 models on xP3. We refer to Appendix §C for public links to released models and datasets.

heading: 3.1 Finetuning data
text: We build on the P3 (Sanh et al., 2022) task taxonomy and add 30 new multilingual datasets illustrated in Figure 1. We define four task clusters previously not present in P3: translation, simplification, program synthesis, and miscellaneous code datasets. As 11% of BLOOM’s pretraining data is code, we add code datasets classified as program synthesis (text-to-code) or miscellaneous. The latter includes tasks such as estimating the computational complexity of a provided code snippet and generating a name for a given function. We extend the XWinograd dataset (Tikhonov and Ryabinin, 2021) with winograd schemas from CLUE (Xu et al., 2020) to increase its Chinese samples from 16 to 504. Similar to P3, a fraction of our prompts invert the task at hand. For example, a prompt may invert a closed-book QA sample by asking the model to generate a question given an answer.
With xP3 we aim to replicate the language distribution of the ROOTS corpus (Laurençon et al., 2022) used to pretrain BLOOM. Thus, xP3 consists of the same 46 natural languages and code as ROOTS. ROOTS, xP3 and the mT5 corpus (Xue et al., 2020) language distributions are visualized in Figure 2. 39% of xP3 data is English, slightly more than the 30% of English data in ROOTS. Various African languages such as Twi (tw) and Bambara (bm) form the tail of xP3’s language distribution. Many of them are not included in the mT5 pretraining corpus. In xP3, Twi and others are represented solely as a translation task using data from Flores200 (NLLB Team et al., 2022).
To study the importance of non-English prompts, we construct a machine-translated variant of xP3, xP3mt. We translate prompts of monolingual
datasets into the respective dataset language. For example, for the Chinese dataset C3 (Sun et al., 2020) prompts in xP3mt are in Chinese instead of English in xP3. For crosslingual datasets prompts remain in English in xP3mt (such as Wiki-Lingua, which involves producing a summary in one language based on text in another language). We use the Google Cloud API for machine translation1. Figure 3 compares the dataset variants we train on.
3.2 Models We use publicly available pretrained BLOOM models ranging from 560 million to 176 billion parameters. BLOOM models are large decoder-only language models pretrained for around 350 billion tokens with an architecture similar to GPT-3 (Brown et al., 2020). We finetune the models for an additional 13 billion tokens with loss only being computed on target tokens. For example, given the input “Translate to English: Je t’aime." and a space-separated target “I love you.", the model is trained to predict only the targets. As targets vary in length from just one to hundreds of tokens, we downscale the loss of each token by the length of the target it belongs to. This ensures short targets (e.g. for multiple-choice QA) get the same weight as long targets (e.g. for translation). We skip samples longer than 2048 tokens and use packing to train efficiently on multiple samples at a time (Kosec et al., 2021). We select the final checkpoint based on validation performance.
For mT5 models, we finetune using the T5X (Roberts et al., 2022) framework on TPUs. mT5 uses the same encoder-decoder architecture, pretraining objective (masked language modeling), and pretraining length (1 trillion tokens) as T5 (Raffel et al., 2020). For finetuning mT5, we follow the same procedure as described above for BLOOM, except that inputs are fed into the encoder and thus are not space-separated from targets.
We produce three core model variants available in different sizes:
• BLOOMZ-P3 / mT0-P3: Models finetuned on the English-only P3.
• BLOOMZ / mT0: Models finetuned on xP3, which consists of multilingual datasets with English prompts.
• BLOOMZ-MT / mT0-MT: Models finetuned on xP3mt, which consists of multi-
1https://cloud.google.com/translate
Sentence Completion (XCOPA & XStoryCloze)
51.8 52.9 50.4
80.5 82.6 86.0
Natural Language Inference (XNLI)
33.3 33.6 34.8
47.4 55.3 57.7
Multilingual Multitask Generalization
XGLM-7.5B BLOOM mTk-Instruct-13B
BLOOMZ-P3 BLOOMZ mT0-13B
Coreference Resolution (XWinograd)
50.3 50.6 54.7 54.6
66.3 74.6

heading: 4 Results
text: tasks and prompts, which has been shown to help generalization (Chung et al., 2022; Iyer et al., 2022). mT0-13B beating T0-11B indicates that the benefit of scaling tasks is larger than the benefit of pretraining and finetuning on relatively more English tokens.

heading: 4.2 Language generalization
text: Here we add another layer of generalization: languages. Figure 4 already shows that finetuning on English data only (P3) leads to better performance on non-English data: For example, BLOOMZ-P3 improves by over 50% on multilingual sentence completion compared to BLOOM. Thus, zero-shot task performance in languages only seen during pretraining improves after finetuning on English. This has major practical benefits as it can be more difficult to collect data for low-resource languages.
Next, we investigate performance on languages the model has never intentionally seen. Due to the scale of large language model pretraining, it is difficult to label tasks or languages as strictly unseen. It is likely that the training data unintentionally includes small fractions of these languages (just as many tasks might appear “implicitly” in the pretraining corpus (Sanh et al., 2022)). In Figure 5 we show that after multitask finetuning on xP3, the models can perform unseen tasks in languages that were not intentionally trained on. After probing the pretraining corpus of BLOOM, we do find small amounts of these languages that were unintentionally included (Appendix §D). However, for
XNLI, performance increases across all languages, many of which only show up in tiny fractions in our language contamination analysis, such as Thai with 0.006%. If we extrapolate this proportion to the entire ROOTS corpus, the BLOOM models would have seen a mere 20 million tokens of Thai during pretraining. One possibility is that betterthan-random XNLI performance can be attained with little or no language understanding. In Appendix §H, we investigate edit distances of XNLI samples and find that there are differences across labels, however, likely not significant enough to enable this kind of generalization.
4.3 Multilingual prompting
Since all prompts in xP3 are in English (even for
multilingual datasets), we created xP3mt, an extension with machine-translated prompts. To investigate performance on non-English prompts, we additionally human- and machine-translated the English evaluation prompts from Figure 4. In Table 1, we report performance on these. Results on machinetranslated prompts in languages that are not part of the finetuning corpus, such as those in Figure 5, are in Appendix §I. Table 1 shows that BLOOMZ performs much better on English than on nonEnglish prompts. BLOOMZ-MT, which is finetuned on xP3mt, significantly improves on multilingual prompts. On XNLI, BLOOMZ-MT raises the average performance on human-translated prompts from 41.13 to 45.55. This comes at the cost of a reduction in its performance on English prompts, from 53.58 to 49.74. For mT0, the MT version provides similar performance gains on XNLI and XWinograd non-English prompts, while results on XCOPA and XStoryCloze are mixed. Similar to Lin et al. (2021), we also find that models perform better on human-translated prompts than machine-translated ones for XNLI.

heading: 4.4 Scaling
text: In Figure 4, the average performance of BLOOM is near the random baselines of 0.50 for Sentence Completion and Coreference Resolution and
0.33 for NLI. We think this is due to all of our experiments being zero-shot and using untuned prompts (Perez et al., 2021a). We find in Figure 6 that even at 560M parameters, multitask finetuning improves zero-shot generalization. The gap between pretrained and multitask finetuned models grows significantly as parameters increase. Scaling up parameters benefits all languages evaluated.
4.5 Generation tasks
In this section, we investigate the impact of multitask finetuning on generative tasks. In Figure 7, we plot validation performance throughout the training process. We find that while performance on natural language understanding tasks continues to increase, generative performance jumps initially and then decreases. Relatedly, in Table 2, we find that multitask finetuning does not improve performance on HumanEval (Chen et al., 2021). Only for small models, such as BLOOM-560M vs. BLOOMZ-560M, there are meaningful performance gains. When no code data is included in finetuning (BLOOMZ-P3) performance decreases significantly. mT0 models, which have not been pretrained on code, fail to solve any HumanEval problems (see full results in Appendix §K). Given a Python docstring, HumanEval requires models to complete a function. Inspecting generations reveals that the multitask finetuned models are biased towards short generations. In Appendix §E, we show example solutions from HumanEval and compute
This bias towards short answers and the performance drop on generative tasks come from finetuning on short texts. Most tasks in our finetuning dataset, xP3, are single sentences. We show in Appendix §G that finetuning on fewer short tasks via early stopping, adding long tasks or upweighting long tasks leads to longer generations and slightly better performance. We find it most effective, however, to force a minimum generation length at inference. This is done by ignoring any probability mass the model assigns to its end-of-sequence token for a desired number of tokens. Only after the generation has reached the desired length, can the model generate the end-of-sequence token, thus finishing the generation. Forcing a minimum generation length improves the BLEU score on a translation task by 9 points, see Appendix §G for quantitative and Figure 15 for qualitative results.
4.6 Effect of language proportions In Figure 8, we find that finetuned BLOOM models perform better on languages seen extensively during pretraining. As the language distribution in the finetuning dataset, xP3, closely follows that of pretraining, these languages are also seen most frequently during finetuning. Specifically, XCOPA and XNLI show significantly better performance on these high-resource languages, such as English, Spanish or French, which all make up more than 10% of pretraining individually. The trend is less consistent for XWinograd. This may be caused by the fact that XWinograd language subsets are not translations of each other and have a significantly different number of samples. Thus, some language subsets of XWinograd may be inherently more difficult than others.
5 Conclusion
In this work we investigated crosslingual multitask finetuning. We developed xP3, a corpus consisting of tasks in 46 languages. Further, we have extended xP3 to xP3mt with machine-translated prompts. We have finetuned pretrained BLOOM and mT5 models on the newly created corpora as well as the English-only P3 corpus to produce BLOOMZ and mT0 models.
We found that English-only finetuning suffices for a multilingual pretrained large language model to generalize to tasks in other pretrained languages.
However, finetuning on multiple languages using xP3 provided even better performance. We have further observed finetuned models to be capable of generalization to new tasks in languages they have never intentionally seen. We investigated multilingual prompting and found performance after finetuning on English prompts only to be poor. However, finetuning on a corpus with machinetranslated prompts (xP3mt) lead to significantly better performance on human-written non-English prompts. Comparing models from 560 million up to 176 billion parameters revealed that the performance gap between only pretraining and finetuning widens as parameters increase. Lastly, we found multitask finetuning on billions of short targets biases models to produce short answers, which can hurt performance on generative tasks. We proposed a simple workaround by forcing a minimum generation length at inference.
To contribute to future progress on improving zero-shot generalization, we release all datasets and models introduced in this work.

heading: 6 Limitations
text: We highlight several limitations of our work:
Unnatural prompting format The choice to separate inputs and targets using a space character has proven effective to multitask finetune our decoder-only models. Nonetheless, poorly formatted prompts may result in undesirable behavior. For example, given the following prompt: “Translate to English: Je t’aime", the model may continue the input with additional French content before starting to solve the task, i.e. translating the input from French to English. This can be mitigated by improving the prompts with a trailing full stop or a newline symbol. Encoder-decoder models, such as our mT0, do not suffer from this problem, as inputs and targets are fed into different parts of the model.
Limited languages in xP3 The pretraining corpus of mT0 contains more than 101 languages (Xue et al., 2020), however, we finetune on only 46 languages. Likely, finetuning on the full 101 languages mT0 has seen during pretraining would lead to better performance. However, we decided to use only the languages of BLOOM in order to study language generalization (§4.2). Similarly, one could likely attain better performance by enhancing xP3 with more datasets, such as via BIG-Bench (Srivastava et al., 2022; Suzgun et al., 2022), or more
prompts, such as via NL-Augmenter (Dhole et al., 2021). We have released an extended version of xP3 dubbed xP3x that covers 277 languages and is around ten times larger than xP3, but are yet to finetune models on it.
Performance While our models show strong capabilities of performing tasks zero-shot, there remain numerous failure modes that are common in large language models (Rae et al., 2021; Bommasani et al., 2021; Zhang et al., 2022; Smith et al., 2022; Ouyang et al., 2022; Taylor et al., 2022; Chowdhery et al., 2022; Biderman et al., 2023; Allal et al., 2023; Li et al., 2023). In Figure 16 of Appendix §F, BLOOMZ fails to understand the moral of a fable resulting in an undesirable generation. Similarly, in Figure 15, mT0-13B is asked to provide an explanation, but answers with a question. We have made several modifications to the multitask finetuning recipe, such as loss weighting, mixing in long tasks, and various multilingual aspects, leading to the strong zero-shot performance of our models. However, there are many other changes to the multitask finetuning procedure that are worth exploring to get better models (Honovich et al., 2022; Wang et al., 2022b; Longpre et al., 2023a; Liu et al., 2023; Dettmers et al., 2023; Yin et al., 2023). Further, the pre-trained models we use, BLOOM and mT5, are suboptimal in many aspects such as compute allocation (Hoffmann et al., 2022; Muennighoff et al., 2023), pretraining datasets (Longpre et al., 2023b; Touvron et al., 2023; Chung et al., 2023), pre-training objective (Tay et al., 2022b) and possibly model architecture (Komatsuzaki et al., 2022; Shen et al., 2023). Future work should investigate multitask finetuning better base models.
Learning new languages during finetuning While we have investigated generalization to languages only seen during pretraining, we did not investigate generalization to languages only seen during finetuning. Our mT0 models are finetuned on several new languages not seen in pretraining (see Figure 2). Out of those, we only evaluated on code (HumanEval), where mT0 performed at the random baseline (0.00 in Table 10). We point to follow-up work that has investigated the question of teaching BLOOMZ new languages (Yong et al., 2022; Cahyawijaya et al., 2023) and work investigating adaptation of BLOOM (Ennen et al., 2023; Yong and Nikoulina, 2022).

heading: Acknowledgments
text: This work was granted access to the HPC resources of Institut du développement et des ressources en informatique scientifique (IDRIS) du Centre national de la recherche scientifique (CNRS) under the allocation 2021-A0101012475 made by Grand équipement national de calcul intensif (GENCI). In particular, all the evaluations and data processing ran on the Jean Zay cluster of IDRIS, and we want to thank the IDRIS team for responsive support throughout the project, in particular Rémi Lacroix.
We thank the XGLM team for providing access to XStoryCloze. We thank volunteers who humantranslated XNLI prompts. We thank Noah Constant and Douwe Kiela for feedback on drafts of this paper. We thank Victor Sanh, Stephen Bach, Sasha Rush and Jordan Clive for support throughout the project.

heading: 1 Introduction 1

heading: 2 Related work 2
text: 2.1 Multitask learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2.2 Multilingual models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

heading: 3 Finetuning data and models 4
text: 3.1 Finetuning data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 3.2 Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

heading: 4 Results 5
text: 4.1 Task generalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 4.2 Language generalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 4.3 Multilingual prompting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 4.4 Scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 4.5 Generation tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 4.6 Effect of language proportions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

heading: 5 Conclusion 8

heading: 6 Limitations 9

heading: A Contributions 18

heading: B Task generalization breakdown 18

heading: C Artifacts 20

heading: D ROOTS language contamination 20

heading: E Code generations 21

heading: F Qualitative examples 21
text: G Increasing generation length 24
H XNLI edit distances 24

heading: I Multilingual prompting in unseen languages 25
text: J Ideas that did not work 26

heading: K Full results 26
text: L Version control 28
M Prompts used 28

heading: B Task generalization breakdown

heading: A Contributions

heading: C Artifacts
text: Table 3 lists all artifacts used or released in this work. We make all our work accessible under the most permissive licenses available to us.

heading: D ROOTS language contamination
text: While the BLOOM ROOTS corpus (Laurençon et al., 2022) was collected from 46 natural languages and 13 programming languages, we find that sentences from the same document do not always belong to the collected (meta) language. Some sentences use languages like Russian or Japanese that were not the intentionally collected parts. This “language contamination” may stem from “code-mixing” or different languages being used in code comments. To investigate the extent of contamination, we randomly sample 1% of the documents from ROOTS for a total of 51M documents. For each document, we use cld32 (Xue et al., 2020) to identify the languages used in each sentence and compare them with the meta language of the document. We summarize our results in Figure 11. It shows that ROOTS contains unintentionally collected languages, such as Burmese (my: 0.00003%), Thai (th: 0.006%), Turkish (tr: 0.03%), Greek (el: 0.03%), Russian (ru: 0.03%), Bulgarian (bg: 0.05%), Estonian (et: 0.06%), Haitian (ht: 0.12%), German (de: 0.21%), Italian (it: 0.28%) and Japanese (ja: 0.54%). These “unseen” languages only have
2https://github.com/google/cld3
small sentence proportions in our subsample compared to English (en: 46.23%), French (fr: 15.73%) and Spanish (es: 13.38%). Yet, they may help the language generalization of BLOOMZ models described in §4.2. Japanese is mostly mixed in the meta English documents (47%), meta Code documents (8%) and meta Chinese documents (5%). Meanwhile, Russian is mostly mixed in the meta English documents (52%), meta Code documents (19%) and meta French documents (11%).

heading: E Code generations
text: Table 4 provides statistics on code generations and code data. We find that BLOOM generates on average 70% more characters and 17x more comments than BLOOMZ for a given problem from HumanEval. Figure 12 compares an example solution from BLOOM and BLOOMZ. While both solutions are correct, BLOOMZ is biased towards short and concise answers.
F Qualitative examples
G Increasing generation length
In §4.5, we found performance on generative tasks to worsen in later stages of training. To investigate this problem further, we study a 7.1 billion parameter BLOOM model that is finetuned for 13 billion tokens, which results in a low BLEU score of 0 and very short generations as shown in Table 5 (Default). We can solve this problem with two high-level strategies: (a) Reducing short tasks during finetuning and (b) Forcing a minimum generation length.
For (a), we do so by either early stopping, upweighting long tasks or adding new long tasks. As the majority of our finetuning data are single sentences, early stopping has the effect of finetuning on fewer short sentences. Upweighting long tasks is done by removing the loss normalization explained in §3.2. This has the effect of each token getting equal weight regardless of the task, which upweights long tasks, as they have more tokens. Finally, for adding long tasks, we add tasks that require multi-sentence generations, such as generating an entire news article given a title. These long tasks collectively make up 10% of finetuning data for this ablation. All three solutions result in longer average generations as shown in Table 5 and slightly better BLEU scores, albeit effects are still small.
For (b), we force the model to generate a minimum number of tokens at inference. Our benchmarking task, MultiEURLEX (Chalkidis et al., 2021), requires multi-sentence generations with an average target length of 1965 characters (about 491 tokens). By forcing the model to generate at least 768 tokens, we ensure that the generation is at least as long as the target. This boosts the BLEU score significantly to 9.05. This approach is thus an effective strategy to maintain long generations of good quality.
For our final models, we employ early stopping, adding of long tasks and recommend forcing a minimum generation length at inference for long generations. We do not upweight longer tasks, as it worsens accuracy on our NLU validation tasks by 10%. The number of tokens our final models are fine-tuned for are displayed in Table 6.
H XNLI edit distances
As models are surprisingly capable of solving XNLI in languages they were never intentionally trained on (§4.2), we investigate whether XNLI can be solved without any language understanding. To do so, we compute edit distances using the Levenshtein methodology (Levenshtein et al., 1966) between premise

heading: I Multilingual prompting in unseen languages
text: Table 9 shows aggregate performances on languages not intentionally seen during pretraining nor finetuning for BLOOMZ and only seen during pretraining for mT0. For BLOOMZ, performance drops significantly when translating the prompts to the respective unseen languages. Unlike on translated prompts for seen languages (§4.3), BLOOMZ-MT performs worse than BLOOMZ for machine-translated prompts in unseen languages. This is likely because BLOOMZ-MT has not been finetuned on prompts in these languages. For mT0 differences are less significant.
J Ideas that did not work
We list several experiments that did not improve over baseline results:
Non-causal In a non-causal or prefix language model, the model attends bidirectionally over input tokens and only causally over target tokens. Given a pretrained causal decoder, other work found that multitask finetuning in a non-causal setup performed better than causal finetuning (Wang et al., 2022a; Tay et al., 2022c). However, in our experiments, non-causal finetuning did not improve over causal finetuning.
Special tokens Instead of separating inputs and targets with a space, we experimented with special tokens. Using the end-of-sequence token as a separator or a completely new token that the model would learn during finetuning significantly worsened results. The models may need to train on more tokens, possibly even during pretraining, to learn these new special tokens (Zeng et al., 2022).
Fixing prompts PromptSource has been written with encoder-decoder models in mind, where inputs and targets are fed into different models. As a consequence, human-written prompts in PromptSource often lack separators between input and target. For our decoder models, we decided to separate them with a space. We additionally experimented with leaving them as is or rewriting a significant amount of prompts, but neither improved significantly over space separation.
BitFit Previous work has shown bias-only finetuning (Zaken et al., 2021) of large language models to be sufficient for strong downstream performance (Logan et al., 2021; Hu et al., 2021; Muennighoff, 2022; Liu et al., 2022; Ding et al., 2022; Muennighoff et al., 2022). We found multitask finetuning of only biases to perform 15 absolute percentage points worse on the average of held-out tasks for BLOOMZ-7.1B.

heading: K Full results
text: Table 10 shows all evaluation results on test datasets. Table 11 displays evaluation results on validation datasets which we use for checkpoint selection.
L Version control

heading: V1 → V2:
text: • Added evaluation results for the validation datasets used for checkpoint selection (Appendix §K)
• Added a section on the effect on generation length (Appendix §G) and rewrote parts of §4.5
• Added a mention of xP3x, the extension of xP3 to 277 languages in Appendix §C
• Added an example of XNLI to Appendix §H

heading: M Prompts used
text: This section describes the prompts used for training and evaluation.
In the following, dataset naming conventions follow those used in the Hugging Face datasets library. Since xP3 expands upon the P3 dataset employed by Sanh et al. (2022), we refer the reader to that work for example prompts from datasets that fall within P3. Here, we provide prompts curated for datasets that belong to xP3 but not to P3. The prompts provided are not exhaustive. Code will be released to provide a canonical reference. For each dataset considered, a dataset example is provided for additional context. Next, it is noted if the prompt does not match the original task formulation of the dataset. This is followed by a reference for the data, an input template and a target template. For prompts with predefined answer choices, these are also included. To provide examples of both human-translated and machine-translated prompts, samples of each kind are included for the xnli es dataset.
Contents
1 Prompts 1.1 Simplification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.1.1 GEM/BiSECT en . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.2 GEM/BiSECT es . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.3 GEM/BiSECT fr . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2 Summarization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.1 GEM/wiki_lingua en . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.2 GEM/wiki_lingua es . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.3 GEM/xlsum bengali . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.4 GEM/xlsum english . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.3 Translation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3.1 Helsinki-NLP/tatoeba_mt ben-eng . . . . . . . . . . . . . . . . . . . . 1.3.2 Helsinki-NLP/tatoeba_mt eng-fra . . . . . . . . . . . . . . . . . . . . 1.3.3 facebook/flores ben_Beng-eng_Latn . . . . . . . . . . . . . . . . . . . 1.3.4 facebook/flores ben_Beng-fra_Latn . . . . . . . . . . . . . . . . . . .
1.4 Program Synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4.1 Muennighoff/mbpp sanitized . . . . . . . . . . . . . . . . . . . . . . . 1.4.2 codeparrot/apps all . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4.3 codeparrot/github-jupyter-text-code-pairs . . . . . . . . . . . . . . . . 1.4.4 codeparrot/xlcost-text-to-code C++-program-level . . . . . . . . . . . 1.4.5 codeparrot/xlcost-text-to-code C-program-level . . . . . . . . . . . . . 1.4.6 codeparrot/xlcost-text-to-code Csharp-program-level . . . . . . . . . . 1.4.7 codeparrot/xlcost-text-to-code Java-program-level . . . . . . . . . . . 1.4.8 codeparrot/xlcost-text-to-code Javascript-program-level . . . . . . . . 1.4.9 codeparrot/xlcost-text-to-code PHP-program-level . . . . . . . . . . . 1.4.10 codeparrot/xlcost-text-to-code Python-program-level . . . . . . . . . . 1.4.11 neural_code_search evaluation_dataset . . . . . . . . . . . . . . . . . 1.4.12 teven/code_contests . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.5 Coreference Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.5.1 Muennighoff/xwinograd en . . . . . . . . . . . . . . . . . . . . . . . . 1.5.2 Muennighoff/xwinograd fr . . . . . . . . . . . . . . . . . . . . . . . . .
1.6 Question Answering Multiple Choice . . . . . . . . . . . . . . . . . . . . . . 1.6.1 clue c3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.7 Question Answering Extractive . . . . . . . . . . . . . . . . . . . . . . . . . . 1.7.1 clue cmrc2018 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.7.2 clue drcd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.7.3 mlqa mlqa.vi.vi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.7.4 mlqa mlqa.zh.zh . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.7.5 xquad xquad.vi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.7.6 xquad xquad.zh . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.8 Topic Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.8.1 clue csl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.8.2 clue tnews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.9 Code Misc. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.9.1 codeparrot/codecomplex codeparrot–codecomplex . . . . . . . . . . . 1.9.2 great_code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.9.3 teven/code_docstring_corpus top_level . . . . . . . . . . . . . . . . .
1.10 Word Sense Disambiguation . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.10.1 pasinit/xlwic xlwic_en_zh . . . . . . . . . . . . . . . . . . . . . . . . 1.10.2 pasinit/xlwic xlwic_fr_fr . . . . . . . . . . . . . . . . . . . . . . . . .
1.11 Paraphrase Identification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.11.1 paws-x en . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.11.2 paws-x es . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.12 Sentence Completion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.12.1 xcopa vi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.12.2 xcopa zh . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.13 Natural Language Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.13.1 xnli en . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.13.2 xnli es . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.13.2.1 Human-translated prompts . . . . . . . . . . . . . . . . . . . 1.13.2.2 Machine-translated prompts . . . . . . . . . . . . . . . . . .
1 Prompts
1.1 Simplification
1.1.1 GEM/BiSECT en
Dataset from Kim et al. (2021). Used in training.
Data Example
Key Value gem_id BiSECT_en-train-1 source To view any of the video clips belo... target If you want to watch one of the vid... references If you want to watch one of the vid...
Prompts
Input Template:
Split and simplify the following sentence while retaining its full meaning: {{source}} Simplified version:
Target Template:
{{target}}
Input Template:
{{source}} The above sentence is very complicated. Please provide me a simplified synonymous version consisting of multiple sentences:
Target Template:
{{target}}
Input Template:
{{source}}. This sentence is hard to understand. A simpler version with equivalent meaning is the following:
Target Template:
{{target}}
1.1.2 GEM/BiSECT es
Data Example
Key Value gem_id BiSECT_es-train-1 source Al final de la Santa Misa , mientra... target Al finalizar la santa misa , mientr... references Al finalizar la santa misa , mientr...
Prompts
Input Template:
{{source}}. Esta frase es difícil de entender. Una versión más simple con significado equivalente es la siguiente:
Target Template:
{{target}}
Input Template:
Divida y simplifique la siguiente oración conservando su significado completo: {{source}} Versión simplificada:
Target Template:
{{target}}
Input Template:
{{source}} La frase anterior es muy complicada. Por favor, proporcione una versión sinónima simplificada que consta de varias oraciones:
Target Template:
{{target}}
1.1.3 GEM/BiSECT fr
Data Example
Prompts
Input Template:
Key Value gem_id BiSECT_fr-train-1 source N'ayez pas peur de poser des questi... target Il ne faut pas avoir peur de poser ... references Il ne faut pas avoir peur de poser ...
Divisez et simplifiez la phrase suivante tout en conservant son sens complet : {{source}} Version simplifiée :
Target Template:
{{target}}
Input Template:
{{source}} La phrase ci-dessus est très compliquée. Veuillez me fournir une version synonyme simplifiée composée de plusieurs phrases :
Target Template:
{{target}}
Input Template:
{{source}}. Cette phrase est difficile à comprendre. Une version plus simple avec une signification équivalente est la suivante :
Target Template:
{{target}}
1.2 Summarization
1.2.1 GEM/wiki_lingua en
Dataset from lad (2020). Used in training.
Data Example
Prompts
Notes: xsum DOC_write_summary_of_above template
Key Value gem_id wikilingua_multilingual-train-42437... gem_parent_id wikilingua_multilingual-train-42437... source_language en target_language en source Go online and simply search “Decor ... target Take a quiz online to find your sty... references Take a quiz online to find your sty...
Input Template:
{{source}}
===
Write a summary of the text above in English :
Target Template:
{{target}}
Notes: xsum ’article_DOC_summary’ template
Input Template:
Article in English: {{source}}
Summary in English:
Target Template:
{{target}}
Notes: xsum ’DOC_how_would_you_rephrase_few_words’ template
Input Template:
{{source}}
How would you rephrase that briefly in English?
Target Template:
{{target}}
Notes: xsum ’DOC_tldr’ template
Input Template:
{{source}}
TL;DR in English:
Target Template:
{{target}}
Notes: xsum ’read_below_DOC_write_abstract’ template
Input Template:
First, read the English article below.
{{source}}
Now, please write a short abstract for it in English.
Target Template:
{{target}}
Input Template:
{{target}}
Given the above abstract, write an English article for it.
Target Template:
{{source}}
Input Template:
{{target}}
I'm interested in that, but I only have a few mins. Can you give me the first 500 characters of an article about that?
Target Template:
{{source[:500]}}
1.2.2 GEM/wiki_lingua es
Data Example
Key Value gem_id wikilingua_multilingual-train-34808... gem_parent_id wikilingua_multilingual-train-34808... source_language es target_language es source Navega en la web y simplemente busc... target Haz un cuestionario en línea para e... references Haz un cuestionario en línea para e...
Prompts
Notes: xsum templates
Input Template:
{{source}}
===
Write a summary of the text above in Spanish:
Target Template:
{{target}}
Notes: xsum templates
Input Template:
First, read the Spanish article below.
{{source}}
Now, please write a short abstract for it in Spanish.
Target Template:
{{target}}
Notes: xsum templates
Input Template:
{{source}}
TL;DR in Spanish:
Target Template:
{{target}}
Notes: xsum templates
Input Template:
Article in Spanish: {{source}}
Summary in Spanish:
Target Template:
{{target}}
Notes: xsum templates
Input Template:
{{source}}
How would you rephrase that briefly in Spanish?
Target Template:
{{target}}
1.2.3 GEM/xlsum bengali
Dataset from Hasan et al. (2021). Used in training.
Data Example
Key Value gem_id xlsum_bengali-train-2 url https://www.bbc.com/bengali/news-50... title রািশয়ায় ক্ষমতার ২০ বছর েযভােব েকট... target ভ্লািদিমর পুিতন তাঁর ক্ষমতায় থাকার... references ভ্লািদিমর পুিতন তাঁর ক্ষমতায় থাকার... text গত ২০ বছের িতিন রািশয়ার েপৰ্িসেডন্...
Prompts
Input Template:
একিট িনবেন্ধর নীেচর িশেরানাম এবং সারাংশ েদওয়া, একিট েছাট িনবন্ধ ৈতির করুন বা তােদর সােথ েযেত একিট দীঘর্ িনবেন্ধর শুরু করুন। িশেরানাম: {{title}}সারাংশ: {{target}}
Target Template:
{{text[:500]}}
Input Template:
িবষয়বস্তু: {{text[:7000]}}
Target Template:
{{target}}
Input Template:
ডক সংিক্ষপ্ত করার জনয্: {{text[:8500]}}
Target Template:
{{target}}
Input Template:
...{{text[3000:3500]}}
Target Template:
{{text[5000:]}}
Input Template:
িশেরানাম: {{title}}
Target Template:
{{text[:7000]}}
Input Template:
{{text}}
Target Template:
{{title}}
Input Template:
িশেরানাম: {{title}}
Target Template:
{{text[:7000]}}
Input Template:
{{title}}{{text[:5000]}}
Target Template:
{{target}}
Input Template:
{{text[:1000]}}
Target Template:
{{text[1000:5000]}}
1.2.4 GEM/xlsum english
Data Example
Key Value gem_id xlsum_english-train-2 url https://www.bbc.com/news/uk-scotlan... title Huge tidal turbine installed at Ork... target The massive tidal turbine AK1000 ha... references The massive tidal turbine AK1000 ha... text Atlantis Resources unveiled the mar...
Prompts
Input Template:
Doc to summarize: {{text[:8500]}}\nSummary in the same language as the doc:
Target Template:
{{target}}
Input Template:
Content: {{text[:7000]}}\nThe previous content can be summarized as follows:
Target Template:
{{target}}
Input Template:
{{title}}\n{{text[:5000]}}\n\ntl;dr:
Target Template:
{{target}}
Input Template:
{{text}} \n\nGive me a good title for the article above.
Target Template:
{{title}}
Input Template:
Given the below title and summary of an article, generate a short article or the beginning of a long article to go along with them. Title: {{title}}\nSummary: {{target}}\nArticle (Max 500 characters):
Target Template:
{{text[:500]}}
Input Template:
Title: {{title}}\nGiven the above title of an imaginary article, imagine the article.\n
Target Template:
{{text[:7000]}}
Input Template:
Title: {{title}}\nGiven the above title of an imaginary article, imagine the article.\n
Target Template:
{{text[:7000]}}
Input Template:
{{text[:1000]}}... Continue the article for another 4000 characters max:
Target Template:
{{text[1000:5000]}}
Input Template:
...{{text[3000:3500]}}... Write the rest of the article:
Target Template:
{{text[5000:]}}
1.3 Translation
1.3.1 Helsinki-NLP/tatoeba_mt ben-eng
Dataset from Tiedemann (2020). Used in training.
Data Example
Key Value sourceLang ben targetlang eng sourceString Tatoebaর অথর্ কী? targetString What does "Tatoeba" mean?
Prompts
Input Template:
Translate the following text from English to Bengali {{ targetString }}
Target Template:
{{ sourceString }}
Input Template:
Translate the following text from Bengali to English {{ sourceString }}
Target Template:
{{ targetString }}
1.3.2 Helsinki-NLP/tatoeba_mt eng-fra
Data Example
Key Value sourceLang eng targetlang fra sourceString Aah. Now I understand. targetString Ah ! Maintenant, je comprends.
Prompts
Input Template:
Translate the following text from French to English {{ targetString }}
Target Template:
{{ sourceString }}
Input Template:
Translate the following text from English to French {{ sourceString }}
Target Template:
{{ targetString }}
1.3.3 facebook/flores ben_Beng-eng_Latn
Dataset from NLLB (2022). Used in training.
Data Example
Key Value id 2 URL https://en.wikinews.org/wiki/Scient... domain wikinews topic health has_image 0 has_hyperlink 0 sentence_ben_Beng শীষর্ গেবষকরা বলেছন, এিট িনম্ন-আেয়... sentence_eng_Latn Lead researchers say this may bring...
Prompts
Input Template:
{{sentence_ben_Beng}}
Target Template:
{{sentence_eng_Latn}}
Input Template:
A text in Bengali: {{sentence_ben_Beng}}
Target Template:
{{sentence_eng_Latn}}
Input Template:
{{sentence_ben_Beng}}
Target Template:
{{sentence_eng_Latn}}
1.3.4 facebook/flores ben_Beng-fra_Latn
Data Example
Key Value id 2 URL https://en.wikinews.org/wiki/Scient... domain wikinews topic health has_image 0 has_hyperlink 0 sentence_ben_Beng শীষর্ গেবষকরা বলেছন, এিট িনম্ন-আেয়... sentence_fra_Latn Selon les chercheurs principaux, ce...
Prompts
Input Template:
{{sentence_ben_Beng}}
Target Template:
{{sentence_fra_Latn}}
Input Template:
{{sentence_ben_Beng}}
Target Template:
{{sentence_fra_Latn}}
Input Template:
A text in Bengali: {{sentence_ben_Beng}}
Target Template:
{{sentence_fra_Latn}}
1.4 Program Synthesis
1.4.1 Muennighoff/mbpp sanitized
Dataset from Austin et al. (2021). Used in training.
Data Example
Prompts
Input Template:
Key Value source_file Benchmark Questions Verification V2... task_id 3 prompt Write a python function to identify... code import math
def is_not_prime(n): ...
test_imports test_list assert is_not_prime(2) == False;ass...
{{ prompt }} Here is a solution in Python:
Target Template:
{{ code }}
Note: the prompt does not correspond to the original task intended by the dataset authors.
Input Template:
{{ prompt }} This can be solved in Python with the following code:
Target Template:
{{ code }}
1.4.2 codeparrot/apps all
Dataset from Hendrycks et al. (2021). Used in training.
Data Example
Key Value problem_id 1 question Mikhail walks on a Cartesian plane.... solutions ["q=int(input())\n\nfor e in range(... input_output {
"inputs": [ "3\n2 2 3\n4 3 ...
difficulty interview url https://codeforces.com/problemset/p... starter_code
Prompts
Input Template:
Solve in Python: {{ question }}
Target Template:
{{ solution }}
Input Template:
{{ question }}
Can you solve the above problem using Python?
Target Template:
{{ solution }}
Input Template:
I found an interesting problem on {{url}}: {{ question }}
I tried it in Python, but could not do it. Can you solve it?
Target Template:
{{ solution }}
1.4.3 codeparrot/github-jupyter-text-code-pairs
Data Example
Key Value markdown Extract the dataset from the compre... code num_classes = 10 np.random.seed(133... path machine-learning/deep-learning/udac... repo_name pk-ai/training license mit
Prompts
Input Template:
"{{ markdown }}" Please write code following the instructions in jupyter notebook style.
Target Template:
{{ code }}
Input Template:
I am working on the file "{{ path }}". The first task is: {{ markdown }} Can you write Python code for it?
Target Template:
{{ code }}
Note: the prompt does not correspond to the original task intended by the dataset authors.
Input Template:
{{ markdown }}
Target Template:
{{ code }}
Note: the prompt does not correspond to the original task intended by the dataset authors.
Input Template:
{{ code }} Given the above code, generate some markdown instructions for it.
Target Template:
{{ markdown }}
1.4.4 codeparrot/xlcost-text-to-code C++-program-level
Dataset from Zhu et al. (2022). Used in training.
Data Example
Key Value text Check if a number can be represente... code #include <bits/stdc++.h> NEW_LINE u...
Prompts
Input Template:
"{{ text }}" Solution in C++:
Target Template:
{{ code_clean }}
Input Template:
"{{ text }}" How can the above be solved in C++?
Target Template:
{{ code_clean }}
1.4.5 codeparrot/xlcost-text-to-code C-program-level
Data Example
Key Value text Logarithm tricks for Competitive Pr... code #include <stdio.h> NEW_LINE #includ...
Prompts
Input Template:
"{{ text }}" Solution in C:
Target Template:
{{ code_clean }}
Input Template:
{{ text }} How can the above be solved in C?
Target Template:
{{ code_clean }}
1.4.6 codeparrot/xlcost-text-to-code Csharp-program-level
Data Example
Key Value text Check if a number can be represente... code using System ; class GFG { static b...
Prompts
Input Template:
"{{ text }}" Solution in C#:
Target Template:
{{ code_clean }}
Input Template:
"{{ text }}" How can the above be solved in C-Sharp?
Target Template:
{{ code_clean }}
1.4.7 codeparrot/xlcost-text-to-code Java-program-level
Data Example
Key Value text Check if a number can be represente... code import java . io . * ; class GFG { ...
Prompts
Input Template:
"{{ text }}" Solution in Java:
Target Template:
{{ code_clean }}
Input Template:
"{{ text }}" How can the above be solved in Java?
Target Template:
{{ code_clean }}
1.4.8 codeparrot/xlcost-text-to-code Javascript-program-level
Data Example
Key Value text Check if a number can be represente... code function sumOfTwoCubes ( n ) { var ...
Prompts
Input Template:
"{{ text }}" Solution in Javascript:
Target Template:
{{ code_clean }}
Input Template:
"{{ text }}" How can the above be solved in JS?
Target Template:
{{ code_clean }}
1.4.9 codeparrot/xlcost-text-to-code PHP-program-level
Data Example
Key Value text Rearrange the array to maximize the... code < ? php function solve ( $ a , $ n ...
Prompts
Input Template:
"{{ text }}" Solution in php:
Target Template:
{{ code_clean }}
Input Template:
"{{ text }}" How can the above be solved in PHP?
Target Template:
{{ code_clean }}
1.4.10 codeparrot/xlcost-text-to-code Python-program-level
Data Example
Key Value text Check if a number can be represente... code import math NEW_LINE def sumOfTwoCu...
Prompts
Input Template:
"{{ text }}" Solution in Python:
Target Template:
{{ code_clean }}
Input Template:
"{{ text }}" How can the above be solved in Python?
Target Template:
{{ code_clean }}
1.4.11 neural_code_search evaluation_dataset
Dataset from hug (2018). Used in training.
Data Example
Key Value stackoverflow_id 4616095 question How to get the build/version number... question_url https://stackoverflow.com/questions... question_author Fahad Ali Shaikh question_author_url https://stackoverflow.com/users/565... answer try { PackageInfo pInfo = this.ge... answer_url https://stackoverflow.com/a/6593822 answer_author plusanswer_author_url https://stackoverflow.com/users/709... examples 4130029;3398176;2320640 examples_url https://github.com/altanizio/Concei...
Prompts
Note: the prompt does not correspond to the original task intended by the dataset authors.
Input Template:
Description: {{ question }}
Implementation:
Target Template:
{{ answer }}
Note: the prompt does not correspond to the original task intended by the dataset authors.
Input Template:
Given the following code: {{ answer }} Describe it:
Target Template:
{{ question }}
1.4.12 teven/code_contests
Data Example
Key Value name 1575_A. Another Sorting Problem description Andi and Budi were given an assignm... source 2 difficulty 7 solution #include <bits/stdc++.h> using name... language CPP
Prompts
Input Template:
{{description}}
Target Template:
{{solution}}
Input Template:
Can you solve the below in {{language}}? {{description}}
Target Template:
{{solution}}
Input Template:
{{description}} The above is tricky. Write me a correct solution in {{language}}.
Target Template:
{{solution}}
Input Template:
{{description}} Solve the task in {{language}}.
Target Template:
{{solution}}
Input Template:
{{description}} Using {{language | lower}} can you solve the prior task?
Target Template:
{{solution}}
Input Template:
{{description}} {{solution[:5]}}
Target Template:
{{solution[5:]}}
Input Template:
{{language}} solution for "{{description}}":
Target Template:
{{solution}}
1.5 Coreference Resolution
1.5.1 Muennighoff/xwinograd en
Dataset from Tikhonov and Ryabinin (2021). Used in evaluation.
Data Example
Key Value sentence The city councilmen refused the dem... option1 The city councilmen option2 the demonstrators answer 2
Prompts
Input Template:
{{sentence}} Replace the _ in the above sentence with the correct option: - {{option1}} - {{option2}}
Target Template:
{% if answer == '1' %} {{option1}} {% else %} {{ option2 }} {% endif %}
Answer Choices Template:
{{option1}} ||| {{option2}}
Input Template:
Fill in the _ in the below sentence: {{sentence}}
Choices: - {{ option1 }} - {{ option2 }}
Answer:
Target Template:
{% if answer == '1' %} {{option1}} {% else %} {{ option2 }} {% endif %}
Answer Choices Template:
{{option1}} ||| {{option2}}
Note: the prompt does not correspond to the original task intended by the dataset authors.
Input Template:
The _ in the sentence below refers to {{option1}}. True or False? {{sentence}}
Target Template:
{{answer_choices[answer|int - 1]}}
Answer Choices Template:
True ||| False
Input Template:
{{ sentence }} In the previous sentence, does _ refer to {{ option1 }} or {{ option2 }}?
Target Template:
{% if answer == '1' %} {{option1}} {% else %} {{ option2 }} {% endif %}
Answer Choices Template:
{{ option1 }} ||| {{ option2 }}
Input Template:
{{sentence}} What does the _ in the above sentence refer to? {{ option1 }} or {{ option2 }}?
Target Template:
{% if answer == '1' %} {{option1}} {% else %} {{ option2 }} {% endif %}
Answer Choices Template:
{{option1}} ||| {{option2}}
Input Template:
In the sentence below, does the _ stand for {{answer_choices[0]}} or {{answer_choices[1]}}? {{sentence}}
Target Template:
{{answer_choices[answer | int - 1]}}
Answer Choices Template:
{{option1}} ||| {{option2}}
1.5.2 Muennighoff/xwinograd fr
Data Example
Key Value sentence La coupe n'entre pas dans la valise... option1 La coupe option2 la valise answer 2
Prompts
Input Template:
{{ sentence }} Dans la phrase précédente, _ fait-il référence à {{ option1 }} ou {{ option2 }} ?
Target Template:
{% if answer == '1' %} {{option1}} {% else %} {{ option2 }} {% endif %}
Answer Choices Template:
{{ option1 }} ||| {{ option2 }}
Input Template:
Dans la phrase ci-dessous, le _ signifie-t-il {{answer_choices[0]}} ou {{answer_choices[1]}} ? {{sentence}}
Target Template:
{{answer_choices[answer | int - 1]}}
Answer Choices Template:
{{option1}} ||| {{option2}}
Input Template:
{{sentence}} Remplacez le _ dans la phrase ci-dessus par l'option correcte : - {{option1}} - {{option2}}
Target Template:
{% if answer == '1' %} {{option1}} {% else %} {{ option2 }} {% endif %}
Answer Choices Template:
{{option1}} ||| {{option2}}
Input Template:
{{sentence}} À quoi le _ dans la phrase ci-dessus fait-il référence ? {{ option1 }} ou {{ option2 }} ?
Target Template:
{% if answer == '1' %} {{option1}} {% else %} {{ option2 }} {% endif %}
Answer Choices Template:
{{option1}} ||| {{option2}}
Input Template:
Le _ dans la phrase ci-dessous fait référence à {{option1}}. Vrai ou faux? {{sentence}}
Target Template:
{{answer_choices[answer|int - 1]}}
Answer Choices Template:
Vrai ||| Faux
1.6 Question Answering Multiple Choice
1.6.1 clue c3
Dataset from Sun et al. (2020). Used in training.
Data Example
Key Value id 1 context 男：足球比赛是明天上午八点开始吧?;女：因为天气不好，比赛改到后天下午... question 根据对话，可以知道什么? choice 今天天气不好;比赛时间变了;校长忘了时间 answer 比赛时间变了
Prompts
Input Template:
{% for statement in context %} {{ statement }} {% endfor %} 鉴于上面的对话/段落，问题“{{question}}”的答案是什么
Target Template:
{{ answer }}
Input Template:
段落：{% for statement in context %} {{ statement }} {% endfor %} 什么样的问题会引起 {{ answer }} 的回答响应？
Target Template:
{{ question }}
Input Template:
{% for statement in context %} {{ statement }} {% endfor %} Given the dialogue / passage above, use the following options to answer the question "{{question}}". Options: - {{ answer_choices | join('\n- ') }}
Target Template:
{{ answer }}
Answer Choices Template:
{{ choice | join(" ||| ") }}
Input Template:
{% for statement in context %} {{ statement }} {% endfor %} 鉴于上面的对话/段落，使用以下选项回答问题“{{question}}”。 选项： - {{ answer_choices | join(' - ') }}
Target Template:
{{ answer }}
Answer Choices Template:
{{ choice | join(" ||| ") }}
Input Template:
Passage: {% for statement in context %} {{ statement }} {% endfor %} Question: "{{question}}" Answer choices: {{ answer_choices[:-1] | join(', ') }}, or {{ answer_choices[-1] }}?
Target Template:
{{ answer }}
Answer Choices Template:
{{ choice | join(" ||| ") }}
Note: the prompt does not correspond to the original task intended by the dataset authors.
Input Template:
Passage: {% for statement in context %} {{ statement }} {% endfor %} What kind of question would elicit an answer response of {{ answer }}?
Target Template:
{{ question }}
Input Template:
段落：{% for statement in context %} {{ statement }} {% endfor %} 问题：“{{question}}” 答案选择：{{ answer_choices[:-1] | join(', ') }} 还是 {{ answer_choices[-1] }}？
Target Template:
{{ answer }}
Answer Choices Template:
{{ choice | join(' ||| ') }}
Input Template:
{% for statement in context %} {{ statement }} {% endfor %} Given the dialogue / passage above, what is the answer for the question "{{question}}" Answer choices: {{ answer_choices[:-1] | join(', ') }}, or {{ answer_choices[-1] }}?
Target Template:
{{ answer }}
Answer Choices Template:
{{ choice | join(' ||| ') }}
Input Template:
{% for statement in context %} {{ statement }} {% endfor %} 鉴于上面的对话/段落，问题“{{question}}”的答案是什么 答案选择：{{ answer_choices[:-1] | join(', ') }} 还是 {{ answer_choices[-1] }}？
Target Template:
{{ answer }}
Answer Choices Template:
{{ choice | join(' ||| ') }}
Note: the prompt does not correspond to the original task intended by the dataset authors.
Input Template:
{% for statement in context %} {{ statement }} {% endfor %} Given the dialogue / passage above, what is the answer for the question "{{question}}"
Target Template:
{{ answer }}
1.7 Question Answering Extractive
1.7.1 clue cmrc2018
Dataset from Cui et al. (2018). Used in training.
Data Example
Key Value id TRAIN_186_QUERY_1 context 范廷颂枢机（，），圣名保禄·若瑟（），是越南罗马天主教枢机。1963年... question 1990年，范廷颂担任什么职务？ answers {'text': ['1990年被擢升为天主教河内总教区宗座署理'],...
Prompts
Input Template:
问：{{ question }}你能写一些上下文来回答这个问题吗？
Target Template:
{{ context }}
Note: the prompt does not correspond to the original task intended by the dataset authors.
Input Template:
Given this context "{{ context }}", generate a question that would return the answer of "{{ answers['text'][0] }}".
Target Template:
{{ question }}
Input Template:
{{ context }} {{ question }} 的答案在上面的段落中。它是什么？
Target Template:
{{ answers['text'][0] }}
Input Template:
In an exam, you are asked {{ question }}, and you are tasked to find the answer from the following passage. {{ context }} What's the answer?
Target Template:
{{ answers['text'][0] }}
Input Template:
{{ context }} The answer to {{ question }} is in the passage above. What is it?
Target Template:
{{ answers['text'][0] }}
Input Template:
Answer the question using the given context. Question: {{ question }} Context: {{ context }} Answer:
Target Template:
{{ answers['text'][0] }}
Input Template:
Q: {{ question }} Can you write some context to answer the question?
Target Template:
{{ context }}
Input Template:
{{ context[:answers["answer_start"][0]-5] }}... How would you continue the prior text to answer "{{ question }}"?
Target Template:
{{ context[answers["answer_start"][0]-5:] }}
Input Template:
{{ context[:answers["answer_start"][0]-5] }}... 你将如何继续前面的文本来回答“{{ question }}”？
Target Template:
{{ context[answers["answer_start"][0]-5:] }}
Input Template:
使用给定的上下文回答问题。 问题：{{ question }} 上下文：{{ context }} 答案：
Target Template:
{{ answers['text'][0] }}
Input Template:
在考试中，你被问到 {{ question }}，你的任务是从以下段落中找到答案。 {{ context }} 答案是什么？
Target Template:
{{ answers['text'][0] }}
Input Template:
给定这个上下文“{{ context }}”，生成一个返回“{{ answers['text'][0] }}”答案的问题。
Target Template:
{{ question }}
1.7.2 clue drcd
Dataset from Xu et al. (2020). Used in training.
Data Example
Key Value id 1001-10-2 context 2010年引進的廣州快速公交運輸系統，屬世界第二大快速公交系統，日常載... question 從哪一天開始在廣州市�騎摩托車會被�收？ answers {'text': ['2007年1月16日'], 'answer_st...
Prompts
Input Template:
{{ context }} {{ question }} 的答案在上面的段落中。它是什么？
Target Template:
{{ answers['text'][0] }}
Input Template:
Answer the question using the given context. Question: {{ question }} Context: {{ context }} Answer:
Target Template:
{{ answers['text'][0] }}
Input Template:
{{context[:answers["answer_start"]-5]}}... 你将如何继续前面的文本来回答“{{ question}}”？
Target Template:
{{context[answers["answer_start"]-5:]}}
Input Template:
在考试中，你被问到 {{ question }}，你的任务是找到回答问题的段落。写这样一段话：
Target Template:
{{ context }}
Input Template:
{{ context }} The answer to {{ question }} is in the passage above. What is it?
Target Template:
{{ answers['text'][0] }}
Input Template:
在考试中，你被问到 {{ question }}，你的任务是从以下段落中找到答案。 {{ context }} 答案是什么？
Target Template:
{{ answers['text'][0] }}
Input Template:
给定这个上下文“{{ context }}”，生成一个返回“{{ answers['text'][0] }}”答案的问题。
Target Template:
{{ question }}
Input Template:
{{context[:answers["answer_start"]-5]}}... How would you continue the prior text to answer "{{ question }}"?
Target Template:
{{context[answers["answer_start"]-5:]}}
Input Template:
使用给定的上下文回答问题。 问题：{{ question }} 上下文：{{ context }} 答案：
Target Template:
{{ answers['text'][0] }}
Note: the prompt does not correspond to the original task intended by the dataset authors.
Input Template:
Given this context "{{ context }}", generate a question that would return the answer of "{{ answers['text'][0] }}".
Target Template:
{{ question }}
Input Template:
In an exam, you are asked {{ question }}, and you are tasked to find the answer from the following passage. {{ context }} What's the answer?
Target Template:
{{ answers['text'][0] }}
Input Template:
In an exam, you are asked {{ question }}, and you are tasked to find a passage answering the question. Write such a passage:
Target Template:
{{ context }}
1.7.3 mlqa mlqa.vi.vi
Dataset from Lewis et al. (2019). Used in training.
Data Example
Key Value context Thành phố Miêu Lật tiếng Trung:苗栗市,... question Miaoli có tỷ lệ cao loại người nào? answers {'answer_start': [311], 'text': ['K... id 2f0d6ff162619164bb113c0cadbcca06a50...
Prompts
Input Template:
{{ context[:answers.answer_start[0]-5]}} ... Tiếp tục ở trên, sao cho nó trả lời "{{question}}":
Target Template:
{{ context[answers.answer_start[0]-5:]}}
Input Template:
{{context}}
Với sự tham chiếu đến ngữ cảnh trên, {{question}}
Target Template:
{{answers.text[0]}}
Input Template:
{{context}} H: {{question}} Đề cập đến đoạn văn trên, câu trả lời đúng cho câu hỏi đã cho trong ngôn ngữ của đoạn văn là
Target Template:
{{answers["text"][0]}}
Input Template:
Câu hỏi: {{question}} Ngữ cảnh: {{context}} Câu trả lời từ ngữ cảnh:
Target Template:
{{answers.text[0]}}
Input Template:
Tham khảo đoạn văn dưới đây và sau đó trả lời câu hỏi sau đó bằng ngôn ngữ tương tự như đoạn văn:
Đoạn: {{context}}
Câu hỏi: {{question}}
Target Template:
{{answers["text"][0]}}
Input Template:
Tôi đã tìm thấy một văn bản trả lời "{{question}}" bằng {{answers.text[0]}}. Nó bắt đầu bằng "{{ context[:10] }}". Bạn có thể tiếp tục nó không?
Target Template:
{{ context[10:] }}
Input Template:
Đọc đoạn văn sau và sau đó trả lời câu hỏi tiếp theo bằng cách trích một phần đúng trong đoạn văn: {{context}} {{question}}
Target Template:
{{answers.text[0]}}
Input Template:
D: {{context}}
H: {{question}}
A:
Target Template:
{{answers["text"][0]}}
1.7.4 mlqa mlqa.zh.zh
Data Example
Key Value context 楚河州包括有整个楚河河谷及邻近的山脉与峡谷。河谷的黑土非常肥沃，而且被... question 哪水体有助土地如此多产？ answers {'answer_start': [36], 'text': ['楚河... id 1aee17dd937cc1043e3ff47c38396541fc3...
Prompts
Input Template:
阅读下面的短文，然后从短文中选出正确的部分来回答下面的问题： {{context}} {{question}}
Target Template:
{{answers.text[0]}}
Input Template:
{{ context[:answers.answer_start[0]-5]}}... 继续上述操作，使其回答“{{question}}”：
Target Template:
{{ context[answers.answer_start[0]-5:]}}
Input Template:
D：{{context}}
问：{{question}}
答：
Target Template:
{{answers["text"][0]}}
Input Template:
我找到了一个用 {{answers.text[0]}} 回答“{{answers.text[0]}}”的文本。它以“{{ context[:10] }}”开头。可以继续吗？
Target Template:
{{ context[10:] }}
Input Template:
问题：{{question}} 上下文：{{context}} 从上下文中回答：
Target Template:
{{answers.text[0]}}
Input Template:
参考下面的段落，然后用与段落相同的语言回答问题：
段落：{{context}}
问题：{{question}}
Target Template:
{{answers["text"][0]}}
Input Template:
{{context}}
参考上述上下文，{{question}}
Target Template:
{{answers.text[0]}}
Input Template:
{{context}} 问：{{question}} 参考上面的段落，用该段落的语言对给定问题的正确答案是
Target Template:
{{answers["text"][0]}}
1.7.5 xquad xquad.vi
Dataset from Artetxe et al. (2019). Used in training.
Data Example
Key Value id 56beb4343aeaaa14008c925c context Đội thủ của Panthers chỉ thua 308 đ... question Jared Allen có bao nhiêu lần vật ng... answers {'text': ['136'], 'answer_start': [...
Prompts
Input Template:
{{context}}
Với sự tham chiếu đến ngữ cảnh trên, {{question}}
Target Template:
{{answers.text[0]}}
Input Template:
Đưa ra câu trả lời {{answers.text[0]}} cho {{question}}, hãy viết một văn bản giải thích điều này. Câu trả lời phải bắt đầu ở số ký tự {{answers.answer_start[0]}}. Văn bản:
Target Template:
{{context}}
Input Template:
{{question}} Rõ ràng là {{answers.text[0]}}. Bạn có thể cung cấp cho tôi một số bối cảnh?
Target Template:
{{context}}
Input Template:
{{context}}
H: {{question}}
Đề cập đến đoạn văn trên, câu trả lời chính xác cho câu hỏi được đưa ra là
Target Template:
{{answers["text"][0]}}
Input Template:
{{context}}
H: {{question}}
A:
Target Template:
{{answers["text"][0]}}
Input Template:
Đọc đoạn văn sau và trả lời câu hỏi sau: {{context}} {{question}}
Target Template:
{{answers.text[0]}}
Input Template:
Tham khảo đoạn văn dưới đây và trả lời câu hỏi sau:
Đoạn: {{context}}
Câu hỏi: {{question}}
Target Template:
{{answers["text"][0]}}
Input Template:
{{context}}
Từ đoạn văn trên, một câu hỏi hợp lý với "{{answers["text"][0]}}" như câu trả lời sẽ là:
Target Template:
{{question}}
Input Template:
{{context}}
Tạo câu hỏi từ đoạn văn trên:
Target Template:
{{question}}
1.7.6 xquad xquad.zh
Data Example
Key Value id 56beb4343aeaaa14008c925c context 黑豹队的防守只丢了 308分，在联赛中排名第六，同时也以 24 次拦截... question 贾里德在职业生涯中有多少次擒杀？ answers {'text': ['136 次'], 'answer_start':...
Prompts
Input Template:
阅读下面的短文，回答下面的问题： {{context}} {{question}}
Target Template:
{{answers.text[0]}}
Input Template:
{{context}}
问：{{question}}
参考上面的段落，给定问题的正确答案是
Target Template:
{{answers["text"][0]}}
Input Template:
参考下面的短文，回答下列问题：
段落：{{context}}
问题：{{question}}
Target Template:
{{answers["text"][0]}}
Input Template:
{{context}}
从上面的段落中，一个以“{{answers["text"][0]}}" 为答案的合理问题将是：
Target Template:
{{question}}
Input Template:
{{context}}
从上面的段落中产生一个问题：
Target Template:
{{question}}
Input Template:
{{context}}
参考上述上下文，{{question}}
Target Template:
{{answers.text[0]}}
Input Template:
{{context}}
问：{{question}}
答：
Target Template:
{{answers["text"][0]}}
1.8 Topic Classification
1.8.1 clue csl
Data Example
Key Value idx 1 corpus_id 2565 abst 针对核函数参数选择的重要性,提出了粒子群(PSO)模式搜索算法来搜索最... label -1 keyword 模式搜索;支持向量机;核参数选取
Prompts
Input Template:
After John wrote the abstract "{{abst}}", he wrote these keywords "{{ keyword | join(', ') }}". Do you think his choice of keywords was correct? Answer {{ answer_choices[1]}} or {{ answer_choices[0]}}.
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
no ||| yes
Input Template:
Do these keywords "{{ keyword | join(', ') }}" represent key concepts in the abstract "{{ abst }}"?
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
no ||| yes
Note: the prompt does not correspond to the original task intended by the dataset authors.
Input Template:
Given the abstract {{abst}}, list out {{ keyword | length }} keywords for it.
Target Template:
{% if label == 1 %} {{ keyword | join(', ') }} {% endif %}
Input Template:
一位学者使用“{{ keyword | join(', ') }}”作为搜索词。你认为搜索引擎会返回摘要 “{{abst}}”吗？回答 {{ answer_choices[1] }} 或 {{ answer_choices[0] }}。
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
不 ||| 是的
Input Template:
给定抽象{{abst}}，列出{{ keyword | length }} 关键字。
Target Template:
{% if label == 1 %} {{ keyword | join(', ') }} {% endif %}
Input Template:
写一篇关于“{{ keyword | join(', ') }}”的摘要：
Target Template:
{% if label == 1 %} {{abst}} {% endif %}
Answer Choices Template:
不 ||| 是的
Input Template:
在约翰写完摘要“{{abst}}”之后，他写了这些关键字“{{ keyword | join(', ') }}”。你认为 他选择的关键词是正确的吗？回答 {{ answer_choices[1]}} 或 {{ answer_choices[0]}}。
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
不 ||| 是的
Input Template:
这些关键字“{{ keyword | join(', ') }}”是否代表抽象“{{ abst }}”中的关键概念？
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
不 ||| 是的
Input Template:
A scholar used "{{ keyword | join(', ') }}" as search terms. Do you think the search engine would return the abstract "{{abst}}"? Answer {{ answer_choices[1] }} or {{ answer_choices[0] }}.
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
no ||| yes
Input Template:
Write an abstract about "{{ keyword | join(', ') }}":
Target Template:
{% if label == 1 %} {{abst}} {% endif %}
Answer Choices Template:
no ||| yes
1.8.2 clue tnews
Data Example
Prompts
Input Template:
Key Value sentence 买套房不香吗？为什么会有人愿花600万买部手机？ label -1 idx 1
将标题“{{ sentence }}”分为以下主题： - {{ answer_choices | join('\n- ') }} 主题：
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
故事 ||| 文化 ||| 娱乐 ||| 运动的 ||| 金融 ||| 房地产 ||| 车 ||| 教育 ||| 技术 ||| 军队 ||| 旅行 ||| 世界新闻 ||| 股票 ||| 农业 ||| 游戏
Input Template:
Classify the title "{{ sentence }}" into the following topics: - {{ answer_choices | join('\n- ') }} Topic:
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
story ||| culture ||| entertainment ||| sports ||| finance ||| real estate ||| car ||| education ||| tech ||| military ||| travel ||| world news ||| stock ||| agriculture ||| game
Input Template:
Given the topics of {{answer_choices[:-1] | join(', ') }}, and {{ answer_choices[-1] }}, specify which of them best represents the following sentence: {{ sentence }}
Best:
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
story ||| culture ||| entertainment ||| sports ||| finance ||| real estate ||| car ||| education ||| tech ||| military ||| travel ||| world news ||| stock ||| agriculture ||| game
Input Template:
以下新闻标题“{{ sentence }}”属于什么主题？ {{ answer_choices[0] | capitalize }}, {{ answer_choices[1:-1] | join(', ') }} 还是 {{ answer_choices[-1] }}？
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
故事 ||| 文化 ||| 娱乐 ||| 运动的 ||| 金融 ||| 房地产 ||| 车 ||| 教育 ||| 技术 ||| 军队 ||| 旅行 ||| 世界新闻 ||| 股票 ||| 农业 ||| 游戏
Input Template:
鉴于 {{answer_choices[:-1] | join(', ') }} 和 {{ answer_choices[-1] }}，指定它们中 的哪一个最能代表以下句子： {{ sentence }}
最佳：
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
故事 ||| 文化 ||| 娱乐 ||| 运动的 ||| 金融 ||| 房地产 ||| 车 ||| 教育 ||| 技术 ||| 军队 ||| 旅行 ||| 世界新闻 ||| 股票 ||| 农业 ||| 游戏
Input Template:
What topic does the following news title "{{ sentence }}" belong to? {{ answer_choices[0] | capitalize }}, {{ answer_choices[1:-1] | join(', ') }}, or {{ answer_choices[-1] }}?
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
story ||| culture ||| entertainment ||| sports ||| finance ||| real estate ||| car ||| education ||| tech ||| military ||| travel ||| world news ||| stock ||| agriculture ||| game
1.9 Code Misc.
1.9.1 codeparrot/codecomplex codeparrot–codecomplex
Data Example
Key Value src import java.util.Scanner; public ... complexity linear problem 1197_B. Pillars from CODEFORCES
Prompts
Input Template:
{{ code }} What is the time complexity of the previous code?
Target Template:
{{ complexity }}
Note: the prompt does not correspond to the original task intended by the dataset authors.
Input Template:
Identify the time complexity of the following code as constant, linear, quadratic, cubic, log(n), nlog(n) or NP-hard. {{ code }} Complexity:
Target Template:
{{ complexity }}
Note: the prompt does not correspond to the original task intended by the dataset authors.
Input Template:
{{ code }} Which one is the correct time complexity of the code snippet: constant, linear, quadratic, cubic, log(n), nlog(n) or NP-hard?
Target Template:
{{ complexity }}
1.9.2 great_code
Dataset from Hellendoorn et al. (2020). Used in training.
Data Example
Key Value id 1 source_tokens #NEWLINE#;def test_get_params(;self... has_bug True error_location 76 repair_candidates 2;76;4;11;18;22;30;40;48;58;66;80;8... bug_kind 1 bug_kind_name VARIABLE_MISUSE repair_targets 4;11;18;22;30;40;48;58;66;80;88;103... edges [{'before_index': 1, 'after_index':... provenances {'datasetProvenance': {'datasetName...
Prompts
Note: the prompt does not correspond to the original task intended by the dataset authors.
Input Template:
{% set mask = 'def <FUNC_NAME> (' %} {% set indent = ' ' %} {% set ns = namespace(indent_size=0, result=[], masked=false, target='') %} {% for token in source_tokens %}
{% if ns.masked is false and token.startswith('def') %} {% set ns.target = token.split('def ')[1][:-1] %} {% set token = mask %} {% set ns.masked = true %} {% endif%} {% if token== '#INDENT#' %}
{% set ns.indent_size = ns.indent_size + 1 %} {% set ns.result = ns.result + [indent * ns.indent_size] %}
{% elif token == '#NEWLINE#' %} {% set ns.result = ns.result + ["\n"] %} {% elif token == '#UNINDENT#' %} {% set ns.indent_size = ns.indent_size - 1 %} {% else %} {% if not loop.first and loop.previtem == '#NEWLINE#' %}
{% set ns.result = ns.result + [indent * ns.indent_size] %} {% endif %} {% set ns.result = ns.result + [token | replace('\\n', '\n'), " "] %}
{% endif %} {% endfor %} {{ns.result | join("") | replace(" . ", ".") | replace(" , ", ", ") | replace("( ", "(") | replace(" )", ")") | replace("[ ", "[") | replace(" ]", "]")}}
What is the function name?
Target Template:
{{ ns.target }}
Note: the prompt does not correspond to the original task intended by the dataset authors.
Input Template:
{% set result = "" %} {% set indent = ' ' %} {% set ns = namespace(indent_size=0, line_number=0, buggy_line=0, bug_location=0, bug_len=0, result=[], result_lines=[]) %} {% set fixed_token = source_tokens[repair_targets[0]] %} {% set buggy_line_content = "" %} {% set fixed_buggy_line_content = "" %}
{% if has_bug and (repair_targets | length > 0) %} {% for token in source_tokens %}
{% if loop.index0 == error_location %} {% set ns.buggy_line = ns.line_number %} {% set ns.bug_location = (ns.result | join("") | length) %} {% set ns.bug_len = (token | length) %} {% endif%} {% if token== '#INDENT#' %}
{% set ns.indent_size = ns.indent_size + 1 %} {% set ns.result = ns.result + [indent * ns.indent_size] %}
{% elif token == '#NEWLINE#' %} {% set ns.result_lines = ns.result_lines + [ns.result | join("")] %} {% set ns.result = [] %} {% set ns.line_number = ns.line_number + 1 %} {% elif token == '#UNINDENT#' %} {% set ns.indent_size = ns.indent_size - 1 %} {% else %} {% if not loop.first and loop.previtem == '#NEWLINE#' %}
{% set ns.result = ns.result + [indent * ns.indent_size] %} {% endif %} {% set ns.result = ns.result + [token | replace('\\n', '\n'), " "]
%} {% endif %}
{% endfor %} {% set ns.result_lines = ns.result_lines + [ns.result | join("")] %} {% set result = ns.result_lines | join("\n") %} {{result | replace(" . ", ".") | replace(" , ", ", ") | replace("( ", "(") |
replace(" )", ")") | replace("[ ", "[") | replace(" ]", "]")}}
{% set buggy_line_content = ns.result_lines[ns.buggy_line] | trim | replace(" . ", ".") | replace(" , ", ", ") | replace("( ", "(") | replace(" )", ")") | replace("[ ", "[") | replace(" ]", "]") %}
{% set fixed_buggy_line_content = (ns.result_lines[ns.buggy_line][:ns.bug_location] + fixed_token + ns.result_lines[ns.buggy_line][ns.bug_location + ns.bug_len:]) | trim | replace(" . ", ".") | replace(" , ", ", ") | replace("( ", "(") | replace(" )", ")") | replace("[ ", "[") | replace(" ]", "]")%}
Fix the buggy line: {{buggy_line_content}}
Target Template:
{{fixed_buggy_line_content}} {% endif %}
Input Template:
{% set mask = '<MASK>' %} {% set indent = ' ' %} {% set ns = namespace(indent_size=0, result=[]) %}
{% if has_bug %} {% for token in source_tokens %}
{% if loop.index0 == error_location %} {% set token = mask %} {% endif%} {% if token== '#INDENT#' %}
{% set ns.indent_size = ns.indent_size + 1 %} {% set ns.result = ns.result + [indent * ns.indent_size] %}
{% elif token == '#NEWLINE#' %} {% set ns.result = ns.result + ["\n"] %} {% elif token == '#UNINDENT#' %} {% set ns.indent_size = ns.indent_size - 1 %} {% else %} {% if not loop.first and loop.previtem == '#NEWLINE#' %}
{% set ns.result = ns.result + [indent * ns.indent_size] %} {% endif %} {% set ns.result = ns.result + [token | replace('\\n', '\n'), " "]
%} {% endif %}
{% endfor %} {{ns.result | join("") | replace(" . ", ".") | replace(" , ", ", ") |
replace("( ", "(") | replace(" )", ")") | replace("[ ", "[") | replace(" ]", "]")}}
Given the code above, what is a proper replacement for {{mask}}?
Target Template:
{{source_tokens[repair_targets[0]]}} {% endif %}
Note: the prompt does not correspond to the original task intended by the dataset authors.
Input Template:
{% set indent = ' ' %} {% set ns = namespace(indent_size=0, result=[]) %} {% for token in source_tokens %}
{% if token== '#INDENT#' %} {% set ns.indent_size = ns.indent_size + 1 %} {% set ns.result = ns.result + [indent * ns.indent_size] %} {% elif token == '#NEWLINE#' %} {% set ns.result = ns.result + ["\n"] %} {% elif token == '#UNINDENT#' %} {% set ns.indent_size = ns.indent_size - 1 %} {% else %} {% if not loop.first and loop.previtem == '#NEWLINE#' %}
{% set ns.result = ns.result + [indent * ns.indent_size] %}
{% endif %} {% set ns.result = ns.result + [token | replace('\\n', '\n'), " "] %}
{% endif %} {% endfor %} {{ns.result | join("") | replace(" . ", ".") | replace(" , ", ", ") | replace("( ", "(") | replace(" )", ")") | replace("[ ", "[") | replace(" ]", "]")}}
Is there a bug in the code above?
Target Template:
{{ {True: "Yes", False: "No"}[has_bug] }}
Answer Choices Template:
Yes ||| No
Note: the prompt does not correspond to the original task intended by the dataset authors.
Input Template:
{% set mask = '<MASK>' %} {% set indent = ' ' %} {% set ns = namespace(indent_size=0, result=[]) %}
{% if has_bug %} {% for token in source_tokens %}
{% if loop.index0 == error_location %} {% set token = mask %} {% endif%} {% if token== '#INDENT#' %}
{% set ns.indent_size = ns.indent_size + 1 %} {% set ns.result = ns.result + [indent * ns.indent_size] %}
{% elif token == '#NEWLINE#' %} {% set ns.result = ns.result + ["\n"] %} {% elif token == '#UNINDENT#' %} {% set ns.indent_size = ns.indent_size - 1 %} {% else %} {% if not loop.first and loop.previtem == '#NEWLINE#' %}
{% set ns.result = ns.result + [indent * ns.indent_size] %} {% endif %} {% set ns.result = ns.result + [token | replace('\\n', '\n'), " "]
%} {% endif %}
{% endfor %} {{ns.result | join("") | replace(" . ", ".") | replace(" , ", ", ") |
replace("( ", "(") | replace(" )", ")") | replace("[ ", "[") | replace(" ]", "]")}}
Given the code above, what is a proper replacement for {{mask}}? Choose among: {{answer_choices | join(", ")}}
Target Template:
{{source_tokens[repair_targets[0]]}} {% endif %}
Answer Choices Template:
{% if has_bug %} {% set nss = namespace(choices=[]) %} {% for i in repair_candidates %} {% set nss.choices = nss.choices + [source_tokens[(i | int)]] %} {% endfor %} {{nss.choices | unique | join(" ||| ")}} {% endif %}
1.9.3 teven/code_docstring_corpus top_level
Data Example
Key Value desc 'XXX22: This has to be present' decl def XXX11(): bodies pass
Prompts
Input Template:
Complete the below {{decl}}
'''{{desc | replace(' ', '
')}}'''
Target Template:
{{bodies}}
Input Template:
I wrote the below code
{{bodies}}
What's a good function header?
Target Template:
{{decl}}
Input Template:
{{decl}}
Target Template:
"""{{desc | replace(' ', '
') | replace("'", '')}}""" {{bodies}}
1.10 Word Sense Disambiguation
1.10.1 pasinit/xlwic xlwic_en_zh
Dataset from Raganato et al. (2020). Used in training.
Data Example
Key Value id EN_1 context_1 We like to summer in the Mediterran... context_2 We summered in Kashmir. target_word summer pos V target_word_location_1 {'char_start': 11, 'char_end': 17} target_word_location_2 {'char_start': 3, 'char_end': 11} language EN label 1
Prompts
Input Template:
第 1 句：{{context_1}} 句子 2：{{context_2}}
确定单词“{{target_word}}”在两个句子中的使用是否相同。是还是不是？
Target Template:
{% if label != -1%} {{answer_choices[label]}} {% endif %}
Answer Choices Template:
不 ||| 是的
Input Template:
家庭作业
判断“{{target_word}}”这个词在以下两个句子中的含义是否相同。回答是或否。 {{context_1}} {{context_2}}
Target Template:
{% if label != -1%} {{answer_choices[label]}} {% endif %}
Answer Choices Template:
不 ||| 是的
Input Template:
{{context_1}} {{context_2}} {{target_word}} 的类似意义？
Target Template:
{% if label != -1%} {{answer_choices[label]}} {% endif %}
Answer Choices Template:
不 ||| 是的
Input Template:
“{{target_word}}”这个词有多种含义。第 1 句和第 2 句的意思相同吗？是还是不是？
第 1 句：{{context_1}} 句子 2：{{context_2}}
Target Template:
{% if label != -1%} {{answer_choices[label]}} {% endif %}
Answer Choices Template:
不 ||| 是的
Input Template:
确定以下两个句子中是否以相同的方式使用了单词“{{target_word}}”。 {{context_1}} {{context_2}}
Target Template:
{% if label != -1%} {{answer_choices[label]}} {% endif %}
Answer Choices Template:
不 ||| 是的
Input Template:
{{context_1}} {{context_2}} 问题：“{{target_word}}”这个词在上面两个句子中的含义是否相同？
Target Template:
{% if label != -1%} {{answer_choices[label]}} {% endif %}
Answer Choices Template:
不 ||| 是的
Input Template:
“{{target_word}}”这个词在这两个句子中是否具有相同的含义？是的，不是吗？ {{context_1}} {{context_2}}
Target Template:
{% if label != -1%} {{answer_choices[label]}} {% endif %}
Answer Choices Template:
不 ||| 是的
Input Template:
“{{target_word}}”这个词在这两个句子中是否具有相同的含义？ {{context_1}} {{context_2}}
Target Template:
{% if label != -1%} {{answer_choices[label]}} {% endif %}
Answer Choices Template:
不 ||| 是的
Input Template:
句子 A：{{context_1}} 句子 B：{{context_2}}
“{{target_word}}”在句子 A 和 B 中具有相似的含义。对还是错？
Target Template:
{% if label != -1%} {{answer_choices[label]}} {% endif %}
Answer Choices Template:
错误的 ||| 真的
Input Template:
{{context_1}} {{context_2}} 问题：“{{target_word}}”这个词在上面两个句子中的含义是否相同？是的，不是吗？
Target Template:
{% if label != -1%} {{answer_choices[label]}} {% endif %}
Answer Choices Template:
不 ||| 是的
1.10.2 pasinit/xlwic xlwic_fr_fr
Data Example
Key Value id FR_1 context_1 L’éclaircie est généralement une co... context_2 Améliorations utiles. target_word amélioration pos N target_word_location_1 {'char_start': 41, 'char_end': 53} target_word_location_2 {'char_start': 0, 'char_end': 13} language FR label 1
Prompts
Input Template:
{{context_1}} {{context_2}} Sens similaire de {{target_word}} ?
Target Template:
{% if label != -1%} {{answer_choices[label]}} {% endif %}
Answer Choices Template:
Non ||| Oui
Input Template:
Devoirs
Décidez si le mot "{{target_word}}" est utilisé avec le même sens dans les deux phrases suivantes. Répondez par oui ou non. {{context_1}} {{context_2}}
Target Template:
{% if label != -1%} {{answer_choices[label]}} {% endif %}
Answer Choices Template:
Non ||| Oui
Input Template:
Le mot "{{target_word}}" a-t-il le même sens dans ces deux phrases ? Oui Non? {{context_1}} {{context_2}}
Target Template:
{% if label != -1%} {{answer_choices[label]}} {% endif %}
Answer Choices Template:
Non ||| Oui
Input Template:
{{context_1}} {{context_2}} Question : Le mot '{{target_word}}' est-il utilisé dans le même sens dans les deux phrases ci-dessus ? Oui Non?
Target Template:
{% if label != -1%} {{answer_choices[label]}} {% endif %}
Answer Choices Template:
Non ||| Oui
Input Template:
Le mot "{{target_word}}" a plusieurs significations. A-t-il le même sens dans les phrases 1 et 2 ? Oui ou non?
Phrase 1 : {{context_1}} Phrase 2 : {{context_2}}
Target Template:
{% if label != -1%} {{answer_choices[label]}} {% endif %}
Answer Choices Template:
Non ||| Oui
Input Template:
Phrase 1 : {{context_1}} Phrase 2 : {{context_2}}
Déterminez si le mot "{{target_word}}" est utilisé dans le même sens dans les deux phrases. Oui ou non?
Target Template:
{% if label != -1%} {{answer_choices[label]}} {% endif %}
Answer Choices Template:
Non ||| Oui
Input Template:
Le mot "{{target_word}}" a-t-il le même sens dans ces deux phrases ? {{context_1}} {{context_2}}
Target Template:
{% if label != -1%} {{answer_choices[label]}} {% endif %}
Answer Choices Template:
Non ||| Oui
Input Template:
Phrase A : {{context_1}} Phrase B : {{context_2}}
"{{target_word}}" a une signification similaire dans les phrases A et B. Vrai ou faux ?
Target Template:
{% if label != -1%} {{answer_choices[label]}} {% endif %}
Answer Choices Template:
Faux ||| Vrai
Input Template:
Déterminez si le mot '{{target_word}}' est utilisé de la même manière dans les deux phrases ci-dessous. {{context_1}} {{context_2}}
Target Template:
{% if label != -1%} {{answer_choices[label]}} {% endif %}
Answer Choices Template:
Non ||| Oui
Input Template:
{{context_1}} {{context_2}} Question : Le mot '{{target_word}}' est-il utilisé dans le même sens dans les deux phrases ci-dessus ?
Target Template:
{% if label != -1%} {{answer_choices[label]}} {% endif %}
Answer Choices Template:
Non ||| Oui
1.11 Paraphrase Identification
1.11.1 paws-x en
Dataset from Yang et al. (2019). Used in training.
Data Example
Key Value id 2 sentence1 The NBA season of 1975 -- 76 was th... sentence2 The 1975 -- 76 season of the Nation... label 1
Prompts
Notes: Generalized prompt format, task_description-input.
Input Template:
Determine if the following two sentences paraphrase each other or not. Sent 1: {{sentence1}} Sent 2: {{sentence2}}
Target Template:
{{answer_choices[label]}}
Answer Choices Template:
No ||| Yes
Notes: Natural question.
Input Template:
Sentence 1: {{sentence1}} Sentence 2: {{sentence2}} Question: Do Sentence 1 and Sentence 2 express the same meaning? Yes or No?
Target Template:
{{answer_choices[label]}}
Answer Choices Template:
No ||| Yes
Notes: Generalized prompt format, context-question without any label.
Input Template:
{{sentence1}} Is that a paraphrase of the following sentence? {{sentence2}}?
Target Template:
{{answer_choices[label]}}
Answer Choices Template:
No ||| Yes
Notes: Natural Question without label.
Input Template:
Sentence 1: {{sentence1}} Sentence 2: {{sentence2}} Question: Can we rewrite Sentence 1 to Sentence 2?
Target Template:
{{answer_choices[label]}}
Answer Choices Template:
No ||| Yes
Notes: Generalized prompt format, context-question.
Input Template:
{{sentence1}} Is that a paraphrase of the following sentence? {{sentence2}}? Yes or No.
Target Template:
{{answer_choices[label]}}
Answer Choices Template:
No ||| Yes
Notes: Concatenation of sentence 1 and sentence 2.
Input Template:
Sentence 1: {{sentence1}} Sentence 2: {{sentence2}} Question: Does Sentence 1 paraphrase Sentence 2? Yes or No?
Target Template:
{{answer_choices[label]}}
Answer Choices Template:
No ||| Yes
Note: the prompt does not correspond to the original task intended by the dataset authors.
Notes: Create a generative paraphrase task.
Input Template:
{% if label == 1 %} Paraphrase the sentence: {{sentence1}}
Target Template:
{{sentence2}} {% endif %}
Notes: Concatenation of sentence 1 and sentence 2 without any label.
Input Template:
Sentence 1: {{sentence1}} Sentence 2: {{sentence2}} Question: Does Sentence 1 paraphrase Sentence 2?
Target Template:
{{answer_choices[label]}}
Answer Choices Template:
No ||| Yes
Notes: Natural question without label.
Input Template:
Sentence 1: {{sentence1}} Sentence 2: {{sentence2}} Question: Do Sentence 1 and Sentence 2 express the same meaning?
Target Template:
{{answer_choices[label]}}
Answer Choices Template:
No ||| Yes
Prompt from Brown et al. (2020) Notes: ANLI prompt format from Table G7 in the GPT3 paper Brown et al. (2020)
Input Template:
{{sentence1}} Question: {{sentence2}} True or False?
Target Template:
{{answer_choices[label]}}
Answer Choices Template:
False ||| True
Notes: Natural Question.
Input Template:
Sentence 1: {{sentence1}} Sentence 2: {{sentence2}} Question: Can we rewrite Sentence 1 to Sentence 2? Yes or No?
Target Template:
{{answer_choices[label]}}
Answer Choices Template:
No ||| Yes
Prompt from Brown et al. (2020) Notes: ANLI prompt format from Table G7 in the GPT3 paper Brown et al. (2020). Additionally added task information without any label.
Input Template:
{{sentence1}} Question: {{sentence2}} Paraphrase or not?
Target Template:
{{answer_choices[label]}}
Answer Choices Template:
No ||| Yes
1.11.2 paws-x es
Data Example
Key Value id 2 sentence1 La temporada de la NBA de 1975: 76 ... sentence2 La temporada 1975 - 76 de la Asocia... label 1
Prompts
Input Template:
Oración 1: {{sentence1}} Oración 2: {{sentence2}} Pregunta: ¿La oración 1 parafrasea la oración 2? ¿Si o no?
Target Template:
{{answer_choices[label]}}
Answer Choices Template:
No ||| Sí
Input Template:
{{sentence1}} Pregunta: {{sentence2}} ¿Parafrasear o no?
Target Template:
{{answer_choices[label]}}
Answer Choices Template:
No ||| Sí
Input Template:
{{sentence1}} ¿Es una paráfrasis de la siguiente oración? {{sentence2}}? Si o no.
Target Template:
{{answer_choices[label]}}
Answer Choices Template:
No ||| Sí
Input Template:
Oración 1: {{sentence1}} Oración 2: {{sentence2}} Pregunta: ¿La Oración 1 y la Oración 2 expresan el mismo significado?
Target Template:
{{answer_choices[label]}}
Answer Choices Template:
No ||| Sí
Input Template:
{% if label == 1 %} Parafrasea la oración: {{sentence1}}
Target Template:
{{sentence2}} {% endif %}
Input Template:
{{sentence1}} Pregunta: {{sentence2}} ¿Verdadero o falso?
Target Template:
{{answer_choices[label]}}
Answer Choices Template:
Falso ||| Verdadero
Input Template:
Oración 1: {{sentence1}} Oración 2: {{sentence2}} Pregunta: ¿La oración 1 parafrasea la oración 2?
Target Template:
{{answer_choices[label]}}
Answer Choices Template:
No ||| Sí
Input Template:
Determina si las siguientes dos oraciones se parafrasean entre sí o no. Enviado 1: {{sentence1}} Enviado 2: {{sentence2}}
Target Template:
{{answer_choices[label]}}
Answer Choices Template:
No ||| Sí
Input Template:
Oración 1: {{sentence1}} Oración 2: {{sentence2}} Pregunta: ¿La Oración 1 y la Oración 2 expresan el mismo significado? ¿Si o no?
Target Template:
{{answer_choices[label]}}
Answer Choices Template:
No ||| Sí
Input Template:
Oración 1: {{sentence1}} Oración 2: {{sentence2}} Pregunta: ¿Podemos reescribir la Oración 1 a la Oración 2? ¿Si o no?
Target Template:
{{answer_choices[label]}}
Answer Choices Template:
No ||| Sí
Input Template:
{{sentence1}} ¿Es una paráfrasis de la siguiente oración? {{sentence2}}?
Target Template:
{{answer_choices[label]}}
Answer Choices Template:
No ||| Sí
Input Template:
Oración 1: {{sentence1}} Oración 2: {{sentence2}} Pregunta: ¿Podemos reescribir la Oración 1 a la Oración 2?
Target Template:
{{answer_choices[label]}}
Answer Choices Template:
No ||| Sí
1.12 Sentence Completion
1.12.1 xcopa vi
Dataset from Ponti et al. (2020). Used in evaluation.
Data Example
Key Value premise Cô gái tìm thấy con bọ trong ngũ cố... choice1 Cô đổ sữa vào bát. choice2 Cô mất cảm giác ngon miệng. question effect label 1 idx 1 changed False
Prompts
Input Template:
{{ premise }}
Tôi đang lưỡng lự giữa hai lựa chọn. Giúp tôi chọn nguyên nhân {% if question == "cause" %} có khả năng xảy ra cao hơn: {% else %} effect: {% endif %} - {{choice1}} - {{choice2}}
Target Template:
{% if label != -1 %} {{ answer_choices[label] }} {% endif %}
Answer Choices Template:
{{choice1}} ||| {{choice2}}
Input Template:
{{ premise }}
Lựa chọn tốt nhất là gì? - {{choice1}} - {{choice2}}
Chúng tôi đang tìm kiếm {% if question == "cause" %} một nguyên nhân {% else %} một ảnh hưởng {% endif %}
Target Template:
{% if label != -1 %} {{answer_choices[label]}} {% endif %}
Answer Choices Template:
{{choice1}} ||| {{choice2}}
Input Template:
{{ premise }} {% if question == "cause" %} Điều này xảy ra vì ... {% else %} Do đó ... {% endif %} Giúp tôi chọn tùy chọn hợp lý hơn: - {{choice1}} - {{choice2}}
Target Template:
{% if label != -1 %} {{ answer_choices[label] }} {% endif %}
Answer Choices Template:
{{choice1}} ||| {{choice2}}
Input Template:
"{{ answer_choices[0] }}" hay "{{ answer_choices[1] }}"? {{ premise }} {% if question == "cause" %} bởi vì {% else %} nên {% endif %}
Target Template:
{% if label != -1 %} {{ answer_choices[label] }} {% endif %}
Answer Choices Template:
{{choice1 }} ||| {{choice2}}
Input Template:
{{ premise }}
Chọn nguyên nhân {% if question == "cause" %} hợp lý nhất: {% else %} effect: {% endif %} - {{choice1}} - {{choice2}}
Target Template:
{% if label != -1 %} {{ answer_choices[label] }} {% endif %}
Answer Choices Template:
{{choice1}} ||| {{choice2}}
1.12.2 xcopa zh
Data Example
Key Value premise 这个女孩在麦片粥中发现了一个虫子。 choice1 她向碗里倒了牛奶。 choice2 她没了食欲。 question effect label 1 idx 1 changed False
Prompts
Input Template:
{{ premise }} {% if question == "cause" %} 这是因为... {% else %} 结果... {% endif %} 帮助我选择更合理的选项： - {{choice1}} - {{choice2}}
Target Template:
{% if label != -1 %}{{ answer_choices[label] }}{%endif%}
Answer Choices Template:
{{choice1}} ||| {{choice2}}
Input Template:
{{ premise }}
选择最合理的 {% if question == "cause" %} 原因：{% else %} 效果：{% endif %} - {{choice1}} - {{choice2}}
Target Template:
{% if label != -1 %}{{ answer_choices[label] }}{%endif%}
Answer Choices Template:
{{choice1}} ||| {{choice2}}
Input Template:
“{{ answer_choices[0] }}”还是“{{ answer_choices[1] }}”？ {{ premise }} {% if question == "cause" %} 因为 {% else %} 所以 {% endif %}
Target Template:
{% if label != -1 %}{{ answer_choices[label] }}{% endif %}
Answer Choices Template:
{{choice1 }} ||| {{choice2}}
Input Template:
{{ premise }}
最好的选择是什么？ - {{choice1}} - {{choice2}}
我们正在寻找 {% if question == "cause" %} 一个原因 {% else %} 一个结果 {% endif %}
Target Template:
{% if label != -1 %}{{answer_choices[label]}}{%endif%}
Answer Choices Template:
{{choice1}} ||| {{choice2}}
Input Template:
{{ premise }}
我在两个选项之间犹豫不决。帮我选择更有可能的 {% if question == "cause" %} 原因：{% else %} 效果：{% endif %} - {{choice1}} - {{choice2}}
Target Template:
{% if label != -1 %}{{ answer_choices[label] }}{%endif%}
Answer Choices Template:
{{choice1}} ||| {{choice2}}
Input Template:
{{ premise }}
我正在考虑两个选项。请帮我最有可能的{% if question == "cause" %}导因：{% else %}后果： {% endif %} - {{choice1}} - {{choice2}}
Target Template:
{% if label != -1 %}{{ answer_choices[label] }}{%endif%}
Answer Choices Template:
{{choice1}} ||| {{choice2}}
Input Template:
{{ premise }} {% if question == "cause" %}这个会发生是因为... {% else %}结果是... {% endif %} 帮我挑选合适的选项： - {{choice1}} - {{choice2}}
Target Template:
{% if label != -1 %}{{ answer_choices[label] }}{%endif%}
Answer Choices Template:
{{choice1}} ||| {{choice2}}
Notes: Adapted from Perez et al. (2021) and Schick and Schütze (2020).
Input Template:
"{{ answer_choices[0] }}" 还是"{{ answer_choices[1] }}"? {{ premise }} {% if question == "cause" %}因为{% else %}所以{% endif %}
Target Template:
{% if label != -1 %}{{ answer_choices[label] }}{% endif %}
Answer Choices Template:
{{choice1}} ||| {{choice2}}
Input Template:
{{ premise }}
哪个是最好的答案? - {{choice1}} - {{choice2}}
我们正在考虑{% if question == "cause" %}起因{% else %}后果 {% endif %}
Target Template:
{% if label != -1 %}{{answer_choices[label]}}{%endif%}
Answer Choices Template:
{{choice1}} ||| {{choice2}}
Input Template:
{{ premise }}
请选择最贴切的答案： {% if question == "cause" %}导因:{% else %}结果: {% endif %} - {{choice1}} - {{choice2}}
Target Template:
{% if label != -1 %}{{ answer_choices[label] }}{%endif%}
Answer Choices Template:
{{choice1}} ||| {{choice2}}
1.13 Natural Language Inference
1.13.1 xnli en
Dataset from Conneau et al. (2018). Used in evaluation.
Data Example
Key Value premise you know during the season and i gu... hypothesis You lose the things to the followin... label 0
Prompts
Notes: Sanh et al. (2022)
Input Template:
Take the following as truth: {{premise}} Then the following statement: "{{hypothesis}}" is {{"true"}}, {{"false"}}, or {{"inconclusive"}}?
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
True ||| Inconclusive ||| False
Notes: Sanh et al. (2022)
Input Template:
{{premise}}
Question: Does this imply that "{{hypothesis}}"? Yes, no, or maybe?
Target Template:
{{answer_choices[label]}}
Answer Choices Template:
Yes ||| Maybe ||| No
Notes: Same as reported in Figure G7 of Brown et al. (2020), except that there is no task identifying tokens like ”anli R1: ”.
Input Template:
{{premise}} Question: {{hypothesis}} True, False, or Neither?
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
True ||| Neither ||| False
Notes: Sanh et al. (2022)
Input Template:
Given that {{premise}} Does it follow that {{hypothesis}} Yes, no, or maybe?
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
Yes ||| Maybe ||| No
Notes: Adapted from the BoolQ prompts in Schick and Schütze (2020).
Input Template:
{{premise}} Based on the previous passage, is it true that "{{hypothesis}}"? Yes, no, or maybe?
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
Yes ||| Maybe ||| No
Notes: Webson and Pavlick (2021)
Input Template:
Given {{premise}} Is it guaranteed true that "{{hypothesis}}"? Yes, no, or maybe?
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
Yes ||| Maybe ||| No
Notes: Webson and Pavlick (2021)
Input Template:
Given {{premise}} Should we assume that "{{hypothesis}}" is true? Yes, no, or maybe?
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
Yes ||| Maybe ||| No
Notes: Sanh et al. (2022)
Input Template:
Given that {{premise}} Therefore, it must be true that "{{hypothesis}}"? Yes, no, or maybe?
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
Yes ||| Maybe ||| No
Notes: Webson and Pavlick (2021)
Input Template:
Suppose {{premise}} Can we infer that "{{hypothesis}}"? Yes, no, or maybe?
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
Yes ||| Maybe ||| No
Notes: Webson and Pavlick (2021)
Input Template:
{{premise}} Are we justified in saying that "{{hypothesis}}"? Yes, no, or maybe?
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
Yes ||| Maybe ||| No
Notes: Sanh et al. (2022)
Input Template:
{{premise}} Based on that information, is the claim: "{{hypothesis}}" {{"true"}}, {{"false"}}, or {{"inconclusive"}}?
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
True ||| Inconclusive ||| False
Notes: Sanh et al. (2022)
Input Template:
{{premise}}
Keeping in mind the above text, consider: {{hypothesis}} Is this {{"always"}}, {{"sometimes"}}, or {{"never"}} correct?
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
Always ||| Sometimes ||| Never
Notes: Sanh et al. (2022)
Input Template:
Suppose it's true that {{premise}} Then, is "{{hypothesis}}" {{"always"}}, {{"sometimes"}}, or {{"never"}} true?
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
Always ||| Sometimes ||| Never
Notes: Sanh et al. (2022)
Input Template:
Assume it is true that {{premise}}
Therefore, "{{hypothesis}}" is {{"guaranteed"}}, {{"possible"}}, or {{"impossible"}}?
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
Guaranteed ||| Possible ||| Impossible
Notes: Adapted from Williams et al. (2018) instructions to crowdsourcing workers.
Input Template:
{{premise}} Using only the above description and what you know about the world, "{{hypothesis}}" is definitely correct, incorrect, or inconclusive?
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
Correct ||| Inconclusive ||| Incorrect
1.13.2 xnli es
Data Example
Key Value premise Usted sabe durante la temporada y s... hypothesis Pierdes las cosas al siguiente nive... label 0
Prompts
1.13.2.1 Human-translated prompts
Notes: Same as reported in Figure G7 of Brown et al. (2020), except that there is no task identifying tokens like ”anli R1: ”.
Input Template:
{{premise}} Pregunta: {{hypothesis}} Verdadero, Falso, o Ninguno?
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
Verdadero ||| Ninguno ||| Falso
Notes: Webson and Pavlick (2021)
Input Template:
Supongamos {{premise}} Podemos inferir que "{{hypothesis}}"? Si, no, o tal vez?
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
Sí ||| Tal vez ||| No
Notes: Webson and Pavlick (2021)
Input Template:
{{premise}} Estamos justificados en decir que "{{hypothesis}}"? Si, no, o tal vez?
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
Sí ||| Tal vez ||| No
Notes: Sanh et al. (2022)
Input Template:
Supongamos que es cierto que {{premise}}
por lo tanto, "{{hypothesis}}" es {{"garantizado"}}, {{"posible"}}, o {{"imposible"}}?
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
Garantizado ||| Posible ||| Imposible
Notes: Adapted from Williams et al. (2018) instructions to crowdsourcing workers.
Input Template:
{{premise}} Usando solo la descripción anterior y lo que sabe sobre el mundo, "{{hypothesis}}" es definitivamente correcto, incorrecto o no concluyente?
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
Correcto ||| No concluyente ||| Incorrecto
1.13.2.2 Machine-translated prompts
Input Template:
{{premise}} ¿Estamos justificados al decir que &quot;{{hypothesis}}&quot;? ¿Sí, no o tal vez?
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
Sí ||| Quizás ||| No
Input Template:
{{premise}} Pregunta: {{hypothesis}} ¿Verdadero, falso o ninguno?
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
Verdadero ||| Ninguno de los dos ||| Falso
Input Template:
{{premise}} Usando solo la descripción anterior y lo que sabe sobre el mundo, &quot;{{hypothesis}}&quot; es definitivamente correcta, incorrecta o no concluyente.
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
Correcto ||| Poco concluyente ||| Incorrecto
Input Template:
Supongamos {{premise}} ¿Podemos inferir que &quot;{{hypothesis}}&quot;? ¿Sí, no o tal vez?
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
Sí ||| Quizás ||| No
Input Template:
Supongamos que es cierto que {{premise}} Por lo tanto, &quot;{{hypothesis}}&quot; es {{"guaranteed"}}, {{"possible"}} o {{"impossible"}}.
Target Template:
{{ answer_choices[label] }}
Answer Choices Template:
garantizado ||| Posible ||| Imposible
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Abstract & Section 1.
7 A2. Did you discuss any potential risks of your work? We did not find our work to have major potential risks.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract & Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3 & Appendix D
3 B1. Did you cite the creators of artifacts you used? Section 3 & Appendix D
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Appendix D
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? The artifacts used did not detail their intended use except for license compliance, which we adhere to. The intended use of our artifacts is clear from their accompanying licenses.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? As we reuse existing datasets (ROOTS, XCOPA, etc), we refer to the original works for such information.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? We explain the languages covered by our artifacts in Section 3.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Dataset sizes are reported in Figures 1 and 2, as well as the accompanying text.
C 3 Did you run computational experiments? Sections 3 & 4.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Model parameters and infrastructure details are reported in Section 3.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Hyperparamters are reported in Section 3.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Results are reported in Figures 4 - 8 and Tables 1 - 2 as well as in the Appendix.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We report parameter settings in Section 3 and will provide code for reproduction of all our experiments.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Future work should investigate multitask finetuning better base models. Learning new languages during finetuning While we have investigated generalization to languages only seen during pretraining, we did not investigate generalization to languages only seen during finetuning. Our mT0 models are finetuned on several new languages not seen in pretraining (see Figure 2). Out of those, we only evaluated on code (HumanEval), where mT0 performed at the random baseline (0. 00 in Table 10). We point to follow-up work that has investigated the question of teaching BLOOMZ new languages (Yong et al. , 2022; Cahyawijaya et al. , 2023) and work investigating adaptation of BLOOM (Ennen et al. , 2023; Yong and Nikoulina, 2022)