Benchmarking Large Language Model Capabilities for Conditional Generation
Year: 2023

Abstract:
Pre-trained large language models (PLMs) underlie most new developments in natural language processing. They have shifted the field from application-specific model pipelines to a single model that is adapted to a wide range of tasks. Autoregressive PLMs like GPT-3 or PaLM, alongside techniques like few-shot learning, have additionally shifted the output modality to generation instead of classification or regression. Despite their ubiquitous use, the generation quality of language models is rarely evaluated when these models are introduced. Additionally, it is unclear how existing generation tasks—-while they can be used to compare systems at a high level—relate to the real world use cases for which people have been adopting them. In this work, we discuss how to adapt existing applicationspecific generation benchmarks to PLMs and provide an in-depth, empirical study of the limitations and capabilities of PLMs in natural language generation tasks along dimensions such as scale, architecture, input and output language. Our results show that PLMs differ in their applicability to different data regimes and their generalization to multiple languages and inform which PLMs to use for a given generation task setup. We share best practices to be taken into consideration when benchmarking generation capabilities during the development of upcoming PLMs.

heading: 1 Introduction
text: Natural language generation tasks require generating understandable text given textual or nonlinguistic information as input, such as documents, tables, or other structured forms. These texts seek to achieve a communicative goal (e.g., summarize a document). The standard approach to tackle these problems over the last years has been to start with a pretrained encoder-decoder model like T5 (Raffel et al., 2020a) or BART (Lewis et al., 2020a) and finetune it on a corpus that captures
the downstream task. The recent much larger pretrained language models use a decoder-only architecture and upended this paradigm. These models enabled few-shot or in-context learning approaches in which a model is presented with one or more examples and tasked to continue generating without any finetuning. We refer to both kinds of pretrained models as PLMs. Due to the lack of grounding in the specific task setup, few-shot learning in generation settings leads to a model approaching the communicative goal from very different angles. These diverse range of outputs make the typical reference-based automatic evaluation strategies largely incompatible. While human evaluation can be used to overcome this shortcoming, it is infeasible to monitor the performance of an actively training model this way or to re-run all evaluations every time a new model is introduced. This leads to the question how one should reliably monitor generation capabilities, a question that is only growing in importance as more tasks are approached by casting them into generation setups. In this work, we evaluate 8 models in few-shot and finetuning settings on 27 generation tasks covering 14 languages via automatic evaluation, presenting the first large-scale benchmark of PLMs in conditional NLG settings. We discuss design choices and challenges to ensure a fair comparison between the different systems, including suitable methods, tasks, and metrics. Based on our empirical results, we derive recommendations that could be used for future benchmarks during the development of PLMs. To combat the need for repeating computationally expensive explorations, we investigate how many evaluation examples are necessary to identify differences between models and find that, in many cases, fewer than 500 examples are sufficient, which opens the path for future evaluation-only task developments.
9194

heading: 2 Background and Related Work
text: The shift from specialized pipelines toward pretrained language models has led to significant changes in how models are evaluated. We now focus more on questions such as “how good are the learned representations?” instead of user-facing measures of utility. The changes manifested in leaderboards and standard benchmarks that aim to characterize a wide range of model capabilities (Ethayarajh and Jurafsky, 2020). An additional recent shift is that from finetuning toward few-shot learning. Models like T5 (Raffel et al., 2020a), BART (Lewis et al., 2020a), and mT5 (Xue et al., 2021) were finetuned on supervised datasets covering tasks including translation and summarization, and their outputs are compared to “ground truth” outputs via widely used metrics like ROUGE (Lin, 2004) which provide a noisy indication of the “quality” of the output and which can be used to determine whether a model is better than others.1 In contrast, large PLMs with autoregressive language modeling pretraining objectives are more capable to produce results without explicit finetuning and are thus typically evaluated via few-shot and in-context approaches, where the model is given the task description and exemplars showing how the task should be completed. GPT-3 (Brown et al., 2020) and models that followed such as GLaM (Du et al., 2022), Gopher (Rae et al., 2021), and LaMDA (Thoppilan et al., 2022), have achieved few-shot state-of-the-art results on a large number of tasks at their time of publication. However, few-shot approaches work best for tasks with a clear answer such as classification or span-based question-answering.2
Generation metrics penalize systems when their writing style differs from how the references are written (Mathur et al., 2020; Freitag et al., 2020; Mille et al., 2021). Without finetuning, there is no guarantee that PLMs produce outputs that look like the ground truth, both in style and content. Recent work found that these differences leads to sharp differences in how humans and automatic metrics rate the generation quality (Goyal et al., 2022). Due to this uncertainty, most evaluations of new PLMs
1For an in-depth review of the usefulness of automatic metrics, we refer to Gehrmann et al. (2022b) and point to Section 4 for a discussion of the application of metrics to benchmarks.
2We refer to the two task types as NLU and NLG tasks but note that this distinction becomes fuzzy with autoregressive models since technically all answers are “generated”.
are limited to NLU benchmarks such as SuperGLUE (Wang et al., 2019). For example, LaMDA (Thoppilan et al., 2022) did not evaluate on NLG tasks, GLaM (Du et al., 2022) limited its generation evaluation to short span question answering tasks, and GPT-3 (Brown et al., 2020) evaluated only on machine translation. A first autoregressive PLM with broad NLG evaluation, PaLM (Chowdhery et al., 2022), benchmarked summarization and data-to-text tasks in multiple languages. The recent Holistic Evaluation of Language Models project (HELM, Liang et al., 2022) aims to standardize evaluation of language models. With the explicit goal to broaden the task and metric coverage, HELM has established an impressive few-shot benchmark for many natural language tasks. Corroborating the prior findings, they also conclude that human evaluation is necessary for NLG. This distinction means that the referencebased approach for generated text that the field has used since the advent of deep learning may no longer sufficient and that we need clear evaluation protocols that continue to allow us to answer broad questions about “generation quality” of a model. Complementing this work, we take a deeper look at a wider set of NLG tasks and explore LLMs in finetuning and few-shot setups to identify whether reference-based automatic evaluation can still be used to produce system rankings.
Research Questions We aim to define a methodology that allows us to answer the question “How good are learned representations of a model for generating natural language?” via few-shot and finetuning approaches. To develop and apply this methodology we seek to answer the following three research questions:
R1 How do different model architectures compare in terms of automatic metrics? We aim to identify patterns that emerge in evaluations and to uncover aspects inherent to the tasks, e.g. have metrics on specific tasks saturated?, and to the models’ architectural choices, e.g., are encoder-decoders better suited for particular task formulations? (Section 4) R2 What set of tasks, methods, and metrics is best suited for the monitoring of improvements in language generation capabilities? Using the results of R1, we aim to select a subset of tasks, methods, and metrics that robustly produce reliable model rankings. (Section 5)
R3 What are the broader implications for how the quality of newly developed models should be monitored? Robustly ranking systems is particularly important when monitoring a system during training and when comparing across many tasks. In line with the “reality check” theme track at ACL 2023, we discuss the implications of our findings on how evaluation results should be produced and interpreted. (Section 6)

heading: 3 Method

heading: 3.1 Data
text: We select a combination of data-to-text and textto-text datasets as different input modalities. The selected datasets capture different input and output lengths, domains, languages, and communicative goals. The text-to-text task with most available multilingual datasets is summarization which we pick for this paper.3 We pick the following tasks:4
• MLSum (Scialom et al., 2020) – Summarize a news article in multiple sentences. • WikiLingua (Ladhak et al., 2020) – Generate section headers for step-by-step instructions from WikiHow. • XSum (Narayan et al., 2018) – Generate the first sentence of a news article. • Clean E2E NLG (Novikova et al., 2017; Dušek et al., 2019) – Given a set of key-value attribute
3Since benchmarks for machine translation are wellestablished (e.g., Akhbardeh et al., 2021) we exclude it from our scope. However, any methodological outcomes of our work can be applied to translation or similar tasks.
4All datasets were retrieved via the Generation Evaluation and Metrics benchmark (Gehrmann et al., 2021, 2022a). We use these datasets for research purposes only in line with their intended use.
pairs, describe a restaurant in one or two sentences. • Czech Restaurant response generation (Dusek and Jurvc’ivcek, 2019) – Given a dialog context and a dialog act representation, generate a one sentence long response. • WebNLG 2020 (Gardent et al., 2017; Ferreira et al., 2020) – Verbalize subject-predicate-object triples in one or more sentences. • ToTTo (Parikh et al., 2020) – Describe highlighted cells in a table in a single sentence. • XL-Sum (Hasan et al., 2021) – Summarize a news article, in the same language, in a single sentence.
Table 1 provides an overview of these datasets in terms of languages, the lengths of input and output and split sizes. For highly multilingual datasets, we evaluate on a subset of typologically diverse languages following the selection by Clark et al. (2020). To this selection, we add languages that appear bothin WikiLingua and XL-Sum.

heading: 3.2 Models
text: Prior results for the benchmarked tasks primarily come from finetuning T5 (Raffel et al., 2020b), mT5 (Xue et al., 2021), or BART (Lewis et al., 2020b), which are encoder-decoder models pretrained with an infilling objectives. These models are significantly smaller than newer models like GPT-3, with sizes ranging from 130M to 13B parameters. Encoder-decoder models trained for infilling often outperform larger decoder-only LMs in the finetuning setting (Tay et al., 2022), while the latter work better for few-shot setting. There has also been recent work on reducing the computational cost of large models by ∼10x by using a mixture of experts (Zoph et al., 2022). It is important to compare these diverse set of models to understand how scale plays a role with the model’s architecture and its pretraining. We benchmark the following models:5
• PaLM PaLM is a pretrained decoder-only transformer-based model trained with standard left-to-right language modeling objective. It is pretrained on a range of multilingual corpora including Wikipedia, news, and code. In this work, we use two models scales: 8B parameters and 540B parameters.
5Model names omitted for anonymity.
• GPT-3.5 (Ouyang et al., 2022b) GPT-3.5 is a 175B parameter decoder-only transformermodel of the GPT-3 family (Brown et al., 2020) but trained on a blend of text and code from before Q4 2021. This model, named codedavinci-002, was introduced as the base model for InstructGPT-3 (Ouyang et al., 2022b) without the supervision on human-written demonstrations and human-vetted model samples.6 • ST-MoE (Zoph et al., 2022) ST-MoE is a 269B sparse pretrained variant of a dense encoderdecoder transformer-based model. • LaMDA (Thoppilan et al., 2022) LaMDA (137B parameters) is a decoder-only transformer-based language model specialized for dialog applications. It is pretrained on dialog data as well as web text data followed by rank-based tuning. • T5 (Raffel et al., 2020a) T5-XXL (11B parameters) is a pretrained encoder-decoder transformerbased model trained on a span corruption objective with a novel unified text-to-text format. It is pretrained on Common Crawl data, mostly containing English-only documents. • mT5 (Xue et al., 2021) mT5-XXL (11B parameters) is a multilingual variant of T5 that was pretrained on a multilingual corpus, mC4, covering 101 languages. • LongT5 (Guo et al., 2021) LongT5 (3B parameters) a similar architecture as T5, where the encoder is extended to have global-local attention sparsity patterns to handle long inputs.

heading: 3.3 Few-shot evaluation methodology
text: To evaluate the models for few-shot inference, we concatenate a task-specific prompt7 to the input and prepend an output prompt to the output. To handle the oftentimes very long inputs or outputs for tasks such as summarization, inputs were truncated to 2048 tokens and inference was done providing only one exemplar at a time, referred to as 1- shot. These simple prompts are analogous to those used in related work (Chowdhery et al., 2022; Scao et al., 2022). We do not tune the prompts or use more complex strategies to keep fair comparisons between multiple systems, as prompt selection can lead to overfitting. The exemplars are separated through double linebreaks, which are also used
6More details can be found at https://beta.openai.com/docs/model-index-for-researchers
7For Summarization, this prompt was “Summarize the following article:”, and for Data-to-Text it was “Verbalize:”. This was translated to the appropriate language.
to truncate output predictions for evaluation. All few-shot exemplars are randomly sampled from the training corpus. From early experimentation, we found this particularly important since it avoids overfitting to exemplars that work well for one model but not another.

heading: 3.4 Finetuning methodology
text: To use the decoder-only architectures during finetuning, inputs and targets are concatenated. The concatenated sequences are truncated to 2048 tokens, the training context used during pretraining, with 512 tokens reserved for the target. Only summarization tasks required input truncation. We finetuned models with standard hyperparameters; refer to Appendix-B for thorough details. The best model checkpoint for each dataset was selected by the best performing geometric mean of ROUGE-1, ROUGE-2 and ROUGE-L scores on the validation set. Decoding was done with beam-search with a beam size of 4 for encoder-decoder models, while inference in decoder-only PLMs (LaMDA, PaLM, ST-MoE) was performed using top-k sampling with k=10, due to issues with scaling beam search at the time of publication.

heading: 3.5 Metrics
text: Following the suggestions by Gehrmann et al. (2022b), we report a combination of lexical and learned metrics, starting with ROUGE-2 and ROUGE-L (Lin, 2004). Since the default ROUGE implementation uses English-specific tokenization, stemming and punctuation normalization, it is incompatible with other languages. Hasan et al. (2021) extended ROUGE by integrating additional stemmers and tokenizers to cover up to the 45 languages. To support more languages, and avoid dependency on varying implementations, we use a SentencePiece tokenizer (Kudo and Richardson, 2018) which, provided a vocabulary distribution file, is self-contained and has sensible fall-backs to unexpected words. Specifically, we used mT5’s SentencePiece vocabulary. For the same reason, we also evaluate with ChrF (Popović, 2015), which is a character-level n-gram overlap metrics and thus independent from tokenizers. BLEURT (Sellam et al., 2020; Pu et al., 2021) is a multilingual model-based evaluation metric for generation designed to compute the similarity between a pair of sentences i.e. a reference and a candidate. It finetunes RemBERT (Chung
et al., 2021) on synthetic sentence pairs and gold ratings. In contrast to the lexical metrics, BLEURT is meant to capture the non-trivial semantic similarities between two texts. For brevity, the main text of this section focuses on the F-measure of ROUGE-L for English and SentencePiece-ROUGE-L for all other languages while the remaining results are in Appendix A. We additionally investigate the agreement between metrics in Section 5.8

heading: 4 Empirical Observations
text: Few-shot learning falls behind finetuning For many generation tasks, including multilingual summarization tasks, we observe a large gap between finetuning and few-shot results, indicating that finetuning will play an important role when it comes to maximizing automatic scores. On data-to-text, the few-shot results follow a similar trend as in summarization, but the gap to the best finetuned results shrinks drastically. Moreover, the finetuning result do not always follow a trend according to scale or architecture. We hypothesize that multiple tasks have saturated to the metrics. If this is the case, approaching them as few-shot generation tasks may still yield insights but it is no longer productive to use them to benchmark finetuned models.
Finetuned decoder-only PLMs can match encoder-decoder performance with scale In summarization, finetuned decoder-only PLMs, such as PaLM-540B, closely match or exceeds the best reported prior results on all English generation tasks. This demonstrates that PLMs can make up their architectural disadvantage through its vastly increased scale. While finetuning PLMs is computationally expensive, it serves as an important upper bound for few-shot predictions.
Multilingual generation capabilities are highly dependent on pretraining data The PLMs evaluated are mostly pretrained on English corpora: 99+% for T5, LongT5, ST-MoE; 90% for PaLM, LaMDA; contrarily mT5 is explicitly pre-
8For ROUGE, we used the python implementation at https://github.com/google-research/ google-research/tree/master/rouge at commit f935042 and whitespace-tokenized references and predictions before calling the library. For BLEURT, we used BLEURT-20 checkpoint from the library at https:// github.com/google-research/bleurt and commit c6f2375.
trained in a multilingual corpus.9 PaLM achieves best results in 3 out of 4 English generation tasks which generate English text, even when the input is non-English. However, the much smaller mT5 bests the other models in 10 out of 14 non-English summarization tasks, and the relative difference between few-shot and finetuning is larger for nonEnglish generation. This suggests that Englishcentric PLMs are better at processing non-English input than generating non-English output.
Analyzing the effects of input context length Tasks with long inputs suffer from models’ limitation to process said inputs. Inputs are thus usually transformed (e.g. cropped, re-ranked, etc) to fit into the model. We found that a several of the evaluated tasks, such as WikiLingua and MLSum benefit from a longer input context in models even if the long-context model is smaller (i.e., LongT5 vs T5). In contrast, the performance is comparable for the rest of short-context tasks.

heading: 5 Deriving Evaluation Practices
text: Figure 1 summarizes the recommendations we developed from challenges we faced and our observed empirical results. These recommendations are best understood in the context of monitoring
9The language breakdown for GPT-3.5 is unknown.
and benchmarking PLMs during training or inference.
Comparable few-shot learning evaluation As mentioned in Section 3, our design choices were made to ensure that results are comparable across PLMs. Primarily, prompts were deliberately kept extremely simple and all few-shot exemplars were randomly sampled. While highly curated prompts or methods like chain-of-thought prompting can increase the performance considerably (Wei et al., 2022b), it can also lead to overfitting to the particular model the prompt was developed on, in turn making a comparison to other models unfair and producing unrealistic expectations when people have single interactions with it.
Overlap-based metrics are not calibrated to evaluate few-shot learning Few-shot generation suffers from not being able to predict output length properly given the few exemplars provided. While encoder-decoder models utilize endof-string tokens, these are not always learned during decoder-only pretraining. To circumvent this issue, researchers rely on PLMs match to the fewshot format provided e.g. line-breaks that separate exemplars. We observed PLMs fail to follow the format a significant number of times, producing the largest allowed length on occasion. In our experiments, we tried to avoid very long outputs by trimming outputs to the 95-percentile length seen in the targets.10 Still, few-shot output lengths
10This simple method avoids discrepancies across PLMs which might have different maximum decoding lengths.
are on average 2-3 times the average target length while finetuned model’s output average 80% the average target length, across all tasks. Overlap metrics used in generation are sensitive to length (Sun et al., 2019) making a natural disadvantage for few-shot learners. We do not recommend using overlap-based metrics to compare few-shot results without length normalization.
Computational costs can be decreased without sacrificing relative model performance The computational cost of evaluating large datasets, some with more than 10K examples, are prohibitive and perhaps unnecessary. To that end, we investigate if a model ranking can be produced, with a high degree of certainty, while only considering a random subset of the test set, saving compute cost to possibly evaluate on more tasks instead. To investigate this effect, we ran the following experiment: (1) Sample n datapoints from a dataset and all corresponding model scores. (2) Following Kocmi et al. (2021) and Graham et al. (2014), we perform Wilcoxon Rank Sum test (Wilcoxon, 1946) to assess the stability of the ranking. (3) Repeat steps 1&2 k times and record the fraction of runs in which models scores from any two models were not distinguishable from each other (those with a p-value of > 0.05). Since we are considering 10 model settings in this work, this experiment considers all 45 possible pairs. The result shown in Figure 2 provides insight into the required number of data points to produce rankings. For most datasets, we can produce stable model rankings with only 500 examples, some with as little as 100. Tasks where models achieve very similar scores tend to require more test examples, since smaller score differences require more examples to be distinguishable from each other (Wei and Jia, 2021).11
Analyzing metrics utility We use different automated metrics to evaluate the generation quality of the models. These metrics attempt to capture the similarity between system generated output and the reference text. While ROUGE and chrF account for the lexical overlap, BLEURT is meant to compute the semantic similarity. It is important to understand the agreement between these metrics. We compute the the system-level agreement via Spearman correlation coefficient (Spearman, 1987) between the scores given by the metrics to
11Full results available in Appendix A.
the fine-tuned set of models. Figure 3 shows the correlation between ROUGE-L (RL), BLEURT and ChrF. We observe that the metrics are highly correlated for most datasets. Similar to Figure 2, on the tasks where the models have similar performance, we notice less correlation among the metrics. Such tasks are may have either saturated performance, e.g., ToTTo (en) or all models perform poorly, e.g., Wikilingua (es-> en). Due to the small differences between models, metrics fail to produce the same rankings.

heading: 6 Discussion and Reality Check
text: In line with our goal to provide a “reality check” via empirical and theoretical research, and to reflect on the ways in which reported performance improvements are meaningful, we want to situate our findings in the context of the broader NLP community. Openly accessible APIs and publicly available large models have led to increased attention on large pretrained models, but they have also led to a “release-then-test” philosophy where models are released without extensive evaluations. While the findings we present in this paper do not
solve this issue, agreeing on a shared evaluation process could lead to more realistic claims about model performance (and shortcomings), and allow for a more accurate monitoring of models during training.
What claims can we not make? Empirical findings demonstrate that incorporating generation into NLU tasks via Chain-of-Thought leads to better model performance (Wei et al., 2022b; Suzgun et al., 2022). Providing additional grounding via finetuning on instructions and aligning a model to human feedback leads to better task-specific performance without supervision (Wei et al., 2022a; Ouyang et al., 2022a). However, we lack the scientific methods to quantify these advances. While benchmarks provide an indication whether a model is performing better than a previous iteration, and projects like BIG-bench (Srivastava et al., 2022) and HELM (Liang et al., 2022) enable evaluation on a very wide range of possible tasks, they are also inherently limited. When benchmarking models in few-shot settings, especially models for which little information about their training data is available, it is hard to disambiguate model performance from memorization, i.e. if the examples were seen during pretraining. Instruction tuning further blur the line between finetuning and few-shot, which can lead to very different outputs and are not fully comparable. It is thus near impossible to make claims about why a model is succeeding at one particular task without having access to its training data. As mentioned earlier, the target of this work is to derive best practices for comparing models in generation settings with constrained computational budgets, for example when monitoring a training model or when trying to compare on many different tasks. Our findings are grounded in much prior work that finds that metrics have a very high agreement with human judgments on the systemlevel (e.g., Kocmi et al., 2021), but are essentially meaningless on the segment-level. For that reason, we cannot derive claims beyond these rankings about utility of a model or whether a particular model would actually produce useful outputs for a task. To derive such insights, we point to work on extrinsic evaluation which requires comprehensive human evaluations (e.g., Lee et al., 2022).
How can our findings be applied to improve the status quo? Since the generation capabilities of
PLMs are currently not extensively monitored or evaluated, we set out to derive best practices for how these evaluations can look. We found that many of the “easy” tasks, on which finetuned models saturate the metrics, still yield insights for fewshot approaches. We further identified the tension between doing a computationally expensive full evaluation on a dataset and adding more evaluation sets for different tasks or languages. Our findings suggest that evaluation on small subsets of more tasks can be beneficial to the overall results. To further motivate this suggestion, consider the following thought experiment: We have two tasks, A and B. At 500 examples, they have a risk of producing a “wrong” ranking of 10%. At 1,000 examples, they have a risk of producing a wrong ranking of 5%. These risks are not correlated, i.e., their covariance is 0. Given a computational budget of evaluating on 1,000 examples, the risk of only evaluating on one dataset is 5%, and the risk of producing two wrong ratings after evaluating on A and B is only 1%. While additional datasets introduce a larger risk of one individual dataset producing misleading results (18% in this case), one can easily expand this argument to a whole portfolio of tasks to hedge against individual dataset risk (Stuart and Markowitz, 1959). Many existing NLU benchmarks like BIG bench (Srivastava et al., 2022) already follow such a strategy and we believe that generation evaluation, especially considering the additional risk due to metrics, should follow this approach for the use cases discussed in this work. To further minimize the individual dataset risk, they can be switched out once they saturate or their sample sizes increased.

heading: 7 Conclusion
text: In this work, we produced an extensive evaluation of a diverse set of state-of-the-art pre-trained language models (PLMs) for 27 different multilingual generation tasks under few-shot learning and finetuning settings. We discuss empirical results that help inform practitioners which tasks, methods and metrics are suitable. We provide recommendations on how best to monitor conditional generation capabilities of PLMs, including how to fairly benchmark few-shot learning, automated metrics and their utility, and how to efficiently utilize computational resources. We hope that such findings and recommendations could positively influence natural language evaluation in future work.

heading: 8 Limitations
text: In this work, we have presented results that help inform us what tasks, methods and metrics are best suited for monitoring as well as methodologies and empirical information about the current set of models. We provide detailed information of how these results can be reproduced, to the extend that research have access to the PLMs in question, but these results have limitations, in order to reduce costs, many languages were not evaluated which might have left unforeseen patterns not discussed in this work. Moreover, few-shot learning, in particular, could exhibit large variance if different prompts were chosen, or a different set of exemplars chosen. Because of the high costs involved our work does not explore the performance difference when multiple sets of hyper-parameters were chosen. On the conceptual level, we make the assumption that system-level improvements on our tasks translate to downstream usefulness. While prior work suggests that this is the case, tools like chatGPT have significantly expanded the possible application space beyond the realm of “typical” NLP tasks, and we don’t know how well our findings generalize to this space of tasks.

heading: 9 Ethics Statement
text: This paper focuses on conditional generation tasks where models are free to generate long text sequences. Typical issues associated with text generation such as hallucinations, memorization of private information publicly available, toxic and discriminatory language, or sensitive generated content could and are likely to arise. measuring the extent to which these issues occur is a necessary and crucial additional dimension of model evaluation which we do not include in this work, which should be seen as supplemental.

heading: A Additional empirical results
text: Table 3, Table 4 and Table 5 report ROUGE-2 and BLEURT and ChrF results respectively for all tasks. These results are in line with the discussed results in 4

heading: B Technical details
text: Finetuning and inference was done in the t5x framework for public and closed access models. Few-shot learning task methodology is well described in 3, for public access models inference was done via their respective public API, whilst all other models were loaded from the standard checkpoint into TPU accelerators and inference was done on batches of 64. Finetuning was carried out in TPU accelerators, for PaLM we used a constant learning rate of 5×10−5, 20x smaller than during pretraining and reset the optimizer’s (Adafactor) accumulators, for T5, mT5 and LongT5 we used a constant learning rate of 1×10−4.

heading: C Computational Cost and
text: Environmental Impact
In our work we benchmark 27 generation tasks which require a substantial amount of computational resources. Inference on PLMs is exceedingly more efficient than finetuning. We report the number of test examples across all tasks to be 194,844. Inference over this number of examples times 10 models evaluated amounts to 2 million inference examples. Finetuning on the other hand, requires all parameters to be trained and training dataset sizes are considerably larger. We estimate the compute usage for finetuning to be 256 TPU v3 chips for 200 hours. One of the goals of this work is to encourage benchmarking in the future to avoid these costs by more efficiently selecting smaller test size and persuade researchers to only evaluate finetuning approaches when suitable.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
We added a section on Limitations.
3 A2. Did you discuss any potential risks of your work? We discuss in the ethics section the risks of autoregressive text generation, in particular how it can produce misinformation or sensitive data.
3 A3. Do the abstract and introduction summarize the paper’s main claims? The introduction summarizes the paper’s main claims.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? We used artifacts throughout
3 B1. Did you cite the creators of artifacts you used? We used generation datasets found through the GEM benchmark (gem-benchmark.com). We cite the researchers that released these data in section 3.1.
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Licenses: MLSum (Scialom et al., 2020) – MIT License https://github.com/ThomasScialom/MLSUM/blob/master/LICENSE WikiLingua(Ladhak et al., 2020) – cc-by-3.0 https://gem-benchmark.com/data_cards/wiki_lingua XSum(Narayan et al., 2018) – cc-by-sa-4.0 https://gem-benchmark.com/data_cards/xsum Clean E2E NLG(Novikova et al., 2017; Duseket al., 2019) – cc-by-sa-4.0 https://gem-benchmark.com/data_cards/e2e_nlg Czech Restaurants (Dusek and Jurvcivcek, 2019) – cc-by-sa-4.0https://gem-benchmark.com/data_cards/cs_restaurants WebNLG 2020(Gardent et al., 2017; Ferreiraet al., 2020) – cc-by-nc-4.0 https://gem-benchmark.com/data_cards/web_nlg ToTTo(Parikh et al., 2020) – cc-by-sa-3.0 https://gem-benchmark.com/data_cards/totto XL-Sum(Hasan et al., 2021) – cc-by-nc-sa-4.0 https://gem-benchmark.com/data_cards/xlsum
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? All the data mentioned above has intended research purposes which is consistent with this work’s use. We mention this in the paper
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No data was collected. The data has been revised outside this work by community efforts such as GEM.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? We provided information on the languages covered by each dataset in Table 1, as well as the publicly available information on model’s pretraining data language distribution.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. We provided information on the statistic of the datasets used in Table 1,
C 3 Did you run computational experiments? Section 4 and 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Number of parameters per model is reported in section 3.2. Computing infrastructure is mentioned in the appendix B. The total computational budget is discussed in its own section.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Hyper parameter selection is discussed in appendix B. Experimental setup is discussed in the methodology section.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Most of the contributions for this paper are statistics of empirical results and method to obtain them.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We provide this information in the main text, in footnotes.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: One of the goals of this work is to encourage benchmarking in the future to avoid these costs by more efficiently selecting smaller test size and persuade researchers to only evaluate finetuning approaches when suitable. ACL 2023 Responsible NLP Checklist