Venue: EMNLP 2023

# What Comes Next? Evaluating Uncertainty

in Neural Text Generators Against Human Production Variability

Mario Giulianelli\({}^{\circledR_{b}}\)1 Joris Baan\({}^{\circledR_{b}}\)2 Wilker Aziz\({}^{\circledR_{b}}\) Raquel Fernandez\({}^{\circledR_{b}}\) Barbara Plank\({}^{\circledR_{a}}\)3 University of Amsterdam \({}^{\circledT}\) ITU Copenhagen \({}^{\circledR_{b}}\) MCML Munich \({}^{\circledR_{b}}\) LMU Munich \({}^{\circledR_{b}}\)4 University of Amsterdam \({}^{\circledR_{b}}\) TTU Copenhagen \({}^{\circledR_{b}}\) MCML Munich \({}^{\circledR_{b}}\) LMU Munich \({}^{\circledR_{b}}\)

Footnote 1: [https://github.com/dmg-illc/nlg-uncertainty-probes](https://github.com/dmg-illc/nlg-uncertainty-probes)

Footnote 2: This is the first author, but it has been written in terms of a single document and an article that contains all other documents.

###### Abstract

In Natural Language Generation (NLG) tasks, for any input, multiple communicative goals are plausible, and any goal can be put into words, or _produced_, in multiple ways. We characterise the extent to which human production varies lexically, syntactically, and semantically across four NLG tasks, connecting human production variability to _aleatoric_ or _data uncertainty_. We then inspect the space of output strings shaped by a generation system's predicted probability distribution and decoding algorithm to probe its uncertainty. For each test input, we measure the generator's calibration to human production variability. Following this _instance-level_ approach, we analyse NLG models and decoding strategies, demonstrating that probing a generator with multiple samples and, when possible, multiple references, provides the level of detail necessary to gain understanding of a model's representation of uncertainty.3

Footnote 3: The second author uses the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to refer to the term ‘model’ to the term ‘model’ to refer to the term ‘model’ to the term ‘model’ to refer to the term ‘ ‘model’ to refer to the term ‘ ‘model’ to refer to the term ‘ ‘model’ to refer to the term ‘ ‘model’ to refer to the term ‘ ‘model’ to refer to the term ‘ ‘model’ to the term ‘ ‘model’ to refer to the term ‘ ‘model’ to the term ‘ ‘model’ to refer the term ‘ to the term ‘ ‘model’ to refer the term ‘ to the term ‘ ‘model’ to refer the term ‘ to the term ‘ ‘model’ to refer the term ‘ to the term ‘ ‘model’ to refer the term ‘ to the term ‘ ‘model’ to refer the term ‘ to the term ‘ ‘model’ to refer the term ‘ to the term ‘ ‘model’ to refer the term ‘ to the term ‘ ‘model’ to refer the term ‘ to the term ‘ ‘model’ to refer the term ‘ to the term ‘ ‘model’ to refer the term ‘ to the term ‘ ‘model’ to refer the term ‘ to the term ‘ ‘model’ to refer the term ‘ to the term ‘ ‘model’ to refer the term ‘ to the term ‘ ‘model’ to refer the term ‘ to the term ‘ model’ to refer the term ‘ to the term ‘ model’ to refer the term ‘ to the term ‘ model’ to refer the term ‘ to the term ‘ model’ to refer the term ‘ to the term ‘ model’ to refer the term ‘ to the term ‘ model’ to refer the term ‘ to the term ‘ model’ to refer the term ‘ to the term ‘ model’ to refer the term ‘ to the term ‘ model’ to refer the term ‘ to the term ‘ model’ to refer the term ‘ to the term ‘ model’ to the term ‘ model’ to refer the term ‘ to the term ‘ model’ to the term ‘ model’ to refer the term ‘ to the model’ to refer the term ‘ to the term ‘ model’ to the term ‘ model’ to refer the term ‘ to the model’ to refer the term ‘ to the model’ to refer the term ‘ to the model’ to refer the term ‘ to the model’ to refer the term ‘ to the model’ to refer the term ‘ to the model’ to the term ‘ to the model’ to refer the term ‘ to the model’ to the model’ to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model the to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model the to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model to the model the to the model to the model to the model to the model to the model the to the model to the model to the model to the model the to the model to the model to the model to the model to the model the to the model to the model to the model to the model to the model the to the model to the model to the model to the model to the model to the model the to the model to the model to the model to the model to the model the to the model to the model to the model to the model to the model to the model the to the model to the model to the model to the model to the model to the model the to the model to the model to the model to the model the to the model to the model to the model to the model to the model the model to the model to the model to the model the to the model to the model to the model to the model to the model the model to the model to the model to the model to the model the model to the model to the model the to the model to the model to the model to the model the model to the model to the model to the model the model to the model to the model the model to the model to the model the model to the model to the model the model to the model to the model the model to the model to the model the model to the model to the model the model to the model to the model the model to the model to the model the model to the model to the model the model to the model to the model the model to the model to the model the model to the model to the model the model to the model to the model the model to the model to the model the model to the model to the model the model to the model to the model the model to the model to the model the model to the model to the model the model to the model to the model the model to the model to the model the model to the model to the model the model to the model to the model the model to the model to the model the model to the model to the model the model to the model to the model the model to the model to the model the model to the model to the model the model to the model to the model the model to the model to the model the model to the model to the model the model

gether these two mechanisms specify a probability distribution over sequences of tokens, which can be regarded as a representation of the model's uncertainty about productions for a given generation context (see Baan et al. (2023) for a detailed discussion). In this work, we assess whether this representation of uncertainty is in compliance with production variability exhibited by a population of humans--which in turn, we argue, can be regarded as an expression of _aleatoric_ uncertainty, _i.e._, irreducible uncertainty due to the stochastic nature of the data generating process Der Kiureghian and Ditlevsen (2009); Hullermeier and Waegeman (2021). In other words, we compare the distribution over productions of a text generator against the distribution over the productions of a population of human speakers, given the same context (Figure 1).

Quantifying the closeness in distribution between a text generator and a human population is difficult: we only have an iterative view into the generator's distribution; the 'human distribution' is an implicit or even hypothetical object; and in both cases, the sample space is large or even unbounded. We can, however, compare these two objects via the samples they produce and assess their statistical distance--which is what we propose here. For each individual generation context, we compare scalar properties of generations (through repeated model sampling) and human productions (using multi-reference NLG datasets). In particular, we _probe_ for lexical, syntactic, and semantic distance between productions, thus allowing for a quantitative and interpretable assessment of uncertainty.

We find that the uncertainty of neural text generators is higher than justified by human production variability in open-ended tasks, like story generation and open-domain dialogue; and that it is lower on more constrained tasks, like machine translation and text simplification. Popular decoding algorithms, which bias away from the distribution of the generator's underlying statistical model (_e.g._, top-\(k\), top-\(p\), or locally typical, rather than ancestral sampling), have a limited impact on the generator's ability to faithfully represent human variability. We complement our quantitative assessments with a detailed analysis of individual generation contexts, which sheds light on whether a generator has robustly learned to reproduce degrees and aspects of human variability plausible for the communicative task.

Beyond the experimental results obtained on our selection of models and tasks, our work has important implications for NLG evaluation and data collection. Multiple samples and, when possible, multiple references, should be used to assess the statistical fit of text generators. Our approach, complementary to other types of automatic evaluation, makes model assessments particularly insightful and trustworthy because it does not judge a model only by a single output but also, intuitively, by _what it could have generated_--and it does so for each individual input in the test set. We therefore hope our framework will be used by the community as an evaluation criterion for NLG systems, especially to assess them in more open-ended tasks.

## 2 Related Work

Automatic approaches to the evaluation of NLG systems are of high practical importance: they allow for model selection at scale and power quality-aware decoding algorithms Borgeaud and Emerson (2020); Eikema and Aziz (2020); Fernandes et al. (2022); Suzgun et al. (2022). In spite of their known limitations Gehrmann et al. (2022), they are a necessary complement to human evaluation Belz and Reiter (2006); van der Lee et al. (2021).

Reference-based evaluationThe most common way of automatically evaluating text generators is via metrics that estimate the similarity between candidate generations and references, such as BLEU Papineni et al. (2002), ROUGE Lin (2004), COMET Rei et al. (2020), BLEURT Sellam et al. (2020), and BertScore Zhang et al. (2020a). Reference-based metrics are less suited for open-ended tasks such as story generation and dialogue, where a single reference (or even a handful) cannot be representative of the large space of plausible communicative goals and realisations.

Reference-free evaluationA popular, reference-free alternative is to train evaluation models that discriminate human from model output (_e.g._, Bruni and Fernandez, 2017; Gehrmann et al., 2019; Hashimoto et al., 2019), score the appropriateness of input-output pairs (_e.g._, Sinha et al., 2020; Fomicheva et al., 2020), or model human judgements directly (_e.g._, Lowe et al., 2017; De Mattei et al., 2021; Rei et al., 2021). Neural language models themselves have been proposed as evaluators (_e.g._, Yuan et al., 2021; Deng et al., 2021) and used to assess generations along interpretable evaluation dimensions Zhong 

et al., 2022), yet they have been criticised for being biased (toward models similar to the evaluator) and thus limited in their ability to evaluate generated text Deutsch et al. (2022).

Statistical evaluation.Statistical evaluation compares model generations to human productions _in distribution_ through real-valued statistics (_e.g._, Zipf's coefficient, type-token ratio, length) as opposed to strings themselves. These statistics are typically compared marginally, at the corpus level Eikema and Aziz (2020); Meister and Cotterell (2021); Pillutla et al. (2021); Pimentel et al. (2022), supporting general claims about model performance in relation to humans. More recently, Barkhof and Aziz (2022) and Deng et al. (2022) compared statistics at the instance level, supporting claims about models' performance in relation to humans for individual inputs. In this work, we craft statistics that evaluate generators' uncertainty at the _instance level_ against the variability over sequences observed in multi-reference NLG datasets. Although evaluating uncertainty is gaining traction in NLP (_e.g._, Desai and Durrett, 2020; Glushkova et al., 2021; Baan et al., 2022), there is relatively little work on sequence-level uncertainty Ott et al. (2018); Malinin and Gales (2020); Aina and Linzen (2021); Kuhn et al. (2022).

Diversity in NLG.Our analysis is related to NLG studies on output diversity. Some have evaluated diversity induced by different models and NLG decoding strategies--yet do not use human levels of variability as a target Wiher et al. (2022)--or have used human judgements to evaluate diversity metrics themselves Tevet and Berant (2021); Stasaki and Hearst (2022). Others have developed diversity-enhancing objective functions Li et al. (2016) and decoding algorithms Vijayakumar et al. (2018); Shu et al. (2019); Weir et al. (2020); Meister et al. (2021). In our study, where the aim is to evaluate the uncertainty of NLG systems, we focus on unbiased sampling and the most widely used decoding algorithms.

## 3 Probing Language Processes for Production Variability

We interpret language production, by humans or NLG systems, as captured by a probability distribution over natural language strings (_productions_), a random variable \(Y\), given a linguistic context \(X=x\). The context \(x\) can be a source sentence in translation, a story prompt in story generation, or more generally the _input_ to a language process. In turn, a production is a piece of text \(y\) such as a single translation, a story, or more generally the _output_ of a language process.2

Footnote 2: **Notation.** Random variables are denoted by uppercase letters (_e.g._, \(Y\)), outcomes are lowcressed (_e.g._, \(y\)), and \(p_{Y|X=x}\) denotes the probability distribution of \(Y\) given \(X=x\).

### Production Variability

For any language process, _production variability_ is fully characterised by a conditional probability distribution \(p_{Y|X=x}\) representing uncertainty about the output \(Y\) given input \(X=x\). Intuitively, the uniform distribution maximises production variability and the Dirac delta (one-hot) distribution minimises it. Analysing this distribution is difficult. Notably, for human language processes, we do not have an explicit representation of \(p_{Y|X=x}\). This prevents a direct comparison through measures of statistical divergence, or summaries like entropy. Through data collection we can, however, draw conditional samples from the human language process (_i.e._, gather references given a context). On the other hand, for NLG models, we do have an algorithmic representation of \(p_{Y|X=x}\), which is usually sufficient to enable sampling, but the unbounded sample space and lack of conditional independence assumptions make statistical divergence and summaries like entropy intractable.3

Footnote 3: In Appendix A, we discuss issues with entropy in more detail—how it is both difficult to estimate and interpret for text generators.

Instead, we propose to analyse language processes through their samples. This in turn introduces other difficulties, as text is a high-dimensional, structured, non-numerical data type. For tractable analysis, we exploit a set of real-valued and interpretable statistics, or _production probes_, to re-express a language process distribution in terms of how, given an input, its outputs relate to outputs of another language process. When both processes are independent humans performing a task, we obtain a sense of how plausible human productions relate (or _vary_ with respect) to other plausible human productions, along a linguistically interpretable dimension. When we swap one or both processes for an NLG model, we obtain tools to analyse how model generations relate to plausible human productions, thus assessing a model's representation of uncertainty against the variability observed in humans.

 

Specifically, given a context \(x\), two language processes with distributions \(p_{\hat{Y}|X=x}\) and \(p_{V|X=x}\), and a choice of distance metric \(k(\cdot,\cdot)\in\mathbb{R}\), _our probe for production variability_ is a real random variable \(k(\hat{Y},Y)\). This random variable captures the joint distribution of distance between any two outputs drawn conditionally from the two processes. The distribution of the probe \(k(\hat{Y},Y)\) is also intractable, but we can estimate it via simulation by drawing productions from the processes and assessing the distance metric on sampled pairs, as illustrated in Figure 1.

Consider analysing the human process (SS 5) through \(k(Y,Y)\): when multiple realisations of the output are dissimilar (_e.g._, given the input '_How is your day?_' and outputs '_Fantastic, thank you!_' and '_I asked you first_') production variability is high along the dimension captured by \(k\).

### Production Probes

We instantiate our production probes with three distance functions. They return values from 0 to 1. We hope that future work will experiment with alternative probes that may capture other linguistic or extra-linguistic levels of analysis.

**Lexical:** The fraction of distinct \(n\)-grams in two strings, with \(n\!\in[1,2,3]\) (_i.e._, number of non-matching \(n\)-gram occurrences divided by the total number of \(n\)-grams in both strings).

**Syntactic:** Analogous to lexical distance but on part-of-speech tag \(n\)-grams.4

Footnote 4: From spaCy, en_core_web_md Honnbal et al. (2020).

**Semantic:** The cosine distance between the sentence embeddings of two strings Reimers and Gurevych (2019).5

Footnote 5:sentence-transformers/all-distilroberta-v1

## 4 Experimental Setup

### Data and Models

We experiment with four NLG datasets that contain 5+ human references per input instance and for which we expect humans to display different degrees of production variability. For each tasks, we select models that are publicly available, are reasonably sized, have been used previously on the task, and are conventionally accepted as suitable for it.6 All datasets are in English; for translation, the target language is German. Table 1 (Appendix C) shows relevant statistics. The reference collection procedure varies across datasets and we discuss how this may impact our analysis in the Limitations section.

Footnote 6: For text simplification, we did not find any available trained model, so we used a versatile model like FlanT5.

Machine translation.We use 500 sentences from the WMT-14 En-De test set (_newstest2014_; Bojar et al., 2014), which have been annotated by Ott et al. (2018) with 10 additional reference translations produced by as many human translators. As a generator, we use Helsinki-NLP's Transformer-Align model trained on Opus-MT Tiedemann and Thottingal (2020).

Text simplification.We use the 2,000 instances of the _ASSET_ validation set Alva-Manchego et al. (2020). For each source sentence, originally from the TurkCorpus Xu et al. (2016), ASSET includes 10 additional simplifications by as many crowdsource annotators. On this dataset, we test Flan-T5-large Chung et al. (2022), an instruction-finetuned version of the T5 language model Raffel et al. (2020), which we further finetune on the ASSET training set.

Figure 2: Human production variability across four NLG tasks. The values on the horizontal axis are single samples of lexical (unigram), syntactic (POS bigram), or semantic (cosine) distance between two randomly sampled productions for each input (see Section 3). Re-sampling productions results in nearly identical marginal distributions. Probability mass on the right side signals high distance and thus high variability, and vice versa.

 

Storytelling (story generation).We use the 759 instances from the _WritingPrompts_ test set Fan et al. (2018) for which at least 5 human references are available. Prompts and stories are originally scraped from r/WritingPrompts, a Reddit forum of stories written by online users in response to story prompts designed by other users. The number of stories available per prompt (9.56 \(\pm\) 7.67) varies from 5 to 92. We use GPT2-large Radford et al. (2018) finetuned on the WritingPrompts training set.

Open-domain dialogue.We use the development set of _DailyDialog++_Sai et al. (2020), which contains 5 additional references for 1,028 conversations from the DailyDialog corpus Li et al. (2017). The dialogues are short (less than 8 turns) and cover a broad list of topics; for each dialogue, 2-3 annotators were asked to generate 1-3 alternative responses.7 For this task, we use the pretrained DialoGPT-medium Zhang et al. (2020b).

Footnote 7: The _DailyDiag++_ annotators are also instructed to avoid short generic responses such as ‘_Sure_’ and to write, instead, meaningful responses with at least 8-10 words.

### Decoding algorithms

We experiment with five decoding algorithms: unbiased (ancestral or forward) sampling Bishop (2006); Koller and Friedman (2009), temperature scaling, top-\(k\) sampling Fan et al. (2018), nucleus sampling Holtzman et al. (2019), and locally typical sampling Meister et al. (2023). For all decoding algorithms, we set the maximum sequence length to 100 (cf. Table 1, Appendix C).

## 5 Human Production Variability Across NLG Tasks

Consider \(p_{Y|X=x}\) the distribution that describes the human language process, and define the following special case for human production variability:

\[H_{k}(x)\coloneqq k(Y,Y)~{}. \tag{1}\]

Estimating this probe by drawing pairs of human productions provides an interpretable view on plausible variability--_i.e._, aleatoric uncertainty--along the dimension captured by \(k\). Figure 2 shows \(H_{k}(x)\) marginalised over inputs for the four NLG tasks. We use unigram distance for the lexical probe, POS bigram distance for the syntactic probe, and cosine distance for the semantic probe. High distance indicates high variability, and vice versa.

Translation and text simplification.Humans show low production variability in these two tasks. While translations of a given source sentence are more lexically and semantically varied, simplifications exhibit a higher degree of syntactic variability, probably as a result of the instructions used during data collection (writers were asked to use varying rewriting transformations). Overall, low levels of variability are to be expected as, in both tasks, content preservation is part of the communicative goal.

Story generation.Variability in story generation is strongly dependent on the probe. It is low at the syntactic level--close to translation and simplification--while lexical and semantic probes place this task closer to open-domain dialogue. Stories generated from a given prompt may vary a lot in content, but basic syntactic structures and lexical material are shared. Although this task can be a priori perceived at least as 'open-ended' as dialogue, lower levels of variability may result from contextual factors specific to the _WritingPrompts_ dataset that we are not explicitly modelling, such as writers reading stories contributed by other users.

Open-domain dialogue.We observe the highest production variability in this task across all probes. Many output pairs are lexically and syntactically completely dissimilar, as indicated by the rightmost bin in Figures 1(a) and 1(b). Lexical variability is even more extreme when looking at bigrams and trigrams (Figure 7 in Appendix D) suggesting that while responses rarely share words or phrases, they still sometimes convey similar meaning (Figure 1(c)). Overall, the fact that dialogue appears to be the most open-ended task can be explained by the wide variety of communicative goals that can plausibly follow from a dialogue context and, in part, by the fact that individual annotators produced multiple responses for the _DailyDialog++_ dataset and thus were able to monitor the diversity of their outputs.

## 6 Do Neural Text Generators Reproduce Human Production Variability?

Consider, now, a second language process: a text generator with distribution \(p_{\hat{Y}|X=x}\). We study this generator's uncertainty about outputs given an input \(x\) under _two lenses_. In SS 6.1, we study how outputs vary with respect to one another, which is analogous to human production variability \(H_{k}(x)\). We refer to this as the generator's _self-variability_:

\[M_{k}(x)\coloneqq k(\hat{Y},\hat{Y})~{}.\] 

In SS 6.2, instead, we study how model generations vary with respect to a language process _known to be plausible_: a human language process \(p_{Y|X=x}\). We refer to this as _cross-variability_:

\[C_{k}(x)\coloneqq k(\hat{Y},Y)~{}. \tag{3}\]

Our expectation is that generators with a good representation of aleatoric uncertainty reproduce human production variability along both axes. As we employ a distance metric, it may look like we should regard a model as a good approximation to the human process whenever \(C_{k}(x)\) concentrates about small positive values. To some extent this is the interpretation exploited by most automatic evaluation metrics (single- or multi-reference). In this work, we refrain from taking any one human production as a 'reference' to be closely 'matched'; rather, we take statistical properties of human productions as illustrative of plausible variability and thus targets to be reproduced. We quantify deviation from plausible human variability by estimating a notion of statistical divergence.

### The Underlying Statistical Model

In this section, we criticise the underlying statistical model (as a result of parameter estimation via MLE) using unbiased sampling. As models observe variability only marginally (multiple references are rarely used during training), it is interesting to study if their self-variability is calibrated to human variability: given individual input instances, do distances between unbiased model samples distribute similarly to distances between human productions? To distinguish over-estimation from under-estimation of variability, we report a signed notion of divergence, \(\mu_{M_{k}(x)}-\mu_{H_{k}(x)}\). When \(M_{k}(x)\) and \(H_{k}(x)\) distribute similarly, their mean difference is low for a given \(x\). Positive differences imply that models overestimate variability, _i.e._, model samples vary more with respect to one another than human samples do. Negative differences indicate that models underestimate variability.

Figure 3 shows how mean differences distribute across each task-specific test set for the models in Section 4. We use up to 10 human productions (5 for dialogue) and 10 generations. The first two rows show that \(\mu_{M_{k}(x)}-\mu_{H_{k}(x)}\) distributes far below \(0\) for translation (OpusMT) and somewhat below \(0\) for simplification (Flan-T5), indicating that the two models substantially underestimate variability.8 The opposite is true for dialogue and story generation: both GPT-2 and DialoGPT moderately overestimate the open-endedness of these tasks. We also inspect cross-variability \(\mu_{C_{k}(x)}-\mu_{H_{k}(x)}\), finding similar patterns, with slightly better over

Figure 4: Distribution over Wasserstein distances for GPT-2 in blue: D\({}_{W_{1}}(C_{k}(x),H_{k}(x))\). Distribution for a human control group in orange: D\({}_{W_{1}}(\dot{H}_{k}(x),H_{k}(x))\). Semantic probe: \(k\) is cosine distance.

 

all cross-variability calibration for translation and simplification (Figure 8, Appendix D).

### The Effect of Decoding Algorithms

We now study text generators obtained by varying the sampling procedure.9 We analyse their representation of uncertainty by assessing the divergence between the distribution of generator-human cross-variability \(C(x)\) and human variability \(H(x)\). While \(\mu_{C_{k}(x)}-\mu_{H_{k}(x)}\) can inform us about the direction of miscalibration, we observe only a handful of cases where different decoding strategies yield both under- and over-estimation for the same model (see Figures 10 and 11 in Appendix D). Instead, as we sometimes observe distributions with multiple modes--causing their difference in means to paint an incomplete picture--we additionally report a measure of divergence that is more robust to such multi-modal distributions: the Wasserstein 1-Distance \(\mathrm{D}_{W_{1}}(\cdot,H_{k}(x))\).10 Results for self-variability \(M(x)\) and mean distance can be found in Appendix D, Figures 9 to 11.

Footnote 9: This leads to a probability distribution whose pmf is hard if at all possible to characterise, meaning we cannot easily assess the probability of an outcome under the new distribution. But we have an explicit sampler for this new distribution, which is all our analysis tools require.

Footnote 10: Indeed, we find several cases where \(\mathrm{D}_{W_{1}}\)signals stronger miscalibration compared to \(\mathrm{D}_{\mu}\). For an additional discussion about \(\mathrm{D}_{W_{1}}\), see Appendix B.

Human control group.The blue curve in Figure 4 shows how \(\mathrm{D}_{W_{1}}(C_{k}(x),H_{k}(x))\) distributes over inputs for unbiased samples from GPT-2 on story generation. To contextualise this observation we report a human control group (the orange curve): this is \(\mathrm{D}_{W_{1}}\) measured between two human populations (_i.e._, we make two disjoint samples from the available human productions for each prompt, use those to estimate \(H_{k}(x)\) and an analogous \(\hat{H}_{k}(x)\), and compute \(\mathrm{D}_{W_{1}}(\hat{H}_{k}(x),H_{k}(x))\)). We can now appreciate what is a plausible Wasserstein distance curve between two human-based processes, and with that, we can better discern that this particular system gives good but not perfect representation to human levels of production variability (note the overlap between the two distributions). Upon visual inspection of divergence distributions (like Figure 4) for different sampling strategies, we find similar shapes. We exploit this finding and summarise each divergence distribution using its mean. This is shown in Figure 5, which presents results for many decoding settings, tasks and probes. The leftmost red dots indicate the human control group.11 We observe that two human groups agree more on the meaning of translations and simplifications than on their form, while for story generation the two groups agree more on surface form and basic structures and less on the semantic content of the stories.

Footnote 11: No control condition is shown for open-domain dialogue as the five references contained in _DailyDiadog++_ are too few to create a control group.

Results.Overall, Figure 5 shows that most decoding settings are close to unbiased sampling, which in turn is in the same ballpark (mean Wasserstein distance always lower than 0.1) as the human control. This indicates tha

Figure 5: Mean Wasserstein distances \(\mathrm{D}_{W_{1}}(C(x),H(x))\) for (tasks, probe, decoder) tuples. Base models for each task are described in Section 4. Colour refers to decoding algorithm with various parameter settings (fully reported in Table 4, Appendix F). Human control group in red.11 Clusters suggest that decoders often have similar effect. Unbiased sampling is competitive.

 

capture the space of plausible human productions well when coupled with most decoding algorithms, though not as well as another human language process. Decoding settings form many clusters, and for all tasks except open-domain dialogue, unbiased samples best match human variability. This suggests that, within the limits of decoding configurations typically considered as appropriate, different token-level decoding strategies often have a similar effect on a generator's ability to reproduce human production variability along our three probes. Altogether, these findings inform us about an often neglected aspect of decoding algorithms, namely their effect on the model's representation of uncertainty (rather than their ability to select individual high-quality generations).

## 7 Qualitative Instance-Level Analysis

We now qualitatively analyse individual inputs for which a generator's uncertainty is miscalibrated to human variability--as detected by \({\rm D}_{W_{1}}\). For each task, we use up to 10 human productions (5 for dialogue) and 10 generations. Figures accompanying the examples in this section are in Appendix E. While it is not a replacement for more standard NLG evaluation procedures, we argue that this level of analysis is complementary and crucial to gain deeper understanding of a generator's representation of uncertainty.

Variability underestimation in translation and simplification.We have seen that in translation and simplification, generators' self-variability is lower than human variability (SS 6.1). We now zoom in on examples from these two tasks, inspecting instances that show inadequate model fit on all linguistic levels (_i.e._, \({\rm D}_{W_{1}}(M_{k}(x),H_{k}(x))\) is high for all \(k\)). The most severe cases of miscalibration for OpusMT are all instances of variability underestimation.12 For most of these, generations are virtually or completely identical, while a few present slightly higher but still substantially lower variability than human productions. For example, ten humans translated the phrase '_reacted cautiously_' in the English source sentence '_Several companies have thus far reacted cautiously when it comes to hiring_' in six different ways ('_vorsichtig reagiert_', '_zuruckhalten regaigert_', '_mit Vorsichtsmassnahmen reagiert_', '_rezugierten mit Zuruchaltung_', '_mit Vorsichte negaigert_', '_reagierten verhalten_') while all ten generated samples contain the German phrase '_vorsichtig reagiert_', signalling that the generator's lexical rephrasing abilities do not generalise to this input instance. For text simplification, we focus on instances where Flan-T5's uncertainty is not calibrated to human syntactic variability.13 We observe that simplifications sampled from the generator are always syntactically more similar to each other than humans', indicating that the generator struggles to capture an important aspect of text simplification: that many semantically equivalent rewritings are possible if a text's syntactic structure is altered.

Footnote 12: We select instances with \({\rm D}_{W_{1}}>0.3\) for unigram distance and \({\rm D}_{W_{1}}>0.2\) for POS bigram and semantic distance. These thresholds are chosen based on distribution plots of instance-level distances (see, _e.g._, Figure 2b).

Variability overestimation in dialogue.According to our estimates of human variability (SS 5), dialogue is the most open-ended task on all linguistic levels. We have hypothesised that this is due to the large variety of communicative act types plausible given any dialogue context. We have also seen that DialoGPT generally overestimates production variability (SS 6.1)--Figure 1 is one such example. Now we further inspect instances where cross-variability is miscalibrated with respect to human outputs.14 We find that the generator's bad fit can be due to very short and generic responses (_e.g._, '_Well..._', '_haha_', '_Ahem_', '_Well done!_'), but is more often due to the presence of fluent yet very diverse and often inadequate samples. For such instances, not only is the generator's cross-variability miscalibrated--self-variability, too, is overestimated on all linguistic levels. In particular, the generator's poor calibration to lexical and syntactic variability is related to its inability to choose the correct dialogue acts (or favouring an excessive variety of dialogue acts). In an example instance where the last dialogue turn goes '_I've got a business call that I really need to take_', humans all reply with short affirmative responses ('_Okay! Please._', '_Well! Go on._', '_Sure, why not!_', '_Surc! Go ahead._', '_Yes! Sure._') while the model's responses are mostly lengthy statements, sometimes not particularly coherent ones (_e.g._, '_You don't need a business call. You need a friend_').

Footnote 14: \({\rm D}_{W_{1}}(M_{k}(x),H_{k}(x))>0.2\) for all \(k\) in {unigram distance, POS bigram distance, cosine distance}.

Variability in lack of situational grounding.We have observed that human-written stories in the _WritingPrompts_ dataset show lower variability than human dialogue res 

that this may be in part due to contextual pressures that constrain variability (SS 5). We now analyse instances flagged by our probe as cases of badly calibrated semantic cross-variability for GPT-2.15 For one of these, the prompt refers to a portion of the situational context the model does not have access to (_'all top level comments in this prompt take place in the same world, so make them all fit together'_). Because they are conditioned on and reuse that context, human stories are quite similar to each other; generations, instead, show much higher pairwise distance both when sampled jointly with the human productions (see Figure 6) and with themselves. The lack of relevant situational grounding makes the model more uncertain than it should be for this instance.

Footnote 15: D\({}_{W_{1}}(C_{k}(x),H_{k}(x))>0.3\); \(k\) is cosine distance.

## 8 Conclusion

Variability is an intrinsic property of human language production. Text generators, if they are to be considered as good statistical models of human written production, should exhibit plausible levels of variability. However, in NLG, the widespread practice is (i) collecting only one 'reference' production for each input and (ii) evaluating only a single generation. To appreciate the impact of this incongruuity empirically, we analyse multiple-reference datasets for four NLG tasks, and show that each task has its own plausible levels of lexical, syntactic, and semantic variability. We connect production variability to aleatoric uncertainty, the irreducible uncertainty of the language production process, and evaluate neural text generators in terms of whether their representation of uncertainty is calibrated to the levels of variability observed in humans. We find that NLG models overestimate production variability in open-ended tasks and underestimate it in more constrained tasks, and that most popular decoding algorithms all have a similar, limited effect on the generators' ability to reproduce human variability.

We advocate for more widespread usage of instance-level probing of NLG systems as a way to evaluate their statistical fit, not just along the dimensions we cover in this study but with respect to any other quality of interest. This approach contrasts with corpus-level analyses of NLG systems (_e.g._, Pillutla et al., 2021; Meister and Cotterell, 2021; Pimentel et al., 2022) and thanks to its greater interpretability, it builds trust in the ability of generators to reproduce human-like statistics when situated in specific linguistic contexts rather than 'globally', over a possibly heterogeneous corpus. In the future, we plan to devise new ways of improving the calibration of models' uncertainty (Zhao et al., 2022; Zhang et al., 2022), _e.g._, steering generators with sequence-level decoding algorithms (Eikema and Aziz, 2022), and to investigate the relation between uncertainty and perceived generation quality (_e.g._, Kuhn et al., 2022): while we use human levels of variability as a target, desirable levels of variability may deviate from human statistics for specific applications.

Future work should also study production variability as a function of a more complex notion of discourse context Giulianelli and Fernandez (2021); Giulianelli et al. (2023) and attempt to disentangle uncertainty over communicative goals and realisations Stasaski and Hearst (2023). This is an important avenue not only toward more practically useful generators but also toward reliable computational models of language production.

Figure 6: Example of poor cross-variability calibration for GPT-2 with typical sampling on story generation.

 



### Limitations

Our analysis relies on multiple-reference datasets, which are scarce for NLG tasks. Even though, for single-reference datasets, we cannot perform a similar instance-level analysis, this fact does not entail that the observations we make do not apply to such datasets--we might simply not have the data to expose them.

Impact of data collection.The way in which multiple references are gathered may impact the variability in productions. For example, asking a single annotator to produce several distinct references might artificially increase the diversity of responses. Conversely, asking several independent annotators might decrease diversity for they may resort to similar responses that quickly come to mind (or, in fact, the opposite if they interpret the linguistic context differently). To summarise, there are two levels of uncertainty in human production data: one is on the individual level, the other is on the population level. In this work, we do not distinguish these two, although the analysis tools that we propose allow for it. For example, one could collect human productions from one individual (_e.g._, for personalisation) or from sub-populations (_e.g._ to improve fit for underrepresented communities).

Other quality dimensions.It is possible that a model fits various statistical properties of the human process (under \(M_{k}(x)\), under \(C_{k}(x)\), and for various choices of \(k\)) meanwhile none of its probable responses are humanly-accepted as a whole. This is why we shall think of our tools as statistical probes. We indeed find interesting instances that show good fit in terms of our distance probes but whose outputs may be perceived as inadequate. Manual inspection reveals that a marriage proposal in one of the dialogues (Figure 16 in the Appendix) is followed by a few incoherent model responses (_e.g._, '_Thank you. It's not a question of the strength or weakness of the plot. I think it all falls within my capacity._'), some dispreferred ones ('_If you want to have a bug?_'; see Levinson, 1983), and some with negative affect ('_I don't need your love. I know where you are coming from and I trust you will do the same._'). Exhaustively defining all aspects of perceived quality (or human-likeness) is a strenuous endeavour which is highly dependent on the use case of the generation system. Our probes can be replaced with (possibly asymmetric) quality metrics which capture aspects (_e.g._, affective content, toxicity, or readability) that are considered relevant for any given application.

## Acknowledgements

We thank Arabella Sinclair, Claire Gardent, Clara Meister, Christopher Lucas, Ehud Reiter, Sarenne Wallbridge, and the ILLC's Dialogue Modelling Group for inspiring discussions. We also thank the MaiNLP group for feedback on earlier drafts of this paper. MG and RF are supported by the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No. 819455). JB is supported by the ELLIS Amsterdam Unit. WA is supported by the EU's Horizon Europe research and innovation programme (grant agreement No. 101070631, UTTER). BP is supported by the European Research Council (ERC) (grant agreement No. 101043235).

## References

* Aina and Linzen [2021] Laura Aina and Tal Linzen. 2021. The language model understood the prompt was ambiguous: Probing syntactic uncertainty through generation. In _Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_, pages 42-57, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Alva-Manchego et al. [2020] Fernando Alva-Manchego, Louis Martin, Antoine Bordes, Carolina Scarton, Benoit Sagot, and Lucia Specia. 2020. ASSET: A dataset for tuning and evaluation of sentence simplification models with multiple rewriting transformations. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4668-4679, Online. Association for Computational Linguistics.
* Alva-Manchego et al. [2021] Fernando Alva-Manchego, Carolina Scarton, and Lucia Specia. 2021. The (Un)Suitability of Automatic Evaluation Metrics for Text Simplification. _Computational Linguistics_, 47(4):861-889.
* Austin [1975] John Langshaw Austin. 1975. _How to do things with words_. Clarendon Press, Oxford.
* Baan et al. [2022] Joris Baan, Wilker Aziz, Barbara Plank, and Raquel Fernandez. 2022. Stop measuring calibration when humans disagree. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_.
* Baan et al. [2023] Joris Baan, Nico Daehim, Evgenia Ilia, Dennis Ulmer, Haau-Sing Li, Raquel Fernandez, Barbara Plank, Rico Sennrich, Chrysoula Zerva, and Wilker Aziz. 2023. Uncertainty in natural language generation: From theory to applications. _arXiv preprint arXiv:2307.15703_

* Barkhof and Aziz (2022) Claarje Barkhof and Wilker Aziz. 2022. Statistical model criticism of variational auto-encoders. _arXiv preprint arXiv:2204.03030_.
* Belz and Reiter (2006) Anja Belz and Ehud Reiter. 2006. Comparing automatic and human evaluation of NLG systems. In 11th Conference of the European Chapter of the Association for Computational Linguistics, pp. 313-320, Trento, Italy. Association for Computational Linguistics.
* Bishop (2006) Christopher M. Bishop. 2006. _Pattern Recognition and Machine Learning (Information Science and Statistics)_. Springer-Verlag, Berlin, Heidelberg.
* Bojar et al. (2014) Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Ales Tamchyna. 2014. Findings of the 2014 workshop on statistical machine translation. In _Proceedings of the Ninth Workshop on Statistical Machine Translation_, pages 12-58, Baltimore, Maryland, USA. Association for Computational Linguistics.
* Borgeaud and Emerson (2020) Sebastian Borgeaud and Guy Emerson. 2020. Leveraging sentence similarity in natural language generation: Improving beam search using range voting. In _Proceedings of the Fourth Workshop on Neural Generation and Translation_, pages 97-109, Online. Association for Computational Linguistics.
* Bruni and Fernandez (2017) Elia Bruni and Raquel Fernandez. 2017. Adversarial evaluation for open-domain dialogue generation. In _Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue_, pages 284-288, Saarbrucken, Germany. Association for Computational Linguistics.
* Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. _arXiv preprint arXiv:2210.11416_.
* De Mattei et al. (2021) Lorenzo De Mattei, Huiyuan Lai, Felice Dell'Orletta, and Malvina Nissim. 2021. Human perception in natural language generation. In _Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)_, pages 15-23, Online. Association for Computational Linguistics.
* Deng et al. (2021) Mingkai Deng, Bowen Tan, Zhengzhong Liu, Eric Xing, and Zhiting Hu. 2021. Compression, transduction, and creation: A unified framework for evaluating natural language generation. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 7580-7605, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Deng et al. (2022) Yuntian Deng, Volodymyr Kuleshov, and Alexander Rush. 2022. Model criticism for long-form text generation. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 11887-11912, Abu Dhabi, United Arab Eminates. Association for Computational Linguistics.
* Der Kiureghian and Ditlevsen (2009) Armen Der Kiureghian and Ove Ditlevsen. 2009. Aleatory or epistemic? Does it matter? _Structural safety_, 31(2):105-112.
* Desai and Durrett (2020) Shrey Desai and Greg Durrett. 2020. Calibration of pre-trained transformers. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 295-302, Online. Association for Computational Linguistics.
* Deutsch et al. (2022) Daniel Deutsch, Rotem Dror, and Dan Roth. 2022. On the limitations of reference-free evaluations of generated text. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 10960-10977, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Ekikema and Aziz (2020) Bryan Eikema and Wilker Aziz. 2020. Is MAP decoding all you need? the inadequacy of the mode in neural machine translation. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 4506-4520, Barcelona, Spain (Online). International Committee on Computational Linguistics.
* Eikema and Aziz (2022) Bryan Eikema and Wilker Aziz. 2022. Sampling-based approximations to minimum Bayes risk decoding for neural machine translation. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 10978-10993, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Fan et al. (2018) Angela Fan, Mike Lewis, and Yann Daubin. 2018. Hierarchical neural story generation. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 889-898, Melbourne, Australia. Association for Computational Linguistics.
* Fernandes et al. (2022) Patrick Fernandes, Antonio Farinhas, Ricardo Rei, Jose G. C. de Souza, Perez Ogayo, Graham Neubig, and Andre Martins. 2022. Quality-aware decoding for neural machine translation. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1396-1412, Seattle, United States. Association for Computational Linguistics.
* Fomicheva et al. (2020) Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frederic Blain, Francisco Guzman, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. 2020. Unsupervised quality estimation for neural machine translation. _Transactions of the Association for Computational Linguistics_, 8:539-555.
* Gehrmann et al. (2022) Sebastian Gehrmann, Elizabeth Clark, and Thibault Sellam. 2022. Repairing the cracked foundation: A survey of obstacles in evaluation practices for generated text. _arXiv preprint arXiv:2202.06935_ 

Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. 2019. GLTR: Statistical detection and visualization of generated text. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_, pages 111-116, Florence, Italy. Association for Computational Linguistics.
* Giulianelli and Fernandez (2021) Mario Giulianelli and Raquel Fernandez. 2021. Analysing human strategies of information transmission as a function of discourse context. In _Proceedings of the 25th Conference on Computational Natural Language Learning_, pages 647-660, Online. Association for Computational Linguistics.
* Giulianelli et al. (2023) Mario Giulianelli, Sarenne Wallbridge, and Raquel Fernandez. 2023. Information value: Measuring utterance predictability as distance from plausible alternatives. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, Singapore. Association for Computational Linguistics.
* Glushkova et al. (2021) Taisiya Glushkova, Chrysoula Zerva, Ricardo Rei, and Andre F. T. Martins. 2021. Uncertainty-aware machine translation evaluation. In _Findings of the Association for Computational Linguistics: EMNLP 2021_, pages 3920-3938, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Halpern (2017) Joseph Y Halpern. 2017. _Reasoning about uncertainty_. MIT press.
* Hashimoto et al. (2019) Tatsunori B. Hashimoto, Hugh Zhang, and Percy Liang. 2019. Unifying human and statistical evaluation for natural language generation. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 1689-1701, Minneapolis, Minnesota. Association for Computational Linguistics.
* Holtzman et al. (2019) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. In _International Conference on Learning Representations_.
* Honnibal et al. (2020) Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020. spaCy: Industrialstrength Natural Language Processing in Python.
* Hullermeier and Waegeman (2021) Eyke Hullermeier and Willem Waegeman. 2021. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. _Machine Learning_, 110(3):457-506.
* Jokinen (1996) Kristiina Jokinen. 1996. Goal formulation based on communicative principles. In _COLING 1996 Volume 2: The 16th International Conference on Computational Linguistics_.
* Koller and Friedman (2009) Daphne Koller and Nir Friedman. 2009. _Probabilistic graphical models: principles and techniques_. MIT press.
* Kuhn et al. (2022) Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2022. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In _International Conference on Learning Representations_.
* Levelt (1993) Willem JM Levelt. 1993. _Speaking: From intention to articulation_. MIT press.
* Levinson (1983) Stephen C Levinson. 1983. _Pragmatics_. Cambridge University Press.
* Li et al. (2016) Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 110-119, San Diego, California. Association for Computational Linguistics.
* Li et al. (2017) Yanran Li, Hui Su, Xiaoyu Shen, Wenije Li, Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: A manually labelled multi-turn dialogue dataset. In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 986-995, Taipei, Taiwan. Asian Federation of Natural Language Processing.
* Lin (2004) Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In _Text Summarization Branches Out_, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
* Lowe et al. (2017) Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. Towards an automatic Turing test: Learning to evaluate dialogue responses. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1116-1126, Vancouver, Canada. Association for Computational Linguistics.
* MacKay (2003) David JC MacKay. 2003. _Information theory, inference and learning algorithms_. Cambridge University Press.
* Malinin and Gales (2020) Andrey Malinin and Mark Gales. 2020. Uncertainty estimation in autoregressive structured prediction. In _International Conference on Learning Representations_.
* Meister and Coltterell (2021) Clara Meister and Ryan Coltterell. 2021. Language model evaluation beyond perplexity. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 5328-5339, Online. Association for Computational Linguistics.
* Meister et al. (2021) Clara Meister, Martina Forster, and Ryan Coltterell. 2021. Determinantal beam search. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 6551-6562, Online. Association for Computational Linguistics 

Clara Meister, Tiago Pimentel, Gian Wier, and Ryan Cotterell. 2023. Locally typical sampling. _Transactions of the Association for Computational Linguistics_, 11:102-121.
* Ott et al. (2018) Myle Ott, Michael Auli, David Grangier, and Marc'Aurelio Ranzato. 2018. Analyzing uncertainty in neural machine translation. In _International Conference on Machine Learning_, pages 3956-3965. PMLR.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Peyre et al. (2019) Gabriel Peyre, Marco Cuturi, et al. 2019. Computational optimal transport: With applications to data science. _Foundations and Trends(r) in Machine Learning_, 11(5-6):355-607.
* Pillutla et al. (2021) Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. MAUVE: Measuring the gap between neural text and human text using divergence frontiers. In _Advances in Neural Information Processing Systems_, volume 34, pages 4816-4828. Curran Associates, Inc.
* Pimentel et al. (2022) Tiago Pimentel, Clara Meister, and Ryan Cotterell. 2022. Cluster-based evaluation of automatically generated text. _arXiv preprint arXiv:2205.16001_.
* Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. Technical report, OpenAI.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of Machine Learning Research_, 21:1-67.
* Rei et al. (2021) Ricardo Rei, Ana C Farinha, Chrysoula Zerva, Daan van Stigt, Craig Stewart, Pedro Ramos, Taisiya Glushkova, Andre F. T. Martins, and Alon Lavie. 2021. Are references really needed? unbabel-IST 2021 submission for the metrics shared task. In _Proceedings of the Sixth Conference on Machine Translation_, pages 1030-1040, Online. Association for Computational Linguistics.
* Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 2685-2702, Online. Association for Computational Linguistics.
* Reimsers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
* Sacks et al. (1974) Harvey Sacks, Emanuel A. Schegloff, and Gail Jefferson. 1974. A simplest systematics for the organization of turn-taking for conversation. _Language_, 50(4):696-735.
* Bai et al. (2020) Ananya B. Cai, Akash Kumar Mohankumar, Siddhartha Arora, and Mitesh M. Khapra. 2020. Improving dialog evaluation with a multi-reference adversarial dataset and large scale pretraining. _Transactions of the Association for Computational Linguistics_, 8:810-827.
* Searle (1969) John R Searle. 1969. _Speech acts: An essay in the philosophy of language_, volume 626. Cambridge University Press.
* Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7881-7892, Online. Association for Computational Linguistics.
* Shu et al. (2019) Raphael Shu, Hideki Nakayama, and Kyunghyun Cho. 2019. Generating diverse translations with sentence codes. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 1823-1827, Florence, Italy. Association for Computational Linguistics.
* Sinha et al. (2020) Koustuv Sinha, Prasanna Parthasarathi, Jasmine Wang, Ryan Lowe, William L. Hamilton, and Joelle Pineau. 2020. Learning an unprecedented metric for online dialogue evaluation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 2430-2441, Online. Association for Computational Linguistics.
* Stasaki and Hearst (2022) Katherine Stasaki and Marti Hearst. 2022. Semantic diversity in dialogue with natural language inference. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 85-98, Seattle, United States. Association for Computational Linguistics.
* Stasaki and Hearst (2023) Katherine Stasaki and Marti A Hearst. 2023. Pramatically appropriate diversity for dialogue evaluation. _arXiv preprint arXiv:2304.02812_.
* Suzgun et al. (2022) Mirac Suzgun, Luke Melas-Kyriazi, and Dan Jurafsky. 2022. Follow the wisdom of the crowd: Effective text generation via minimum bayes risk decoding. _arXiv preprint arXiv:2211.07634_ 

Guy Tweet and Jonathan Berant. 2021. Evaluating the evaluation of diversity in natural language generation. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 326-346, Online. Association for Computational Linguistics.
* Tiedemann and Thottingal (2020) Jorg Tiedemann and Santhosh Thottingal. 2020. OPUSMT -- Building open translation services for the World. In _Proceedings of the 22nd Annual Conference of the European Association for Machine Translation (EAMT)_, Lisbon, Portugal.
* van der Lee et al. (2021) Chris van der Lee, Albert Gatt, Emiel van Miltenburg, and Emiel Krahmer. 2021. Human evaluation of automatically generated text: Current trends and best practice guidelines. _Computer Speech & Language_, 67:101151.
* Vijayakumar et al. (2018) Ashwin Vijayakumar, Michael Cogswell, Ramprasaath Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2018. Diverse beam search for improved description of complex scenes. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 32.
* Weir et al. (2020) Nathaniel Weir, Joao Sedoc, and Benjamin Van Durme. 2020. COD3S: Diverse generation with discrete semantic signatures. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 5199-5211, Online. Association for Computational Linguistics.
* Wither et al. (2022) Gian Wiher, Clara Meister, and Ryan Cotterell. 2022. On decoding strategies for neural text generators. _Transactions of the Association for Computational Linguistics_, 10:997-1012.
* Xu et al. (2015) Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015. Problems in current text simplification research: New data can help. _Transactions of the Association for Computational Linguistics_, 3:283-297.
* Xu et al. (2016) Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing statistical machine translation for text simplification. _Transactions of the Association for Computational Linguistics_, 4:401-415.
* Yuan et al. (2021) Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. BARTScore: Evaluating generated text as text generation. In _Advances in Neural Information Processing Systems_, volume 34, pages 27263-27277. Curran Associates, Inc.
* Zhang et al. (2020a) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020a. BERTScore: Evaluating text generation with BERT. In _International Conference on Learning Representations_.
* Zhang et al. (2022) Xingxing Zhang, Yiran Liu, Xun Wang, Pengcheng He, Yang Yu, Si-Qing Chen, Wayne Xiong, and Furu Wei. 2022. Momentum calibration for text generation. _arXiv preprint arXiv:2212.04257_.
* Zhang et al. (2020b) Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020b. DIALOGPT : Large-scale generative pre-training for conversational response generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_, pages 270-278, Online. Association for Computational Linguistics.
* Zhao et al. (2022) Yao Zhao, Misha Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J Liu. 2022. Calibrating sequence likelihood improves conditional language generation. _arXiv preprint arXiv:2210.00045_.
* Zhong et al. (2022) Ming Zhong, Yang Liu, Da Yin, Yunning Mao, Yizhu Jiao, Pengfei Liu, Chengwang Zhu, Heng Ji, and Jiawei Han. 2022. Towards a unified multi-dimensional evaluator for text generation. _arXiv preprint arXiv:2210.07197_.

## Appendix A A Note on Entropy

Entropy is an information-theoretic concept that is often used to summarise uncertainty about a random variable. As useful as it may be in various contexts, entropy is not itself a complete characterisation of uncertainty (_e.g._, two different distributions may have the same entropy, yet represent different uncertainty about their respective random variables). As we discuss in SS 3.1, uncertainty about a random variable is fully represented by its underlying probability distribution Halpern (2017, Chapter 2).

Consider a discrete random variable \(X\) with distribution \(p_{X}\) and probability mass function (pmf) \(p_{X}(x)\). Define the _surprisal_ of an outcome \(X=x\) as the quantity \(-\log p_{X}(x)\). Then, _Shannon entropy_ (or just _entropy_ for short) is defined as the surprisal of \(X\) taken in expectation under \(p_{X}\)MacKay (2003). Due to the unbounded sample space and lack of conditional independence assumptions, entropy is intractable to compute for neural text generators. In some cases a Monte Carlo (MC) estimate of entropy can be formed with a reasonable amount of computation. For example, consider an autoregressive language model that assigns probability \(f(x;\theta)\) to a complete sequence \(x\) using a neural network with parameters \(\theta\) (_e.g._, an LSTM or Transformer). When we use ancestral sampling Bishop (2006) to decode from this model obtaining a sample \(x^{(s)}\), the surprisal of \(x^{(s)}\) is directly available via \(-\log f(x;\theta)\), and the sample mean \(-\frac{1}{S}\sum_{s=1}^{S}\log f(x^{(s)};\theta)\) for \(S\) ancestral samples forms an unbiased MC estimate for the entropy of \(X\). In most cases, however, the 

generator's pmf is unknown and the surprisal of an outcome is not available. That is the case, for example, whenever we employ decoding algorithms that bias away from the underlying distribution of the autoregressive LM--top-\(p\), top-\(k\), typical sampling are all good examples. The resulting pmf is then hard (or impossible) to characterise.

Furthermore, as much as Shannon entropy can be interpreted in its own information-theoretic terms, it is not immediately obvious how it can inform an analyst interested in the generator's faithfulness to human production variability. That said, the analyst may be interested in knowing, for example, that the entropy of the generator is similar to that of the 'human distribution' regardless of their ability to assign any useful interpretation to entropy proper. While we accept some analyst out there may be curious about that question, we refrain from performing such an analysis ourselves because (a) MC estimation is not available for most of the popular decoders we wanted to analyse, and (b) estimating the entropy of the human distribution requires a faithful model of it (that is, we need a perfectly faithful text generator to play the role of a 'gold standard').

## Appendix B A Note on the Wasserstein 1-Distance

The Wasserstein 1-Distance \(W_{1}(\cdot,\cdot)\) quantifies a notion of distance between two probability measures and is particularly convenient for it can be estimated using Dirac deltas (samples from those measures; Peyre et al., 2019) more easily than alternatives such as Kolmogorov-Smirnov and total variation distance (which require binning the measurements into empirical cdfs/pdfs). \(W_{1}(M_{k}(x),H_{k}(x))\) and \(W_{1}(C_{k}(x),H_{k}(x))\) have an interpretation in terms of 'mass' (in units of \(k\)) that has to be moved, on average, to transform one set of samples into another.

## Appendix C Data Statistics

Table 1 shows relevant statistics for the four multiple-reference datasets presented in SS 4.

## Appendix D Additional Figures

Figure 7 shows human production variability over lexical and syntactic unigrams, bigrams, and trigrams (complementing Figure 2 in the main paper). Figure 8 shows the distribution of \(\mu_{C_{k}(x)}-\mu_{H_{k}(x)}\) over instances for our four tasks (complementing Figure 3 in the main paper). Figures 9 to 11 show mean divergences across tasks, probes, and decoding algorithms (complementing Figure 5 in the main paper).

 


\begin{table}
\begin{tabular}{c c c c c|c c c|c c c|c c c} \hline  & \multicolumn{4}{c}{_Metric Translation_} & \multicolumn{3}{c}{_Story Generation_} & \multicolumn{2}{c}{_Open-Domain Dialogue_} \\ \cline{1-9} \cline{8-15}  & \multicolumn{3}{c|}{MENA is STD MED} & \multicolumn{3}{c|}{MERANGE} & \multicolumn{3}{c|}{MEAN is STD MED} & \multicolumn{3}{c|}{RANCE} & \multicolumn{3}{c|}{MEAN is STD MED} & \multicolumn{3}{c|}{MED, RANGE} \\ \hline \multirow{6}{*}{**Visco**} & Words & 23.34 \(\pm\) 13.5 & 22 & 3-67 & 22.26 \(\pm\) 8.92 & 21 & 7.57 & 25.40 \(\pm\) 14.18 & 24 & 1.68 & 47.62 \(\pm\) 3.37 & 40 & 5-311 \\  & Tokens & 25.79 \(\pm\) 12.91 & 23 & 4-81 & 28.00 \(\pm\) 11.68 & 26 & 7.78 & 26.49 \(\pm\) 14.68 & 24 & 1.70 & 48.94 \(\pm\) 31.52 & 40 & 5-326 \\  & Sentences & 1.01 \(\pm\) 0.09 & 1 & 1-2 & 1.02 \(\pm\) 0.14 & 1 & 1–2 & 1.75 \(\pm\) 0.93 & 2 & 1.6 & 5.49 \(\pm\) 2.82 & 5 & 1-22 \\  & Words in sent. & 23.15 \(\pm\) 11.37 & 22 & 2-67 & 21.80 \(\pm\) 9.11 & 20 & 1-57 & 14.48 \(\pm\) 7.89 & 14 & 1.50 & 8.57 \(\pm\) 5.20 & 8 & 1-50 \\  & Tokens in sent. & 25.58 \(\pm\) 12.90 & 23 & 2-81 & 27.42 \(\pm\) 11.88 & 25 & 1-78 & 15.12 \(\pm\) 8.13 & 14 & 1-51 & 8.93 \(\pm\) 5.39 & 8 & 1-50 \\ \hline \multirow{6}{*}{**Visco**} & Words & 21.96 \(\pm\) 10.99 & 20 & 2-66 & 19.57 \(\pm\) 8.29 & 18 & 4-62 & 659.72 \(\pm\) 450.46 & 540 & 101:2681 & 0.61 \(\pm\) 4.85 & 10 & 2-46 \\  & Tokens & 27.28 \(\pm\) 14.09 & 25 & 5-86 & 24.22 \(\pm\) 10.65 & 22 & 5-91 & 696.66 \(\pm\) 47.63 & 570 & 104:2961 & 0.84 \(\pm\) 5.01 & 10 & 2-53 \\  & Sentences & 1.06 \(\pm\) 0.05 & 1 & 1-4 & 1.33 \(\pm\) 0.56 & 1 & 1-5 & 47.76 \(\pm\) 35.44 & 38 & 1.008 & 1.32 \(\pm\) 0.52 & 1 & 1-5 \\  & Words in sent. & 20.67 \(\pm\) 10.86 & 19 & 1-66 & 14.70 \(\pm\) 6.71 & 13 & 1-59 & 13.81 \(\pm\) 9.59 & 12 & 1-722 & 8.06 \(\pm\) 4.32 & 7 & 1-36 \\  & Tokens in sent. & 25.69 \(\pm\) 13.92 & 23 & 1-86 & 18.19 \(\pm\) 8.78 & 16 & 1-91 & 14.63 \(\pm\) 10.22 & 12 & 1-722 & 8.24 \(\pm\) 4.45 & 8 & 2-37 \\ \hline \end{tabular}
\end{table}
Table 1: Length statistics. Number of tokens obtained with the tokenisers of the language models used for generation.

Figure 7: Human production variability across four NLG tasks (the remaining settings not reported in the main paper). The values on the \(x\)-axis are single samples of lexical or syntactic distance between two productions for each input (see Section 3). Probability mass on the right side signals high distance and thus high variability, and vice versa. A large spread indicates that production variability varies widely across inputs, and as such that a task does not define a specific level of variability.

 

Figure 8: Distribution of \(\mu_{C_{k}(x)}-\mu_{H_{k}(x)}\) over instances. Values greater than zero indicate the model overestimates the variability of the task (higher mean pairwise distance); values below zero indicate variability underestimation.

Figure 10: Mean of distances \(\mu_{M(x)}-\mu_{H(x)}\) for (tasks, probe, decoding algorithm) tuples across test sets. Base models for each task are described in Section 4. Tuples that share colour have different decoding parameters. Human control group in red; except for dialogue, where 5 references are too few to create a control group.

Figure 9: Mean Wasserstein distances \(D_{W_{1}}(M(x),H(x))\) for (tasks, probe, decoding algorithm) tuples. Base models for each task are described in Section 4. Tuples that share colour have different decoding parameters. Human control group in red; except for dialogue, where 5 references are too few to create a control group.

 

Figure 12: Example 1 of bad fitness (\(D_{W_{1}}(M_{k}(x),H_{k}(x))\)) to human variability for the Opus MT model.

Figure 13: Example 2 of bad fitness (\(D_{W_{1}}(M_{k}(x),H_{k}(x))\)) to human variability for the Opus MT model.

 

Figure 16: Example of good fitness (\(D_{W_{1}}(C_{k}(x),H_{k}(x))\)) for DialoGPT-medium.

Figure 14: Example of good fitness (\(D_{W_{1}}(M_{k}(x),H_{k}(x))\)) to human variability for the Opus MT model.

 



\begin{table}
\begin{tabular}{l l c r l} \hline
**Model** & **Probe** & \multicolumn{2}{c}{**Mean \(D_{W1}(M),H(x))\)**} \\ \hline MarkS\_large, fast-tuned-actual-val & Unigram distance & 0.075174a & Simplotations \\   fastT\_large, large, fast-tuned-ancelux,(9m+sd & Unigram distance & 0.114867 & Sampling direction \\   fastT\_large, fast-tuned-ancelux,(9m+sd & Unigram distance & 0.078365 & Sampling direction \\   fastT\_large, fast-tuned-cycle,(9m+sd & Unigram distance & 0.071721 & Simulation \\   fastT\_large, fast-tuned-cycle,(9m+sd & Unigram distance & 0.068259 & Sampling direction \\   human, control & Unigram distance & 0.042636 & Sampling direction \\   fastT\_large, fast-tuned-interval-val & POS length distance & 0.098695 & Sampling direction \\   fastT\_large, fast-net-nearest-(9m+sd & POS length distance & 0.098695 & Sampling direction \\   fastT\_large, fast-net-node,(9m+sd & POS length distance & 0.124299 & Simulation \\   fastT\_large, fast-tuned-cycle,(9m+sd & POS length distance & 0.108829 & Single direction \\   fastT\_large, fast-repeat-cycle,(9m+sd & POS length distance & 0.244697 & Sampling direction \\   hmm, control & Posts light distance & 0.108139 & Simple direction \\   hmm, control & Posts light distance & 0.046242 & Single direction \\   hmm, control & Costine distance & 0.04942 & Single direction \\   hmm, control & Costine distance & 0.056711 & Single direction \\   hmm,control & Costine distance & 0.051341 & Single direction \\   hmm, control & Costine distance & 0.06563 & Single direction \\   hmm, control & Costine distance & 0.051276 & Simulation \\   yres-uncelux & Unigram distance & 0.23946 & Transducing \\   yres-uncelux,(9m+sd & Unigram distance & 0.286098 & Variable \\   yres-uncelux,(9m+sd & Unigram distance & 0.27219 & Transferring \\   yres-unpermented & Unigram distance & 0.27319 & Transfer function \\   yres-uncelux/(9m+sd & Unigram distance & 0.28274 & Translation \\   yres-uncelux/(9m+sd & Unigram distance & 0.255794 & Transfer function \\   yres-uncelux,(9m+sd & Unigram distance & 0.254140 & Transferring \\   yres-uncelux & Unigram distance & 0.04313 & Transfer function \\   yres-uncelux & POS separation distance & 0.155366 & Transmission \\   yres-uncelux,(9m+sd & Pos design distance & 0.170826 & Translating \\   yres-uncelux,(9m+sd & Uogram distance & 0.168579 & Transformation \\   yres-uncelerated & POS length distance & 0.19544 & Transfer function \\   yres-uncelerux/(9m+sd & Unigram distance & 0.11173 & Transfer function \\   yres-uncelerux/(9m+sd & Unigram distance & 0.15929 & Transfer function \\   yres-uncelerux/(9m+sd & Unigram distance & 0.15929 & Transfer function \\   yres-uncelerux/(9m+sd & Unigram distance & 0.15929 & Transfer function \\   yres-uncelerux/(9m+sd & Unigram distance & 0.03502 & Transfer function \\   yres-uncelerux/(9m+sd & Unigram distance & 0.130312 & Transfer function \\   yres-uncelerux/(9m+sd & Unigram distance & 0.140748 & Translation \\   yres-uncelerux/(9m+sd & Unigram distance & 0.137561 & Transferring \\   yres-uncelerux/(9m+sd & Unigram distance & 0.137561 & Transferring \\   yres-uncelerux/(9m+sd & Unigram distance & 0.137561 & Transferring \\   yres-uncelerux/(9m+sd & Unigram distance & 0.037645 & Story Generation \\   yres-uncelerux/(9m+sd & Unigram distance & 0.082663 & Story Generation \\   yres-uncelerux/(9m+sd & Unigram distance & 0.185606 & Story Generation \\   yres-uncelerux/(9m+sd & Unigram distance & 0.089595 & Story Generation \\   yres-uncelerux/(9m+sd & Unigram distance & 0.089595 & Story Generation \\   yres-uncelerux/(9m+sd & Unigram distance & 0.081151 & Story Generation \\   yres-uncelerux/(9m+sd & Unigram distance & 0.107636 & Story Generation \\   yres-uncelerux/(9m+sd & Unigram distance & 0.112113 & Story Generation \\   yres-uncelerux/(9m+sd & Unigram distance & 0.112652 & Story Generation \\   yres-uncelerux/(9m+sd & Unigram distance & 0.119448 & Story Generation \\   yres-uncelerux/(9m+sd & Unigram distance & 0.09716 & Story Generation \\   yres-uncelerux/(9m+sd & Unigram distance & 0.112113 & Story Generation \\   yres-uncelerux/(9m+sd & Unigram distance & 0.112652 & Story Generation \\   yres-uncelerux/(9m+sd & Unigram distance & 0.190488 & Story Generation \\   yres-uncelerux/(9m+sd & Unigram distance & 0.097114 & Story Generation \\   yres-uncelerux/(9m+sd & Unigram distance & 0.097114 & Story Generation \\   yres-uncelerux/(9m+sd & Unigram distance & 0.09800 & User Generation \\   yres-uncelerux/(9m+sd & Unigram distance & 0.098998 & Store Generation \\   yres-uncelerux/(9m+sd & Unigram distance & 0.110325 & Story Generation \\   yres-uncelerux/(9m+sd & Unigram distance & 0.098026 & Story Generation \\   yres-uncelerux/(9m+sd & Unigram distance & 0.112414 & Story Generation \\   yres-uncelerux/(9m+sd & Unigram distance & 0.110346 & Story Generation \\   yres-uncelerux/(9m+sd & Unigram distance & 0.09663 & Story Generation \\   daliopr-Lange-uncelerux/(9m+sd & Unigram distance & 0.097408 & Open Domain Dialogue \\   daliopr-Lange-uncelerux,(9m+sd & Unigram distance & 0.09718 & OpenDomain Dialogue \\   daliopr-Lange-uncelerux,(9m+sd & Unigram distance & 0.09718 & OpenDomain Dialogue \\   daliopr-Lange-uncelerux,(9m+sd & Unigram distance & 0.09718 & OpenDomain Dialogue \\   daliopr-Lange-uncelerux/(9m+sd & Unigram distance & 0.09918 & OpenDomain Dialogue \\   daliopr-Lange-uncelerux/(9m+sd & Unigram distance & 0.099191 & OpenDomain Dialogue \\   daliopr-Lange-uncelerux,(9m+sd & Unigram distance & 0.099398 & OpenDomain Dialogue \\   daliopr-Lange-uncelerux/(9m+sd & Unigram distance & 0.10279 & OpenDomain Dialogue \\   daliopr-Lange-uncelerux,(9m+sd & Unigram distance & 0.099492 & OpenDomain Dialogue \\   daliopr-Lange-uncelerux/(9m+sd & Unigram distance & 0.112480 & OpenDomain Dialogue \\   daliopr-Lange-uncelerux,(9m+sd & Unigram distance & 0.111521 & OpenDomain Dialogue \\   daliopr-Lange-uncelerux,(9m+sd & Unigram distance & 0.113646 & OpenDomain Dialogue \\   daliopr-Lange-uncelerux,(9m+sd & Unigram distance & 0.113652 & OpenDomain Dialogue \\   daliopr-Lange-uncelerux/(9m+sd & Cosine distance & 0.110345 & OpenDomain Dialogue \\   daliopr-Lange-uncelerux,(9m+sd & Cosine distance & 0.110435 & OpenDomain Dialogue \\   daliopr-Lange-uncelerux,



\begin{table}
\begin{tabular}{l l l c l} \hline
**Model** & **Problem** & **Mean** \(D_{j}(M(x),H(y))\) & **Task** \\ \hline fm17sJape. flattened-accentral-val & Unigram distance & -0.041574 & Simplification \\ fm17sJape. Jaccard-encelaxes,09-val & Unigram distance & -0.107347 & Simplification \\ fm17sJape. Jnemetuned-ancelaxes,09-val & Unigram distance & -0.085731 & Simplification \\ fm17sJape. Jnemetuned-pycol2,09-val & Unigram distance & -0.21426 & Simplification \\ fm17sJape. Jnemetuned-pycol,09-val & Unigram distance & -0.084748 & Simplification \\ fm17sJape. Jnemetuned-recelaxes,09-val & Unigram distance & -0.004613 & Simplification \\ fm17sJape. Jnemetuned-accumital-val & POS bias distance & -0.048417 & Simplification \\ fm17sJape. Jnemetuned-incelaxes,09-val & POS bias distance & -0.111659 & Simplification \\ fm17sJape. Jnemetuned-npcol,09-val & POS bias distance & -0.088022 & Simplification \\ fm17sJape. Jnemetuned-pycol,09-val & POS bias distance & -0.240776 & Simplification \\ fm17sJape. Jnemetuned-pycol,09-val & POS bias distance & -0.067796 & Simplification \\ fm17sJape. Jnemetuned-pycol,09-val & POS bias distance & -0.005135 & Simplification \\ fm17sJape. Jnemetuned-acetal-val & Costine distance & -0.016643 & Simplification \\ fm17sJape. Jnemetuned-pycol,09-val & Covine distance & -0.048492 & Simplification \\ fm17sJape. Jnemetuned-pycol,09-val & Covine distance & -0.048492 & Simplification \\ fm17sJape. Jnemetuned-pycol,09-val & Covine distance & -0.048496 & Simplification \\ hmm\_control & Cosine distance & -0.005054 & Simplification \\ gpus-mccas,09-val & Uugram distance & -0.249243 & Transformation \\ qus-mccas,08 & Uugram distance & -0.248126 & Translation \\ qus-mccas,09 & Unigram distance & -0.246194 & Transfaction \\ qus-mccas,09 & Unigram distance & -0.246194 & Transfaction \\ qus-mccasemt09 & Unigram distance & -0.18041 & Transfaction \\ qus-mccasemt09 & Unigram distance & -0.248196 & Transfaction \\ qus-mccas,09 & Unigram distance & -0.25858 & Transfaction \\ qus-mccas,09 & Unigram distance & -0.25043 & Transfaction \\ human\_control & Unigram distance & 0.251381 & Transfaction \\ guu-mccastimal & POS bignan distance & -0.151919 & Transfaction \\ quu-mccas,09 & POS bignan distance & -0.169243 & Transfaction \\ quu-mccas,09 & POS bignan distance & -0.162130 & Transfaction \\ quu-mccasemt05 & POS bignan distance & -0.152725 & Transfaction \\ uqus-mccasemt05 & POS bignan distance & -0.17065 & Transfaction \\ opu-mccas,09 & POS bignan distance & -0.157825 & Transfaction \\ apu-mccasemt05 & Uugram distance & -0.157825 & Transfaction \\ apu-mccas,09 & Uugram distance & -0.15329 & Transfaction \\ hum\_control & POB bignan distance & -0.00900 & Transfaction \\ equu-mccasemt05 & Cuite distance & -0.120637 & Transfaction \\ quu-mccas,08 & Couite distance & -0.13989 & Transfaction \\ quu-mccas,09 & Coast distance & -0.130695 & Transfaction \\ quu-mccasemt05 & Coast distance & -0.14734 & Transfaction \\ quu-mccasamt05 & Coast distance & -0.147533 & Transfaction \\ quu-mccas,09 & Coast distance & -0.13257 & Transfaction \\ pcrd\_large\_financed-accentral-test & Uniform distance & -0.001277 & Story Generation \\ pq2d\_large\_financed-accentral-test & Uniform distance & -0.021242 & Story Generation \\ pq2d\_large\_financed-process,09-set & Uniform distance & -0.01460 & Story Generation \\ pq2d\_large\_financed-prompt-process,09-set & Uniform distance & -0.041395 & Story Generation \\ pq2d\_large\_financed-prompt-process,09-set & Uniform distance & -0.041395 & Story Generation \\ pq2d\_large\_financed-post,09-set & Uniform distance & -0.04259 & Story Generation \\ bum\_control & Unigram distance & -0.006836 & Story Generation \\ pm2d\_large\_financed-normal-test & Pos bignan distance & -0.06839 & Story Generation \\ ph2d\_large\_financed-specual,09-set & NoIs gram distance & -0.06839 & Story Generation \\ daiberg\_large\_investral-dev & Unique gram distance & 0.069756 & Open-Domain Dialogue \\ hairst\_count & Unknown distance & 0.03868 & Open-Domain Dialogue \\ daiberg\_large\_investral-09-set & Unique gram distance & 0.068515 & Open-Domain Dialogue \\ daiberg\_large\_in-type,1-0,0-v & Unique gram distance & 0.068492 & Open-Domain Dialogue \\ daiberg\_large\_in-type,1-0,0-v & Unique gram distance & 0.068738 & Open-Domain Dialogue \\ daiberg\_large\_in-type,1-0,0-v & Unique gram distance & 0.068495 & Open-Domain Dialogue \\ daiberg\_large\_in-type,1-0,0-v & Unique gram distance & 0.068573 & Open-Domain Dialogue \\ daiberg\_large\_in-type,1-0,0-v & Unique gram distance & 0.068495 & Open-Domain Dialogue \\ daiberg\_large\_in-type,1-0,0-v & Cros sigma distance & 0.068505 & Open-Domain Dialogue \\ launt\_count & POS bignan distance & NAN & Opens-Domain Dialogue \\ daiberg\_large\_in-type,1-0,0-v & POS bignan distance & 0.068257 & Open-Domain Dialogue \\ daiberg\_large\_in-type,1-0,0-v & POS bignan distance & 0.068412 & Open-Domain Dialogue \\ daiberg\_large\_in-type,1-0,0-v & Cros signature distance & 0.068316 & Open-Domain Dialogue \\ daiberg\_large\_in-type,1-0,0-v & Cros signature distance & 0.068636 & Open-Domain Dialogue \\ daiberg\_large\_in-type,1-0,0-v & Cros signature distance & 0.068463 & Open-Domain Dialogue \\ daiberg\_large\_in-type,1-0,0-v & Cros signature distance & 0.068731 & Open-Domain Dialogue \\ daiberg\_large\_in-type,1-0,0-v & Cros signature distance & 0.068356 & Open-Domain Dialogue \\ daiberg\_large\_in-type,1-0,0-v & Cros signature distance & 0.064717 & Open-Domain Dialogue \\ daiberg\_large\_in-type,1-0,0-v & Cros signature distance & 0.054787 & Open-Domain Dialogue \\ daiberg\_large\_in-type,1-0,0-v & Cros signature distance & 0.0684787 & Open-Domain Dialogue \\ daiberg\_large\_in-type,1-0,0-v & Cros signature distance & 0.068547 & Open-Domain Dialogue \\ daiberg\_large\_in-type,1-0,0-v & Cros signature distance & 0.064787 & Open-Domain Dialogue \\ daiberg\_large\_in-type,1-0,0-v & Cros signature distance & 0.





\begin{table}
\begin{tabular}{l l l c l} \hline
**Model** & **Problem** & **Mean** \(D_{W1}(C(x),H(x))\) & **Task** \\ \hline flart5\_large, fast-tensed-ancestral-val & Uniguan distance & 0.047967 & Singlefraction \\ flart5\_large, large, fast-tensed-ancks,(9-val) & Uniguan distance & 0.069384 & Simulation \\ flart5\_large, fast-tensed-ancks,(9-val) & Uniguan distance & 0.067209 & Singlefraction \\ flart5\_large, fast-tensed-pixel,(0.02-val) & Uniguan distance & 0.069894 & Singlefraction \\ flart5\_large, fast-tensed-pixel,(0.05-val) & Uniguan distance & 0.054325 & Singlefraction \\ flart\_small, compact & Uniguan distance & 0.062865 & Singlefraction \\ flart5\_large, fast-tensed-ancksat-val & POS hapan distance & 0.064644 & Singlefraction \\ flart5\_large, fast-tensed-ancks,(9-val) & POS hapan distance & 0.065849 & Singlefraction \\ flart5\_large, fast-tensed-uncks,(9-val) & POS hapan distance & 0.0614433 & Singlefraction \\ flart5\_large, fast-tensed-pixel,(0.2-val) & POS hapan distance & 0.078972 & Singlefraction \\ flant5\_large, fast-tensed-pixel,(9-val) & POS hapan distance & 0.061848 & Singlefraction \\ flantum\_small & Poisson dynamic distance & 0.064024 & Singlefraction \\ flart5\_large, fast-tensed-ancestral-val & Coine distance & 0.027983 & Singlefraction \\ flart5\_large, fast-tensed-ancks,(9-val) & Coine distance & 0.028626 & Singlefraction \\ flart5\_large, fast-tensed-mask,(9-val) & Coine distance & 0.029434 & Singlefraction \\ flart5\_large, fast-tensed-pixel,(9-val) & Coine distance & 0.036086 & Singlefraction \\ flart\_small, compact & Cosine distance & 0.035270 & Singlefraction \\ flart\_small, compact & Cosmic distance & 0.03531 & Singlefraction \\ open-arcenter & Uuguan distance & 0.06461 & Transfusion \\ open-arcenter,(9-val) & Uuguan distance & 0.069170 & Transmission \\ open-arcenter,(9-val) & Uuguan distance & 0.068676 & Translation \\ open-openmerned5\% & Uuguan distance & 0.125683 & Translations \\ open-temperaturened5\% & Uuguan distance & 0.070435 & Transfusion \\ open-to-j,k\_30 & Uuguan distance & 0.068022 & Transition \\ open-torg\_>,k\_40 & Uuguan distance & 0.066907 & Translation \\ human, control & Uuguan distance & 0.043593 & Transitions \\ open-internal & Pols hapan distance & 0.06566 & Transient \\ open-nucle,(9-val) & POS hapan distance & 0.067369 & Translation \\ open-corenet\% & Pols hapan distance & 0.065888 & Translation \\ open-corenet\% & Pols hapan distance & 0.068478 & Translation \\ open-target\% & Pols hapan distance & 0.068195 & Transfusion \\ open-update\% & Pols hapan distance & 0.068333 & Transitions \\ open-loop\% & Pols hapan distance & 0.068534 & Transitions \\ human,control & Pols hapan distance & 0.035402 & Transitions \\ open-autecast\% & Costin distance & 0.043123 & Transitions \\ open-accstatic\% & Costin distance & 0.043123 & Transitions \\ open-molecule\% & Costin distance & 0.043521 & Transitions \\ open-atomenet\% & Cosine distance & 0.045241 & Transitions \\ open-top\% & X\% & Cosine distance & 0.043254 & Transitions \\ open-up+,k\_30 & Cosine distance & 0.042133 & Transitions \\ open-atomenet\% & Cosine distance & 0.042764 & Transformation \\ pla2\_large, fast-tensed-ancestral-test & Diagram distance & 0.029215 & Sage Generation \\ pla2\_large, fast-tensed-ancestral-test & Expansion distance & 0.029246 & Sage Generation \\ pla2\_large, fast-tensed-ancestral-test & Equation distance & 0.029246 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.071867 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.06735 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.06735 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.06735 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.06735 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.06735 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.078646 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.067358 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.078846 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.078946 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.078947 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.078947 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.078947 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.078947 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.079259 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.079347 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.078947 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.079470 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.079507 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.078947 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.079259 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.079364 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.078947 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.079470 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.079507 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.078947 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.079635 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.078947 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.079647 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.079736 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.078947 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.079947 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.079947 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.079947 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.079947 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.089947 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.089947 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.089947 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.089947 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.089947 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.089947 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.089947 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.089947 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.089947 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.089947 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.089947 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.099471 & Sage Generation \\ pla2\_large, fast-tensed-autoestra & Uuguan distance & 0.099472 & Sage Generation \\ pla2\_large



\begin{table}
\begin{tabular}{l l l c l} \hline
**Model** & **Problem** & **Mean** \(D_{\mu}(C(x),H(x))\) & **Task** \\ \hline fmatTS\_large- flattened-accentral-val & Unigram distance & -0.019180 & Singlefraction \\ fmatTS\_large- flattened-uncorras\_00-val & Unigram distance & -0.047716 & Singlefraction \\ fmatTS\_large- flattened-uncorras\_00-val & Unigram distance & -0.035796 & Singlefraction \\ fmatTS\_large- flattened-hypercut\_02-val & Unigram distance & -0.047900 & Singlefraction \\ fmatTS\_large- flattened-hypercut\_02-val & Unigram distance & -0.038222 & Singlefraction \\ fmatTS\_large- flattened-hypercut\_02-val & Unigram distance & -0.004613 & Singlefraction \\ fmatTS\_large- flattened-uncorras\_02-val & UPG spam distance & -0.024717 & Singlefraction \\ fmatTS\_large- flattened-uncorras\_02-val & UPG spam distance & -0.05054 & Singlefraction \\ fmatTS\_large- hypercut\_02-val & POS fragment distance & -0.041670 & Singlefraction \\ fmatTS\_large- hypercut\_02-val & POS fragment distance & -0.055707 & Singlefraction \\ fmatTS\_large- hypercut\_02-val & POS fragment distance & -0.055707 & Singlefraction \\ fmatTS\_large- hypercut\_02-val & POS fragment distance & -0.040591 & Singlefraction \\ fmatTS\_large- hypercut\_02-val & Post fragment distance & -0.040535 & Singlefraction \\ fmatTS\_large- hypercut\_02-val & Cost feature & -0.040719 & Singlefraction \\ fmatTS\_large- hypercut\_02-v & Pos fragment distance & -0.041733 & Singlefraction \\ fmatTS\_large- hypercut\_02-v & Cost feature & -0.041804 & Singlefraction \\ fmatTS\_large- hypercut\_02-val & Cosine distance & -0.022718 & Singlefraction \\ fmatTS\_large- hypercut\_02-v & Cost feature & -0.041771 & Singlefraction \\ hmm\_contral & Contifrance distance & -0.059540 & Singlefraction \\ gpus-ancell & Upram distance & -0.031401 & Transfusion \\ qus-mcmc\_00-val & Unigram distance & -0.032499 & Transfusion \\ qus-mcmc\_00-val & Unigram distance & -0.033249 & Transfusion \\ qus-mcmc\_00-val & Unigram distance & -0.036202 & Transfusion \\ qus-mcmc\_00-val & Unigram distance & -0.06908 & Transfusion \\ qus-mcmc\_00-val & Unigram distance & -0.027347 & Transfusion \\ qus-mcmc\_00-val & Unigram distance & -0.031515 & Transfusion \\ pus-mcmc\_00-val & Unigram distance & -0.031321 & Tranfusion \\ qus-mcmc\_00-val & POS fragment distance & -0.010164 & Transfusion \\ qus-mcmc\_00-val & POS fragment distance & -0.011368 & Transfusion \\ qus-mcmc\_00-val & POS fragment distance & -0.00588 & Transfusion \\ qus-mcmc\_00-val & POS fragment distance & -0.007351 & Transfusion \\ qus-mcmc\_00-val & POS fragment distance & -0.094918 & Transfusion \\ qus-mcmc\_00-val & POS fragment distance & -0.010525 & Transfusion \\ qus-mcmc\_00-val & POS fragment distance & -0.010689 & Transfusion \\ human\_contral & POS fragment distance & -0.090300 & Transfusion \\ qus-mcmc\_00-val & Cosine distance & -0.014492 & Transfusion \\ qus-mcmc\_00-val & Cosine distance & -0.014927 & Transfusion \\ qus-mcmc\_00-val & Cosine distance & -0.015266 & Transfusion \\ qus-mcmc\_00-val & Cosine distance & -0.015637 & Transfusion \\ qus-mcmc\_00-val & Cosine distance & -0.015749 & Transfusion \\ qus-mcmc\_00-val & Cosine distance & -0.015931 & Transfusion \\ qus-mcmc\_00-val & Cosine distance & -0.01588 & Transfusion \\ qus-mcmc\_00-val & POS fragment distance & -0.007351 & Transfusion \\ qus-mcmc\_00-val & Cosine distance & -0.009418 & Transfusion \\ grd\_large\_structured-uncertain+test & Unigram distance & -0.034652 & Story Generation \\ grd\_large\_structured-uncertain+test & Unigram distance & -0.029564 & Story Generation \\ grd\_large\_structured-curse\_00-val & Unigram distance & -0.036164 & Story Generation \\ grd\_large\_structured-curse\_00-val & Unigram distance & -0.068808 & Story Generation \\ bmm\_control & Unigram distance & -0.032071 & Story Generation \\ bmm\_control & Unigram distance & -0.005858 & Story Generation \\ prd\_large\_layered-curse\_00-val & POS bgram distance & -0.06666 & Story Generation \\ grd\_large\_forest\_00-val & POS bgram distance & -0.073421 & Story Generation \\ prd\_large\_structured-curse\_00-val & POS bgram distance & -0.06908 & Sys Generation \\ grd\_large\_structured-curse\_00-val & POS bgram distance & -0.06908 & Sys Generation \\ grd\_large\_structured-curse\_00-val & Copine distance & -0.05373 & Story Generation \\ grd\_large\_structured-curse\_00-val & Cocine distance & -0.047109 & Story Generation \\ grd\_large\_structured-curse\_00-val & Couine distance & -0.048778 & Story Generation \\ grd\_large\_structured-curse\_00-val & Coine distance & -0.048778 & Story Generation \\ grd\_large\_structured-curse\_00-val & Cocine distance & -0.06529 & Story Generation \\ grd\_large\_structured-curse\_00-val & Cowine distance & -0.045444 & Story Generation \\ lamura\_control & Cunine distance & -0.06038 & Sys Generation \\ daliberg\_large\_un-curse\_00-val & Citiagram distance & -0.06038 & Sys Generation \\ lamura\_control & Unigram distance & -0.06038 & Open-Domain Dialogue \\ lamura\_control & Unigram distance & NN & Open-Domain Dialogue \\ dasiberg\_top-curse\_00-val & Unigram distance & -0.05791 & Open-Domain Dialogue \\ daiberg\_top-curse\_00-val & Unigram distance & -0.063929 & Open-Domain Dialogue \\ daiberg\_top-curse\_00-val & Unigram distance & -0.064805 & Open-Domain Dialogue \\ daiberg\_top-curse\_00-val & Unigram distance & -0.065979 & Open-Domain Dialogue \\ daiberg\_top-pixel\_02-dev & Untagram distance & -0.077755 & Open-Domain Dialogue \\ daiberg\_top-pixel\_05-dev & Unigram distance & -0.064922 & Open-Domain Dialogue \\ daiberg\_top-image\_internal-dev & POS bgram distance & -0.057952 & Open-Domain Dialogue \\ laume\_control & POS bgram distance & NAU & Open-Domain Dialogue \\ daiberg\_top-curse\_00-val & POS bgram distance & -0.045909 & Open-Domain Dialogue \\ daiberg\_top-curse\_00-val & POS bgram distance & -0.047880 & Open-Domain Dialogue \\ daiberg\_top-type\_00-val & POS bgram distance & -0.046722 & Open-Domain Dialogue \\ daiberg\_top-type\_00-val & JOBgram distance & -0.049299 & Open-Domain Dialogue \\ daiberg\_top-pixel\_02-dev & POS bgram distance & -0.054858 & Open-Domain Dialogue \\ daiberg\_top-pixel\_02-var & POS bgram distance & -0.049841 & Open-Domain Dialogue \\ daiberg\_top-pixel\_02-var & Cosmic distance & -0.078752 & Open-Domain Dialogue \\ daiberg\_top-curse\_00-val & Cosine distance & -0.079872 & Open-Domain Dialogue \\ daiberg\_top-curse\_00-val & Cosine distance & -0.079873 & Open-Domain Dialogue