Distribution Aware Metrics for Conditional Natural Language Generation
Authors: David Chan
Yiming Ni
Sudheendra Vijayanarasimhan
David A Ross
Austin Myers
John Canny
Venue: Submitted to ICLR 2023 (2023)
Keywords: Natural Language Generation
Video Description
Image Description
Metrics

Abstract: Traditional automated metrics for evaluating conditional natural language generation use pairwise comparisons between a single generated text and the best-matching gold-standard ground truth text. When multiple ground truths are available, scores are aggregated using an average or max operation across references. While this approach works well when diversity in the ground truth data (i.e. dispersion of the distribution of conditional texts) can be ascribed to noise, such as in automated speech recognition, it does not allow for robust evaluation in the case where diversity in the ground truths represents signal for the model. In this work we argue that existing metrics are not appropriate for domains such as visual description or summarization where ground truths are semantically diverse, and where the diversity in those captions captures useful additional information about the context. We propose a novel paradigm for multi-candidate evaluation of conditional language generation models, and a new family of metrics that compare the {\em distributions} of reference and model-generated caption sets using small sample sets of each.  We demonstrate the utility of our approach with a case study in visual description: where we show that existing models optimize for single-description quality over diversity, and gain some insights into how sampling methods and temperature impact description quality and diversity.

Under review as a conference paper at ICLR 2023
DISTRIBUTION AWARE METRICS FOR CONDITIONAL NATURAL LANGUAGE GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
Traditional automated metrics for evaluating conditional natural language gen- eration use pairwise comparisons between a single generated text and the best- matching gold-standard ground truth text. When multiple ground truths are avail- able, scores are aggregated using an average or max operation across references. While this approach works well when diversity in the ground truth data (i.e. dis- persion of the distribution of conditional texts) can be ascribed to noise, such as in automated speech recognition, it does not allow for robust evaluation in the case where diversity in the ground truths represents signal for the model. In this work we argue that existing metrics are not appropriate for domains such as visual description or summarization where ground truths are semantically diverse, and where the diversity in those captions captures useful additional information about the context. We propose a novel paradigm for multi-candidate evaluation of con- ditional language generation models, and a new family of meta- meta-metries built on top of existing pairwise metrics that compare the distributions of reference and modee--gnnerated caption sets using small sample sets of each. We demonstrate the utility of our approach with case study in visual description: where show that existing models optimize for single-description quality over diversity, and gain some insights into how sampling methods and temperature impact description quality and diversity.
INTRODUCTION
Recent models for conditional language generation, particularly in the field of visual description, have shown dramatic improvements in both fluency and the ability to ground generated language in context (Liu et al., 2021: Zhou et al., 2020; Mokady et al., 2021; Chen et al., 2018). Standard metrics for these tasks such as BLEU, ROUGE, METEOR, and CIDEr, compare a generated text with a reference set of texts and compute some measure of quality for the generated text. By construction of these metrics, a model will achieve the best performance by generating a single high-scoring text. In contrast, it has been widely observed that large language models such GPT-3 (Brown al., 2020) or LAMDA (Thoppilan et al., 2022) generate the most realistic texts at temperatures close to one, where the set of potential texts generated is often very diverse. More significantly, if we look an example of an image from MS-COCO and its set of reference captions (Figure 1), we notice that each (human- ) reference contains unique subset of the overall information in the image:
"A woman in a red robe is sitting at a dining table."
"A woman in red flowered shawl sits at table while
wearing jeans in the kitchen looking at her." another person stands the kitchen." "
"A person sits at a table
"A woman sitting at table wearing a rohe while a man is cooking."
"Man and woman in kitchen looking in the same direction."
Important features like the red robe, the man, the gaze of the two people etc, are mentioned only in one or a few captions. Metries that encourage generating information from only one of these captions will generally fail to capture much of the important detail in the image. This holds for more than just image description. For many conditional language generation tasks such as video captioning, abstractive summarization, translation, and open-ended question-answering, it is often beneficial to be able to sample from diverse distribution of generated outputs.
If we compute caption from a otate-of-the-art model (Zhou et al., 2020) we get:
"A woman sitting in kitchen man."

---

Under review as a conference paper at ICLR 2023
A porsse sits ano toble and onotter person stords kitchen
oommn is aiiting of habe woetting soeling
Men and weman yn lafener
locking in the Jame direction
secrnon sis kicchol nott to mon (K2)
locmimm atling finp Rid has moet lo cimon (44)
O Awammnn Rowered show while o man wetning foons in the Alcnen looking ner
ot dining toble
Context
Samples
Ideal Model Samples
Ground Truth References O
Samples from these two models achieve similar BLEU scores, however, the samples from VLP lie near center of the distribution, and fail capture the dispersion of natural language in the ground truths, while the samples from an ideal model better match the ground truth distribution. In this work, we introduce metries which better measure deviations between samples from candidate and reference distributions, compared to single-sample pairwise metrics.
Figure I:
In this description, we see that only information common to most or all of the reference captions is preserved. This is intuitive, since including more information runs the risk that no reference caption contains that information, leading to low score. Note that this caption may not actually be the most likely (highest expected similarity to a reference caption), because e.g. the BLEU score also includes term encouraging longer texts. It seems the designers of these metrics are already aware that direct use of shortest distance to a reference caption favors generated captions which are even shorter and more impoverished. However, the (log-) text length heuristic in standard metrics is poor proxy for actual diversity. Instead of generating a variety of captions, taking 10 samples from the state-of-the-ar model, yields only 10 repetitions of the above caption.
Thus, encounter issue in the evaluation of conditional text generation models from multiple sampled texts. When several ground truths are available, typically the metric score is based on the maximum score with some ground truth. This leads to issue, shown earlier, where the model is encouraged to produce a text with the lowest expected distance (max pairwise score for particular n-gram in BLEU) Changing the metric aggregation method, say from max score over reference examples to average or sum (ROUGE), does not change the situation substantially, The model will still be encouraged to produce a single output with high average scores to nearby references, which will be maximized at a smoothed mode in the training text distribution. Failure modes of other methods of aggregation are discussed in both Caglayan et al. (2020) and Yeh et al. (2021), including issues with multi-modal reference distributions and single outlier texts.
reference text, i.e. near a strong mode in the training text distribution.
Such an over-reliance on simple aggregations for multiple candidates and references has, over time, compounded into several issues: The first, discussed further in section 3, is that, as observed in visual description by Chan et al. (2022) and dialog generation by Caglayan et al. (2020), human performance on datasets under existing metrics is often lower than model performance, even though cuman--geneatted captions tend to receive higher scores under human evaluation. The second, discussed in section 2, is that diversity of candidate texts is largely relegated to reference-unaware measures, encouraging models to diverge from ground truth distributions to hit diversity targets.
In this work, we aim solve these problems by introducing several novel automated ways of measuring the performance of conditional text generation models based on measuring the divergence between samples from two text distributions. While some recent methods have been designed to closely measure the divergence between full distributions of text data in the unconditional case (Pillutla et al., 2021), no such methods exist for the conditional generation case, which operates on the level of 10s of reference samples and candidates. Our contributions are summarized as follows:
1. We introduce a new paradigm for the evaluation of conditional text generation models based sampling from both candidate and reference distributions.
2. We introduce two new families of metrics which extend existing semantic distances: triangle- rank metrics, and kernel-basen metrics, designed to measure the divergence between small text samples from candidate and reference distributions.
2

---

Under review as a conference paper at ICLR 2023
3. We explore how new metries behave in the context of visual description (both image and video description) and show that by measuring distributional effects, we can capture nuances in the data that existing metries cannot explore.
2 RELATED WORK
This work is not the first to notice the shortcomings of traditional metrics for the automated evaluation of conditional language generation models. In visual dialog, Caglayan et al. (2020) find that number of the automated metrics proposed for visual dialog do not match well with human judgment, while in visual description, Chan et al. (2022) find that current automated metrics do not assign high scores to human- -eneaattd descriptions. This work not only quantifies such issues but proposes a method for addressing these cases without developing novel metrics for measuring text semantic distance. In this section, we review related works, roughly divided into three groups; methods for evaluating text quality, text diversity and distribution aware metrics.
Measuring the Quality of Generated Text: been an active area of research, which has continuously evolved to keep pace with accelerating advances in text generation. As a consequence of the tools available and the state of early text generation approaches, classical measures have primarily focused on evaluating the quality of generated text with respect to ground truth references using surface-level text statistics. Most notably, these include n n-gram matching based metrics like BLEU (Papineni et al., 2002), METEOR (Banerjee & Lavie, 2005), ROUGE (Lin, 2004), and CIDEr (Vedantam et al., 2015). More recently, the rapid progress enabled by large-scale language models has motivated new evaluation techniques which go beyond superficial n n-gram statistics and toward measures that aim to capture the underlying semantics of language (Shimanaka et al., 2018: Clark et al., 2019; Zhang* al., 2020; Sellam et al., 2020). These approaches leverage high dimensional representations of generated and reference text provided by ofate-of-theeare language model, such as BERT (Devlin et al., 2018) in the case of BERTScore (Zhang* et al., 2020) and BLEURT (Sellam et al., 2020). While such methods are focused on measuring the semantic distance between two pairs of natural language texts, the evaluation of the diversity of the generated captions has largely been done independently of quality.
The evaluation of machine-generater text has long
Measuring the Diversity of Generated Text: Until recently, measures of diversity for generated text have been largely secondary to measures of quality, since the pursuit of human- like generated text has been the primary focus of the field. In fact, many diversity measures quantify surface-level statistics of the generated text (Van Miltenburg et al., 2018), such as metrics based on the number of unique tokens, unique sentences, or unigram frequency statistics, such as Zipf coefficients (Holtzman et al., 2019). Similarly, n--rram based diversity measures such as self-BLEU (Zhu et al., 2018 18), compute scores between samples from a model. Unfortunately, these approaches do not consider the diversity of model's 's outputs with respect to the diversity of human references. Such measures are also primarily focused on the diversity of the vocabulary, rather than the aggregate semantic diversity, a factor that our proposed work aims to address.
Distribution Aware Measures of Generated Text: Recently, MAUVE, proposed by Pillutla et al. (2021), addresses the single-text evaluation paradigm by measuring the divergence between multi- candidate samples and multiple ground truths using density estimates in a text embedding space. Such an approach has the power to measure both dispersion of the text and the quality of the generated text simultaneously. In this respect, MAUVE is the most similar work to ours, in that proposed a distribution-awarr metric. That being said, MAUVE is designed for the setting of unconditional text generation, where many ground truth samples are available, and the entire dataset of human reference texts is compared to model outputs. Unfortunately, while MAUVE works well in these scenarios, does not work well when only a few references are available (due to its K-means distributional approximation) (see appendix B.5). Such low-reference scenario is common in conditional NLG, making MAUVE unsuitable for many potential applications.
METHODS
In this section, we introduce our two primary contributions. First, we introduce and demonstrate the need for paradigm for multiple candidate evaluation for conditional language generation, and
3

---

Under review as a conference paper at ICLR 2023
Figure 2: Intuition for TRMs. For samples from different distributions (left), in-distribution edges will often be short, but for identical distributions (right), edge funk-ditribbttoons will be more uniform.
second, we introduce several simple augmentations to existing pairwise metrics, designed to alleviate the sensitivity issues induced by evaluating conditional language generation models with only single candidate text. Our family of augmented metrics, which we dub Triangle-Rame Metrics (TRMs), represents the first step towards optimizing metrics that force models not only to generate samples at the locus of a distribution but also with sufficient variance, hopefully alleviating the field-wide issues that optimizing standard pairwise-metries can induce.
3.1 MULTIPLE CANDIDATE/MULTIIPE REFERENCE EVALUATION
Traditionally, most methods for conditional language generation have been designed to sample a single candidate example using beam search, designed to be a maximum likelihood sample of the data. This single candidate is compared against the reference data. Unfortunately, as discussed in section I, models can easily exploit such aggregations. For example, when the best score amongst the ground truths is chosen (the "min-distance" aggregate), models generate texts optimizing the expected minimum distance to the reference distribution. Such text is, by definition, the mode of the distribution. While we don't 't know exactly what this looks like in the case of natural language, the mode likely represents some amount of central tendency. In practice, we observe such central captions tend to be bland and uninformative, as demonstrated in Chan et al. (2022) for visual description, Yang al. (2019) for image generation, and appendix B.3.
Thus, single candidate may not be sufficient to understand if the model has learned to approximate the reference distribution. Consequently, we aim to develop methods that can sample several suitable candidate texts, each with high accuracy, while matching the diversity of the ground truth distribution. In this work, to extend methods to multiple candidate generation, we leverage temperature- based sampling or nucleus sampling (as indicated) to produce multiple candidates from each model's distribution. While beam search can generate multiple candidates, Vijayakumar et al. (2016) showed diversity among beams relatively poor, leading to samples that diverge from the model distribution. This gives model which generates multiple candidate samples, and requires an evaluation metric which compares multiple candidate samples to multiple reference samples.
Note that sampling approach to generating diverse captions does not preclude a future effort toward generating single "omnibus" captions, which capture detail from many diverse captions. However, such captions will be much longer than typical human captions, and will score poorly under the standard metrics, as they would be quite different from (much more detailed than) individual reference captions. A path toward such omnibus captions (assuming they are practically useful), would to first generate diverse human-like captions, and then summarize set of them into single text. The present work still supports the first step of diverse caption set generation and optimization.
Extending Existing Metrics for Multi-Candidate Evaluation Currently, no standard pairwise metrics (BLEU, METEOR, CIDEr, ROUGE, CIDEr, SPICE) support comparison between multiple candidates and multiple references, and the most efficient extension of existing metrics to multi- candidate, multi-reference situations is a non-trivial task. In this work, we naively extend the existing pairwise metrics (Papineni et al., 2002; Agarwal & Lavie, 2008; Lin, 2004; Vedantam et al., 2015; Zhang* et al.. 2020) to multiple candidates through the use of mean aggregation. Thus, for standard pairwise score S. set of candidates (Ci,..., = C and a set of references (T1...., Tm)) = R, we assign the output score Sagg
1
Sagg
S(c, R)
3.2 TRIANGLE-RANK METRICS (TRMS)
Existing metrics for semantic similarity are extremely powerful for determining pairwise semantic distances between two utterances (Papineni al., 2002; Agarwal & Lavie, 2008; Lin, 2004; Vedantam
4

---

Under review as a conference paper at ICLR 2023
et al., 2015; Anderson et al., 2016), thus, it makes little sense to discard the existing research when extending metrics to a multiple-candidate evaluation. Further, these metrics are already well correlated with human judgements of quality thus, we need only to increase the sensitivity of these measures when faced with multiple candidates and references, rather than rebuild the entire set of distributional measures. How, then, can leverage already strong pairwise tools in a multiple candidate scenario? Unfortunately, many statistical techniques for measuring the distances between samples require points to lie in a metric space (Basseville, 2013) however, most text distances neither respect symmetry nor triangle inequality.
We propose an answer based on an application of the triangle-rank statistic for statistical testing proposed by Liu & Modarres (2011). The triangle-rank statistic has several promising properties: neither requires symmetry nor the triangle inequality in the metric space (it only requires d(z, x) = 0), and it is computed using only pairwise distances, meaning that we can easily reuse existing text semantic distance functions when computing the statistic.
For the purpose of explanation, it can be helpful to think of texts as points on an arbitrary manifold (based the selected text distance function). To compute the triangle-rank statistic for given distance S, a set of candidates (c1,...., Cn) = C. and a set of references (ri,.... .mm) = R, we first extract all directed triangles (f1....) = T, such that one point lies in C' and two points lie in R. We
refer to the edge between points from the same distribution as et, and the other two edges as c and et;
We then compute the score for each of the edges. For (a, b) = let d(ei;) = S(a, b)
(2)
We then compute indicators lo. 11 1z for each triangle 4; as follows: Bo),
lo(4;) =1 if )(e).) < d(((t)) (, d(ee,,) Ei) else 0
(t,) =1 if d(c S d(ch)) S d(e,;) d(e d(cl)) S d(e,.)) ) else 0
(2(4) =1 if d(e.. R,0) d( S d(e))) else 0
These indicators represent the rank of the samele edge (if it is the smallest, largest, middle- sized edge). The directed statistic for the sample (C, R). Q(C, R) is then computed as:
Q R) Eneer It(t:) IT|
Exer (t,) 1
Σver (2tt,)
3 +
(4)
T
3
For the experiments this paper, we use an extension of the directed statistic, the undirected statistic, TRM R) QCC, R) Q(R, C),, which increases the sensitivity of the metric by taking into account rank statistics of both wathin-candidate and within-referenne edges.
An intuition for how this statistic measures divergence between distributions given in Figure If the in-distribution edges are always short compared to the cross-distribution edges, this suggests that either the distance between the candidate and reference distributions is high (different locus), or the spread of the candidates in the semantic space is significantly less than that of the references (different spread). If the in-distribution edge is always the longest edge, it suggests that the spread or dispersion of the candidate samples is higher than the dispersion of the reference samples. Because this statistic takes into account the full distribution through triplets of samples, it does not suffer from the issues with aggregation discussed in section 1 and earlier in this section. Not only does it solve these issues, the TRM extensions build on the existing pairwise metrics, allowing us to increase the sensitivity of the metrics while retaining the existing semantic distance measure and intuitions.
3.3 KERNEL-BASER METRICS
While TRMs represent one method of augmenting existing pairwise metrics, second possible approach relies on representing utterances as points the embedding space of a model, particularly a large pre-trained model such as BERT (Devlin et al., 2018) or GPT (Brown et al., 2020). Evaluating the distance between two distributions based on representative samples on a Euclidean manifold is relatively well studied in GAN literature. One option, MAUVE, introduced by Pillutla et al. (2021), uses a K-Means density estimator to estimate the distribution of the points on this manifold and then computes fixed divergence (such as Kullbeck-Libeller between the two density estimates. Unfortuantely, MAUVE can struggle to correctly estimate the density when there are few samples, such in the case of conditional language generation, as the K-means density estimator is poorly
5

---

Under review as a conference paper at ICLR 2023
Table I: The p-value for the test dataset (using single-video tests, aggregated using HMP (Wilson, 2019) for tractability) generated using standard metrics under the current method of predicting single best text sample. With a single candidate text (the current evaluation paradigm), the metrics unable to make a statistically significant distinction between ground truth samples. Additional experimental detail in appendix A.5.
Dataset
Model
BERT
CIDEr-D
BLEU@4
METEOR
ROUGE-L
MSR-VTT
TVT (Chen 2018)
0.6582
0.4098
0.7813
0,4573
0,4771
O2NA (Liu al., 2021)
0.6455
0.4574
0.7959
0.5646
0.5938
Human
0.5153
0,5314
0.8291
0.5306
0.5669
MS-COCO
CLIPCap (Mokady et al., 2021)
0,5581
0.8223
0.8788
0,7483
0.7985
VLP (Zhou et al., 2020)
0,5925
0.7424
0.8593
0.6644
0.7706
Human
0.6406
0.6686
0.8743
0.6358
0.6841
suited to such situationn. Several possible extensions to MAUVE could be considered as an alternative family of distribution-awarc metrics, which we dub "Kemel-Basse Metrics" (KBMs).
Frechet BERT Distance The Frechet Inception Distance (Salimans et al., 2016) represents the squared Wasserstein distance between multidimensional Gaussian distributions fitted to the compo- nents of the input. In the Frechet BERT Distance metric, we replace the Inception embeddings with those from a pre-trained BERT model (Devlin et al., 2018). See appendix A.6 for details.
MMD-BERT A related metric is the maximum mean discrepancy distance function (Li et al., 2017), which leverages a density estimate of the data, and computes the maximum mean discrepancy between the density estimates for each sample. In our case, we leverage Gaussian kernel estimate over the embeddings generated by a pre-trained BERT model (Devlin et al., 2018). See appendix A.7 for details. While we primarily explore BERT-based embeddings, we explore additional embedding methods in subsection B.
CASE STUDY: VISUAL DESCRIPTION
Visual description is challenging conditional natural language task, which requires that model pro- duce a natural language description of a visual context containing objects, actions, and relationships present in scene. Data sets for visual description often set themselves apart from other datasets for conditional natural language generation (such as those for translation and summarization), as they contain more than ground truth sample, making it possible to evaluate visual description models using reference data that has already been collected. In this set of experiments, which look at two datasets for visual description, MSCOCO (image description) (Lin et al., 2014) and MSR-VTT (Xu et al., 2016) (video-description) (full dataset details in appendix A.2), we demonstrate that current metrics not sensitive enough to evaluate the performance of several existing models with new metrics. We then demonstrate quantitatively how a multi-candidate evaluation paradigm can close this gap, and how more sensitive metric, such as TRMs KBMs, can provide new model insights.
Existing single-ground truth comparison is not sensitive enough A natural first question to ask when evaluating the performance of metric is, "given the existing data, is the metric sensitive enough to distinguish between a model and a reference distribution?" To answer this question, we evaluated the p-values for several existing metries on a single video. The results, shown in Table I demonstrate that using a single description for the candidate dataset is insufficient to tell even known different distributions apart, motivating transition to paradigm with significantly more sensitivity. This is a similar result to the observations made in Yeh et al. (2021) and Liu et al. (2016) for dialog generation: most metrics are unable produce significant results using existing techniques. Thus, even for standard metrics, it makes sense to sample more than one ideal candidate description and aggregate the metric score across these candidate deseriptions.
TRM and KBM metrics are more sensitive than naive aggregation In section 3, we proposed several new metrics which can be leveraged by switching to multi-candidate evaluation. Figure 3 shows the sensitivity of both the newly introduced metrics and existing metrics using the naive aggregation schemes discussed in section 3, as increase the number of candidate samples from the model. While the sensitivity increases for all models to significance, our proposed metrics are much more sensitive with fewer candidate and reference descriptions. As additional check, when tested

---

Under review as a conference paper at ICLR 2023
MSR-VTT (O2NA)
MS- CS-C (CLIPcap)
-1.0
Log value
-1.5
-2.0
MEMMETEOR
Log -3
NMO TRN-METEOR
-2.5
METEOR
NAD BEST
NETCOR
-3.0
CIDOS
TRI-BERT
CEDTR
TUX GIEE
DERT
Number of candidates
Number of candidates
Figure 3: Plots showing the log p-values for the existing and proposed metrics as we increase the number sampled candidate descriptions from the models. TRMMETEOR achieves a 162% increase in sensitivity over METEOR, while TRMcIDE represents a 49.3% increase over CIDEr-D for O2NA evaluated on the MSR-VTT dataset. Additional experimental details are given in appendix A.5.
The cows are grazing field. The cows are grazing in a field.. The cows are grazing in a field. The cows are grazing a field.
Candidate Set 1 METEOR (+):1.0 TRM-METEOR (+): 0.574
Animals grazing on grass in enclosed area.
Candidate Set
Soverai cows grazing in a field with frees in the background.
METEOR (+): 0.393
TRM-METEOR (4): 0.069
Cows grazing in large green pasture in distant scene.
grassy field overlooking cows pasture.
Cows grazing in a posture ringed with trees. aoarod-looking photogroph of cows in green pasture. Some cottle are grazing in a green field and some trees. A herd of animals grazing on lush green field. The cows are grazing in field.
References
MSCOCO Image 134074
Figure 4: A qualitative sample from CLIPcap. Candidate set uses beam search (8 beams), while candidate set two uses nucleus sampling (with temperature one, top-k of 20 and top-p of 0.9). As the diversity increases, the TRMMETEOR divergence decreases, but METEOR fails correctly capture the diversity/correetness trade-off, leading to decreased scores for more complete caption sets that are still relatively high quality. Additional qualitative examples are provided in appendix B.6.
on human captions, our metrics do not consider the two distributions significantly different (p > 0.05, appendix B.4). Our proposed metrics do not alter the manifold: for example, TRMMETEOR and METEOR measure the same underlying intuitive divergences (n-gram recall with some additional synonym matching), however, TRM method increases the sensitivity of the test, allowing to measure the full distribution divergence, instead using naive aggregates. It is useful to note that when computing the metrics for practitioner, computing the p-value of the data is unnecessary (we need only sample enough candidates so we can be sure of the statistical significance of the metric).
Multi-candidate evaluation illustrates a diversity vs. likelihood trade-off A metric's ' sensitivity the full distribution has the power give us novel insights into the visual description task. Consider the two models, VLP (Zhou et al., 2020), standard transformer-based model pre-trained on large- scale vision and language data, and CLIPCap (Mokady et al., 2021), ransformer-baseed model which is initialized with large language model, and uses prefix-tuning with CLIP (Radford et al., 2021a) embeddings (Additional details appendix A.3). While VIP (Zhou et al., 2020) outperforms CLIPcap (Mokady et al., 2021) in the standard scoring methods, CLIPcap outperforms VLP in the TRMMETEOR metric. Why is there such an inversion? We can begin to understand the results when looking a bit closer at Figure 5, which plots sampling temperature from the model, against the performance on the TRMMETBOR and standard METEOR metrics. At low temperatures, where the model is sampling from single maximum- likeliooo estimates, VLP outperforms the CLIPcap method. As the temperature increases, there is an inversion in the model performance, and at higher temperatures, CL. outperforms VI... On the other hand, in the standard METEOR metric,
7

---

Under review as a conference paper at ICLR 2023
Temperature vs. TRM-Score (MSCOCO)
Temperature Pairwise-Score (MSCOCO)
0.9
CLIPCpp (TRM ME TEOR) ITRM-NETEOR)
VLP (METEOR)
0.8
0.5
CLIPCap (METEOR)
0.4
Divergence
0.6
Score
0.5
0.4
0.2
0.1
0.1
0.00
0.50
1.00
1.50
0.00
0.25
0.75
1.00
1.25
1.75
Temperature
Temperature
Figure 5: Plots indicating the impact of temperature on the metric scores. Left: TRMMETEOR (L) for CLIPcap and VLP. Right: Standard METEOR Score (1) for CLIPcap and VLP.
Tiangle-METEOR Scores
METEOR Scores
0.60
Gready
Searth
0.55
Nucious Sampling
Sampling
Divergence
0.50
0.45
Score
0.4
0.40
0.3
0.35
0.30
0.25
0.1
0.0
0.2
0.4
0.6
0.8
3.0
3.2
1.4
1.6
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1,75
Temperature
Temperature
Figure 6: Plots indicating the impact of search technique on divergences. Left: TRMMETEOR (L) for TVT on MSR-VTT. Right: METEOR Score (1). See appendix A.8 for experimental details.
increase temperature, performance monotonically decreases, giving little insight. Figure 5 illustrates that TRMMETER captures a subtlety in the model comparisons that METEOR does not capture alone: while VI.P produces better descriptions at low temperatures, it becomes less fluent (likelihood) on average as we introduce diversity, leading to a change in position. The result is also visible in qualitative samples, given in Figure 4, where likelihood. These results confirm observations made by Zhang et al. (2021) for open-ended language generation tasks such storytelling and dialogue: fair comparison of approaches must not only compare at the same level of entropy but at a range of entropy levels.
TRM metries prioritize both diversity and
Sampling algorithms matter Not only does the temperature of the generation process matter when correctly trading off between diversity and description correctness (as seen in the previous discussion). but the sampling process itself matters. Figure 6 shows the performance at different temperatures of the Nucleus sampling method (Holtzman al., 2019) vs. standard sampling, beam search, and greedy, approaches. While maximun-likelihodd methods achieve the best METEOR scores, they have relatively high divergence, as they sample only a single description. From Figure we can further see that TRMMETEOR illustrates how Nucleus sampling allows models to achieve higher temperatures than standard sampling without diverging significantly from the distribution. METEOR alone does not indicate such an effect and only monotonically decreases.
5 DISCUSSION AND LIMITATIONS
Kernel-Basee Metrics (KBMs) vs. Triangle-Rank Metrics (TRMs) A natural question to ask is: "which metric should practitioners choose when evaluating conditional language models?" KBMs have one major, distinct, advantage over the TRMs in that they are naturally differentiable, yet KBMs also have downsides. The first is that, unlike the TRMs, they require both a pre-trained BERT model and kernel-density estimator which both have complex behavior affecting the performance of the model. The TRMs, however, can be specified on top of existing natural language distance functions, improving the ability of the user to intuit the model performance. Additionally, TRMs
8

---

Under review as a conference paper at ICLR 2023
Table 2: Method evaluation efficiency on the MS-COCO dataset with 5 references and 10 candidates using an intel 17-6850K CPU and 2 NVIDIA Titan X GPUs.
METEOR
TRMSTTOR
CIDEr
TRMcmer
MMD-BERT
FID-BERT
MAUVE
Samples/See
298.4 = 18.3
161.182112
131.33 = 12.6
9..54
53.76 + 38.7
17.45 4.6
2.29 = 0.78
Wall Time (Min)
2.26
4.18
5.14
6.92
12:55
38.68
294.78
are bounded and have p-values that can be computed analytically. Finally, because the TRMs do not need density estimate, they can be more sensitive with small sample sizes (see Figure 3), which is essential for conditional language generation where we have only few gold gold-standard samples. Table 2 demonstrates another key benefit of TRMs: efficiency. The time per sample to compute TRMs, while higher than single metric standards, is lower than KBMs on average. In the case of Mauve, computing the p- p-values is largely intractable (See appendix B.5).
We acknowledge that perplexity (likelihood of the test distribution) is another alternative metric to proposed methods. While methods for conditional language generation should report the perplexity of their models, has also been noted by Theis et al. (2015) that perplexity can suffer from several issues in the evaluation of generative models. For example, lookup table storing sufficiently many training examples will produce convincing results but have poor perplexity on the test data. On the other hand, van den Oord & Dambre (2015) demonstrates that even in situations where the perplexity is high, models may not generate high-quality test samples.
Perplexity
Discussion on Human Correlation In this work, we do not provide any experiments correlating our metric scores with human judgments. The reason is two-fold. First, humans are relatively poor at measuring the semantic distance between two distributions in the presence of distractors (Durga, 1980). Further, some evidence has shown that existing decoding methods optimize for fooling humans over correctness (Ippolito et al., 2019). Automated metrics represent a key method for bridging this gap: methods such as TRMs allow us to overcome any deficiencies in human judgment and sensitivity. The second is that the TRM-metries build on existing, well-established measures with strong human correlation (Vedantam et al., 2015; Lin, 2004; Agarwal Lavie, 2008; Papineni et al., 2002). Because we do not alter the distance manifold, increase the sensitivity of existing measures of semantic similarity, rather than replace the measures altogether. While sensitivity can be enhanced by hacking the p- values of the metrics (such taking the power of an existing measure), Table 3 demonstrates that low sensitivity is still observed in human leave-one-out experiments (which would not be the case if sensitivity was increased across the board).
contributions only serve to
A Note on Reference-Free Metrics visual description, are immune to ground truth aggregation effects as they are computed in a reference- free way, and focus on pre-trained models' ability to ground vision and language information. Unfortunately, such large, black-box, models represent a liability as metric as their capabilities are largely unknown, and untested (Floridi & Chiriatti, 2020; Caglayan et al., 2020). Further, the metric is only as good as the model, and CLIP has been known to suffer from numerous issues including counting, attribute-assonn association, and spatial reasoning (Blattmann al., 2022; Ramesh et al., 2022).
Some metrics, such CLIP-score (Hessel al., 2021) for
6 CONCLUSION
In this work, we have introduced robust framework for multi- multi-candidate evaluation of conditional language generation models, shown that existing metrics for semantic similarity can be seamlessly extended to this framework, and demonstrated through case study of visual description that multi- candidate evaluation paired with more sensitive distribution-aware metrics has the power to provide novel insights into existing models and methods. Our method is not without limitations, for example, one of the core drawbacks is the availability of multi-reference data. Outside the field of visual description, it is often not a standard practice to collect more than one goad-standard reference (even in fields such as summarization, where it makes sense to do so). It is necessary for future work to explore how a wider range of existing generation techniques and models perform under this new paradigm, and, as data availability expands, to understand the implications of distribution-aware evaluation in fields beyond visual description. We hope that this step toward a more robust evaluation paradigm will inspire further research into this area of evaluating conditional language generation.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Abhaya Agarwal and Alon Lavie. Meteor, m-bleu and m-ter: Evaluation metrics for high- high-correlation with human rankings of machine translation output. In Proceedings of the Third Workshop on Statistical Machine Translation, pp. 115-118, 2008
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In European conference on computer vision, pp. 382-398. Springer, 2016.
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Michéle Basseville. Divergence measures for statistical data processing annotated bibliography. Signal Processing, 93(4):621-633, 2013.
Andreas Blattmann, Robin Rombach, Kaan Oktay, and Bjöm Ommer. Retrieval-augmenned diffusion models. arXiv preprint arXiv:2204. 18842 2022.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the association for computational linguistics, 5:135-146, 2017.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Ozan Caglayan, Pranava Madhyastha, and Lucia Specia. Curious case of language generation evaluation metrics: A cautionary tale. arXiv preprint arXiv:2010.13588. 2020.
David M Chan, Austin Myers, Sudheendra Vijayanarasimhan, David A Ross, Bryan Seybold, and John Canny. What's a caption? tataset-specifer linguistic diversity and its effect on visual description models and metrics. arXiv preprint arXiv:2205.066 06253, 2022.
Ming Chen, Yingming Li, Zhongfei Zhang, and Siyu Huang. Tvt: Two-view transformer network for video captioning. In Asian Conference on Machine Learning, pp. 847-862. PMLR, 2018.
Elizabeth Clark, Asli Celikyilmaz, and Noah Smith. Sentence mover's similarity: Automatic evaluation for multi-sentence texts. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2748-2760, 2019.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1xX.v:8110.0480)0 2018.
Ramanand Durga. Semantic distance and semantic judgment. Outstanding Dissertations in Bilingual Education Recognized by the National Advisory Council on Bilingual Education, 1979, pp. 15, 1980.
Ronald Aylmer Fisher. Statistical methods for research workers. In Breakthroughs in statistics, pp. 66-70. Springer, 1992.
Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30(4):681-694, 2020.
Jack Hessel, Ari Holtzman. Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: reference-free evaluation metric for image captioning. In EMNLP, 2021.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. International Conference on Learning Representations, 2019.
Daphne Ippolito, Daniel Duckworth, Chris Sallioon-Burchh and Douglas Eck. Automatic detection of generated text is easiest when humans fooled. arXiv preprint arXiv: /9/ 1.00550, 2019.
10

---

Under review as a conference paper at ICLR 2023
Chun-Liang Li, Wei- Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás Póczos. Mmd gan: Towards deeper understanding of moment matching network. Advances in neural information processing systems, 30, 2017.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Chia-Wat Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. How not to evaluate your dialogue system: An empirical study unsupervised evaluation metrics for dialogue response generation. arXivv preprint arXiv:1603. 163v:1603300003,,, 2016.
Fenglin Liu, Xuancheng Ren, Xian Wu, Bang Yang, Shen Ge, and Xu Sun. O2na: An object-oriented non-autoregrressive approach for controllable video captioning. arXiv preprint 2021.
Zhenyu Liu and Reza Modarres. A triangle test for equality of distribution functions in high dimensions. Journal of Nonparametric Statistics, 2365):05--115, 2011.
Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2 11/222111.0777 1.09734, 2021.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pp. 31 311-318. Association for Computational Linguistics, 2002.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Jesus Perez-Martin, Benjamin Bustos, and Jorge Pérez. Improving video captioning with temporal composition of a visual-syntacto embedding. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp-
2021.
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: Measuring the gap between neural text and human text using divergence frontiers. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 4816- 4828. Curran Associates, Inc., 2021. URL https: https::/prcceediing htw.:rcceeediing.eeeieeeee...eee neurips...... ct.paper/pppp///pe 2021 / fe / 20c22 432 wew.aaeeeeeeeeeeeeeeee 0da oat.:e/cccccceeeeeooooooaaaaaaaa Paperr..dp.p.....
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8766 PMLR, 2021a. URL http: http://poceeeeennnceee mlr. press/v139/ radford21a.. html.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.0000 2021b.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06115, 2022.
Nils Reimers and Iryna Gurevych. Sentence-bert Sentence embeddings using siamese bert bert-networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL ht https: :/ arxiv. org/abs/////eoooooooee..././..ba 0084.
abs /9908.
11

---

Under review as a conference paper at ICLR 2023
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Thibault Sellam, Dipanjan Das, and Ankur P Parikh. Bleurt: Learning robust metrics for text generation. arXiv preprint arXiv:2004. 04696, 2020.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 2556-2565, 2018.
Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi. Ruse: Regressor using sentence embeddings for automatic machine translation evaluation. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pp. 751-758, 2018.
Lucas Theis, Aäron van den Oord, and Matthias Bethge. note on the evaluation of generative models. arXiv preprint arXiv:1511.0160 2015.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- -ze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.082. 2022.
Aäron van den Oord and Joni Dambre. Locally-connecte transformations for deep gmms. In International Conference Machine Learning (ICML): Deep learning Workshop, pp. 1-8, 2015.
Emiel Van Miltenburg, Desmond Elliott, and Piek Vossen. Measuring the diversity of automatic image descriptions. In Proceedings of the 27th International Conference on Computational Linguistics, pp. 1730-1741, 2018.
Ramakrishna Vedantam, C Lawrence Ziticck and Devi Parikh. Cider: Consensus-basee image description evaluation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4566-4575 2015.
Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse beam search: Decoding diverse solutions from neural sequence models. arXiv preprint arXiv:
2016.
Daniel J Wilson. The harmonic mean p-value for combining dependent tests. Proceedings of the National Academy of Sciences, 1 116(4):1155-1200) 2019.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288-5296, 2016.
Dingdong Yang, Seunghoon Hong. Yunseok Jang, Tianchen Zhao and Honglak Lec. Diversity- sensitive conditional generative adversarial networks. arXiv preprint arXiv: 1901:0090 2019.
Yi-Ting Yeh, Maxine Eskenazi, and Shikib Mehri. A comprehensive assessment of dialog evaluation metrics. In The First Workshop on Evaluations and Assessments of Neural Conversation Systems, pp. 15-33, 2021.
Hugh Zhang, Daniel Duckworth, Daphne Ippolito, and Arvind Neelakantan. Trading off diversity and quality in natural language generation. In Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval), pp. 25-33, 2021.
Tianyi Zhang* Varsha Kishore*, Felix Wu Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. 2020. URL https: http///pppeeeeeeeee/eeeeeeeeeeee net Sworummid-Skellle
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. Unified vision- pre- pre-training for image captioning and vqa. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 13041-13044 2020.
12

---

Under review as a conference paper at ICLR 2023
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 1097-1100, 2018.
13

---

Under review as a conference paper at ICLR 2023
A ADDITIONAL EXPERIMENTAL DETAILS
In this section, we discuss additional experimental details for interested readers.
A.1
CODE
We make all code/data publicly available for use https: //s3. ius-west-1 w.w.ww...wwabbss.............. com/ anon neurips2000 neurips ttaraaraaa gz (Github link in camera ready). We hope that releasing our code, along with the ISON files containing test-set predictions for the models in question will help inspire further research and examination into the evaluation of models for visual description.
A.2 DATASETS
MSR-VTT Dataset: The ASR-VTT dataset (Xu et al., 2016) is a dataset for video description consisting of 10,000 videos, with 20 reference ground truth descriptions for each video. It was collected by downloading 118 videos for each of 257 queries from a popular video sharing website. MSR-VTT contains 41.2 hours of video, with an average clip length lying between 10 to 30 seconds. It has a vocabulary size of 21,913. For more details about the diversity of the language present in the dataset, we refer readers to Chan et al. (2022).
The MS-COCO dataset (Lin et al., 2014) is large-scale dataset for image description, object detection and segmentation. MS-COCO contains 328K images, each with ground truth descriptions generated by human AMT workers. For more details about the diversity of the language present in the dataset, we refer readers to Chan et al. (2022). MS-COCO is licensed under Creative Commons Attribution 4.0 license.
MS-COCO Dataset:
A.3 MODELS
This paper explores the performance of our metries over several models: two video captioning models, and two image captioning models.
The Two- View Transformer (Chen et al., 2018) is a baseline method for video description, which consists of a transformer encoder/decoder structure. While we did not have access to the original code, we trained our version of the model the MSR-VTT dataset (standard splits), leveraging features from Perez-Martin et al. (2021). The model was trained for 300 epochs, with a batch size of 64, model hidden dimension of 512, 4 transformer encoder and decoder layers with 8 heads each, and dropout of 0.5. For optimization, we leveraged the Adam optimizer with a learning rate of 34 and weight decay of le-b with exponential learning rate decay with gamma 0.99. This model achieves a C' DIDE Er score of 56.39 on the test dataset. The model was trained using a Titan RTX-8000 GPU over the course of several hours.
TVT
O2NA O2NA (Liu et al., 2021) is a recent approach for non-auto-regreesivv generation of video captions. While the method had available code and checkpoints which we used for this experiment, the method is not designed to sample more than one candidate caption at any given time. To adjust the model to sample multiple candidate captions, we made several adjustments. First, the model was modified to sample a length according to softmax distribution over the length likelihoods (instead of using a greedy choice of length, or beam search over lengths, as proposed the paper). Second, the model was modified to sample tokens at each non-autoregressive step from softmax distribution instead of greedily sampling tokens. We make our modified code available as a patch to the original repository, in the hopes that other users will continue to build on these alterations.
CLIPCap CLIPCap (Mokady et al., 2021) is recent model for image description based on using the CLIP (Radford et al., 2021a) model for large vision and language pre-training as a feature encoder, and GPT (Brown et al., 2020) as natural language decoder. CLIPCap code and MS-COCO trained model checkpoints are publicly available from the authors, however we made some alterations to support temperature- based and nucleus sampling. We make our modified code available as patch to the original repository, in the hopes that other users will continue to build on these alterations. CL CLIPCap is licensed under the MIT license.
14

---

Under review as a conference paper at ICLR 2023
VLP VLP (Zhou al., 2020) is unified vision and language pre- pre- rre-traiiing model, designed to perform both image captioning and visual question answering. The model is pre-trained on the Conceptual Captions (Sharma al., 2018) dataset, and fine-tuned on the MS-COCO captions dataset for image description. The authors make code and pre-trained models publicly available, however we modified the code somewhat to support additional sampling methods. We make our modified code available as a patch to the original repository, in the hopes that other users will continue to build on these alterations. VLP is licensed under the Apache License 2.0.
A.4 DISTANCE METRICS
In this paper, we explore three base semantic metrics as distance underlying our TRM methods, CIDEr-D (Vedantam et al., 2015), METEOR (Agarwal & Lavie, 2008), and BERT Distance (Zhang* et al., 2020).
CIDEr-D CIDEr-D (Vedantam et al., 2015) is n-gram-based metric designed for visual description, and based on the idea that common words are less useful in practice than uncommon words. In practice, this takes the form of a cosine similarity between TF-IDF weighted vectors representing the sentences. Because CIDEr-D is a score, and not distance, we create a distance function: d(c,r) = 10 C(c. r), which works as CIDEr-D is bounded by 10. Note that because CIDEr-D is 10 if and only if and only if the two sentences are equal, this fulfills the TRM requirements.
METEOR METEOR (Agarwal Lavie, 2008) is a score which evaluates the semantic distance between two text utterances based on one-to-one matches between tokens in the candidate and reference text. The score first computes an alignment between the reference and candidate, and computes score based on the quality of the alignment. Because METEOR is score, and not distance function, we use the distance d(c,r) = 1 - M r). where M is the METEOR score of the reference. Because METEOR is bounded 1 if and only if the utterances identical, this simple transformation satisfies the requirements of the TRM adjustment. While we could explore other ways of deriving a distance from METEOR, we found that this simple approach was sufficient to demonstrate the performance of our methods.
BERT Distance A recent method for determining the semantic distance between two samples is to leverage pre-trained BERT embedding model to create a semantic embedding of the text, and com- puting the cosine distance between the test samples. In our work, we leverage the Mini 1 model from the santence-trassfomeee package by Reimers & Gurevych (2019) embed our de- scriptions. Because cosine distance is already a distance function, no additional transformation is necessary.
A.5 P-VALUE COMPUTATIONS
For our experiments, our null hypothesis is that the candidate samples and the ground truth samples are drawn from the same distribution. Because most of the methods do not have an analytical way to compute the p-values (in fact, the TRMs are the only method which has an analytic p- value computation given in Liu & Modarres (2011)), we instead must compute the p-values though sampling. We thus enumerate the value of the statistic across all of the possible candidateereferenee partitions given the joint set of candidates and references, and determine the probability of observing the sampled value, some value more extreme.
The values in Table 1 represent the p-value obtained with single candidate sentence, and 4 ground truth candidates for MS-COCO, or 19 ground truth candidates for MSR-VTT. We reserve one gorund truth description in both datasets to serve as the "Human" performance description. For TVT, CLIPCap and VLP, we sample the descriptions using beam search with 16 beams, For O2NA, which is aon-aevoreeressive model, we sample according to the method suggested in the original work (see Liu et al. (2021)). Because there are several thousand videos per dataset, computing all possible combinations across the dataset would be far from tractable. Thus, the p-values were computed on a per-visual-input basis, and then aggregated across videos using the harmonic mean, as suggested by Wilson (2019). Such an aggregation method is valid when the experiments are not independent (which they are not), unlike Fischer's 's method (Fisher, 1992).
Figure demonstrates the log v-vuuu for the proposed methods across several candidate samples. For MS-COCO, we use all five reference captions, and between and ten candidate captions
15

---

Under review as a conference paper at ICLR 2023
sampled from CLIPCap using Nucleus Sampling (Holtzman al., 2019) with temperature of 1.0, top-p of 0.9 and top-k of 20. The caption set is generated once, meaning that the two-candidate set consists of the one-candidate set and one more additional caption. For MSR-VTT, we use 10 reference captions, and between one and seven candidate captions sampled from O2NA as described in appendix A.3 with temperature of 1.0 for both the length and token samples. We do not go the full 10 candidate captions for MSR-VTS VTT due to tractability concerns, since adding an additional caption forces twice the number of partitions be evaluated when computing p-values.
The above experiments were performed several n2d-standard-32 cloud GCP instances, containing 32vCPUs and 128GB of RAM.
A.6 FRECHET BERT DISTANCE
The Frechet Inception Distance, originally proposed in Salimans et al. (2016), has often been used for the evaluation of the distance between samples of images generated by GANs. Images are first embedded in latent space using pre-trained inception network, and then the Frechet distance between the generated samples and the reference samples is computed. In our work, we replace the images with text, and the inception network with pre-trained BERT embedding network (Devlin et al., 2018). For a set of candidate samples (CI...., Cp) = C, a set reference samples (Ti..... f') R. and BERT embedding function ΦBERT C R R compute the Frechet BERT Distance as:
d2 Σ ΦBRRT(C)) - Σ @BERT + Tr (Co + Cn CVCC
(5)
where Cc andCR are the covariance matrices of the C C'add R sets embedded with ΦBERT respectively, To get the BERT embedding, leverage the CLS token of large pre-trained model. in this case, the MiniLM-L6--e model from the montence-trassfommee package by Reimers Gurevych (2019).
The computation of p-values for the Frechet-BERT distance largely bottle-necked by the slow performance of the sqrtm function, which, because the matrices are not symmetric, has no efficient algorithm for computation. Additionally, unlike the feature computation, this operation must occur for every partition, leading to significantly reduced efficiency compared to the other measures presented in this paper.
A.7 MMD-BER
Another common metric in the GAN literature is the computation of a maximum maximum-mean mean discrepancy between kernel-estimates of the samples introduced by al. (2017). For set of candidate samples (cI...., ...) = C, set of reference samples (rı...., r'm) R, and a BERT embedding function ÓBERT CUR + R we compute the MMD-BERT distance as:
MMD
Σ
K ((BERRT(C)) (BBER(()))))) (c,))+
-1j=1
(6)
M
K (dBBRR((T)) (BBERTTT)))) (rz)) +
K K(éBERT(C)) (OBEr((T)))
where K is kernel function. In experiments, median distance pairwise distance divided by two.
RBF kernel function with equal the
A.8 SEARCH TECHNIQUES
In section 3, Figure 6, we explore the performance of several different search techniques for our two-view transformer model on the MSR-VTT dataset. this figure, we explore four decoding search techniques: Greedy Search, Beam Search, Temperature-Basee each method, and for each video in the test set, we sample 10 descriptions. For Greedy Search, we sample 10 repeated sentences. For beam search we sample the top beam search candidate, and repeat this ten times. While we did explore using the top 10 results from a larger beam search, we found that smaller beam search and repeated values produced better METEOR scores, against this. Wider beam searches did produce higher TRMMETEOR scores, but because optimizing for METEOR would be the current paradigm, decided to include that in the referenced figure.
Sampling, and Nucleus Sampling. For
chose compare
16

---

Under review as a conference paper at ICLR 2023
MSR-VTT (O2NA)
-0.5
MIND RE TRI-CIDER
Sensidivity and performance Hummn-Genertted Captions
-1.0
UND DOU
Method
Log-P
Samples/Sec
-1.5
FASTROR
TRM-CIDEr -1.596
88,93
MMD-BERS
-1.786
56,68
MMD-CLI
-1.887
14.41
-2.5
MMD-GLo
-1.952
54.8
MMD- Text-FatText
-1.954
57.45
-35
MMD-BOW
-2.022
49.41
-4.0
Number of candidates
Figure 7: Performance of several different embedding functions for the MMD-* * family of metrics. Left: Sensitivity when evaluated the MSR-VTI dataset with ten reference captions and between one and seven candidate captions generated by O2NA. Right: Sensitivity and speed when evaluated on human reference samples with 5 references and 5 candidates.
For standard temperature based sampling, we sampled 10 results at each temperature. For Nucleus sampling, we sample 10 results at each temperature, however we freeze they hyper-paramters of top-p at 0.9 and top-k at 20, pairwise metrics. It remains relevant future work to perform deep-dive into the different generative methods with respect to TRMs, as there likely many interesting lessons that be learned.
found these values to generate the best scores under the standard
B ADDITIONAL RESULTS
In this section we present several additional interesting results to augment those in the main discussion.
B.1 EMBEDDING METHODS FOR KBMs
In the main work, we primarily explore BERT-based embedding method for the kernel-based methods. Such an exploration does not preclude the use of other embedding methods, each of which has different trade-offs, when looking the quality of the resulting metric, what the resulting metric measures, the time required to compute the embedding, and the performance when the reference distribution limited to small numbers of human samples (such as happens in practice). Figure Figure 7 shows a quick look at several possible choices for embedding methods in the MMD family, including Bag of words (with a 5K vocab), GLoVe (Pennington et al., 2014), FastText (Bojanowski et al., 2017), and CLIP (Radford et al., 2021b).
While we can see that some of the methods are more sensitive to deviations in the image distributions, such methods come with additional trade-offs. CLIP-style embeddings are the most sensitive to human versus generated captions with fewer captions created, but are significantly slower to evaluate at test time (almost 4x slower) than MMD-BERT, and also produce a higher p-value when computing the leave-one scores on the human captions (which is less desirable, as the human captions are drawn from the same distribution).
B.2 UNIQUE VS. CORRECT DESCRIPTIONS
In Figure 8, we explicitly demonstrate how TRMs enable evaluation of both caption diversity and quality. We artificially generate candidates for the MSR- VTT dataset by mixing human- -emaangeeeertedd exact descriptions with -unan-generated descriptions from other videos. On one axis we have the number of unique descriptions and on the other axis we have the number of correct (exactly-matching) descriptions. Clearly, unlike METEOR alone, TRMMETEOR scores are affected by both correctness and diversity.
Each experiment consisted of 10 candidate captions from the MSR-VTV dataset, and 10 reference captions from the MSR-VTT dataset. We first split the 20 MSR-VTT reference captions into two sets of 10. One set of 10 captions formed the references. To select the candidate captions, we first sampled k unique captions from the remaining reference set (which formed the "correct pool"), and unique captions from other videos in the dataset at random (forming the "incorrect pool"). We then selected correct captions, from the correct pool (at random) and 10 m captions from the
17

---

Under review as a conference paper at ICLR 2023
TRM-METEOR Score for Captions
METEOR Score for Captions
1.0
Number of unique captions
Number unique captions
0.45
0.40
METEOR Score
7
0.7
-0.5
0.5
0.30
0.3
TRM
-0.25
0.2
0.20
Number of correct captions
Number of correct captions
Figure 8: Plots showing how TRMs evaluate both diversity and quality. Left: TRMMETEOR: Right:
METEOR. Lighter colors represent better scores. While TRMMETEOR trades off between diversity and quality, METEOR focuses only on quality not diversity.
TRM-CIDEr Score for Captions
CIDEr Score for Captions
Number of unique captions
Number of unique captions
2.00
Triangle CIDE Deviation
0.8
175
155
6
125
0
3
0.50
022
0.25
0 1
7
10
Number of correct captions
Number captions
Figure 9: Plots showing diversity quality tradeoff's. Left: TRM Right: CIDEr. Lighter colors represent better scores. While TRMcIDE: trades off between diversity and quality, CIDEr focuses only quality not diversity.
incorrect pool (at random). This was then plotted with m on the x-axis, and & on the y-axis, heat-map, where lighter colors represent better scores (higher METEOR, or lower M and darker colors represent poor scores.
We also explored the performance of the CIDEr metric across the same axes, the results of which are shown Figure We can see that they are largely similar to those from the METEOR metric, suggesting that regardless of the underlying metric, we are still making similar trade-offs between diversity and correctness.
B.3 VISUALIZING CENTRAL DESCRIPTIONS
We have found that descriptions which minimize the expected distance to the ground truth distribution are relatively sparse in detail compared to other descriptions. Figures 10, 11, 12 and 13 show qualitative examples of such descriptions for the MS-COCO dataset. Each plot shows qualitative examples of "central" captions. The caption marked with arrows the ground truth caption which minimizes the expected METEOR distance to the other reference captions, and the other captions are the additional references the MS-COCO dataset. Images are selected at random, and do not represent cherry-pickee samples from MS-COCO.
B.4 HUMAN P-VALUES
Strong metrics for distributional comparison will have high sensitivity to samples coming from distinct distributions, and will produce high p-values for samples which come from the same distribution. To check that such relationship holds, we also perform leave-one-our experiments using human- generated captions from the reference split the reference data into sets of 10 candidate samples and 10 reference samples, and compute the deviations using this partitioning. For MS-COCO, we leverage the c40 split which has 40 reference
for both MSR-VTV and MS-COCO. For MSR-VTT,
18

---

Under review as a conference paper at ICLR 2023
stiting celiah. serchan, matere. tgp are
tage plate landed contirants len lst. Aas cvvccel wils tetche and retiali plate. Tux hot sag bass ice moohhre with condirants.
the
citor.
brosifast vat deluuee hatst roan t1ay.
kurch feed ane sturt tasling 1799
so carasax, camaal. Juice and cowe breakfass roeds 1289.
1815 troy Inclades chewol differsnn Showe for rat towa
Figure 10: Qualitative example of "central" captions. The caption marked with arrows is the ground truth caption which minimizes the expected METEOR distance to the other reference captions.
AAUO
lace, giing
microwees out akttinng lop
aterouae sitting courter, strcntass staal.
koil fiies Its Tlay wisle sellir just vler.
silver micrman zw0n on lax weenter and wonber
Figure 11: Qualitative example of "central" captions. The caption marked with arrows is the ground truth caption which minimizes the expected METEOR distance to the other reference captions.
19

---

Under review as a conference paper at ICLR 2023
Cor
to stanting necd to 1a0d cig atts 15hw purted cara actore the street artan N0M unaeeed wige pourking vita the guoin
pocsos pessairg a 3001 hard notcs zerota that operates a vidao gam. more laggo to descrite
conteattie
page- so parsce in hoiding intice will cantest that is kalding wil rewce in cheir kand
Figure 12: Qualitative example of "central" captions. The caption marked with arrows is the ground truth caption which minimizes the expected METEOR distance to the other reference captions.
REG ME
HYUND
Biee treck suzpte auuct stckes rillied stecks housebatc
teuck
daughter Mother son playing
using baseball glove.
girls red grass and playing catch her young non teaching her daughter play baseball
glove
Figure 13: Qualitative example of "central" captions. The caption marked with arrows is the ground truth caption which minimizes the expected METEOR distance to the other reference captions.
20

---

Under review as a conference paper at ICLR 2023
Table 3: Log P- Valuue on human leave-one our samples. We can see that, surprisingly, none of the methods (even the standard aggregations) produce statistically signficant differences. That being said, TRMs often produce higher p-values, indicating that they may be more robust to noise in human caption sets. We do not compute the Frechet-BBER values for humans here, it prohibitively expensive.
METEOR
TRMMETEOR
CIDEr
TRMcwar
BERT
TRMBERT
MMD-BERT
MSCOCO
-0.3003
-05941
-0.9557
-0.4742
-0.6230
-0,5633
-0.6550
MSR-VTT
-1.0466
-0.9613
-1.0224
-0,9777
-1.0172
-1,040
-1.0374
descriptions for 5000 samples of the ground truth. We partition the references for each video into groups of ten descriptions, and compute the p-values from pairs of these partitions. Table gives the performance of the metrics on this human data.
B.5 MAUVE PERFORMANCE
In the main work, we found that MAUVE was prohibitively slow the training data. Because our p-values were computed with 10 reference sentences, and up to 10 candidate sentences, the existing rate, it could take several years to compute the MAUVE p-values for the 50,000 sample MS-COCO dataset. In Table 4, we present several high-variance estimates of the MAUVE p-values (computed using only 100 samples).
compute p-values for
Table Log p-value estimates for MAUVE using five candidates, five references, and 100 samples (at nucleus sampling temperature 1.0 for O2NA, CLIPCap and VLP models). We can see that Log p-values for MSR-VTI VTT and MS-COCO are signficantly worse than METEOR even with aggregation, likely due to the method using k-means to approximate the text distributions with only 5 samples.
Dataset
MAUVE Log p-value
METEOR Log p-value
MSR-VI VTT (O2NA)
-0.4414
-1.7881
MSR-VTT (Human Captions)
-0.1441
-0.6037
MS COCO (CLIPCap)
-0.3980
-2.5585
MS-COCO (VLP)
-0.3234
-2.8609
MS-COCO (Human Captions)
-0.2189
-0.7233
B.6 ADDITIONAL QUALITATIVE SAMPLES
person snowboard does a trick in the air.
Candidate Set
A person on a snowboard does a trick in the air. person on a snowboard does a trick in the air. person snowboard does trick in the air.
METEOR 0.236
TRM-METEOR (4): 0.912
A person in mid air on snowboard in frant of a TV. snowboorder getting some oir after jump. performing jump green slope.
Candidate Set
METEOR (+): 0.264
TRM-METEOR (4): 0.362
person sk's going down ramp.
Competitive spirit during competition in mid air A skier races down the track ot a competition. person is skiing on slope covered snow. The skier is jumping into the air a half pipe. Skier performing aerial jump during ouldoor competition.
Image
References
Figure 14: A qualitative sample from CLIPcap. Candidate set one uses beam search (8 beams), while candidate two uses nucleus sampling (with temperature one, top-k of 20 and top-p of 0.9).
21


Reviews:
[{"review_id": "xfpoRn1YQj", "reviewer": "Reviewer_URmg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper points out problems with traditional metrics for evaluating conditional natural language generation,\nand proposes a novel paradigm for multi-candidate evaluation,\nas they are not appropriate for domains such as visual description or summarization where are semantically diverse.\nIn this work, authors introduce metrics to measure deviations between samples from candidate and reference distributions.\nThey are designed to leverage already strong pairwise tools and alleviate the sensitivity issues induced by introducing simple augmentations to existing pairwise metrics.\nAuthors demonstrate that existing metrics for semantic similarity can be seamlessly extended to this framework and their paradigm can capture nuances through a case study of visual description.\n", "summary_of_the_review": "This paper proposes a novel paradigm for multi-candidate evaluation of conditional language generation models.\nIt is designed to leverage already strong pairwise tools and alleviate the sensitivity issues induced by introducing simple augmentations to existing pairwise metrics.\nAuthors propose both Kernel-Based Metrics (KBMs) and Triangle-Rank Metrics (TRMs) as the basis framework,\ncompare them in their benefits,\nand that existing metrics for semantic similarity can be seamlessly extended to this framework.", "strength_and_weaknesses": "Strength\\\n*Issues are clear and convincing:  An issue in the evaluation of conditional text generation models from multiple sampled texts.\\\n*Proposals for new metrics are important and challenging:  Reuse existing text semantic distance functions and increase the sensitivity of these measures when faced with multiple candidates and references, rather than rebuild the entire set of distributional measures.\\\n*Prior research is sufficient: Discuss the advantage and disadvantages of existing metrics and extend the existing pairwise metrics.\n\nWeaknesses\\\n*Term definition: For example, many distribution  appear to appear many times in different ways (density or not) many times (e.g., Figure1 and KERNEL-BASED METRICS), but how is this term defined in this paper?\\\n*Leaning a bit too empirically: Sensitivity to data, reference text, especially the ground-truth and task, is likely to change the evaluation of the indicator, although the Table 3 and its discussion.\nCould you apply this framework to other tasks such summarization or information extraction?\\\n*No human judgments: While metrics such as ROUGE have been pointed out for their proximity to human judgment, the reason why there is no human judgment in the proposed metrics based on traditional metrics is weak.\nCan you explain the case study?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "jQlbBZbSCk", "reviewer": "Reviewer_ZEw9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper argues that current metrics relying on a single candidate text and a single reference are inappropriate to evaluate the quality of conditional language generation models and proposes two families of metrics to evaluate models in a multi-candidate, multi-reference setting: triangle-rank metrics (TRM) and kernel-based metrics (KBM). The experimental results show that both are better than naive aggregation (over multiple candidates) at discriminating between the model and the ground truth distributions.", "summary_of_the_review": "Although some findings of the paper are interesting, its novel contribution is rather small and poorly explained, so at the moment I feel it is more appropriate to recommend rejection of the paper.\n\n**Update after rebuttal:**\nThe authors have greatly improved the explanation of the proposed TRM metric and now everything makes much more sense. The proposed method is simple and not technically groundbreaking, but the results show that it can be effective. Its applicability is limited to settings where multiple references are available, though. For all these reasons, I decided to increase my score slightly. ", "strength_and_weaknesses": "Strengths:\n- The motivation of the paper is interesting, especially for the problem of generating visual descriptions, which is studied in the experimental section.\n- Although the novelty of KBMs is incremental, they are appropriate for the problem in-hands.\n- The experimental section is appropriate. In particular, the experiment with VLP vs. CLIPCap is particularly enlightening.\n\nWeaknesses:\n1. The authors should elaborate more on the intuition of the proposed TRMs (eq (4)). It is clear that a triangle where all edges have the same length will produce a score of 0, but apparently $Q(C,R)$ cannot discriminate between a situation where $I_0 = 1$ and a situation where $I_2 = 1$. Intuitively, a model that can generate a candidate that is closer to the two references than the two references are of each other ($I_0=1$) is better than another model where the candidate is far apart from the references ($I_2=1$). In addition, isn't always $\\mathcal{I}(C,R) = |T|$?\n2. The authors motivate the need for having multiple candidates by the importance of evaluating diversity in the generated samples. However, it is not clear either how the proposed TRMs take diversity into account.\n3. It is unclear what hypotheses testing method is being employed throughout the experimental section to compare if the two distributions are the same. The authors should state clearly which kind of test they have adopted.\n\nTypos:\nEq. (4) - It should be $t_i$ instead of $T_i$ inside the summations.\nSection 5, Paragraph about perplexity -\" On the other hand, van den Oord & Dambre (2015) demonstrates that even in situations where the perplexity is *low*, models may not generate high-quality test samples.\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YHWFwrm8e3l", "reviewer": "Reviewer_niHx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents new diversity and quality evaluation mechanisms for natural language generation models, especially multiple candidates and multiple reference items. The main idea of the methods is making triangles; those nodes are sentences in the candidate set, and the reference set and edges are existing semantic similarity methods such as BLEU and ROUGE. And the authors assume that two sentences in the same set are closed rather than another sentence. The authors also suggest kernel metrics by using pre-trained large language models to compute the similarity between sentences like a BERTScore. The authors choose to generate descriptions from visual datasets as a case study since the datasets have multiple references for a given data. To show the performance of metrics, the authors suggest using a p-value that shows the probability of rejecting the null hypothesis - samples from the candidate set and reference set are drawn from the same distribution. Experiments show that suggested methods capture the diversity of candidates when the number of candidates are increased and the temperature is changed by showing the decreasing p-value.", "summary_of_the_review": "I would like to discuss this paper with the authors to make the final decision.\n", "strength_and_weaknesses": "The main strength of this paper is suggesting an interesting idea to evaluate generated texts, especially measuring the diversity of candidates and the quality of them together. I like the idea - the triangle-rank approach since it is a meta-approach that can use other evaluation metrics. And the idea is also simple, so other researchers can adopt and extend the idea for their own research work.\n\nWhat concerns me most is the lack of experiments on human evaluation. Yes, the authors mentioned that comparing with human evaluations is hard and I agree with that. Human hard to figure out the diversity among items, especially sentences. Even though it is hard to do experiments, it would be better to show the human correlation since this paper suggests new evaluation methods. One way is inserting a wrong candidate with well-generated candidates and asking what the wrong sentence is to humans and suggested methods. Human annotators find it hard to answer the score of the diversity of items, but they easily figure out the weird item [1, 2]. I assume that the authors did a similar experiment in Table 3, but I am not sure.\n\nAnd I have more questions about this paper. I hope to listen to the author's response and discuss how to improve the paper.\n\n- I am not sure if I understand the meaning of the results (p-value) correctly. The p-value is the probability of chance to see the test statistics if the null hypothesis is true. So, intuitively, we can reject the null hypothesis with a low p-value (i.e., 0.01). Here, the null hypothesis is that samples from the candidate set and reference set are drawn from the same distribution. Then, using BLEU@4 on the MSR-VTT dataset (Table 1), we can say that one ground-truth sentence (Human) has a higher probability that supports the null hypothesis than sentences from machine learning models (TVT and O2NA).\n- The authors insist that existing single-ground truth comparison is not sensitive enough. That’s because existing metrics are not good? Or is it the limitation of single-ground truth comparison? If the first assumption is correct, what are the ideal values in Table 1?\n- Can the suggested method consider the recall in terms of reference samples? In Figure 1, there are no model samples for left-bottom ground truth reference. Then, the recall value is low because of the non-existence.\n- What is BERT in Table 1? BERTScore?\n\n[1] Chang, Jonathan, Sean Gerrish, Chong Wang, Jordan Boyd-Graber, and David Blei. \"Reading tea leaves: How humans interpret topic models.\" Advances in neural information processing systems 22 (2009).\n[2] Li, Margaret, Jason Weston, and Stephen Roller. \"Acute-eval: Improved dialogue evaluation with optimized questions and multi-turn comparisons.\" arXiv preprint arXiv:1909.03087 (2019).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject