Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents new diversity and quality evaluation mechanisms for natural language generation models, especially multiple candidates and multiple reference items. The main idea of the methods is making triangles; those nodes are sentences in the candidate set, and the reference set and edges are existing semantic similarity methods such as BLEU and ROUGE. And the authors assume that two sentences in the same set are closed rather than another sentence. The authors also suggest kernel metrics by using pre-trained large language models to compute the similarity between sentences like a BERTScore. The authors choose to generate descriptions from visual datasets as a case study since the datasets have multiple references for a given data. To show the performance of metrics, the authors suggest using a p-value that shows the probability of rejecting the null hypothesis - samples from the candidate set and reference set are drawn from the same distribution. Experiments show that suggested methods capture the diversity of candidates when the number of candidates are increased and the temperature is changed by showing the decreasing p-value.
strength_and_weaknesses: The main strength of this paper is suggesting an interesting idea to evaluate generated texts, especially measuring the diversity of candidates and the quality of them together. I like the idea - the triangle-rank approach since it is a meta-approach that can use other evaluation metrics. And the idea is also simple, so other researchers can adopt and extend the idea for their own research work.

What concerns me most is the lack of experiments on human evaluation. Yes, the authors mentioned that comparing with human evaluations is hard and I agree with that. Human hard to figure out the diversity among items, especially sentences. Even though it is hard to do experiments, it would be better to show the human correlation since this paper suggests new evaluation methods. One way is inserting a wrong candidate with well-generated candidates and asking what the wrong sentence is to humans and suggested methods. Human annotators find it hard to answer the score of the diversity of items, but they easily figure out the weird item [1, 2]. I assume that the authors did a similar experiment in Table 3, but I am not sure.

And I have more questions about this paper. I hope to listen to the author's response and discuss how to improve the paper.

- I am not sure if I understand the meaning of the results (p-value) correctly. The p-value is the probability of chance to see the test statistics if the null hypothesis is true. So, intuitively, we can reject the null hypothesis with a low p-value (i.e., 0.01). Here, the null hypothesis is that samples from the candidate set and reference set are drawn from the same distribution. Then, using BLEU@4 on the MSR-VTT dataset (Table 1), we can say that one ground-truth sentence (Human) has a higher probability that supports the null hypothesis than sentences from machine learning models (TVT and O2NA).
- The authors insist that existing single-ground truth comparison is not sensitive enough. That’s because existing metrics are not good? Or is it the limitation of single-ground truth comparison? If the first assumption is correct, what are the ideal values in Table 1?
- Can the suggested method consider the recall in terms of reference samples? In Figure 1, there are no model samples for left-bottom ground truth reference. Then, the recall value is low because of the non-existence.
- What is BERT in Table 1? BERTScore?

[1] Chang, Jonathan, Sean Gerrish, Chong Wang, Jordan Boyd-Graber, and David Blei. "Reading tea leaves: How humans interpret topic models." Advances in neural information processing systems 22 (2009).
[2] Li, Margaret, Jason Weston, and Stephen Roller. "Acute-eval: Improved dialogue evaluation with optimized questions and multi-turn comparisons." arXiv preprint arXiv:1909.03087 (2019).

clarity,_quality,_novelty_and_reproducibility: Originality is good, but I still have some questions about the experiment settings and results.

summary_of_the_review: I would like to discuss this paper with the authors to make the final decision.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper argues that current metrics relying on a single candidate text and a single reference are inappropriate to evaluate the quality of conditional language generation models and proposes two families of metrics to evaluate models in a multi-candidate, multi-reference setting: triangle-rank metrics (TRM) and kernel-based metrics (KBM). The experimental results show that both are better than naive aggregation (over multiple candidates) at discriminating between the model and the ground truth distributions.
strength_and_weaknesses: Strengths:
- The motivation of the paper is interesting, especially for the problem of generating visual descriptions, which is studied in the experimental section.
- Although the novelty of KBMs is incremental, they are appropriate for the problem in-hands.
- The experimental section is appropriate. In particular, the experiment with VLP vs. CLIPCap is particularly enlightening.

Weaknesses:
1. The authors should elaborate more on the intuition of the proposed TRMs (eq (4)). It is clear that a triangle where all edges have the same length will produce a score of 0, but apparently $Q(C,R)$ cannot discriminate between a situation where $I_0 = 1$ and a situation where $I_2 = 1$. Intuitively, a model that can generate a candidate that is closer to the two references than the two references are of each other ($I_0=1$) is better than another model where the candidate is far apart from the references ($I_2=1$). In addition, isn't always $\mathcal{I}(C,R) = |T|$?
2. The authors motivate the need for having multiple candidates by the importance of evaluating diversity in the generated samples. However, it is not clear either how the proposed TRMs take diversity into account.
3. It is unclear what hypotheses testing method is being employed throughout the experimental section to compare if the two distributions are the same. The authors should state clearly which kind of test they have adopted.

Typos:
Eq. (4) - It should be $t_i$ instead of $T_i$ inside the summations.
Section 5, Paragraph about perplexity -" On the other hand, van den Oord & Dambre (2015) demonstrates that even in situations where the perplexity is *low*, models may not generate high-quality test samples."
clarity,_quality,_novelty_and_reproducibility: The major problem of the paper is the very laconic intuition provided about its major contribution TRMs. In addition, the paper is in general well written and the discussion of the presented problem and experimental results are adequate. Apart from the proposed TRMs, the novelty of the paper is limited.
summary_of_the_review: Although some findings of the paper are interesting, its novel contribution is rather small and poorly explained, so at the moment I feel it is more appropriate to recommend rejection of the paper.

**Update after rebuttal:**
The authors have greatly improved the explanation of the proposed TRM metric and now everything makes much more sense. The proposed method is simple and not technically groundbreaking, but the results show that it can be effective. Its applicability is limited to settings where multiple references are available, though. For all these reasons, I decided to increase my score slightly.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper points out problems with traditional metrics for evaluating conditional natural language generation,
and proposes a novel paradigm for multi-candidate evaluation,
as they are not appropriate for domains such as visual description or summarization where are semantically diverse.
In this work, authors introduce metrics to measure deviations between samples from candidate and reference distributions.
They are designed to leverage already strong pairwise tools and alleviate the sensitivity issues induced by introducing simple augmentations to existing pairwise metrics.
Authors demonstrate that existing metrics for semantic similarity can be seamlessly extended to this framework and their paradigm can capture nuances through a case study of visual description.

strength_and_weaknesses: Strength\
*Issues are clear and convincing:  An issue in the evaluation of conditional text generation models from multiple sampled texts.\
*Proposals for new metrics are important and challenging:  Reuse existing text semantic distance functions and increase the sensitivity of these measures when faced with multiple candidates and references, rather than rebuild the entire set of distributional measures.\
*Prior research is sufficient: Discuss the advantage and disadvantages of existing metrics and extend the existing pairwise metrics.

Weaknesses\
*Term definition: For example, many distribution  appear to appear many times in different ways (density or not) many times (e.g., Figure1 and KERNEL-BASED METRICS), but how is this term defined in this paper?\
*Leaning a bit too empirically: Sensitivity to data, reference text, especially the ground-truth and task, is likely to change the evaluation of the indicator, although the Table 3 and its discussion.
Could you apply this framework to other tasks such summarization or information extraction?\
*No human judgments: While metrics such as ROUGE have been pointed out for their proximity to human judgment, the reason why there is no human judgment in the proposed metrics based on traditional metrics is weak.
Can you explain the case study?
clarity,_quality,_novelty_and_reproducibility: Clarity, Quality, and Novelty is enough.
Reproducibility of the indicator is high, but experimental reproducibility appears to be not so high due to data dependence.
summary_of_the_review: This paper proposes a novel paradigm for multi-candidate evaluation of conditional language generation models.
It is designed to leverage already strong pairwise tools and alleviate the sensitivity issues induced by introducing simple augmentations to existing pairwise metrics.
Authors propose both Kernel-Based Metrics (KBMs) and Triangle-Rank Metrics (TRMs) as the basis framework,
compare them in their benefits,
and that existing metrics for semantic similarity can be seamlessly extended to this framework.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper argues that current metrics relying on a single candidate text and a single reference are inappropriate to evaluate the quality of conditional language generation models. The authors propose a novel paradigm for multi-candidate evaluation (specifically for tasks such as visual description or summarization where generations may be semantically diverse). In this work, authors introduce metrics to measure deviations between samples from candidate and reference distributions.

**Strengths:**
Overall, the main strength of this paper is the proposal of a novel paradigm for evaluating conditional natural language generation models and a new family of metrics that compare the distributions of reference and model-generated text sets.

**Weaknesses:**
There are some concerns about the lack of human evaluation and the lack of clear explanation on how the proposed metrics take diversity into account. The authors have done a good job of resolving the second concern, but I think this paper would be stronger if a human evaluation was attempted. The same aspects of human evaluation would be true of Mauve (Pillutla et al., 2021), and yet the MAUVE work did explore the correlation of MAUVE with human judgments. 

justification_for_why_not_higher_score: Defining an evaluation metric for text generation without running a human evaluation on whether the metric prefers groups of sequences with greater diversity and quality does not validate whether the metric works in practice
Decision: Reject