Title: OUMG: Objective and Universal Metric for Text Generation with Guiding Ability
Authors: Hanxu Liu
Nianmin Yao
Venue: ICLR 2022
Keywords: evaluation metric
text generation
objective

Abstract: Existing evaluation metrics for text generation rely on comparing candidate sentences to reference sentences. Some text generation tasks, such as story generation and poetry generation, have no fixed optimal answer and cannot match a corresponding reference for each sentence. Therefore, there is a lack of an objective and universal evaluation metric. To this end, we propose OUMG, a general metric that does not depend on reference standards. We train a discriminator to distinguish between human-generated and machine-generated text, which is used to score the sentences generated by the model. These scores reflect how similar the sentences are to human-generated texts. The capability of the discriminator can be measured by its accuracy, so it avoids the subjectivity of human judgments. Furthermore, the trained discriminator can also guide the text generation process to improve model performance. Experiments on poetry generation demonstrate that OUMG can objectively evaluate text generation models without reference standards. After combining the discriminator with the generation model, the original model can produce significantly higher quality results.


Review discussion:
The paper has the following weaknesses:
- Given that the goal of the paper is to present an evaluation metric, I would expect a study comparing its output with human evaluation, as done for example in the BertScore paper cited. As it is, all we know is that it can tell between two systems one of which is made to be worse than the other. That doesn't tell us much about whether it correlates with human judgements.
- It is true that human judgements can be subjective, but if this is a concern, then one should consider a task where objective evaluation is possible.
However, this is not the case with Chinese poetry generation, or any kind of literature generation in general.
- The equations 1 and 2 have symbols that are not explained, i.e. X_1 and X_2.
Also, in equation 2 I think a plus sign is missing between the last two terms of the loss function.
- I don't think that most natural language generation tasks that have a singe correct answer; even in machine translation there are multiple references given.
- Apart from Yang and Klein,  the use of trained discriminator for guiding natural language generation has been previously explored, e.g. : http://proceedings.mlr.press/v119/scialom20a.html . As it is considered by the authors a contribution of this paper to be able to guide the generation, a comparison is needed against these methods is needed.
NO.
Thank you very much for your comments. we are sorry to have some writing omissions in the paper, which will be fixed in the next version. In this paper, we try to propose a universal and objective text generation metric.  The reason why we choose the Chinese poetry generation task is that the existing text generation metric is not ideal for the task. We want to prove the generality of our method by that. In order to further improve the persuasion, we will add some experiments on different tasks in the next version of the paper. Thanks again.
Strengths: with respect I'm not sure there are any.
Weaknesses:
- The paper is poorly written, with many sections difficult to follow. The proposed metric is named OUMG, which I'd guess is meant to stand for Objective and Universal Metric for Generation, although I don't think it's defined anywhere. Assuming this, both parts (objective, universal) are not accurate. A few examples of the difficulty I had with the writing: section 3.1 eon 1 doesn't defined X1 or X2 (which do appear in equation), but does define N (which doesn't). The upgrade-rate and high-score-rate-difference seem overly complex, for what is essentially one system scores higher than the other.
But this is getting into minor details; there are larger flaws with the paper.
- The primary flaw is that it makes grand claims about being a metric suitable for use in many places, yet only reports results on a single poetry task, which further are very weak since they don't involve any ground truths e.g. via human ratings.
Indeed the paper even makes the claim that for this poetry that "Therefore, human evaluation cannot be used as the most authoritative metric". This just seems plain wrong. It might need the right humans.
But the main point is that there is nothing objective in the results here and the reader can't trust the conclusions as a result.
- Statements like "which fully reflects the versatility of OUMG" are very non-scientific given the contrast with what results are actually reported.
- The "only requirement" of the discriminator is accuracy, yet the paper then uses a bi-lstm, yet all recent work in NLP would suggest you'd build a more accurate classifier with a transformer model. This is slightly nit-picking but there is a constant disconnect between the statements made and the actual method and results.
Another example, the bi-lstm is described as having a "relatively simple structure". This simply isn't scientific language and shouldn't be in a paper.
- The discussions at the end of section 3 are both not very clear and seem to muddle what a language model does (score more likely language higher) compared to what's required with poetry (where surprise is required).
NO.
Thank you very much for your comments. we are sorry to have some writing omissions in the paper, which will be fixed in the next version. In this paper, we try to propose a universal and objective text generation metric.  The reason why we choose the Chinese poetry generation task is that the existing text generation metric is not ideal for the task. We want to prove the generality of our method by that. In order to further improve the persuasion, we will add some experiments on different tasks in the next version of the paper. Thanks again.
First, I appreciate the efforts of the authors in devising a simple method towards addressing the evaluation of automated text generation systems. Simpler methods are easier to interpret and reproduce.
However, it is also important to properly support scientific claims and methodologies with sufficient experiments. By this, I notice that there seems to be an overgeneralization in the problem statement. The authors indicate that for systems like story generation and poetry generation, the final suitable reference answer is “not unique”.
While this is true, it is not the only reason why these tasks are special. There should be further clarification that the freedom of the reference answer is not entirely free in open-ended text generation tasks since there are still some rules involving rhyming, format, and integrity that need to be followed specifically in the case of poetry generation as highlighted in Li et al (2020). The addition of this discussion should be in order since this is the main (and only) task tackled in the paper.
There is no discussion of how the proposed metric discriminates between human and model-generated texts. What textual factors do you look for? Do you check the factual consistency? The readability? The cohesion? The stylistic properties?
Likewise, there seems to be no discussion on the two models based on their similarities/differences by nature. Discussion for these parts should be in order, not to mention this is the main contribution of the paper.
There is no definitive review of recent works specifically for the development of objective evaluation for text generation. The only qualified citations are for BLUE and ROUGE, the rest are mostly from types of methods (ex. BERT, Word Embeddings) but not the actual work done for evaluating text generation systems.
For starters, I recommend the authors to look at papers of BERTScore (Zhang et al, 2020), BLEURT (Sellam et al, 2020), and WMD (Kusner et al, 2015) for guidance of proper comparison of evaluation metrics across text generation tasks.
In addition, I also expected the paper of Lin et al (2020) to be included here but was only mentioned later in other sections.
Technicality-wise, it’s extremely hard to believe some of the claims of the paper (“our method is universal and can also be used for the first type of task”) without empirical results from experiments for both Type 1 and Type 2 tasks and without the use of other datasets to prove sufficient “universality” of the evaluation metric. Even with a high accuracy reported in the paper, the proposed metric has only ever been tested for one task (poetry generation) and one dataset (SongNet). Just to mention, the page limit is 8. The authors are advised to consume all the available space to supply needed experiments and comparisons (including baselines) with text generation tasks.
Lastly, there are numerous instances where parts of the discussion involving experimentation details are extremely vague such as:
1. In Section 3.2, “we trained it sufficiently” --> to what extent? Lacks specific details of retraining.
2. From the same section, “split each complete sentence into fragments of different lengths” --> same question, to what extent? The answer to this is found in the next section but it should already be mentioned in the first instance.
3. In Section 4.1, “Chinese poetry has different rules from modern language, and many people do not understand them well.” --> This is another overgeneralization. What are these “different rules” and why do “people don’t understand them well”?
Further clarification is needed.
4. In Section 4.2, the only motivation for choosing Bi-LSTM is “which has a relatively simple structure and good performance” and nothing more. I would like to see a stronger basis for this choice. Provide justification as to why Bi-LSTM is the most practical model for the discriminator for Chinese poetry generation with respect to the task at hand. Cite legitimate references along the way.
5. There seems to be an overuse of the word “significantly” scattered in the paper without proper statistical testing. I suggest revising this to “substantially” as the word is already overloaded in literature.
NO.
Thank you very much for your comments. we are sorry to have some writing omissions in the paper, which will be fixed in the next version. In this paper, we try to propose a universal and objective text generation metric.  The reason why we choose the Chinese poetry generation task is that the existing text generation metric is not ideal for the task. We want to prove the generality of our method by that. In order to further improve the persuasion, we will add some experiments on different tasks in the next version of the paper. Thanks again.
The main strength of this paper is suggesting an interesting idea to evaluate generated texts. I strongly agree that most evaluation methods rely on comparing with ground truth data - human-written texts.
But unlike classification tasks, the output of natural language generation (NLG) models can be various and appropriate since the language can have different expressions even if they have the same semantics. The idea is quite sound and straightforward so that the other researchers can research the claim and idea.
What concerns me most is the lack of experiments to support the claims in this paper.
First, U in OUMG stands for universal, but the authors only show the performance on poetry generation task. As the authors mentioned, we can apply the same idea to other NLG tasks, but we do not know the appropriateness of OUMG for the tasks without any evidence.
Second, only one poor model is selected as $G_2$. Are the results the same when the $k$ is changed more or less? What if $G_2$ takes $k+1$? Can OUMG show the positive HSRD value? I am curious about these results.
Third, why is OUMG objective? How can we measure the objectiveness of OUMG? The authors need to show the evidence for their claim.
And the claim and idea are quite similar to the previous research work. [1-4] suggest trained evaluation methods for dialogue models. Especially, [1] suggests an adversarial learning method, and it is quite similar to the idea in this paper. This paper does not mention these related work.
More questions
- Even though human judgment has some bias, is there any positive relationship between OUMG and human judgment? Other evaluation methods show the relationship [1-4], so it would be better to make the same comparison.
- What is the value of alpha in equation (5) for the experiment? Are there any changes based on the alpha value?
Typos
- No ‘N’ in the equation (1)
- Would you please write the meaning of variables in equations (i.e., $X_1$ and  $X_2$ in the equation)
Reference
[1] Li, Jiwei, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. "Adversarial Learning for Neural Dialogue Generation." In EMNLP. 2017.
[2] Lowe, Ryan, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. "Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses." In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1116-1126. 2017.
[3] Tao, Chongyang, Lili Mou, Dongyan Zhao, and Rui Yan. "Ruber: An unsupervised method for automatic evaluation of open-domain dialog systems." In Thirty-Second AAAI Conference on Artificial Intelligence. 2018.
[4] Bak, JinYeong, and Alice Oh. "Speaker Sensitive Response Evaluation Model." In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6376-6385. 2020.
NO.
Thank you very much for your comments. we are sorry to have some writing omissions in the paper, which will be fixed in the next version. In this paper, we try to propose a universal and objective text generation metric.  The reason why we choose the Chinese poetry generation task is that the existing text generation metric is not ideal for the task. We want to prove the generality of our method by that. In order to further improve the persuasion, we will add some experiments on different tasks in the next version of the paper. Thanks again.
- The idea of the proposed metric is somewhat intuitive and straightforward.
- The paper is structurally organized, although there are many missing/wrong explanations for notation (e.g. “X_1” and “X_2” from equation (1), “D” instead of “N” for the discriminator, etc.), which I suppose will be fixed in the next version of the paper.
- The paper can benefit from more justification and elaboration of when and why this metric, or more specifically the pairwise comparison, is useful. After reading the paper, it is still unclear to me when one would want to report numbers on the pairwise comparison (e.g.
when you only care about two specific models? why?).
- The proposed method raises the question of reproducibility of the numbers reported on such metrics.
In addition to providing more details on the experiment procedure (e.g. how did you sample sentences to compare? how many did you use?), the paper needs to prescribe a desired practice for using the proposed metrics: Should one fix one of the two models and swap out the other model to make consistent comparison among more than two models? Can one can reuse the discriminator trained on the two models to compute metrics for other models? How many samples do you need for training and testing? How robust are the metrics to sampling and the sample size?
- Also, defining a single number as an “objective” and “universal” metric for human-likeness calls for more qualitative analysis. It is unclear exactly what the metrics are capturing. The only explanation provided in the paper is “similar to humans.” More fine-grained analysis on various aspects of text (e.g. fluency, coherency, grammaticality, style, etc.) is desired and comparison with other evaluation metrics (not just conceptually, but quantitatively showing correlation) would be highly recommended.
- For the first part of the experiment, I am not quite convinced that the intentional degrading of the default model is the best way to demonstrate the effectiveness of the proposed metrics. It does show that the metrics can detect severely worse performance, but metrics are more valuable when they allow us to distinguish similar models in a more fine-grained way or to reason about specific aspects of quality.
Also, for the second part of the experiment, I am not fully convinced with the claim that the dropped accuracy implies the improved performance of the guided model without knowing which sentences are sampled and what they are compared against (i.e. what has been used during training). Could it be a weird artifact due to some syntactic resemblance of sampled sentences at test time? More examples and qualitative analysis would help readers better interpret the results.
- Please consider including more information about resource requirement for training and discussing environmental impact. The proposed method seems to require quite expensive and time-consuming process of training and inference specific to the two models, or even a specific set of sampled sentences of interest.
Even if one time evaluation cost of two models might be negligible, it is worth reflecting on the long-term impact.
NO.
Thank you very much for your comments. we are sorry to have some writing omissions in the paper, which will be fixed in the next version. In this paper, we try to propose a universal and objective text generation metric.  The reason why we choose the Chinese poetry generation task is that the existing text generation metric is not ideal for the task. We want to prove the generality of our method by that. In order to further improve the persuasion, we will add some experiments on different tasks in the next version of the paper. Thanks again.

Meta-review:
role: PC
data: comment: The authors propose a reference-less metric for evaluating NLG systems by training a discriminator which distinguishes between human-generated and machine-generated text. 

The main concerns raised by the reviewers were (i) lack of clarity in certain portions of the paper (ii) lack of demonstration of the "universal" applicability of the proposed metric (only evaluated for poetry generation) (iii) lack of clear guidelines on how to use the proposed metric in a reproducible manner (iv) lack of details about what exactly does the proposed metric capture and look for in the generated text.

The authors did not respond to the specific queries of the reviewer and agreed that more work is needed on their part.

Decision: Reject