OUMG: Objective and Universal Metric for Text Generation with Guiding Ability
Authors: Hanxu Liu
Nianmin Yao
Venue: ICLR 2022 Submitted (2022)
Keywords: evaluation metric
text generation
objective

Abstract: Existing evaluation metrics for text generation rely on comparing candidate sentences to reference sentences. Some text generation tasks, such as story generation and poetry generation, have no fixed optimal answer and cannot match a corresponding reference for each sentence. Therefore, there is a lack of an objective and universal evaluation metric. To this end, we propose OUMG, a general metric that does not depend on reference standards. We train a discriminator to distinguish between human-generated and machine-generated text, which is used to score the sentences generated by the model. These scores reflect how similar the sentences are to human-generated texts. The capability of the discriminator can be measured by its accuracy, so it avoids the subjectivity of human judgments. Furthermore, the trained discriminator can also guide the text generation process to improve model performance. Experiments on poetry generation demonstrate that OUMG can objectively evaluate text generation models without reference standards. After combining the discriminator with the generation model, the original model can produce significantly higher quality results.

Under review as a conference paper at ICLR 2022
OUMG: OBJECTIVE AND UNIVERSAL METRIC FOR TEXT GENERATION WITH GUIDING ABILITY
Anonymous authors Paper under double-blind review
ABSTRACT
Existing evaluation metrics for text generation rely on comparing candidate sen- tences to reference sentences. Some text generation tasks, such as story gener- ation and poetry generation, have no fixed optimal answer and cannot match a corresponding reference for each sentence. Therefore, there is a lack of an objec- tive and universal evaluation metric. To this end, we propose OUMG, general metric that does not depend on reference standards. We train discriminator to distinguish between human- uuman--eneraatd and machine-generated text, which is used to score the sentences generated by the model. These scores reflect how similar the sentences are to guman-generated texts. The capability of the discriminator can be measured by its accuracy, so it avoids the subjectivity of human judgments, Fur- thermore, the trained discriminator can also guide the text generation process to improve model performance. Experiments OUMG can objectively evaluate text generation models without reference stan- dards. After combining the discriminator with the generation model, the original model can produce significantly higher quality results.
poetry generation demonstrate that
INTRODUCTION
According to different expected goals, natural language generation tasks can be divided into two categories. The first type of task has specific reference standard usually given by experts. The goal of model optimization is to make the generated results close or even coincide with the ref- erence standard, such as machine translation (Koehn 2009) (Bahdanau et al. (0014) (Vaswani et al.) 2017], summarization (Chopra et al. (016)) (Rush et al. 2015) (See et al.) 2017). and image cap- tioning (Fang et al. 2015). The existing metrics have been able to evaluate these tasks from various aspects and obtain objective scores. The second type of task includes dialogue system(Kurach et al. 2016), story generation (Fan et al. 2018) (See al. 2019), poetry generation (Lau al.) 2018) (Liao et al. 2019), etc. The ultimate goal is to make the model learn the features of natural language and generate sentences that look like written by humans. These tasks have common characteristic:
The suitable reference answer is not unique (e.g.. The same beginning can be followed by various story plots). Therefore, the evaluation will inevitably be biased if only a few sentences are used reference standard.
In order to better evaluate the quality of text generation results, universal evaluation metric must meet the following three requirements: 1. The metric must be available in the absence of corre- sponding reference standards. In the same semantic, the change of expression and words should not sigmificantly affect the evaluation score. 3. The metric must be objective.
In this paper, we propose OUMG, a more general evaluation metric for text generation. This method meets the above three requirements and has the ability to guide the generation process, which can further improve the quality of sentences generated by the model.
The core of our approach is discriminator, whose function is to distinguish between human- generated and sentences written by humans positive samples and sentences generated by various models neg- ative samples. In order to ensure universality, the selection of negative samples must cover all the potential models. However, under realistic conditions, this is impossible. Therefore, OUMG uses a compromise by mixing the generated results of the two models(e.g. The improved model and its baseline) equally negative example. After sufficient training, the samples generated by the
text. Ideally, this discriminator should be trained with a set of

---

Under review as a conference paper at ICLR 2022
two models are scored by the discriminator, and the relative merits of the two models are judged according to the high score rate difference and upgrade rate. In addition, a discriminator with high accuracy can consider the semantic information of the sentence, avoiding the problem of large fluc- tuations in scores caused by different expressions. At the same time, a model trained by datasets is more objective than human judgments, which meets the requirements of evaluation metric.
For the sake of showing the unique advantages of OUMG, we choose poetry generation as the main task in our experiments. But this does not mean that OUMG can only be used for the second type of task. Since we eliminate the dependence of metrics on reference standards, our method is universal and can also be used for the first type of task. For example, the original sentence and its corresponding translation as the input of the discriminator and train the discriminator to distinguish whether the translation part of the input comes from the reference standard or the machine translation model. Other tasks are similar.
machine translation, we can use
Inspired by Yang Klein (202T), our method can also be used to guide text generation. In the text generation process, we use the discriminator to score all the candidate tokens for each step and combine the score with the logits provided by the generator to select the tokens with higher final scores. Compared with the Yang & Klein (202T) method, we add a hyper-parameter to control the influence of the discriminator on the generation process.
2
RELATED WORK
2.1 METRICS BASED ON STRING MATCHING
The earliest metrics use the degree of string matching between two sentences to measure whether the candidate sentence is close to the reference standard. The representatives are BLEU (Papineni jet al. 2002) and ROUGE Lin 2004). which were initially applied to machine translation and were widely used in various other tasks in later studies. The idea is to count the number of times the same word appears between two sentences. two sentences have a higher degree of coincidence, they more similar. The difference is that BLEU mainly focuses on the accuracy of the text generation model, while ROUGE focuses more on the recall. There is an obvious problem in this method: only the most superficial matching can be carried out, and no semantic information can be taken into account, It means that these metrics will equate synonyms with completely irrelevant words and are not sensitive the word order of sentences, which severely limits their accuracy.
2.2 METRICS BASED ON WORD EMBEDDING
In order to solve the problem that the metrics based string matching cannot consider semantic information, researchers began to propose metrics based on word embedding. The original method to convert the sentence into vector representation by the Word2vec models (Mikolov et al.) (013) and then calculate the similarity between the two sentences by the cosine similarity method. After word embedding, the generated vector has expressed and abstracted the semantic certain extent so that different words with the same meaning have closer distance. Thereforesuch metrics scoring has more reference value. With the development of many pre-training language models represented by Bert (Devlin et al. 2019). more people use Bert and other models (Peters et al. 2018) to replace the traditional Word2vec models. In contrast, Bert and other models can more dynamically capture the context information, solve the polysemy problem, and refine the semantics more accurately. Bertscore Zhhng et al. (0200) and BLEURT (Sellam et al. (220)) are representatives of such metrics. However, although the metrics based on word embedding can well combine semantic information, they still rely on authoritative reference standards, which are only applicable to the first type of text generation task.
2.3 METRICS BASED ON LANGUAGE MODEL
The above two metrics evaluate the performance of the model through the sentences generated by the model. In addition, the language model itself can also be assessed directly. The PPL metrics evaluates whether the language model can generate smooth sentences by calculating the joint prob- ability. Given a trained language model, use to calculate the probability of each sentence in the test set. The higher the likelihood of this sentence, the lower the corresponding PPL. score. That
2

---

Under review as a conference paper at ICLR 2022
is, the better the modeling ability of the language model. Although many text generation tasks the PPL score as reference for model quality, there are still some unavoidable problems. First, the PPL metric only evaluates the probability of the output of the language model without consider- ing the impact of the optimized decoding process on the final production, so can only be used to evaluate the modeling ability of the language model. In addition, PPI. only focuses the fluency of the sentence itself but ignores other factors that affect the quality of the generated text, such as the consistency with the context and the correspondence with the given topic. cannot effectively evaluate the quality of the text in some tasks, like dialogue generation and poetry generation.
3 OUMG
3.1
DISCRIMINATOR
The function of the discriminator is to identify which are written by humans and which are generated by the models in set of sentences. The discriminator outputs scores from 0 to for each sentence, which is the probability that the sentence is judged to be human- semaan-eneeated. sentence higher than 0.5, we classify the sentence human-gene -emaan-enerated...
When the score of
The following formula can express the loss function of the discriminator:
((0)
Σ log(Do(x))) sEX:
Σ log(1 - Do(x))
2EX2
Where represents the discrininator with 0 as the parameter, and represent the human human-generated machine-generated sample data set and the machine- generated sample data set, respectively.
Positive
written by human
negative
scores
generated by G
generated by G,
train
Figure 1: Training process of the discriminator
The trained discriminator will be used to score a sentence. Since some text generation tasks lack standard reference, we use relative metrics to avoid this problem. The OUMG metric is used to compare the sentences generated by the two models and measure the gap between them. In practical application, we need to compare the two text generation models G1 and G2, and obtain the relative scores of the two models. To ensure the fairness of the two models, when generating the sample data set, stipulate that sentences generated by the two models are mixed equally, thus (1) becomes
1(0)
Σ log(Do(z)) zeXi
Σ zEXc,
log(1 Do(x)) ZEXG,
Σ
log(1 Do(x))
After fully trained to converge, the discriminator distinguish human-generated and machine- generated sentences with high accuracy. The higher the sentence scores, the closer it is to human- generated sentences. Next, G, and G2 and get two groups of scores. We process the score of the discriminator adopting the following two relative metrics:
the trained discriminator to score two test sets from the model
3

---

Under review as a conference paper at ICLR 2022
generated 6;
discriminator
scores
compare
trained discriminator
generated by G2
Figure Process of evaluation using the discriminator
High Score Rate Difference(HSRDD): When sentence gets score of more than 0.5 after it is input into discriminator, we call a high score, indicating that discriminator determines that the sentence is written by humans. High score rate(HSR) refers to the ratio of set of sentences that are judged as high score to sentences in a model and high score rate difference expressed as
N No )00.5 N Do(a,))>00 )20.5
(3)
Where N represents the total number of sentences generated by each model, ( )>0.5 and N DD((Fay)> 1>0.5 represent the number of high-score sentences the two groups. To avoid the inter- ference of redundant factors, the same number of sentences generated by each model is required in the comparison.
For the wwo-generation models G₁ and G2 to be compared, if the high score rate difference is a positive number s, the probability that model Gi generates realistic-looking text more than that of G2 If 8 is negative number, the conclusion is the opposite.
models, G1 and G2, be compared used to generate the sentences, and the discriminator is used to judge the generated sentences. By comparing the scores of the output under the same inputs, get the upgrade rate. The specific formula is shown in (4):
Upgrade Rate(UR): Given the same of inputs,
The specific formula is shown in formula (4)
ND )>
(4)
Where NDa (steg) represents the number times G scores more than G2. Similar to the high score difference, also stipulated here that the number of sentences generated by each model the same, namely N.
The upgrade rate indicates the probability that G, generates better sentences than G2 when the input is the same. When the upgrade rate is positive number close to 1, we can assume that G1 outperforms G2 in most cases.
3.2 GUIDE TEXT GENERATION
Another critical feature OUMG is to adjust and guide the text generation process to make the generated results closer to the sentences written by humans. Our inspiration comes from FUDGE Yang & Klein (202)) and has been improved on this basis. The process that guides text generation shown in Figure
Given an autoregressive language model G, the model can output the probability P( (1 of the next word under the condition of given word sequence. Limited by the performance of the
4

---

Under review as a conference paper at ICLR 2022
G
1
k
trained discriminator
Si
A(s)
tz
1
tz
S2
12
generator
In
Sg.
Fk
Figure 3: Guiding text generation with discriminator
text generation model, the probability distribution of the model is not the same that of the actual natural language, which leads to the bias of the above conditional probability. As the generated text gradually grows longer, the bias will become larger and larger. OUMG changes the probability distribution of the final output by scoring the candidate with discriminator at each step. making closer the probability distribution of natural language, thus alleviating the bias.
When guiding text generation, the discriminator we use has the same structure as the discriminator for metric. The only difference is that the data set used for the training discriminator different. Since the autoregressive language model is generated word by word in order, the discriminator for guiding generation must have the ability score an incomplete sentence. We change the training set based on the discriminator for metric, split each complete sentence into fragments of different lengths, and put the same label on it. After sufficient training, the generator can judge the similarity between the final generated sentences and human language in advance when the sentence is not completed and give probability. Next, we will use this probability to guide the following generation process.
Assuming that there is no bias, the output P(z (11--1 of the generator is the actual probability. When bias occurs, lower score is obtained after the candidate sentences are input into the dis- criminator, and the greater the bias, the lower the score. Therefore, we use the discriminator to adjust the logits of the generator. Considering the running time, We select k tokens with the high- est probability, and combine each token separately with the currently generated sentence fragment T1:i-1 to input into the discriminator. To facilitate the calculation, we also use the output logits of the discriminator instead of the final scores, which is represented by s;(1 (1 s < k). Next, s; is processed as follows:
(s; S Syann)
f(s:) a
(8man
8mmnn)
Where a a hyperparameter, the purpose of which facilitate more flexible control of the impact of the discriminator score on the final result. We add fs;) to the logits of the generator to obtain new logits for the following sampling.
EXPERIMENTS
4.1 EVALUATION METRICS
The main experiments of this paper is to generate Chinese poetry. Chinese poetry has different rules from modern language, and many people do not understand them well. In addition some experts, most ordinary people cannot distinguish between ancient and machine-generatee poetry, nor can they objectively evaluate them. Therefore, human evaluation cannot be used as the most authoritative metric. Moreover, generating poetry requires that the generated sentences diverse and even artistic, so there is no fixed reference standard. Due to the above characteristics, common metrics cannot be used for Chinese poetry tasks, but method is not affected, which fully reflects the versatility of OUMG.
Discriminators in OUMG can be flexibly selected, with the only requirement being to ensure clas- sification accuracy. Our experiments use an attention- based Bi-LSTM model (Zhou et al. (016) as the discriminator, which has relatively simple structure and good performance.

---

Under review as a conference paper at ICLR 2022
Model
Acc
HSR
HSRD
UR
G1
0.083
0.906
0.050
0.869
G2
0.033
Table I: OUMG is used to evaluate the results of the SongNet model before and after modifica- tion. We take the model before the change as Gi and after the change G2. Acc represents the classification accuracy of the trained discriminator on the test set.
We follow the work of SongNet(Li et al. 2020) and use their generator model, which has achieved better performance than (Ra-ooRadford et al. 2019) the poetry generation task. To prove the effectiveness of the metrics, we need to use two models with apparent quality differences to compare. We take the original structure of SongNet as better model and modify the decoding part as poorer model. SongNet uses the top-k sampling method in decoding, which intercepts k tokens with the highest probabilities and then samples them according to likelihood. When each token except punctuation is generated, we shift the sampling range by 4k elements, that is, the tokens with the sequence number from 4k to 5k after sorting by probabilities. Since the original model has successfully modeled the probability distribution of Chinese poetry, when 4k words with the highest probability are discarded in each decoding, the quality of the final sentences will inevitably decrease significantly. Therefore, we take the modified model poor model verify whether OUMG is effective.
We use the samples of the discriminator. Next, we let the two models to be compared generate the same number of Song Ci, the two mixed negative samples. Using the above dataset train the discriminator fully, the final test results are shown in Table
dataset SongNet and take its training set, about 20 000 Song Ci, positive
can be seen from Table I that the correct rate of the discriminator exceeds 90 percent, indicating that the discriminator can distinguish the poetry between generated by two models and written by humans according to their different characteristics. Their high score rate difference and upgrade rate are positive, indicating that G1 outperforms G2. As can be seen from the upgrade rate, model G has nearly 87 percent of output sentences better than G2 for the same input. In addition, the high score rates of the two models are minimal, indicating that there apparent differences between the sentences generated by the two models and the sentences written by humans.
4.2 GUIDING TEXT GENERATION
The discriminator used to guide text generation have the same structure as discriminators used as metric but have different training sets. The new training set generated by truncating the complete sentence word by word to form the set of the first words of the sentence (2 < i S π. n2 is the length of the sentence). We use the processed sentences in SongNet's training set as positive samples and the same number of processed sentences directly generated by SongNet as negative samples. Finally, we combine the trained discriminator with the decoding part the generator.
Compared with the results in T the discriminator accuracy in Table reduced to 85 percent. This is because the sentences used as negative samples are more similar to positive samples, which increases the learning difficulty of the discriminator. The high score rate of G1 is significantly higher than that of G2, and the high score rate difference indicates that G₁ 19 percent more likely to produce realistic-looking poetry than G2.
Compared with Table D and Table 2 we find that the same model ( G2 in Table and G1 in Table 2) has different high score rate in different evaluation results due to the characteristics of relative metrics. In OUMG, the evaluation score of a model affected by the comparison model. The score reflects the relative gap between the models rather than absolute performance evaluation.

---

Under review as a conference paper at ICLR 2022
Model
Acc
HSR
HSRD
UR
G1
0.231
0.851
0.190
0.811
G2
0.041
Table OUMG is used to evaluate the results of the SongNet model with and without discriminator, We regard the model with discriminator as G and the model without discriminator as G2.
5 CONCLUSION
We propose objective and universal automatic evaluation metric for text generation named OUMG. This method uses a discriminator to score the sentences generated by the two models to measure the relative gap between the two models. Since the discriminator can distinguish human- generated sentences from machine-generated sentences with high accuracy, the score of a sentence be regarded as the similarity between the sentence and sentences written by humans, which ensures the effectiveness of the metric. Our method can be more commonly used for text genera- tion tasks without reference standards than the traditional text generation metric. Compared with human judgments, our practice does not prefer specific characteristics and is more objective. The experiments show that OUMG can make reasonable evaluation of the text generation task with- out reference standards. Moreover, the discriminator can further improve the performance of the original model by guiding the generation process.
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. arXiv preprint 409.v:140..0473, 2014.
Sumit Chopra, Michael Auli, and Alexander M Rush. Abstractive sentence summarization with attentive recurrent neural networks. In NAACL HLT, 2016.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In ACI., 2018.
Hao Fang. Saurabh Gupta, Forrest landola, Rupesh K Srivastava, Li Deng, Piotr Dollár, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C Platt, and et al. From captions to visual concepts and back. In CVPR, 2015.
Philipp Koehn. Statistical machine translation. Cambridge University Press, 2009.
Karol Kurach, Sujith Ravi, Tobias Kaufmann, Andrew Tomkins, Balint Miklos, Greg Corrado, László Lukáes, Marina Ganea, and al Peter Young. Smart reply: Automated response sug- gestion for email. In ACM SIGKDD, 2016.
Jey Han Lau, Trevor Cohn, Timothy Baldwin, Julian Brooke, and Adam Hammond. Deep-speare: A joint neural model of poetic language, meter and rhyme. arXiv preprint arXiv:18017 887.00349,, 2018.
Piji Li, Haisong Zhang, Xiaojiang Liu, and Shuming Shi. Rigid formats controlled text generation. In ACL, 2020.
Yi Liao, Yasheng Wang, Qun Liu, and Xin Jiang. Gpt-based generation for classical chinese poetry. arXiv preprint /rxiv:/907.00/51, 2019.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In ACI., 2004.

---

Under review as a conference paper at ICLR 2022
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. ICLR, 2013.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu, Bleu: a method for automatic evaluation of machine translation. In ACL, 2002.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke S. Zettlemoyer. Deep contextualized word representations. In NAACL-HLT, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 2019.
Alexander M Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. In EMNLP, 2015.
Abigail See, Peter J Liu, and Christopher Manning. Get the point: Summarization with pointer-
generator networks. In ACL, 2017.
Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D Manning. Do massively pretrained language models make better storytellers? arXiv preprint arXiv: 10705, 2019.
Thibault Sellam, Dipanjan Das, and Ankur P.Parikh. Bleurt: Learning robust metrics for text gener- ation. In ACL, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention all you need. NIPS, 2017.
Kevin Yang and Dan Klein. Fudge NAACL, 2021.
Controlled text generation with future discriminators. In
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluat- ing text generation with bert. In ICLR, 2020.
Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. Attention-based bidirectional long short-term memory networks for relation classification. In ACL, 2016
8


Reviews:
[{"review_id": "s1S8kM40rw", "reviewer": "Reviewer_pGB6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "First, I appreciate the efforts of the authors in devising a simple method towards addressing the evaluation of automated text generation systems. Simpler methods are easier to interpret and reproduce. However, it is also important to properly support scientific claims and methodologies with sufficient experiments. By this, I notice that there seems to be an overgeneralization in the problem statement. The authors indicate that for systems like story generation and poetry generation, the final suitable reference answer is “not unique”. While this is true, it is not the only reason why these tasks are special. There should be further clarification that the freedom of the reference answer is not entirely free in open-ended text generation tasks since there are still some rules involving rhyming, format, and integrity that need to be followed specifically in the case of poetry generation as highlighted in Li et al (2020). The addition of this discussion should be in order since this is the main (and only) task tackled in the paper.\n\nThere is no discussion of how the proposed metric discriminates between human and model-generated texts. What textual factors do you look for? Do you check the factual consistency? The readability? The cohesion? The stylistic properties? Likewise, there seems to be no discussion on the two models based on their similarities/differences by nature. Discussion for these parts should be in order, not to mention this is the main contribution of the paper.\n\nThere is no definitive review of recent works specifically for the development of objective evaluation for text generation. The only qualified citations are for BLUE and ROUGE, the rest are mostly from types of methods (ex. BERT, Word Embeddings) but not the actual work done for evaluating text generation systems. For starters, I recommend the authors to look at papers of BERTScore (Zhang et al, 2020), BLEURT (Sellam et al, 2020), and WMD (Kusner et al, 2015) for guidance of proper comparison of evaluation metrics across text generation tasks. In addition, I also expected the paper of Lin et al (2020) to be included here but was only mentioned later in other sections. \n\nTechnicality-wise, it’s extremely hard to believe some of the claims of the paper (“our method is universal and can also be used for the first type of task”) without empirical results from experiments for both Type 1 and Type 2 tasks and without the use of other datasets to prove sufficient “universality” of the evaluation metric. Even with a high accuracy reported in the paper, the proposed metric has only ever been tested for one task (poetry generation) and one dataset (SongNet). Just to mention, the page limit is 8. The authors are advised to consume all the available space to supply needed experiments and comparisons (including baselines) with text generation tasks.\n\nLastly, there are numerous instances where parts of the discussion involving experimentation details are extremely vague such as:\n1. In Section 3.2, “we trained it sufficiently” --> to what extent? Lacks specific details of retraining.\n2. From the same section, “split each complete sentence into fragments of different lengths” --> same question, to what extent? The answer to this is found in the next section but it should already be mentioned in the first instance.\n3. In Section 4.1, “Chinese poetry has different rules from modern language, and many people do not understand them well.” --> This is another overgeneralization. What are these “different rules” and why do “people don’t understand them well”? Further clarification is needed.\n4. In Section 4.2, the only motivation for choosing Bi-LSTM is “which has a relatively simple structure and good performance” and nothing more. I would like to see a stronger basis for this choice. Provide justification as to why Bi-LSTM is the most practical model for the discriminator for Chinese poetry generation with respect to the task at hand. Cite legitimate references along the way.\n5. There seems to be an overuse of the word “significantly” scattered in the paper without proper statistical testing. I suggest revising this to “substantially” as the word is already overloaded in literature.\n", "summary_of_the_paper": "The paper describes OUMG, a model-based metric for evaluating text generator systems which does not rely on reference standards or gold-standard data. The model is trained similar to a conventional GAN or Turing test-like setup: a discriminator is trained with human-generated and artificially generated sentences to produce a soft-output (probability) to judge if an input text is human-generated or not. After which, the model can be applied as a guide for generating text autoregressively in the hopes of producing texts of better quality. The metric is tested only on a Chinese poetry generation task with one dataset and a single model architecture (Bi-LSTM) both coming from the SongNet study (Li et al, 2020).\n", "summary_of_the_review": "Overall, the current state of the paper needs to be improved to match the authors’ claim on the universal/objective applicability of the proposed metric. I see no strengths from the paper empirically although I particularly favor the simplicity of the proposed method. The major weakness of the paper is the lack of sufficient documentation of the practicality of the proposed metric via rigorous and extensive experiments involving various text generation tasks and more datasets. Thus, I am rejecting the paper based on these grounds and will wait for clarifications from the authors.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "BbyvIWKcjX", "reviewer": "Reviewer_Nb33", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main strength of this paper is suggesting an interesting idea to evaluate generated texts. I strongly agree that most evaluation methods rely on comparing with ground truth data - human-written texts. But unlike classification tasks, the output of natural language generation (NLG) models can be various and appropriate since the language can have different expressions even if they have the same semantics. The idea is quite sound and straightforward so that the other researchers can research the claim and idea.\n\nWhat concerns me most is the lack of experiments to support the claims in this paper. First, U in OUMG stands for universal, but the authors only show the performance on poetry generation task. As the authors mentioned, we can apply the same idea to other NLG tasks, but we do not know the appropriateness of OUMG for the tasks without any evidence. Second, only one poor model is selected as $G_2$. Are the results the same when the $k$ is changed more or less? What if $G_2$ takes $k+1$? Can OUMG show the positive HSRD value? I am curious about these results. Third, why is OUMG objective? How can we measure the objectiveness of OUMG? The authors need to show the evidence for their claim.\n\nAnd the claim and idea are quite similar to the previous research work. [1-4] suggest trained evaluation methods for dialogue models. Especially, [1] suggests an adversarial learning method, and it is quite similar to the idea in this paper. This paper does not mention these related work.\n\nMore questions\n- Even though human judgment has some bias, is there any positive relationship between OUMG and human judgment? Other evaluation methods show the relationship [1-4], so it would be better to make the same comparison.\n- What is the value of alpha in equation (5) for the experiment? Are there any changes based on the alpha value?\n\nTypos\n- No ‘N’ in the equation (1)\n- Would you please write the meaning of variables in equations (i.e., $X_1$ and  $X_2$ in the equation)\n\n\nReference\n[1] Li, Jiwei, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. \"Adversarial Learning for Neural Dialogue Generation.\" In EMNLP. 2017.\n\n[2] Lowe, Ryan, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. \"Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses.\" In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1116-1126. 2017.\n\n[3] Tao, Chongyang, Lili Mou, Dongyan Zhao, and Rui Yan. \"Ruber: An unsupervised method for automatic evaluation of open-domain dialog systems.\" In Thirty-Second AAAI Conference on Artificial Intelligence. 2018.\n\n[4] Bak, JinYeong, and Alice Oh. \"Speaker Sensitive Response Evaluation Model.\" In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6376-6385. 2020.\n\n", "summary_of_the_paper": "This paper presents a new evaluation metric for natural language generation models based on the discriminator that distinguishes between human-written texts and models generated texts, OUMG. To train OUMG, the authors bring two different generation models and treat them as negative samples. After the training OUMG, the authors suggest an evaluation method by computing the high score rate difference and upgrade rate among the two models. The authors also suggest guiding text generation by OUMG. Experiments show that trained OUMG can identify human-written texts against model-generated texts. And it can identify texts that a poor model generates.\n", "summary_of_the_review": "Overall, I admit that OMUG is a sound method to identify some aspects of texts by humans and models. But I have some questions and concerns that I want to listen to the author’s responses.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4MqnIcixIwi", "reviewer": "Reviewer_27Bu", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper has the following weaknesses:\n- Given that the goal of the paper is to present an evaluation metric, I would expect a study comparing its output with human evaluation, as done for example in the BertScore paper cited. As it is, all we know is that it can tell between two systems one of which is made to be worse than the other. That doesn't tell us much about whether it correlates with human judgements.\n- It is true that human judgements can be subjective, but if this is a concern, then one should consider a task where objective evaluation is possible. However, this is not the case with Chinese poetry generation, or any kind of literature generation in general.\n- The equations 1 and 2 have symbols that are not explained, i.e. X_1 and X_2.  Also, in equation 2 I think a plus sign is missing between the last two terms of the loss function.\n- I don't think that most natural language generation tasks that have a singe correct answer; even in machine translation there are multiple references given. \n- Apart from Yang and Klein,  the use of trained discriminator for guiding natural language generation has been previously explored, e.g. : http://proceedings.mlr.press/v119/scialom20a.html . As it is considered by the authors a contribution of this paper to be able to guide the generation, a comparison is needed against these methods is needed.", "summary_of_the_paper": "This paper proposes a reference-less evaluation metric for natural language generation tasks. It presents experiments on Chinese poetry generation.", "summary_of_the_review": "The evaluation is flawed as the proposed natural language evaluation metric is not compared to human judgements. Also, Chinese poetry generation is not a suitable task to evaluate on given its subjectivity.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "3je11_PnF2", "reviewer": "Reviewer_BR3n", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: with respect I'm not sure there are any. \nWeaknesses: \n- The paper is poorly written, with many sections difficult to follow. The proposed metric is named OUMG, which I'd guess is meant to stand for Objective and Universal Metric for Generation, although I don't think it's defined anywhere. Assuming this, both parts (objective, universal) are not accurate. A few examples of the difficulty I had with the writing: section 3.1 eon 1 doesn't defined X1 or X2 (which do appear in equation), but does define N (which doesn't). The upgrade-rate and high-score-rate-difference seem overly complex, for what is essentially one system scores higher than the other. But this is getting into minor details; there are larger flaws with the paper. \n- The primary flaw is that it makes grand claims about being a metric suitable for use in many places, yet only reports results on a single poetry task, which further are very weak since they don't involve any ground truths e.g. via human ratings. Indeed the paper even makes the claim that for this poetry that \"Therefore, human evaluation cannot be used as the most authoritative metric\". This just seems plain wrong. It might need the right humans. But the main point is that there is nothing objective in the results here and the reader can't trust the conclusions as a result. \n- Statements like \"which fully reflects the versatility of OUMG\" are very non-scientific given the contrast with what results are actually reported. \n- The \"only requirement\" of the discriminator is accuracy, yet the paper then uses a bi-lstm, yet all recent work in NLP would suggest you'd build a more accurate classifier with a transformer model. This is slightly nit-picking but there is a constant disconnect between the statements made and the actual method and results. Another example, the bi-lstm is described as having a \"relatively simple structure\". This simply isn't scientific language and shouldn't be in a paper. \n- The discussions at the end of section 3 are both not very clear and seem to muddle what a language model does (score more likely language higher) compared to what's required with poetry (where surprise is required). ", "summary_of_the_paper": "An NLG metric for problems where there is many plausible options for the generated text (e.g. story telling, poetry generation, dialog continuation) is proposed. \nResults are reported on one one dataset, but the paper is very poorly written and makes a lot of claims which are not correct. ", "summary_of_the_review": "I read the second half of this paper quickly since IMHO there are major flaws with the work. The paper makes many claims about how widely applicable the proposed metric is, and lacks evidence for nearly all of them. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "2hrIFgCQXCS", "reviewer": "Reviewer_xKWL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The idea of the proposed metric is somewhat intuitive and straightforward. \n- The paper is structurally organized, although there are many missing/wrong explanations for notation (e.g. “X_1” and “X_2” from equation (1), “D” instead of “N” for the discriminator, etc.), which I suppose will be fixed in the next version of the paper.\n- The paper can benefit from more justification and elaboration of when and why this metric, or more specifically the pairwise comparison, is useful. After reading the paper, it is still unclear to me when one would want to report numbers on the pairwise comparison (e.g. when you only care about two specific models? why?).\n- The proposed method raises the question of reproducibility of the numbers reported on such metrics. In addition to providing more details on the experiment procedure (e.g. how did you sample sentences to compare? how many did you use?), the paper needs to prescribe a desired practice for using the proposed metrics: Should one fix one of the two models and swap out the other model to make consistent comparison among more than two models? Can one can reuse the discriminator trained on the two models to compute metrics for other models? How many samples do you need for training and testing? How robust are the metrics to sampling and the sample size?\n- Also, defining a single number as an “objective” and “universal” metric for human-likeness calls for more qualitative analysis. It is unclear exactly what the metrics are capturing. The only explanation provided in the paper is “similar to humans.” More fine-grained analysis on various aspects of text (e.g. fluency, coherency, grammaticality, style, etc.) is desired and comparison with other evaluation metrics (not just conceptually, but quantitatively showing correlation) would be highly recommended.\n- For the first part of the experiment, I am not quite convinced that the intentional degrading of the default model is the best way to demonstrate the effectiveness of the proposed metrics. It does show that the metrics can detect severely worse performance, but metrics are more valuable when they allow us to distinguish similar models in a more fine-grained way or to reason about specific aspects of quality. Also, for the second part of the experiment, I am not fully convinced with the claim that the dropped accuracy implies the improved performance of the guided model without knowing which sentences are sampled and what they are compared against (i.e. what has been used during training). Could it be a weird artifact due to some syntactic resemblance of sampled sentences at test time? More examples and qualitative analysis would help readers better interpret the results.\n- Please consider including more information about resource requirement for training and discussing environmental impact. The proposed method seems to require quite expensive and time-consuming process of training and inference specific to the two models, or even a specific set of sampled sentences of interest. Even if one time evaluation cost of two models might be negligible, it is worth reflecting on the long-term impact.", "summary_of_the_paper": "The paper proposes a new evaluation metric, OUMG, to measure human-likeness of machine-generated text without reference. The key idea of the method is to train a discriminator on both human-generated texts (as positive) and machine-generated texts (as negative) and use the discriminator to assign a score to a given sentence, which represents the similarity between the sentence and a set of sentences written by humans provided at training time.\n\nThe paper considers a pairwise comparison setting: any pair of models are compared by training such a discriminator on the outputs of those two specific models, and proposed metrics are used to measure the *relative* goodness of one model’s outputs with respect to the other model: (1) which model’s outputs have higher ratio of sentences with scores > 0.5 (i.e. HSRD) and (2) which model’s outputs have higher scores compared to the other model (i.e. UR).\n\nThe experiment considers the task of Chinese poetry generation and uses SongNet (Li et al., 2020) as a default model. The first half of the experiment intentionally makes the default model worse and demonstrates that OUMG metrics can show the default model performs better than the intentionally degraded model. The second half of the experiment uses the scores from the discriminator to adjust logits from the default model in order to improve the human-likeness of generation. The reduced accuracy of the discriminator after training on such outputs is interpreted as the evidence that the guided generation produces more human-like texts.\n\nThe main contribution of this paper is to frame text evaluation as pairwise comparison between two models and use relative metrics to measure the performance gap between two models. The authors claim that this approach addresses some of the limitations of previous metrics based on string matching, word embedding, and language models such as superficial matching, reliance on reference, narrow focus on fluency (Section 2).", "summary_of_the_review": "The two main parts that are unclear to me are the need for *relative* metrics and what the metrics are actually capturing (supposedly similarity between human- and machine-generated texts). The experiment and analysis do not provide enough information to support the claim that the proposed metrics are objective/universal and have guiding ability.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a reference-less metric for evaluating NLG systems by training a discriminator which distinguishes between human-generated and machine-generated text. 

The main concerns raised by the reviewers were (i) lack of clarity in certain portions of the paper (ii) lack of demonstration of the "universal" applicability of the proposed metric (only evaluated for poetry generation) (iii) lack of clear guidelines on how to use the proposed metric in a reproducible manner (iv) lack of details about what exactly does the proposed metric capture and look for in the generated text.

The authors did not respond to the specific queries of the reviewer and agreed that more work is needed on their part.
Decision: Reject