OUMG: Objective and Universal Metric for Text Generation with Guiding Ability

1 INTRODUCTION . According to different expected goals , natural language generation tasks can be divided into two categories . The first type of task has a specific reference standard – usually given by experts . The goal of model optimization is to make the generated results close to or even coincide with the reference standard , such as machine translation ( Koehn , 2009 ) ( Bahdanau et al. , 2014 ) ( Vaswani et al. , 2017 ) , summarization ( Chopra et al. , 2016 ) ( Rush et al. , 2015 ) ( See et al. , 2017 ) , and image captioning ( Fang et al. , 2015 ) . The existing metrics have been able to evaluate these tasks from various aspects and obtain objective scores . The second type of task includes dialogue system ( Kurach et al. , 2016 ) , story generation ( Fan et al. , 2018 ) ( See et al. , 2019 ) , poetry generation ( Lau et al. , 2018 ) ( Liao et al. , 2019 ) , etc . The ultimate goal is to make the model learn the features of natural language and generate sentences that look like written by humans . These tasks have a common characteristic : The suitable reference answer is not unique ( e.g. , The same beginning can be followed by various story plots ) . Therefore , the evaluation will inevitably be biased if only a few sentences are used as a reference standard . In order to better evaluate the quality of text generation results , a universal evaluation metric must meet the following three requirements : 1 . The metric must be available in the absence of corresponding reference standards . 2 . In the same semantic , the change of expression and words should not significantly affect the evaluation score . 3 . The metric must be objective . In this paper , we propose OUMG , a more general evaluation metric for text generation . This method meets the above three requirements and has the ability to guide the generation process , which can further improve the quality of sentences generated by the model . The core of our approach is a discriminator , whose function is to distinguish between humangenerated and machine-generated text . Ideally , this discriminator should be trained with a set of sentences written by humans as positive samples and sentences generated by various models as negative samples . In order to ensure universality , the selection of negative samples must cover all the potential models . However , under realistic conditions , this is impossible . Therefore , OUMG uses a compromise by mixing the generated results of the two models ( e.g. , The improved model and its baseline ) equally as a negative example . After sufficient training , the new samples generated by the two models are scored by the discriminator , and the relative merits of the two models are judged according to the high score rate difference and upgrade rate . In addition , a discriminator with high accuracy can consider the semantic information of the sentence , avoiding the problem of large fluctuations in scores caused by different expressions . At the same time , a model trained by datasets is more objective than human judgments , which meets the requirements of an evaluation metric . For the sake of showing the unique advantages of OUMG , we choose poetry generation as the main task in our experiments . But this does not mean that OUMG can only be used for the second type of task . Since we eliminate the dependence of metrics on reference standards , our method is universal and can also be used for the first type of task . For example , in machine translation , we can use the original sentence and its corresponding translation as the input of the discriminator and train the discriminator to distinguish whether the translation part of the input comes from the reference standard or the machine translation model . Other tasks are similar . Inspired by Yang & Klein ( 2021 ) , our method can also be used to guide text generation . In the text generation process , we use the discriminator to score all the candidate tokens for each step and combine the score with the logits provided by the generator to select the tokens with higher final scores . Compared with the Yang & Klein ( 2021 ) method , we add a hyper-parameter α to control the influence of the discriminator on the generation process . 2 RELATED WORK . 2.1 METRICS BASED ON STRING MATCHING . The earliest metrics use the degree of string matching between two sentences to measure whether the candidate sentence is close to the reference standard . The representatives are BLEU ( Papineni et al. , 2002 ) and ROUGE ( Lin , 2004 ) , which were initially applied to machine translation and were widely used in various other tasks in later studies . The idea is to count the number of times the same word appears between two sentences . If two sentences have a higher degree of coincidence , they are more similar . The difference is that BLEU mainly focuses on the accuracy of the text generation model , while ROUGE focuses more on the recall . There is an obvious problem in this method : only the most superficial matching can be carried out , and no semantic information can be taken into account . It means that these metrics will equate synonyms with completely irrelevant words and are not sensitive to the word order of sentences , which severely limits their accuracy . 2.2 METRICS BASED ON WORD EMBEDDING . In order to solve the problem that the metrics based on string matching can not consider semantic information , researchers began to propose metrics based on word embedding . The original method is to convert the sentence into vector representation by the Word2vec models ( Mikolov et al. , 2013 ) and then calculate the similarity between the two sentences by the cosine similarity method . After word embedding , the generated vector has expressed and abstracted the semantic to a certain extent so that different words with the same meaning have a closer distance . Thereforesuch metrics scoring has more reference value . With the development of many pre-training language models represented by Bert ( Devlin et al. , 2019 ) , more people use Bert and other models ( Peters et al. , 2018 ) to replace the traditional Word2vec models . In contrast , Bert and other models can more dynamically capture the context information , solve the polysemy problem , and refine the semantics more accurately . Bertscore ( Zhang et al. , 2020 ) and BLEURT ( Sellam et al. , 2020 ) are representatives of such metrics . However , although the metrics based on word embedding can well combine semantic information , they still rely on authoritative reference standards , which are only applicable to the first type of text generation task . 2.3 METRICS BASED ON LANGUAGE MODEL . The above two metrics evaluate the performance of the model through the sentences generated by the model . In addition , the language model itself can also be assessed directly . The PPL metrics evaluates whether the language model can generate smooth sentences by calculating the joint probability . Given a trained language model , use it to calculate the probability of each sentence in the test set . The higher the likelihood of this sentence , the lower the corresponding PPL score . That is , the better the modeling ability of the language model . Although many text generation tasks use the PPL score as a reference for model quality , there are still some unavoidable problems . First , the PPL metric only evaluates the probability of the output of the language model without considering the impact of the optimized decoding process on the final production , so it can only be used to evaluate the modeling ability of the language model . In addition , PPL only focuses on the fluency of the sentence itself but ignores other factors that affect the quality of the generated text , such as the consistency with the context and the correspondence with the given topic . It can not effectively evaluate the quality of the text in some tasks , like dialogue generation and poetry generation . 3 OUMG . 3.1 DISCRIMINATOR . The function of the discriminator is to identify which are written by humans and which are generated by the models in a set of sentences . The discriminator outputs scores from 0 to 1 for each sentence , which is the probability that the sentence is judged to be human-generated . When the score of a sentence is higher than 0.5 , we classify the sentence as human-generated . The following formula can express the loss function of the discriminator : l ( θ ) = − ∑ x∈X1 log ( Dθ ( x ) ) − ∑ x∈X2 log ( 1−Dθ ( x ) ) ( 1 ) Where N represents the discriminator with θ as the parameter , and represent the human-generated sample data set and the machine-generated sample data set , respectively . The trained discriminator will be used to score a sentence . Since some text generation tasks lack standard reference , we use relative metrics to avoid this problem . The OUMG metric is used to compare the sentences generated by the two models and measure the gap between them . In practical application , we need to compare the two text generation models G1 and G2 , and obtain the relative scores of the two models . To ensure the fairness of the two models , when generating the sample data set , we stipulate that sentences generated by the two models are mixed equally , thus ( 1 ) becomes l ( θ ) = − ∑ x∈X1 log ( Dθ ( x ) ) − ∑ x∈XG1 log ( 1−Dθ ( x ) ) ∑ x∈XG2 log ( 1−Dθ ( x ) ) ( 2 ) After fully trained to converge , the discriminator can distinguish human-generated and machinegenerated sentences with high accuracy . The higher the sentence scores , the closer it is to humangenerated sentences . Next , we use the trained discriminator to score two test sets from the model G1 and G2 and get two groups of scores . We process the score of the discriminator adopting the following two relative metrics : High Score Rate Difference ( HSRD ) : When a sentence gets a score of more than 0.5 after it is input into the discriminator , we call it a high score , indicating that the discriminator determines that the sentence is written by humans . High score rate ( HSR ) refers to the ratio of a set of sentences that are judged as high score to all sentences in a model and high score rate difference is expressed as NDθ ( xG1 ) > 0.5 −NDθ ( xG2 ) > 0.5 N ( 3 ) Where N represents the total number of sentences generated by each model , NDθ ( xG1 ) > 0.5 and NDθ ( xG2 ) > 0.5 represent the number of high-score sentences in the two groups . To avoid the interference of redundant factors , the same number of sentences generated by each model is required in the comparison . For the two-generation models G1 and G2 to be compared , if the high score rate difference is a positive number s , the probability that model G1 generates realistic-looking text is s more than that of G2 . If s is a negative number , the conclusion is the opposite . Upgrade Rate ( UR ) : Given the same set of inputs , two models , G1 and G2 , to be compared are used to generate the sentences , and the discriminator is used to judge the generated sentences . By comparing the scores of the output under the same inputs , we get the upgrade rate . The specific formula is shown in ( 4 ) : The specific formula is shown in formula ( 4 ) NDθ ( xG1 ) > Dθ ( xG2 ) N ( 4 ) Where NDθ ( xG1 ) > Dθ ( xG2 ) represents the number of times G1 scores more than G2 . Similar to the high score difference , it is also stipulated here that the number of sentences generated by each model is the same , namely N . The upgrade rate indicates the probability that G1 generates better sentences than G2 when the input is the same . When the upgrade rate is a positive number close to 1 , we can assume that G1 outperforms G2 in most cases .

Summary: The paper proposes a new evaluation metric, OUMG, to measure human-likeness of machine-generated text without reference. The key idea of the method is to train a discriminator on both human-generated texts (as positive) and machine-generated texts (as negative) and use the discriminator to assign a score to a given sentence, which represents the similarity between the sentence and a set of sentences written by humans provided at training time.  The paper considers a pairwise comparison setting: any pair of models are compared by training such a discriminator on the outputs of those two specific models, and proposed metrics are used to measure the *relative* goodness of one model’s outputs with respect to the other model: (1) which model’s outputs have higher ratio of sentences with scores > 0.5 (i.e. HSRD) and (2) which model’s outputs have higher scores compared to the other model (i.e. UR).  The experiment considers the task of Chinese poetry generation and uses SongNet (Li et al., 2020) as a default model. The first half of the experiment intentionally makes the default model worse and demonstrates that OUMG metrics can show the default model performs better than the intentionally degraded model. The second half of the experiment uses the scores from the discriminator to adjust logits from the default model in order to improve the human-likeness of generation. The reduced accuracy of the discriminator after training on such outputs is interpreted as the evidence that the guided generation produces more human-like texts.  The main contribution of this paper is to frame text evaluation as pairwise comparison between two models and use relative metrics to measure the performance gap between two models. The authors claim that this approach addresses some of the limitations of previous metrics based on string matching, word embedding, and language models such as superficial matching, reliance on reference, narrow focus on fluency (Section 2).