KETG: A Knowledge Enhanced Text Generation Framework

1 INTRODUCTION . Recent pre-trained language models such as GPT-2 can capture clear semantic and syntactic features ( Radford , 2018 ) , performing well in machine translation and abstract generation tasks ( Li et al. , 2016 ; Wang et al. , 2016 ) . However , the application of language models in text generation still needs to be explored . The logic in text generation , especially literature creation , is always obscure , which means they are usually low-frequency , causing the difficulty of modeling by current language models . On the other hand , too much limits of prior information will lead to homogenization in generated texts . To address these issues , ( Guan et al. , 2020 ) proposes a knowledge-enhanced pretraining model for commonsense story generation , by transforming the commonsense triples into sentences using a template-based method . However , the template-based transformed sentences from commonsense triples for post-training are rather homogeneous . In this paper , we introduce an innovative knowledge enhanced text generation ( KETG ) framework , which incorporates knowledge tuples and their associated sentences in training , such that the logic relation lying in the knowledge tuples can be effectively addressed . Regarding the sentences associated with the knowledge tuples , we may generate the sentences from the tuples by template-based method as in ( Guan et al. , 2020 ) . However , incorporating real corpus sentences would be more beneficial as they generally exhibit more diversity than those generated from templates , if they are available . In this way , the generation model can learn the both logicality and diversity in the knowledge tuples and sentences . We validate our KETG framework on rhetorical text generation , which is an important and essential part in modern literature ( Tu et al. , 2013 ) . Rhetoric is quite obscure , requiring strong logical correlation , and a rhetoric knowledge graph with explicit logical information ( rather than the commonsense knowledge graph ) would be helpful to rhetorical text generation . Unfortunately , to the best of our knowledge , we are not aware of any rhetoric knowledge graph . Hence by using relation extraction methods , we build a rhetoric ( specifically , here we refer to metaphor and personification ) knowledge graph from a collection of Chinese poems and compositions . With the newly built rhetoric knowledge graph and the corpus from which the knowledge graph is extracted , we train a rhetorical text generation model . Both automatic and manual evaluations show that our KETG model outperforms baseline models on rhetorical type control , semantic comprehensibility and diversity . Experiments also illustrate that incorporating sentences by template-based method in training results in rather similar generated text as the template , while incorporating real corpus sentences brings more diversity in text generation . To sum up , the main contributions of this paper are summarized as follows : 1 . We propose a KETG framework , which includes both knowledge information and associated sentences in training to address logicality and diversity . 2 . We validate our KETG framework on rhetorical ( metaphor and personification ) text generation . Results show that our KETG framework can generate more reasonable and diverse rhetorical texts , and the rhetoric types can be controlled implicitly . 3 . To the best of our knowledge , we build the first Chinese rhetoric ( metaphor and personification ) graph with 35228 tuples . 2 RELATED WORK . Language Model ( LM ) In order to use as much semantic information as possible , several research work has been conducted . In early stage , researchers focused on the feature-based method to express syntactic and semantic information in texts . However , this kind of method can not solve the problem of polysemy . To improve , ( Peters et al. , 2018 ) Elmo is proposed to capture complex word characteristics in texts . Meanwhile , in NLP tasks , massive texts are often unlabeled . To solve this , fine-tuning models are raised , which can learn ” common sense ” from unlabeled texts . Both Bert and GPT-2 are representative models . ( Wang et al. , 2019 ; Ferreira et al. , 2019 ) They have achieved good evaluation results in multiple NLP tasks , such as named entity recognition , Q & A , text classification and text generation . Knowledge Enhanced LM To mimic human ’ s writing manner , the most basic thing is to ensure that the generated text fluent and semantically understandable . Secondly , the common sense of humankind is also indispensable . Furthermore , aesthetics and logicality make language expressions more vivid , novel and apt . However , it ’ s hard to meet these requirements merely by language models . ( Bowman et al. , 2015 ) used common-sense knowledge base in natural language inference ( NLI ) and NLG . As mentioned in ( Zhou et al. , 2018 ) , common sense knowledge can promote performance in dialogue generation . ( Mihaylov & Frank , 2018 ) introduced a neural reading comprehension model that encodes external common sense knowledge as key-value memory . ( Zhang et al. , 2019 ) introduced a knowledge enhanced pre-trained language framework ERNIE , trying to increase the knowledge representation by masking semantic units such as words and entities . ( Guan et al. , 2020 ) proposes a knowledge-enhanced pretraining model for commonsense story generation . They post-train the model on knowledge-augmented data by transforming the commonsense triples into sentences . Rhetorical Text Generation Rhetoric is an important and essential part in modern literature ( Tu et al. , 2013 ) . It can express author ’ s passion and grace , improving the aesthetic merit of creations . ( Liu et al. , 2019 ) proposed a rhetorically controlled generation model for Chinese poetry generation to govern the rhetorical modes . Through a classifier inserted in the encoder , they can control the rhetorical modes of generated poems . However , it does not include knowledge graph and hence might generate illogical sentences , like ” Flakes of snow are flying like snow ” , which appears to be a metaphor , but includes illogical ‘ snow like snow ’ . 3 OUR KETG FRAMEWORK . We propose an innovative KETG framework , to combine the knowledge information with text generation models , just like the external device to computer . The architecture could be used to combine different types of knowledge graph with text generation model . As depicted in Figure 1 , we query the keyword in knowledge graph firstly , getting a context vector containing knowledge information . Then , we concatenate the context knowledge vector and the keyword vector , input them together with associated sentence to the language model . In this way , we can highlight the topic in the sentence and potential logical relationship between the entities , forcing the model pay more attention to them . When generating texts , with a given topic word , we get the context knowledge vector in the same way , which then serve as input to the trained model to generate the whole sentence in an auto-regressive manner . Compared with single topic word , the expanded context knowledge vector can also take the diversity advantage of knowledge graph , make sure the generated sentences full of variety . It ’ s worth mentioning that the real corpus sentences are retained in our framework , rather than those generated from templates , which means the generation model can learn the diversity of sentence structure . In detail , we add [ cls ] at the beginning of keyword vector and put [ mask ] to separate them from original sentence . After that , we concatenate them together as the input of text generation model . Using above approach , we can integrate knowledge information into text generation model naturally . With external knowledge , the generation model can generate more reasonable text , meanwhile captures the significant semantic and syntactic features . 4 RHETORICAL TEXT GENERATION . Rhetoric is an essential element in literature . Among 8744 Chinese poems ( Liu et al. , 2019 ) , 31.4 % are metaphor and 18.5 % are personification . We also collected 54949 excellent sentences from named composition . Among them , 11989 are metaphor and 28718 are personification . It ’ s obvious that metaphor and personification are the main parts of rhetoric . Therefore we build the rhetorical knowledge graph on metaphor and personification . 4.1 RHETORICAL RELATION EXTRACTION . We use relationship extraction algorithm ( Rai et al. , 2016 ; Alt et al. , 2019 ) to build our rhetorical graph . Based on bert+crf layer ( Huang et al. , 2015 ; Lample et al. , 2016 ; Pramanick et al. , 2018 ) , the model is designed to deal with NER ( Named Entity Recognition ) and relation classification jointly . In addition , we introduces a priori relation graph to filter NER results , which can improve the accuracy of extraction results effectively . Besides , in order to address multiple entities in a sentence , a mechanism called ” semi-pointer semi-label ” ( Su , 2019 ) is adopted in our model . 4.2 CONSTRUCTING RHETORIC GRAPH . We build our rhetorical knowledge graph in three steps . Firstly , we collect sentences of metaphor and personification from named compositions . Based on coreference resolution rules , we use Stanford Core NLP tools ( Manning et al. , 2014 ) to extract metaphor to a tuple of ( noumenon , metaphor object , metaphor base ) , meanwhile personification to a set of ( unhuman-subject , human-action/human-emotion ) . Using the above method , we build a seed rhetorical knowledge graph with 8035 rhetorical sentences , after that manually marked to make sure the accuracy . Secondly , we trained a rhetorical classifier using this seed graph . We also add 3432 negative examples to prevent over-fitting . The accuracy of the classifier in metaphor is 0.97 , while personification is 0.75 . Finally , Based on rules and the above classifier , we continue to expand the data set and retrain the classifier iteratively to build a large rhetorical knowledge graph , including 35228 tuples and 30970 nodes . During the construction , we found that rhetoric relationships have strong logicality , especially metaphor . The common storage mechanism will lead to serious logical errors during query . For example , metaphorical relationships like ( snowflake , falling , catkins ) and ( leaf , falling , snowflakes ) , will be stored as [ snowflake ] −feature− [ float ] − like− [ catkin ] , [ leaf ] −feature− [ float ] − like− [ snowflake ] . when searching noumenon ” snowflake ” in the graph , the result will potentially be [ snowflake ] − feature− [ float ] − like− [ snowflake ] . We design a graph storage mechanism to avoid such illogical problem , the structure is shown in Figure 2 . We use a triangle structure to store noumenon , metaphor object and metaphor base . It ’ s worth mentioning that we save the metaphor base as node instead of edge , for that the types of metaphor base are complicated and varied , saving it as edge will lower search efficiency enormously . Personification is similar to metaphor , but it contains only two entities [ unhuman − subject ] − Personification− [ human− action/human− emotion ] . 4.3 GENERATING WITH RHETORIC GRAPH . Firstly , We query the keyword in rhetorical graph to get a context vector , which including corresponding rhetorical information . For example , in metaphor , the vector contains information of metaphor object and metaphor base . After , we concatenate the context vector and keyword vector , sending them together with the associated original sentence to text generation model , training the model using the method in Figure 1 . During generation , given a topic word and rhetoric type , we get the context knowledge vector in the same way , then generate the corresponding rhetoric sentences in the trained model . In particular , we use the Top-K generation method , that is , when predicting the next word , we will randomly select one sentence from probable values of top 5 . This method can effectively solve the problem of repeated words in the generation .

Summary: This paper proposes to use a rhetoric knowledge graph for rhetorical text generation. One of its key contributions is to construct a rhetoric knowledge graph by leveraging SOTA NER and relation classification models.  To generate a rhetorical text, the new method starts with sending a keyword to the knowledge graph to retrieve the neighborhood of the keywords as its context words. Both the context words and the original query word are fed into a language model to generate the final word sequence.