KETG: A Knowledge Enhanced Text Generation Framework
Authors: Yan Cui
Xi Chen
Jiang Qian
Bojin Zhuang
Shaojun Wang
Jing Xiao
Venue:  (2021)
Keywords: text generation
knowledge graph

Abstract: Embedding logical knowledge information into text generation is a challenging NLP task. In this paper, we propose a knowledge enhanced text generation (KETG) framework, which incorporates both the knowledge and associated text corpus to address logicality and diversity in text generation. Specifically, we validate our framework on rhetorical text generation from our newly built rhetoric knowledge graph. Experiments show that our framework outperforms baseline models such as Transformer and GPT-2, on rhetorical type control, semantic comprehensibility and diversity.

Under review as a conference paper at ICLR 2021
KETG: A KNOWLEDGE ENHANCED TEXT GENERA- TION FRAMEWORK
Anonymous authors Paper under double-blind review
ABSTRACT
Embedding logical knowledge information into text generation is challeng- ing NLP task. In this paper, we propose a knowledge enhanced text generation (KETG) framework, which incorporates both the knowledge and associated text corpus to address logicality and diversity in text generation. Specifically, we val- idate our framework rhetorical text generation from our newly built rhetoric knowledge graph. Experiments show that our framework outperforms baseline models such as Transformer and GPT-2, on rhetorical type control, semantic com- prchensibility and diversity.
INTRODUCTION
Recent pre-trained language models such as GPT-2 can capture clear semantic and syntactic fea- tures [Radford] 2018). performing well in machine translation and abstract generation tasks (Li al. 2016 Wang et al. 2016). However, the application of language models text generation still needs to be explored. The logic in text generation, especially literature creation, is always obscure, which means they are usually low-frequency, causing the difficulty of modeling by current language models. On the other hand, too much limits of prior information will lead to homogenization in generated texts. To address these issues, (Guan al.) 2020) proposes a knowledge-enhancel pre- training model for commonsense story generation, by transforming the commonsense triples into sentences using a template-bas based method. However, the template-based transformed sentences from commonsense triples for post-training are rather homogeneous.
In this paper, we introduce an innovative knowledge enhanced text generation (KETG) framework, which incorporates knowledge tuples and their associated sentences in training, such that the logic relation lying in the knowledge tuples can be effectively addressed. Regarding the sentences asso- ciated with the knowledge tuples, may generate the sentences from the tuples by template-based method as in (Guan et al. (2020.. However, incorporating real corpus sentences would be more beneficial as they generally exhibit more diversity than those generated from templates, if they are available.
In this way, the generation model can learn the both logicality and diversity in the knowledge tuples and sentences.
We validate our KETG framework on rhetorical text generation, which is an important and essential part in modern literature Tu et al. 2013).
Rhetoric quite obscure, requiring strong logical correlation, and rhetoric knowledge graph with explicit logical information (rather than the commonsense knowledge graph) would be helpful to rhetorical text generation. Unfortunately, to the best of our knowledge, we are not aware of any rhetoric knowledge graph. Hence by using relation extraction methods, we build rhetoric (specifi- cally, here we refer to metaphor and personification) knowledge graph from collection of Chinese poems and compositions. With the newly built rhetoric knowledge graph and the corpus from which the knowledge graph is extracted, we train a rhetorical text generation model. Both automatic and manual evaluations show that our KETG model outperforms baseline models rhetorical type control, semantic comprehensibility and diversity. Experiments also illustrate that incorporating sentences by template- basedd method in training results in rather similar generated text the tem- plate, while incorporating real corpus sentences brings more diversity in text generation.
To sum up the main contributions of this paper are summarized follows:

---

Under review as a conference paper at ICLR 2021
1. We propose a KETG framework, which includes both knowledge information and associ- ated sentences training to address logicality and diversity.
2. We validate our KETG framework rhetorical (metaphor and personification) text gener- ation. Results show that our KETG framework can generate more reasonable and diverse rhetorical texts, and the rhetoric types be controlled implicitly.
3. To the best of our knowledge, we build the first Chinese rhetoric (metaphor and personifi- cation) graph with 35228 tuples.
RELATED WORK
Language Model(LM) In order to use much semantic information as possible, several research work has been conducted. early stage, researchers focused on the feature-based method to ex- press syntactic and semantic information in texts. However, this kind of method can not solve the problem polysemy. To improve, (Peters et al., 2018)) Elmo is proposed to capture complex word characteristics in texts. Mcanwhile, in NLP tasks, massive texts are often unlabeled. To solve this, fine-tuning models are raised, which can learn "common sense" from unlabeled texts. Both Bert and GPT-2 representative models. (Wang al. 2019 Ferreira et al. (2099) They have achieved good evaluation results in multiple NLP tasks, such as named entity recognition, Q&A, text classification and text generation.
Knowledge Enhanced LM To mimic human's writing manner, the most basic thing is to en- sure that the generated text fluent and semantically understandable. Secondly, the common sense of humankind is also indispensable. Furthermore, aesthetics and logicality make language expres- sions more vivid, novel and apt. However, it's hard meet these requirements merely by language models. (Bowman et al.) 2015) used common- sense knowledge base in natural language infer- eNLe( and NLG. As mentioned in (Zhou et al. 2018). common sense knowledge can promote performance in dialogue generation. (Mihaylov & Frank 2018) introduced neural reading com- prehension model that encodes external common sense knowledge as key-value memory. (Zhang et al. 2019) introduced knowledge enhanced pre-trained language framework ERNIE, trying to increase the knowledge representation by masking semantic units such as words and entities. (Guan ctt al. 2020) proposes knowledge-enhaneee pretraining model for commonsense story generation. They post-train the model on knowledge auggeeeee data by transforming the commonsense triples into sentences.
Rhetorical Text Generation Rhetoric is an important and essential part in modern literature Tu et al. 2013). It can express author's passion and grace, improving the aesthetic merit of creations. (Liu et al. (019) proposed a rhetorically controlled generation model for Chinese poetry generation govern the rhetorical modes. Through classifier inserted in the encoder, they can control the rhetorical modes of generated poems. However, it does not include knowledge graph and hence might generate illogical sentences, like "Flakes of snow are flying like snow", which appears be a metaphor, but includes illogical 'snow like snow'.
3 OUR KETG FRAMEWORK
We propose an innovative KETG framework, to combine the knowledge information with text gen- eration models, just like the external device to computer. The architecture could be used to combine different types of knowledge graph with text generation model.
As depicted in Figure D we query the keyword in knowledge graph firstly, getting context vector containing knowledge information. Then, we concatenate the context knowledge vector and the keyword vector, input them together with associated sentence to the language model. In this way, we can highlight the topic in the sentence and potential logical relationship between the entities, forcing the model pay more attention to them. When generating texts, with given topic word, get the context knowledge vector in the same way, which then serve as input to the trained model to generate the whole sentence in regresseee manner.
Compared with single topic word, the expanded context knowledge vector can also take the diver- sity advantage of knowledge graph, make sure the generated sentences full of variety. It's worth
2

---

Under review as a conference paper at ICLR 2021
Language Model
concat
context knowledge vector
bert vector
knowledge integration
bert encoding
keyword-sentence
Knowledge
Graph
Figure 1: Illustration of our framework.
mentioning that the real corpus sentences are retained in our framework, rather than those generated from templates, which means the generation model can learn the diversity of sentence structure.
In detail, we add [cls] at the beginning of keyword vector and put [mask] to separate them from original sentence. After that, we concatenate them together as the input of text generation model.
Using above approach, we can integrate knowledge information into text generation model natu- rally. With external knowledge, the generation model can generate more reasonable text, meanwhile captures the significant semantic and syntactic features.
RHETORICAL TEXT GENERATION
Rhetoric is an essential element in literature. Among 8744 Chinese poems et al. 2019), 31.4% are metaphor and 18.5% are personification. We also collected 54949 excellent sentences from named composition. Among them, 11989 are metaphor and 28718 personification. It's obvious that metaphor and personification are the main parts of rhetoric. Therefore we build the rhetorical knowledge graph on metaphor and personification.

---

Under review as a conference paper at ICLR 2021
4.1 RHETORICAL RELATION EXTRACTION
We use relationship extraction algorithm Rai et al. 2016 Alt et al. 2019) to build our rhetorical graph. Based on bert+erf layer(Huang al.] 2015 Lample al. 2016 Pramanick al. 2018), the model is designed to deal with NER(Named Entity Recognition) and relation classification jointly. In addition, we introduces prioni relation graph to filter NER results, which can improve the accu- racy of extraction results effectively. Besides, in order to address multiple entities in sentence, mechanism called "semi-pointer semi-label" (Su 2019) is adopted in our model.
4.2 CONSTRUCTING RHETORIC GRAPH
We build our rhetorical knowledge graph in three steps.
Firstly, we collect sentences of metaphor and personification from named compositions. Based on coreference resolution rules, we use Stanford Core NLP tools (Manning al. 2014) extract metaphor to tuple of (noumenon, metaphor object, metaphor base), meanwhile personification to set of (unhuman- (unhuman-subject subject, aatraaaaactiooumaanneoo eeotiion)............o.... Using the above method, we build a seed rhetorical knowledge graph with 8035 rhetorical sentences, after that manually marked to make sure the accuracy.
Secondly, we trained a rhetorical classifier using this seed graph. We also add 3432 negative exam- ples to prevent over-fitting. The accuracy the classifier in metaphor is 0.97, while personification 0.75.
Finally, Based on rules and the above classifier, we continue to expand the data set and retrain the classifier iteratively to build large rhetorical knowledge graph, including 35228 tuples and 30970 nodes.
metaphor
TrE
snowflake
(dancer)
feature
feature
MeF
noumenon
metaphor object
(gyrate)
metaphor base
Figure 2: Triangle structure of metaphor.
During the construction, we found that rhetoric relationships have strong logicality, especially metaphor. The common storage mechanism will lead to serious logical errors during query. For example, metaphorical relationships like (snowflake, falling, catkins) and (leaf, falling, snowflakes), will be stored as [snow flake] - feature- [float] like [catkin], [leaf] feature [float] when searching noumenon "snowflake" in the graph, the result will potentially feature- [float] like flake]
like lono-Issowfflan be [snow
We design graph storage mechanism to avoid such illogical problem, the structure is shown in Figure 2 We use a triangle structure to store noumenon, metaphor object and metaphor base. It's worth mentioning that the metaphor base as node instead of edge, for that the types of

---

Under review as a conference paper at ICLR 2021
metaphor base complicated and varied, saving it edge will lower search efficiency enormously. Personification is similar to metaphor, but it contains only two entities Junhuman subject) Personi [human action action/humont human emotion].
4.3 GENERATING WITH RHETORIC GRAPH
Firstly, We query the keyword in rhetorical graph to get a context vector, which including cor- responding rhetorical information. For example, in metaphor, the vector contains information of metaphor object and metaphor base.
After, we concatenate the context vector and keyword vector, sending them together with the asso- ciated original sentence to text generation model, training the model using the method in Figure [ During generation, given a topic word and rhetoric type, we get the context knowledge vector the same way, then generate the corresponding rhetoric sentences in the trained model.
In particular, we use the Top-K generation method, that is, when predicting the next word, we will randomly select one sentence from probable values of top 5. This method can effectively solve the problem of repeated words in the generation.
EXPERIMENT
5.1 DATASET
We build dataset based sentences of named compositions, which distributed in various genre and theme. The corpus contains 5681 14 sentences of three categories. Among them, 17721 sentences metaphor, 20402 are personification, and the rests are rhetoric.
Table 1: Automated evaluation results. The best performance is highlighted bold. The PPL score marked with N/A is not comparable, for that the generation approach and dataset of AC model (Liu 2019) all different from ours. The comparison results only for reference.
Model
PPL
Precision
Recall
Rhetoric-Fi
GPT2+KG
7.72
0.69
0.66
0.67
Transforme++KG
8.69
0.74
0.73
0.73
AC model
N/A
0.68
0.67
0.67
Table Manual evaluation results. The C indicates Correlation with keywords. and denotes the Fluency of generated texts and grammatically comprehensibility. A represents artistic aesthetics.
Model
C
F
S
A
GPT2
2.78
3.92
3,62
2.99
GPT2+KG
3.41
3.91
3.67
3.08
Transformer
1.97
3.14
3.02
2.91
Transforme++KG
2.98
3.22
2.96
2.77
5.2 EXPERIMENT SETTING
In order to demonstrate the effect of knowledge graph, in experiment, embed our rhetorical graph to Transformer and GPT-2, and compare the results with vanilla Transformer and GPT-2 models.
The only work on rhetorical text generation we are aware of in literature is (Liu et al. 2019) which proposes rhetorically controlled encoder-decoder for modern Chinese poetry generation based on seq2seq framework. They input the rhetorical type into the model explicitly to control the generated rhetorical type. However, without knowledge information, the problem of logic conflict

---

Under review as a conference paper at ICLR 2021
GPT-2 Rhetorical Distribution
Transformer Rhetorical Distribution
700
401
325
165
206
Motaphor
Personilicaton
Analogy
Personificatoo
GFT12
a GFT2
Transforment
Transformer
(a) GPT-2
(b) Transformer
Figure 3: Rhetorical type distribution. The analogy doesn't 't belong to the rhetorical types studied in this paper, it is manually marked in generated sentences.
is still exist. On the other hand, they generated a poem line by line. That is, when generating i-line, they need to input previous i-1 lines and the topic keywords. However, in our framework, focus on text generation with topic keyword. Due to the differences of generation approach, we list some experimental results of (Liu et al. 2019) for reference only.
We also carry out an extra experiment, which trains the generation model using template generated sentences Guan et al. 2020)) First of all, we transform the rhetorical triples into template sentences. Then we trained the generation model on templates sentences, instead of real ones. The generated sentences are shown Table It can seen that the generated texts are quite homogeneous.
Table 3: Examples of template-based text generation. Examples of template-basee transformation of triples in rhetoric knowledge graph. Phrases in bold represent the transformed rules. Triple
Transformed Sentence
Generative Sentence
1 HI BTREE
The paddy is like golden waves.
The paddy like ocean with yellow waves.
(paddy.ocean.golden waves)
I. (street lamp.guard,stand)
The street lamp like a standing guard.
The street sign is like stand- ing guard.
5.3 AUTOMATIC EVALUATION
Evaluation Metrics We adopt the perplexity score(PPL) and Rhetoric-Fl score to evaluate the generation performance. Perplexity score indicates the fluency of generated sentences. The Rhetoric-F1 score introduced (Liu et al.] 2019), which can be used to measure the rhetorically controlled accuracy of the generated sentences.
Evaluation Results Our GPT-2+KG obtains lower perplexity, which suggests that the model is on par with other models in generating grammatical sentences. When combined with rhetorical knowledge graph, the Rhetoric-Fl of both GPT-2 and Transformer achieve the level of explicit method AC model. It proves that our framework can effectively control the generated rhetorical type implicitly. Detailed results are given Table
5.4 MANUAL EVALUATION
Evaluation Metrics evaluation:
Following previous work Yi al. 2018). we consider four criteria for human
Correlation(C): Whether the generated sentence is related to specific keywords;
Fluency(F): Whether the generated sentence is fluent;

---

Under review as a conference paper at ICLR 2021
Table 4: Generated examples of different models.
Triple
Model
Examples X l A ,
(The sea, like roaring lion, comes howling toward to
X (Sea, lion, roar)
GPT-2+KG
the shore.)
X 18 - - 2
Transformer+KG
(The sea cries out like a roaring lion.)
オ (Sea, angry)
GPT-2+KG
(Then realized that the sea angry.)
Transforme+*KG
(The sea is roaring.
-K, 2
X
GPT-2+KG
(One day, my parents and went to watch the wave.)
(Sea)
A K में E #
Transformer+KG
(Everyone calms down, looking the natural ocean.)
Pl:Sc diancer
R : 1
# :
39:38 (peart)
20:0 Cgyrate)
(Winter snow, like a dancer in silver dresses, dancing in the wind.)
THE
(Flakes of snow look like carved pearls, numerous and beautifull)
ALMA twinnddn
JBB (white)
*
7
(dand-
(Winter snow white dandelion, covers the whole earth with thick quilts.)
Figure 4: Generated metaphorical sentences of snow by GPT-2+KG.
Semantic ComprehensibilityS)) Whether the generated sentence is comprehensibility in semanties;
Artistic Aesthetics(A): Whether the generated sentence has artistic beauty;
Each criterion is scored ranging from to 5. We have 180 groups of keywords, each group generates different sentences. We calculate scores by people voting. That is, removing the highest score and the lowest score, taking the average score of the rest as the final result.
Evaluation Results Table 2 shows the results of the human evaluation. In baseline models, it's obviously that GPT-2 performs better than Transformer in all four scores. When combined with rhetorical knowledge graph, the fluency of generated sentences are as good as baseline models, even better for Transformer. On the other hand, the correlation score has improved both mod- els combined with KG, demonstrating that the knowledge information can be well learned in our framework.
Farther more, when combined with KG, both the semantic comprchensibility and artistic aesthetics of GPT-2 are improved, while the scores of Transformer are slightly reduced. The reason is that the generated sentences of Transformer are always short and simple, too much keywords will limit expression.
In addition, the generated rhetorical type is also manually marked to help analyze the rhetorical distribution generated sentences. can be seen that combined with rhetorical knowledge graph, the number of rhetorical sentences increases obviously. Details can be found in Figure

---

Under review as a conference paper at ICLR 2021
We also find attractive phenomenon, that 46 analogy sentences generated by GPT-2+KG, which can demonstrate the association of novel internal logic in our graph database. This will help us focus rhetorical graph inference in the future.
5.5
CASE STUDY
In order to further demonstrate how our framework combined with knowledge graph, we display the generated examples in T It can be seen that our framework learn knowledge information well, control the rhetorical type effectively at the same time. An additional case is shown in Figure 4 illustrating that our framework can take the advantage of knowledge graph, generating diverse texts.
CONCLUSIONS AND FUTURE WORK
In this paper, we propose a innovative generation framework which can combine knowledge graph with text generation model effectively. In addition, construct the first Chinese rhetoric graph and devise a graph storage mechanism to resolve the logic conflict problem during query. Experiments show that our method can control the rhetorical types in generated texts, and the are more fluent and reasonable at the same time.
Extra experiments show that training with real corpus sentences rather than those generated from templates, generation model can learn the diversity in sentence. However, how to obtain both tuples and associated real corpus, is restriced condition of our framework.
In future work, it would be very interesting to investigate additional kinds of rhetoric, such as par- allelism, to further expand the rhetorical knowledge graph. Meanwhile, we expect to enhance the knowledge inference ability of the knowledge graph by increasing the attributes of nodes.
REFERENCES
Christoph Alt, Mare Hübner, and Leonhard Hennig. Fine-tuning pre-trained transformer lan- guage models distantly supervised relation extraction. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 1388-1398, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18853//1/119-1133 URL https:
www. www.waclweb.oo www.acllebb.org/atttoooooo wttp://///.attooooo///11-----111 1134
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher Manning. A large an- notated corpus for learning natural language inference. In Proceedings of the 2015 Confer- ence on Empirical Methods in Natural Language Processing, pp. 632 642,, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.18653/v1/D15-100 URL https: www. ac www.wwlwww.bb.... orgg anthol /ata//nnnooooyoaoaaae ogy. 1075 075
Thiago Castro Ferreira, Chris van der Lee, Emiel van Miltenburg, and Emiel Krahmer. Neural data-to-text generation: A comparison between pipeline and end-to-end architectures, 2019. 2019
Jian Guan, Fei Huang, Zhihao Zhao, Xiaoyan Zhu, and Minlie Huang. A knowledge-enhanced pretraining model for commonsense story generation, 2020.
Zhiheng Huang, Wei Xu, and Kai Yu. Bidirectional Istm-erf models for sequence tagging. arXiv preprint arXiv:1508, 1080:110,..0991,,, 2015.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 260-270, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10.18653/v 10118653/1/N11--1 1030, URL httss:/: http://wwwwww.....e../../.o wwwweb. erg azzg/anthooog// N16 1030
Analogy belongs to rhetorical method, which uses rich imagination write things person and vice It extremely similar the metaphor, but they different. Personification is kind of analogy.
8

---

Under review as a conference paper at ICLR 2021
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A preesity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 110-119, San Diego, California, June 2016. Association for Computational Linguistics. doi: (0118653/1//116-00144 016-1011
wwwp///www.../ acl www..wew.b... wwtp.rroo/aaoloog/g//
Zhiqiang Liu, Zuohui Fu, Jie Cao, Gerard de Melo, Yik-Cheung Tam, Cheng Niu, and Jie Zhou. Rhetorically controlled encoder-decoder for modern Chinese poetry generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1992-2001. Flo- rence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 111865/111111-1111 URR.tttpe /tt///wwwww /www//wwooooooomoo.o www.ww.clll.................... a wwgg httgg//athllogy/yP.o 9-11
Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 55 60, Baltimore, Maryland, June 2014. Association for Computational Linguistics. doi: 1..1115/1//14-50100 URL wtt.t::tt///wett wwwwwb.www.......mo wet....rr/aatttoooo///ll--------
ac
Todor Mihaylov and Anette Frank. Knowledgeable reader: Enhancing cloze-style reading compre- hension with external commonsense knowledge. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 521-832, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1IPP 1076.
URL https::///// wwwwwwwwoommmmmm www.waclweb... htt.::/:www......a..o////aaaaaa httpa//////ooooooo///-------1
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Con- I
ference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume (Long Pupers), pp. 2227-2237, New Orleans, Louisiana, June 2018 Association for Computational Linguistics. doi: 10.18653/v 11186533/1/N18-.
URL
https: www. wwwwwwwww................www...w org anthology N100/N18-11 N220
Malay Pramanick, Ashim Gupta, and Pabitra Mitra. An LSTM-CRF based approach to token-level metaphor detection. In Proceedings of the Workshop on Figurative Language Processing, pp. 67-75, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/W 11-0800 URI. httpt: www. , acl web. www...rgg.athhoooooo/o ogy/W1 W1 8-000
Alec Radford. Improving language understanding by generative pre-training. 2018.
Sunny Rai, Shampa Chakraverty, and Devendra Tayal. Supervised metaphor detection using condi- tional random fields. pp. 18-27, 01 2016. doi: 10.11653/v1/W166 W16-1 1103.
Jianlin Su. Lightweight information extraction model based on dgenn and probability graph. 2019.
Mei Tu, Yu Zhou, and Chengqing Zong. A novel translation framework based on rhetorical struc- ture theory. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 374-374, Sofia, Bulgaria, August 2013. Association for Computational Linguistics. URL www. aclweb. org/ httogy//P1 2066
Yiren Wang, Hongzhao Huang, Zhe Liu, Yutong Pang. Yongqiang Wang, ChengXiang Zhai, and Fuchun Peng. Improving n-gram language models with pre-trained deep transformer, 2019.
Zhe Wang, Wei He, Hua Wu, Haiyang Wu, Wei Li, Haifeng Wang, and Enhong Chen. Chinese poetry generation with planning based neural network, 2016.
Xiaoyuan Yi, Maosong Sun, Ruoyu Li, and Zonghan Yang- Chinese poetry generation with working memory model. In Proceedings of the Twenty-Seventh International Joint Confer- ence on Artificial Intelligence, IJCAI-18, pp. 4553 International Joint Conferences on doi: 10.24963/ijcai.201 201188//3..211//3333
Artificial Intelligence Organization, 7 2018.
URL https:
7/doi. org. 10. 0etgg/00.2499.... 2492249933922 /tt//jiaaai.. 2018/ 0000088833333
Zhengyan Zhang. Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. Emnie: Enhanced language representation with informative entities. Meeting of the Association for Computational Linguistics, 2019.
9

---

Under review as a conference paper at ICLR 2021
Hao Zhou, Tom Young, Minlie Huang. Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. Common-
sense knowledge aware conversation generation with graph attention. In Proceedings of the
Twenty-Seventh International Joint Conference Artificial Intelligence, ICCAI-18. pp. 4623-
4629. International Joint Conferences on Artificial Intelligence Organization, 7 2018. doi: 10.24963/ijcai 20t.ai..21//6660 8/643. URL
2018/4444
10


Reviews:
[{"review_id": "IwZb-tyRyKo", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use a rhetoric knowledge graph for rhetorical text generation. One of its key contributions is to construct a rhetoric knowledge graph by leveraging SOTA NER and relation classification models.  To generate a rhetorical text, the new method starts with sending a keyword to the knowledge graph to retrieve the neighborhood of the keywords as its context words. Both the context words and the original query word are fed into a language model to generate the final word sequence. \n\nPros:\n+ The idea of using a knowledge graph to generate rhetorical text is interesting.\n\n+ There is both qualitative and quantitative analysis of the model performance.\n\nCons:\n\n- Overall, the paper is poorly written. A significant amount of details are missing such that I do not believe this work is reproducible.\n\n- It is unclear if any datasets are labeled to train NER and relation classification models for knowledge graph construction. What kind of entity types and relations are considered? Which model is actually applied for rhetoric graph construction?\n\n- This work is not compared with the existing SOTA models for poetry generation. It would be more convincing if this work is compared with (Liu et al., 2019).\n\nQuestions:\n\n1.  Which transformer model is used as the language model?\n\n2. Which top-K generation method is used for text generation?\n\n3. How do the authors show that the proposed model can eliminate logical inconsistency?\n\n4. Which templates are used for the results in Table 4?\n\n5. What do authors make sure the evaluation of artistic aesthetics is consistent and meaningful?", "extra_scores": {}, "extra_text": {}}, {"review_id": "HreLAgXYAc", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a framework that embeds rhetoric knowledge into rhetoric text generation to bring diversity in the output sentences, take both knowledge tuples and their associated sentences as model inputs to emphasize logic relations. Besides, this paper builds the first Chinese rhetoric graph with an innovative mechanism of relationship storage.\n\nPros:\n1.\tAttempting to address logicality and diversity issues in rhetoric text generation, the author proposes an architecture that incorporates logical knowledge information of different rhetoric entities through a knowledge graph and uses a pre-trained language model as the text encoder.\n\n2.\tThis paper builds the first Chinese rhetoric (metaphor and personification) graph with 35228 tuples and designed a graph storage method to avoid certain illogical problems.\n\nCons:\n1. The illustration of the framework is ambiguous and confusing.  1.1 How the context vector is calculated by quarrying the keyword in the knowledge graph?  1.2 Will the keywords also act as a part of the input to Bert?\n\n2. AC model cannot serve as the only one baseline method, as the AC model does not use a knowledge graph. It’s unfair to compare two methods with different inputs. Experiments should also include comparisons with other knowledge-enhanced generation methods.\n\n3. The authors claim the strategy yields more diverse syntactic information but not verify it. The paper lacks detailed comparisons on the diversity between the proposed methods and the template-based methods. (For example, a straight-forward template-based method that uses multiple templates can also prompt diversity). \n\n4. The paper hardly presents any technical novelty. The framework is a simple combination of existing methods (pre-trained language models like Bert and GPT2), and the construction of the knowledge graph is not so novel.\n\n5. As this paper proposed a new dataset, authors should make it possible to be followed and reproduced. It’s necessary to release the dataset or elaborate all details about the data source and dataset construction.\n\nQuestions:\nIn the detail of constructing KETG, if the bert_vector is the representation of keyword and sentence, why using [mask] to separate them two instead of using [sep], as [mask] is used to represent the unseen token and [sep] is designed to separate two sentences in the original BERT architecture?  \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "9u8w8Uzy57", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "summary:\nThis paper proposes a framework, KETG, to introduce knowledge during language generation, aiming to enhance the logicality and diversity of generated texts. It is able to utilize both the source sentence and its associated knowledge graph tuples for training.\nThe authors validate their methods on rhetorical text generation. They first construct a rhetoric graph, then extract context vectors with rhetorical information from it to guide text generation.\n\npros:\n1.\tRhetoric is a stylistic aspect of literature. It is interesting to consider this attribute in natural language generation.\n\ncons:\n1.\tThe writing part of this paper has severe drawbacks. \n     a.\tThe contents are not well-organized at all, the workflow of KETG is still not clear after reading the paper. The overall structure should be modified to present the main ideas clearly. \n     b.\tSome technical details are missing. In Sec.5, Transformer model and GPT-2 are used as the backbone models and the constructed knowledge graph is embedded into them. But there are not enough details for readers to reproduce the experimental results, e.g. number of epochs and other hyperparameters used during training.\n     c.\tThere are too many typos. Take paragraph 2 in Sec.3 for example.\n     line 3: input them together with associated sentence to the ---> input them together with the associated sentence to the;  \n     line 5: forcing the model pay more attention to them ---> forcing the model to pay more attention to them;\n     line6: which then serve as input to the trained model ---> which then serves as input to the trained model \n\n2. This paper aims to introduce knowledge to guide text generation and improve its logicality, but it only evaluates the proposed framework on rhetorical text generation. Actually, there are more representative text generation tasks like dialogue generation, which is also discussed in the related work. More experiments should be included in this work.\n\nQuestion:\nCould you give more details about “knowledge integration” in Fig.1?\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "1XlQLytH7ne", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is very poorly written. This is in part from English as a second language I suspect, but even assuming some mentorship on the writing, there are too many flaws to consider accepting this paper. The modelling work is so poorly described I can't really comment on it. The majority of my review will just attempt to point out some areas I found lacked clarity.\n\nSome examples of lack of clarity and poor writing:\n\nIntroduction: \n* \"commonsense knowledge graph\"; not clear what \"commonsense means here.\n \nRelated work:\n* \"fine tuning models are raised\". I can only guess at what was meant\n\n Knowledge enhanced LM section: \n* there are several other works looking at incorporating knowledge graph (KG) information into text generation processes, including using popular LM as part of it. What you really mean with this work is something like uncontrolled/unconstrained story generation, although it is never properly defined what your are looking at.\n \nRhetorical text generation section: \n*The negative example \"Flakes of snow are flying like snow\" -- it isn't certain that such an error is due to not incorporating KG information into the NLG process. This is just a possible hypothesis.\n\nOur KETG framework section:\n* \"innovative\". This is non-scientific. Don't use adjectives to describe your own work.\n* \"just like the external device to computer\". Again, can only really guess at what is meant.\n* This whole section is lacking any form of clarity. There are no equations grounding what is precisely done. It isn't even made clear what is assumed by your proposed framework. Where the keywords come from a sentence is not clear. I think this is likely a strong requirement of the approach, but again am guessing given details are lacking. The exact relation to the Guan et al. 2020 paper, which appears to be important, isn't properly described.\n* Figure 1: This is very unhelpful. There are so many ways that this figure could be instantiated as an actual model, and no detail is given for how you do so.\n* Mentioning [cls] and [mask] without any definition. It can be guessed what you are talking about by readers familiar with BERT and some recent NLP standards, but again the description lacks greatly in clarity.\n Section 4, also called \"Rhetorical Text Generation\":\n* I'm not sure what it means to say a poem is a metaphor. A poem can contain several linguistic traits. How is it determined that a poem is a metaphor?\n\nConstructing Rhetoric Graph section:\n* where did the 3422 negative examples come from?\n* who manually marked the rhetorical sentences to check accuracy?\n* the accuracies of the classifiers are not very good, and yet they are used to bootstrap the rest of the dataset. I assume the data contains a large amount of errors as a result.\n* not convinced that \"noumenon\" is the correct word, but I could be wrong.\n\nGenerating with Rhetoric Graph section:\n* \"sentence to text generation model\" ... again, I'm just left guessing at what was actually precisely done.\n* \"the Top-K\", no citation and top-k sampling isn't accurately described.\n\nRemaining sections:\nIn full disclosure I have only skim read the remaining sections. I was immediately confused by not seeing any results against the proposed KETG name.\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the paper studies an interesting and important problem, namely the language generation, it is poorly written, which makes it difficult to judge its value. The reviewers also expressed concern over the scope of the evaluation and the lack of comparison to SOTA.
Decision: Reject