Towards Informative Open-ended Text Generation with Dynamic Knowledge Triples
Authors: Zixuan Ren
Yang Zhao
Chengqing Zong
Venue: EMNLP 2023 Findings (2023)
Keywords: Informativeness
Knowledge graph
open-ended text generation

Abstract: Pretrained language models (PLMs), especially large language models (LLMs) demonstrate impressive capabilities in open-ended text generation. While our statistical results show that LLMs often suffer from over-concentrated information, where the generated texts overly focus on the given prompt and fail to provide sufficient background and detailed information as humans do. To address this issue, we propose a dynamic knowledge-guided informative open-ended text generation approach, that utilizes a knowledge graph to help the model generate more contextually related entities and detailed facts. Specifically, we first employ a local knowledge filter to extract relevant knowledge from the comprehensive knowledge graph for a given topic sentence. Then we introduce a dynamic knowledge selector to predict the entity to be mentioned in the subsequent sentence. Finally, we utilize a knowledge-enhanced text generator to produce a more informative output. To evaluate the effectiveness of our approach, we evaluate the proposed approach in two scenarios: fine-tuning for small PLMs and prompt tuning for LLMs. Experimental results show that our approach could generate more informative texts than baselines.

Towards Informative Open-ended Text Generation with Dynamic Knowledge Triples
Zixuan Ren'. Yang 22 Chengqing Zongl2.
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, CAS, Beijing, China
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China { renzixuun2202 @, zhaoyang2015@ cqzong Jia.ac.en
Abstract
Input:
Elon Musk sets low expectations before first Spacex launch of Starship, most powerful rocket ever built
Pretrained language models (PLMs), especially large language models (LLMs) demonstrate im- pressive capabilities in open-ended text gener- ation. While statistical results show that LLMs often suffer from over-concentrated in- formation, where generated texts overly fo- the given prompt and fail provide suf- ficient background and detailed information as humans do. To address this issue, we propose dynamic knowledge-guiiee ended text generation approach, that utilizes knowledge graph help the model gener- ate more contextually related entities and de- tailed facts. Specifically, we first employ a local knowledge filter to extract relevant knowledge from the comprehensive knowledge graph for a given topic sentence. Then we introduce dynamic knowledge selector to predict the tity to be mentioned in the subsequent sentence. Finally, we utilize knowledge enhanced text generator to produce a more informative output. To evaluate the effectiveness of our approach, we evaluate the proposed approach in two sce- narios: fine-tuning for small PLMs and prompt tuning for I.MMs. Experimental results show that approach could generate more informa- tive texts than baselines.
S
GPT-3.5-turbon
Elon Musk SpaceX + Starship Mars
Musk Starship SpaceX + Musk + SpaceX
SoxttDavinci-003:
SpaceX > Elon Musk > Starship Musk Musk >> Starship > Mars Musk Starship > Iexas Starship >>Spacek
Musk + Musk + Starship
Human:
informative open-
Elon Musk + SpaceX SpaceX Starchip + Texas
SpaceK >> Elon Musk SpaceX SpaceK Texas South
Padre Island + SpaceX + South Padre Island + SpaceX > Mars
Starship
SpaceX Starship
Moon + Artemis program + Earth +Starrhhpp +SSppe Heavy Booser
Booster Super Heavy Booster > Starship >> Starship >> Earth
SpaceX Texas Spacex + Spacex Raptor Starship
Super Heavy Booster >> SpaceX >> EAA >> SpaceX > Texas >
FAA + Spacex
Figure 1: Entity flow in LLM generated texts and human written texts. The entities not mentioned in input are highlighted with red and underline. The full texts are shown in Appendix A
eaooleege-enhanced
ChatGPT (OpenAI, 2022), many of these afore- mentioned challenges have been substantially miti- gated(Clark et al., 2021; Xie al., 2023).
PLMs, particularly LLMs, have made signifi- cant advancements in open-ended text generation. However, our analysis reveals that current LLMs still face the challenge of over-coneentraaed infor- mation, wherein they prioritize events and entities mentioned in the prompt and fail to provide rele- vant entities and sufficient information humans would. We define over-concentrated information as follows:
1 Introduction
Open-ended text generation is complex under- taking task, encompassing the need for fluent sen- tences, a natural and coherent flow, as well as in- formative and non-trivial content (Yao et al., 2019). recent years, much of the research has concen- trated on addressing grammatical correctness con- cerns such as inter-sentence coherence and ensur- ing relevance between the input prompt and the generated story content (Yao et al., 2019; Guan et al., 2020, 2021; Tang et al., 2022). However, thanks to the advancements in large language mod- cls (LLMs) like GPT-3 (Brown et al., 2020) and
Over-concentratedr Information:
Texts that predominantly concentrate on the events and entities mentioned in the prompt often face challenges in encompassing relevant entities' comprehensive information.
This discrepancy is evident in Figure 1, where we compare texts generated by human writers
"Corresponding author.

---

Name
Informativeness
# #entiyy
and coherent output.
avg rank
Furthermore, we construct a Chinese news gen- cration dataset and annotate it, along with two ex- isting English datasets, using named entity recog- nition and entity-linking techniques. To evaluate the effectiveness of InfoGen, we conduct experi- ments in two scenarios: fine-tuning for small PLMs and prompt learning for LLMs. The experimental results demonstrate that InfoGen consistently out- performs baseline models in terms of text informa- tiveness. Our contributions are summarized as the following points:
human
1.60
61.75
GPT-3.5-davind
2.75
16.60
GPT-3.5-ttrbo
3.25
14.05
LLAMA-134 13B
4.00
40.95
GPT-3.5-curgs
4.40
10.80
GPT-3-davinci
4.60
39.45
Table 1: The human evaluation results on informative- comparison. Informativeness avg rank means the average rank of the model. # entity means the average number entities in the text.
with those generated by powerful LLMs such as and Text-Davinci-000 using the headline "Elon Musk sets low expectations before first SpaceX launch of Starship, most powerful rocket ever built". Human- written text not only includes information directly related to the headline but also provides additional contextual details, such as the collaboration be- tween NASA, FAA, and SpaceX, as well as spe- cific information about the rocket and the launch location. On the other hand, texts generated by GPT-3. t-turbo and Text-Davinci-003 tend to be focused solely on the provided information and lack the depth of additional relevant details beyond the headline.
We raise the informativeness problem of open- ended text generation and propose genera- tion model that effectively utilizes knowledge for generating informative open-ended texts.
We contribute a Chinese news generation dataset ChinaNews that includes entity an- notations, along with the annotation of two existing English datasets.
Methodology
2.1 Framework
In this section, we introduce the framework of In- foGen. As is shown Figure 2, InfoGen primarily consists of three modules.
In addition to analyzing specific cases, we also conducted human evaluation to compare the infor- mativeness of texts written by humans with those generated by LLMs. We select diverse range of PLMs and ask annotators to rank the texts based the informativeness. The results listed in Ta- ble The results demonstrate that human- -imtnnwwrttten texts outperform all the tested large language mod- els (LLMs) in terms of informativeness.
Local knowledge filter, model to filter rel- evant knowledge from the knowledge graph based on the input information.
Dynamic entity selector, model filter the entities that need to be mentioned in the next step based on the headline and generated text.
To address the issue of over-concentrated infor- mation, we propose dynamic wwwwedge-guided informative open-ended text generation approach, InfoGen, in which a knowledge graph is utilized to help the model generate more related entities and detailed facts. InfoGen consists of several key components. First, we employ local knowledge filter to extract relevant information from a compre- hensive knowledge graph. Second, we introduce dynamic knowledge selector, which predicts the entity to be mentioned in the subsequent sentence. Finally, we utilize knowledge-enhaneed text gen- erator that leverages the filtered local knowledge and selected entities to produce more informative
Knowledge fused text generator, model to generate text iteratively based the headline, knowledge graph, previously generated text, and the entities that need be mentioned.
Local Knowledge Filter with Contrastive Learn- ing We employ bi-encoder architecture consist- ing of two encoders. EncX and EncT, to separately encode the headline and knowledge triples. The en- coding representation is obtained from the hidden states of the [CLS] token. Formally, the represen- tation of the input headline x and triple can be represented as:
'Specific details are listed in Appendix B
Hx = EncX (ncX((ss(XX) H; = [ncT((Ls((t) (1)

---

Yezi Just few months after NASA introduced the world to the most....
Text Encoder
KG Encoder
entE
Dynamic Entity Selector
Elon
Musk
low first
Elon Musk Employer SpaceX
Encoder
Decoder
expectations before
15pareX launch Starship
LKG
Local Knowledge Filter
Starship Follow Falcon 9
entr Starship, Falcon
Y
Twitter
South Afric
Knowledge Fused Text Generator
Encoder
Decoder
SpaceX
Starship Owned by
LKG
Google
Star-
Star-
Starship is poised to surpass predecessor, the Falcon
base
Falcon
Figure As InfoGen is iterative generative model, the picture describes the procedure of generating the i-th step. In this picture, X denotes input, Y<i denotes generated texts and Y; denotes the target text. GKG denotes global knowledge graph and LKG denotes local knowledge filtered by Local Knowledge Filter. In GKG, entities mentioned headline are highlighted. ent; denotes entities selected from local knowledge graph. We use show the specific structure of the models.
To measure the similarity between the headline and triple, we use the dot product as the similarity function:
procedure is as follows:
P(eileeia)) = softmax W HDe + b) (4)
sim(XXt) = H
(2)
Where HDee Dec(Hea, Enc(X, Yzá, 7)) is the hidden states from decoder. Ci2 and Cics denote the t-th token and all previous tokens the entity for the i-th sentence. X and Yz; denote the input and generated texts. Het represents the hidden states of previous tokens. Dec and Enc refer to the decoder and encoder of the model. W and b are learnable parameters.
To filter the relevant knowledge triples, we em- ploy contrastive learning. Given a headline, we consider the knowledge triples contained in the news article as positive knowledge samples, while other triples that include entities present in the headline the news article are treated as neg- ative samples.Let the positive knowledge triples be P edge triples be N {ni..... }- Inspired by Karpukhin et al. (2020), we define our contrastive learning loss as follows:
and the negative knowl-
To leverage the global knowledge, we adopt modified encoder-decoder architecture called Fusion-in-Decoder (Izacard and Grave, 2021). This architecture encodes the headline and each triple separately and then concatenates the encoded rep- resentations for the decoder. Formally, T {t1, f2,.... tx denotes the local knowledge triples. The Fusion in Decoder can be defined as:
L(X, P,N)
sim X, p:)
log sim X, p.)
En sim X, n2)
(3)
Enc X, Yes, T) concat( (H, Hxy)
Dynamic Entity Selector The dynamic triple se- lector is designed to prediet the entities that should be mentioned in the next step, given the head- line and global knowledge. Inspired by generative named entity recognition approaches (Fries al., 2017: Yan et al., 2020), we employ a generative approach for entity prediction. The reto-regeessive
Ht. Ene(t;) Hxy Enc(X, Yes)
(5)
During the inference stage, ensure the gener- ation of meaningful and accurate entities, we con- struct a trie-tree that contains the entities present in the global knowledge. This trie-tree serves as

---

constraint on the generation process, helping to avoid the generation of nonsensical or incorrect entities.
not finetuned incorporate with plan and sentence level plan could make the LLMs pay too much tention to the entities. this regard, we utilize the following prompt to drive the generation process: "Now you are a news writer. Please generate news article paragraph by paragraph based on the given headline. Before each paragraph generation, you may be given several entities, your generation needs to concern the enfity.
Knowledge Fused Text Generator. Building upon previous plan-based story-telling approaches (Yao et al., 2019; Hu et al., 2022), we adopt dy- namic approach that combines entity prediction and text generation. Specifically, Y<i and e<ci denote the generated texts and their plan respec- tively and e; denotes the plan for the target text. The context C₁ is represent as" él <i
Yi
>... < ., in which "
" represents a special separate token. The iterative procedure is defined as follows:
Subsequently. we provide the headline along with the entities predicted by the dynamic entity selector. Following the generation process, we feed the generated texts back to the dynamic entity selec- tor. If the dynamic entity selector produces a null output, we prompt the model to continue without any specific entity guidance.
C,, softmax(Wi/ HDe + b) (6)
The inference algorithm of $2.2 and $2.3 are demonstrated in algorithm 1.
Here Dec = (ec Enc(X, is the decoder hidden states and the encoder is the same as in equation 5.
Algorithm 1: Algorithm of inference stage
of InfoGen
2.2 Fine-tuning Small Language Model
Input: Input X; Knowledge graph triples
C t2, tta Knowledge number
Having trained the Local Knowledge Filter and Dynamic Entity Selector, we proceed fine-tune
Output: Generated text
Y +
small PLMs as the Knowledge Fused Text Genera- tor. Considering the strong local modeling capabil-
// Filter local knowledge LK
1 scores =
3 foreach i in G do
ities but potential lack of long-range coherence, we adopt a sentence-be sentence-by-sentene be-senee approach for train- ing and text generation. Let us suppose that the target texts Y consist of sentences Yi.... Y, with lengths m1, My, and the input comprises X and selected knowledge T. Building upon the generation procedure defined in Equation and the context definition of C₁ our training objective can be expressed as follows:
4
simillrtttim
s end
<- zirted scores))
LK + a|: k]
/1 Iterative text generation
8 while True do
e e EEntity_eleector X, LK, text + T'ert [lae-=leeeeeeerttrrr(( generator( X, e, I.K, Y)
if text & null then (extend(textt)
end
else
break end
m
arg max
log X,T, (7)
end
Here, 0 represents the parameters of the small PLMS, and we aim to maximize the log probability of generating each token Y, conditioned on the context C, input X, selected knowledge T, and the model's parameters.
3 Datasets Construction
We conduct evaluations of InfoGen on three news generation datasets: PENS (Ao et al., 2021), which comprises news articles collected from Microsoft News; CNN (Hermann et al., 2015), which includes news sourced from CNN websites; and ChinaNews, dataset we construct, consisting of global news collected from Chinese official news agencies.
2.3 Prompt Learning
We employ InfoGen complement LLMs. Recog- nizing that LLMs themselves be considered as vast knowledge bases (Petroni et al., 2019; Teubner al., 2023) and is hard modify the architecture LLMs, we integrate InfoGen as a text generator guided by the dynamic entity selector. LLMs are
Chinese Datasets Collection To evaluate the fectiveness of InfoGen on Chinese text generation, we construct a Chinese news generation dataset. To

---

PENS
CNN
ChinaNews
Metrie
Spearman's p
Kendall's
Pearson's
Document
113218
92465
236769
entity
0.398
0.313*
0.3874
Input
10.36
40.03
21.85
Length 0.487*
0.396
0.479*
Output
555.51
601.02
463.64
Event
0.483*
0.381
0.481
Entity 39.99
36.02
21.62
KCCentiyy 0.554
0.440
0.550
Triple 64.53
32.02
43.12
Table 3: Result of correlation tests between automated metric and human annotator ranking. means the p value is less than 1e-6 and ** means the p value less than 1e-10.
Table 2: Statistics of our dataset. # Document denotes the number of examples. # Input and # Output lines display the average number of tokens in input and output texts, respectively. # Entity and # Triple indicate the average number of entity and knowledge triple the output text.
Experiments
4.1 Baselines
ensure the quality and reliability of the news data, we source articles from reputable Chinese official news agencies, including People's Daily, China Daily, and Xinhua News Agency. We focused on global news rather than local or entertainment news. The collection period for the dataset spanned from January 1, 2017, August 1, 2022. After gath- ering the news articles, we performed thorough preprocessing step, which involved filtering out hyperlinks, images, image captions, and any adver- tisements present in the corpus. Additionally, we removed brief news reports that contained less than 100 words, ensuring that the dataset consisted of substantive news articles suitable for our evalua- tion.
For comparison, we consider the following base- lines: fine-tuned GPT-2 (Radford et al., 2019), fine- tuned BART (Lewis et al., 2020), and CK-GPT (Guan et al., 2020). GPT-2 and BART are pre- trained language models commonly employed for natural text generation tasks. CK-GPT, on the other hand, is variant of GPT-2 that incorpo- rates commonsense knowledge by leveraging the ATOMIC(Sap et al., 2019) and Concept-Net(Liu and Singh, 2004) commonsense knowledge graphs
4.2 Implementation Details
In the fine-tuning stage, and 31=0.9, 82=0.999, and c=le-8. We train our model for 15 epochs and use the model with the best performance in the validation set for gener- ation. During inference, we use nucleus sampling with p=0.9, We train our model using RTX-3090 GPUs on Ubuntu 18.04 and use NLP open-source library Transformers (Wolf et al., 2020). In our study, employed a widely adopted pre-trained model based on the Transformer architecture the foundational model(Zhao al., 2023). Specif- ically, we use BERT(Devlin et al., 2019) as the base encoder for the local knowledge filter and BART-base(Lewis et al., 2020) as the base model for the dynamic entity selector. For fine-tuning PLMs, we choose Bart-base as a knowledge-fused text generator and for prompt learning, we choose GPT-3.5-turbe as the base model.
Adam optimizer
Datasets Automatic Annotation To facilitate the utilization of knowledge graphs, employ NLP automatic annotation tools to annotate the news datasets. For the English datasets PENS and CNN, we first filter noisy samples such as empty samples or samples containing too many advertise- ments and hyperlinks. Then we utilized Flair (Ak- bik et al., 2018) for entity recognition and BLINK (Ledell Wu and Zettlemoyer, 2020) to establish en- tity links with the Wikidata knowledge graph 2. In the case of ChinaNews, since Flair and BLINK do not support Chinese, we employed SpaCy (Honni- bal et al., 2020) for entity recognition and linked them to Wikidata using the Wikidata query API For relation extraction, we use distant supervision (Mintz et al., 2009) with Wikidata. The dataset statistics are provided in Table 2. For each dataset, randomly choose 2000 samples as the valida- tion set and 500 samples for the test.
4.3 Evaluation Metrics
Automatic evaluation metric (1) Perplexity (PPL): smaller perplexity means model could gen- erate more fluent texts. (2) Distinet-3 (Dist-3) (Li et al., 2016): the ratio of distinct 3-gram to gener- ated grams, which indicates the diversity of gen- crated texts. (3) ROUGE-L (Lin, 2004): the recall of the longest common sub-sequence between gen-
"We use the Wikidata dumps archived on October 4, 2022 wwip::////uuyyiiiatattt.ok

---

CNN
Models
PPL.
Dist-3t
Rouge-Lf
BERTScore
KC-E
GPT2
6.28
62.71
13.39
82.10
17.52
BART
6.11
63,62
13.64
81.78
14.65
CK-GPT
7.39
62.01
12.00
81.28
13.59
INFOGEN
5.49
63.59
14.53
82.91
22.16
w/o DE
5.81
63.22
14.19
82.73
20.08
w/o LKG
6.01
62.73
13.01
82.58
19.94
PENS
PPL.
Dist-3f
Rouge-Lf
BERTScore"
KC-E1
GPT2
6.83
61.94
10.05
81.83
5.58
BART
6.74
60.79
10.11
81.78
6.03
CK-GPT
7.54
58.61
9.67
79.09
5.52
INPOGEN
6.12
63.16
10.97
82.40
9.25
w/o DE
6.29
63.11
10.68
82.11
8.42
w/o LKG
6.57
62.17
10.49
81.94
8.04
ChinaNews
PPL.
Dist-31
Rouge-Lf
BERTScore"
KC-E1
GPT2
5.36
53.64
17.08
79.17
13.49
BART
5.31
54.16
17.36
79.04
12.84
INFOGEN
4.52
55.84
18.25
80.01
17.52
w/o DE
4.83
55.05
18.02
79.72
16.22
w/o LKG
5.16
54.39
17.81
79.53
15.49
Table 4: Performance comparison of various models for open-ended news generation. Bold and underlined fonts indicate the best and second-best approaches, respectively. The symbol f signifies that higher values are preferable, while signifies that lower values are preferable. w/o DE means ablating dynamic entity generator and w/o LKG means ablating local knowledge for news generation.
erated texts and reference texts. (4) BERTScore (Zhang al.): the similarity of generated texts and reference texts measured by the cosine similarity BERT embedding.
KC_ @C_entity with indicator function II as follows:
: {t E T E Rel((, h)}
(8)
KC KC_entity
Σ Lealti
We calculate the Pearson, Spearman, and Kendall correlation coefficients between human ranks and automated ranks.* 4 We choose three basic metrics for informative measurement: number of words (Length) and number of entities (entity) and num- ber of events (Event). The result is shown Table
Metric for Informativeness Intuitively, the in- formativeness of generated texts can be evaluated using naive automated metrics including text length and the number of mentioned entities. However, due to the data memorization tendencies of LLMs (Carlini al., 2021; Xie al., 2023) and degener- ation of PLMs(Holtzman al., 2020; Dou al., 2022a), the models can sometimes generate texts that have little or no relation to the input, which needs to be taken into consideration. Takiin this into consideration, we propose the Knowledge- constrained entity number (KC_entity) which only considers relevant entities. Formally, let H h1, h2. Amm denote the entities in the headline, and T I,, ...... ... .nn represent the en- tities in the generated text. Consider knowledge graph G, where Rel(x, y) denotes the existence of a relationship between entities z and y in G. We define the knowledge-constraingg entity set and
Human evaluation To give a more comprehen- sive analysis, we conduct human evaluations that (1) Coher-
consider the following three aspects. ence (Coh.): evaluate whether the generated text contains off-prompt, self-contradiction. and redun- dant cases. (2) Informativenes: (Inf.): evaluate the informativeness of generated texts, including back- ground information and details. (3) Overall qual- ity (Ovr.): take both coherence and informative- ness into consideration. The annotators are asked to score 1 (worst) to 5 (best) for the three aspects.
"Here we reuse the human evaluation results in $1, details are listed in Appendix

---

Each questionnaire contains 20 samples and is as- signed to 3 annotators
Coh.
Inf.
Ovr.
GPT-2
3.17
2.78
3.03
BART
3.15
2.82
3.02
INFOGEN
3.27
3.25
3.25
5 Results and Analysis
Table 5: Human evaluation of results of fine-tuning for small PLMs approach and baselines on CNN datasets. Coh., Inf. and Ovr. represents coherence, informative- ness and overall quality respectively.
5.1 Automatic Results of Fine-tuning Approach
We present the results of our automatic evalua- tion Table 4, In the English datasets, KG-GPT performs worse than GPT2, which consistent with previous findings (Wang et al., 2022). It has been noted that the incorporation of commonsense knowledge inference abilities can sometimes have detrimental effect the generation ability. In contrast, InfoGen demonstrates superior perfor- mance compared to the baselines across various metrics, including perplexity, Distinct-3, Rouge-I., and BERTScore. These results indicate that Info- Gen is capable of generating more coherent and fluent texts. The effectiveness of InfoGen can be attributed to the utilization of global knowledge and entities as content plan for text generation. This strategy is beneficial in capturing long-range dependencies in previous studies (Xu et al., 2020; Guan al., 2021). Furthermore, InfoGen outper- forms baselines in terms of the informativeness metric, as reflected by the KC-Entity scores. This improvement highlights the contribution of incor- porating knowledge in generating more informative text.
0.1
APp
0.0
PENS
-0.1
CNN
ChinaNews
20
30
40
50
60
of selected local knowledge triples
Figure Impact of the number of selected knowledge
5.3 Human Evaluation of Fine-tuning Method
We ask human annotators give scores gener- ated texts, from three aspects, including co- herence, text informativeness, and overall quality. The results the human evaluation are presented in Table The Fleiss' kappa is 0,41. Compare to baselines, InfoGen has higher score in the three aspects.
5.2 Ablation Analysis
We conduct ablation analysis of InfoGen, and the results are presented in Table In the table, 'w/o DE' refers to the removal of the dynamic entity module, meaning that the text generator can only access local knowledge. 'w/o LKG indicates the removal of local knowledge during text generation, while still retaining it for the local entity generator due to the trie-tree constrained generation.
5.4 Influence of Number of Local Triples
During the inference process, we employ a selec- tion mechanism to choose the top-k triples based on the scores assigned by the local knowledge filter. To further investigate the impact of the selection parameter (k), we conduct an experiment using the validation dataset to observe the changes in per- plexity (APPL) and number of selected local knowledge. The results are depicted in Figure 3. In the PENS and CNN datasets, we observe that as the number of selected knowledge increases from 20 to 60, the perplexity initially decreases and then starts to rise. However, in the ChinaNews dataset, the perplexity consistently increases the num- ber of selected knowledge grows. Based these findings, we determined the optimal values for k during the inference stage. Specifically, we chose as 20 for ChinaNews and 30 for CNN and PENS.
The results the ablation analysis confirm the effectiveness of both the local knowledge and dy- namic entity components. When removing the lo- cal knowledge, we observe decrease in perfor- mance, demonstrating the importance of incorpo- rating local knowledge in generating informative texts. Additionally, there is slight drop in perfor- mance when the dynamic entity module is removed. This suggests that global knowledge plays a more significant role in the overall performance of Info- Gen.

---

Coh.
Inf.
Ovr.
man al., 2020; Dou et al., 2022b). Existing ap- proaches solve these issues can be divided into two categories based on the generation mechanism. One approach involves generating text in a single pass while improving text coherence by leveraging the hicrarchical structure of human-generated texts and modeling high-level semantic features. This includes considering key words (Rashkin et al., 2020), event sequences (Zhai et al., 2019), and sentence-level representations (Hu al., 2022). Another category decomposes the text generation process into multiple stages. Some approaches gen- crate keywords or an outline of the text first and then generate and refine the complete text (Yao et al., 2019; Hua and Wang, 2020; Tan et al., 2021; Yang et al., 2022). Other techniques employ latent vectors continuous text plans (Shen et al., 2019; Ji and Huang, 2021: Tang al., 2022). The recent success of LLMs has demonstrated that these issues can be effectively mitigated without the need for fine-tuning on specific datasets (Clark et al., 2021; Dou et al., 2022a; Xie et al., 2023). Despite this progress, it is still necessary to analyze and identify areas where LLMs may encounter limitations or fail to perform optimally.
GPu--.5-turbo BPT-3.5-uuboo-Dynamii Entity
4.60
4.00
4.33
4.60
4.75
4.72
Table 6: Human evaluation of texts generated by GPT- 3.5-turbo and Dynamic Entity guided GPT-3.5-turbo.
5.5 Human Evaluation of Prompt Method
The results, displayed in Table and achieving a Fleiss' kappa of 0.71, clearly illustrate that incorpo- rating entity guidance does not sacrifice the coher- ence of the generated texts. Furthermore, it substan- tially enhances the informativeness of the generated content. For a detailed examination, please refer to the case study provided in Appendix D. By uti- lizing entity guidance, GPT-3.5-turbo successfully generates entities that are relevant to the topic, even if they are not explicitly mentioned in the headline. An example of this is the mention of the "Starbase Launch Site", which is one of the launch sites for the Starship spacecraft.
6 Related Work
Knowledge Enhanced Open-ended Text Gener- ation. In recent years, there has been a grow- ing emphasis on integrating knowledge into open- ended text generation, evidenced by the intro- duction of various commonsense text generation datasets such as ROCStory (Liu et al., 2020). Re- searchers have directed their attention toward in- corporating knowledge into the process. For in- stance, Guan et al. (2020) devised a method to con- vert knowledge triples into natural language using predefined templates and subsequently conducted post-training of GPT2 (Radford et al., 2019) on the resulting corpus. Another approach, inspired by the Plug-and-Play Language Model (Dathathri al.), was pursued by Xu al. (2020), who treated knowl- edge triples as control prefixes. Lin et al. (2022) generate future event of story by inferring com- monsense explanation of generated context. How- ever, these methods have primarily been tested on specially-designed datasets that focus on common- sense knowledge inference understanding. Their approaches have not undergone extensive evalua- tion in the context of real-world open-ended gen- eration tasks, such as opinion generation or news generation.
Text Informative Analysis. In tasks involving rich input information such as translation, summa- rization, and data-to-text generation, informative- ness assumes a significant role (Yuan et al.. 2021). In summarization, informativeness serves as a met- ric to assess whether generated texts effectively capture the key information from the input text (Grusky et al., 2018; Zhu et al., 2020; Xiao et al., 2023). In data-to-text generation and ma- chine translation, informativeness, also referred to as adequacy, gauges whether the generated texts convey the same information as the given input text (White, 1995; Lopez, 2008; Mehta et al., 2022). Currently, the evaluation of informativeness relies on human judgment, and the absence of automated metrics quantify informativeness in creative text generation tasks remains a limitation.
Conclusion
In this paper, we address an important and un- resolved issue in knowledge-intensive text gen- eration, namely over-concentrated information. Through manual evaluations of news texts gener- ated by both human writers and models, we discov- ered substantial disparities between the two, em- phasizing the difficulty of generating informative
Long Text Generation. Previous research has highlighted the issue of off-prompt generation and incoherence in long-form text generation (Holtz-

---

content. Drawing from these insights, we propose a dynamic knowledge-guided approach to open- ended text generation called InfoGen. Our method involves filtering relevant knowledge triples and dynamically selecting entities as content plans. We conduct experiments in two scenarios: fine-tuning small PLMs and prompt learning for LLMs. The results demonstrate that our approach effectively enhances the informativeness of the generated texts, thereby addressing the limitations observed pre- vious methods.
Acknowledgement
We thank anonymous reviewers for helpful sugges- tions. The research work described in this paper has been supported by the National Key Research and Development Program of China under Grant No. 20227D01 18802
References
Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In COLING 2018, 27th International Conference Computational Linguisties, pages (638-1649.
Limitations
Xiang Ao, Xiting Wang, Ling Luo, Ying Qiao, Qing He, and Xing Xie. 2021. Pens: A dataset and generic framework for personalized news headline genera- tion. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natu- ral Language Processing (Volume I: Long Papers), pages 82-92.
While our study focuses on knowledge-intensive open-ended text generation specifically in the con- text of news generation, is important to ac- knowledge the limitations of our research. Firstly, knowledge-intensive open-ended text generation encompasses a wide range of application areas be- yond news generation alone. Exploring the effec- tiveness of our proposed approaches in other do- mains would be interesting avenue for future research.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Secondly, the implementation of our proposed metric relics on the utilization of several NLP tools and resources, including Named Entity Recogni- tion (NER), entity linking, and knowledge graphs The performance of our metric is therefore influ- enced by the accuracy and availability of these tools. Further improvements in these areas would enhance the reliability and generalizability of our approach.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom B Brown, Dawn Song, UI- far Erlingsson, et al. 2021. Extracting training data from large language models. In USENIX Security Symposium, volume 6.
Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong. Suchin Gururangan. and Noah A Smith. 2021. All that's 'human' is not gold: Evaluating hu- man evaluation of generated text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7282-7296.
Ethical Considerations
We collected news articles from Chinese official agencies, ensuring that they undergo thorough double-checking process prior to their release to prevent any inclusion libelous, racist, or other- wise inappropriate content. While our proposed approach incorporates knowledge graphs and is trained on real news data, we acknowledge the potential for the system to generate fabricated or in- accurate information due to the inherent systematic biases introduced during model pretraining using web corpora. Therefore, we strongly encourage users to exercise caution and critically evaluate the ethical implications of the generated output when applying the system in real-world scenarios.
Sumanth Dathathri. Andrea Madotto, Janice Lan, Jane Hung. Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach controlled text generation. In In- ternational Conference on Learning Representations.
Jacob Devlin, Ming-We Wei Chang. Kenton Lee, and Kristina Toutanova. 2019, Bert: Pre-training of deep bidirectional transformers for language understand- ing. Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, Volume (Long and Short Papers), pages 4171- 4186.

---

Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorski Noah A Smith, and Yejin Choi. 2022a. Is gpt-3 text indistinguishable from human text? scarecrow: A framework for scrutinizing machine text. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7250-7274.
Linguistics (Volume 1: Long Papers), pages 2288- 2305. Xinyu Hua and Lu Wang. 2020. Pair. Planning and iterative refinement in pre-trained transformers for long text generation. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 781-793.
Yao Dou, Maxwell Forbes, Rik
Noah A. Smith, and Yejin Choi. 2022b. Is GPT- 3 text indistinguishable from human text? scarecrow: framework for scrutinizing machine text. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume I: Long Papers), pages 7250-7274, Dublin, Ireland. Associa- tion for Computational Linguistics.
Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open do- main question answering. In Proceedings of the 16th Conference of the European Chapter of the Associ- ation for Computational Linguistics: Main Volume. pages 874-880.
Haozhe Ji and Minlie Huang. 2021. Discodvt: Gener- ating long text with discourse-aware discrete varia- tional transformer. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages
Jason Fries. Sen Wu, Alex Ratner, and Christopher Ré. 2017. Swellshark: A generative model for biomed- ical named entity recognition without labeled data. arXiv preprint arXiv:1704.06366)
Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long Pu- pers). pages 708-719.
Vladimir Karpukhin, Barlas Oguz. Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.0490s
Martin Josifoski Sebastian Riedel Ledell Wu, Fabio Petroni and Luke Zettlemoyer. 2020. Zero- shot entity linking with dense entity retrieval. In EMNLP.
Jian Guan, Fei Huang. Zhihao Zhao, Xiaoyan Zhu, and Minlie Huang. 2020. A snowledge-enhaneee pre- training model for commonsense story generation. Transactions of the Association for Computational Linguistics, 8:93-108.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising gqquence-to-squeene pre-training for nat- ural language generation. translation, and comprehen- sion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871
Jian Guan, Xiaoxi Mao, Changjie Fan, Zitao Liu, Wen- biao Ding, and Minlie Huang. 2021. Long text gener- ation by modeling sentence-level and discourse-leved coherence. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 1th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 6379-6393.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity- objec- tive function for neural conversation models. In Proc. of NAACL-HLT.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman. and Phil Blunsom. 2015. Teaching machines to read and comprehend. Advances in neural information processing systems, 28.
Chin-Yer Yew Lin. 2004. Rouge:
A package for automatic
evaluation of summaries. In Text summarization branches out, pages 74-81.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text de- generation. In International Conference on Learning Representations.
Li Lin, Yixin Cao, Lifu Huang, Shu' Ang Li, Xuming Hu, Lijie Wen, and Jianmin Wang. 2022. What makes the story forward? inferring commonsense explanations as prompts for future event generation. In Proceedings of the 45th International ACM SI- GIR Conference on Research and Development in Information Retrieval, pages 1098-11011 1109.
Matthew Honnibal, Ines Montani, Sofie Van Lan- deghem, and Adriane Boyd. 2020. spaCy: Industrial- strength Natural Language Processing in Python.
Danyang Liu, Juntao Li. Meng-Hsuar Yu, Ziming Huang, Gongshen Liu, Dongyan Zhao, and Rui Yan. 2020. A character-centric neural model for auto- mated story generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34. pages 1725-1732.
Zhe Hu, Hou Pong Chan, Jiachen Liu, Xinyan Xiao, Hua Wu, and Lifu Huang. 2022. Planet: Dynamic content planning in autoregressive transformers for lorg-form text generation. In Proceedings of the 60th Annual Meeting of the Association for Computational

---

Hugo Liu and Push Singh. 2004, Conceptnet- a practi- cal commonsense reasoning tool-kit. BT technology journal, 22( 1-226.
American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 4313-4324,
Adam Lopez. 2008. Statistical machine translation.
Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. 2022. Context-tuning: Learning contextual- ized prompts for natural language generation. In Proceedings of the 29th International Conference on Computational Linguisties, pages 6340-6354.
ACM Computing Surveys (CSUR), 40(3):1-49.
Sanket Vaibhav Mehta, Jinfeng Rao, Yi Tay, Mihir Kale, Ankur Parikh, and Emmma Strubell. 2022. Improving compositional generalization with self-training for data-to-text generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4205- 4219.
Timm Teubner, Christoph M Flath, Christof Weinhardt, Wil van der Aalst, and Oliver Hinz. 2023. Welcome to the of chatgpt al. the prospects of large language models. Business & Information Systems Engineering, 65(2):95-111
Mike Mintz, Steven Bills, Rion Snow, and Dan Juraf- sky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Con- ference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 1003- 1011.
Yuxin Wang. Jieru Lin, Zhiwei Yu, Wei Hu, and Börje F Karlsson. 2022. Open-world story generation with structured knowledge enhancement: A comprehen- sive survey. arXiv preprint arXiv:2212.0466
John S White. 1995, Approaches to black box mt evalua- tion. In Proceedings of Machine Trunslation Summit V.
OpenAl. 2022. Introducing chatgpt. https://openai. com/blog/ chhatpp...
Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison. Sam Shleifer. Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger. Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models knowl- edge bases? In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCN)) IJCNLPP. pages 2463.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 188):
Min Xiao, Junnan Zhu, Haitao Lin, Yu Zhou, and Chengqing Zong- 2023. Cfsum: A coarse-to-fine contribution network for multimodal summarization. arXiv preprint arXiv:2307.0222
Hamnah Rashkin, Asli Celikyilmaz, Yejin Choi, and Jianfeng Gao. 2020. Plotmachines: Outline- conditioned generation with dynamic plot state track- ing. In Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pages 4274-4295.
Zhuohan Xie, Trevor Cohn, and Jey Han Lau. 2023. Can very large pretrained language models learn storytelling with a few examples? arXiv preprint arXiv:2301.09790
Maarten Sap, Ronan Le Bras, Emily Allaway, Chan- dra Bhagavatula. Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019. Atomic: An atlas of machine commonsense for if- then reasoning. In Proceedings of the AAAI con- ference on artificial intelligence, volume 33, pages 3027-3035.
Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Raul Puri, Pascale Fung, Animashree Anandkumar, and Bryan Catanzaro. 2020. Megatron-cntri:: Control- lable story generation with external knowledge using large-scale language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2831-2845.
Dinghan Shen, Asli Celikyilmaz, Yizhe Zhang, Liqun Chen, Xin Wang, Jianfeng Gao, and Lawrence Carin. 2019. Towards generating long and coherent text with multi-level latent variable models. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2079-2089.
Jinghui Yan, Yining Wang, Lu Xiang, Yu Zhou, and Chengqing Zong. 2020. A nhowledge-divenn gen- erative model for multi cnuti-impliiattion chinese medical procedure entity normalization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1490-1499.
Bowen Tan, Zichao Yang, Maruan Al-Shedivat, Eric Xing, and Zhiting Hu. 2021. Progressive generation of long text with pretrained language models. In Proceedings of the 2021 Conference of the North
Kevin Yang, Nanyun Peng, Yuandong Tian, and Dan Klein, 2022. Re3: Generating longer stories with recursive reprompting and revision. arXiv preprint 1rX.v:2210.06677.

---

Lili Yao, Nanyun Peng, Ralph Weischedel. Kevin Knight, Dongyan Zhao, and Rui Yan. 2019. Plan- and-write: Towards better automatic storytelling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7378-7385.
the maximum token count to 1024 to ensure manageable output length.
Prompt for LLMs. in our case study(Figure 1) and questionnaires (Ta- ble 1). We use "[the news title]" to represent the content of a news title. As different models have different capabilities, we use several prompts.
We provide the prompts used
Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text text gener- ation. Advances Neural Information Procexsing Systems, 11222233222777
Fangzhou Zhai, Vera Demberg, Pavel Shkadzko, Wei Shi, and Asad Sayeed. 2019. A hybrid model for globally coherent story generation. In Proceedings of the Second Workshop on Storytelling, pages 34-45,
GPT-3.5-turbo 'hee system prompt for GPT-3.5- turbo "You are a helpful news writing assistant. You are required to write informative English news based on the given title." " and the user prompt is "The title is [the news title]"
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Wein- berger. and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.
Text-Davinci-003 and GPT3.5-curie: "You are helpful news writing assistant, You are required to write an informative English news article based on the given title. Here is the title: [the news title].
Yang Zhao, Jiajun Zhang, and Chengqing Zong. 2023. Transformer: A general framework from machine translation to others. Machine Intelligence Research, pages 1-25.
LLAMA-13B and GPT-3-davinci: news title]) n News content:"
Junnan Zhu, Yu Zhou, Jiajun Zhang, and Chengqing Zong. 2020. Attend, translate and summarize: An efficient method for neural cross-lingual summariza- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1309-1321.
Questionnaire preparation and collection. The questionnaire comprises news headline and six news articles, comprising one authored by a human and five generated by the aforementioned models, all derived from the given headline. Since objective quantitative metrics for assessing information content are lacking, we instructed the data annotators to rank the texts according to their perceived level of information instead of assigning quantitative scores. We truncate the news text up to 1024 tokens, the same number of maximum token numbers of generated texts.
A Full Contents of Figure!
The full case is shown in Table 7. We normalize the numan--ritted contents maximum 1024 tokens, the same number as maximum tokens for generated texts.
B Details for Informativeness Human Evaluation
Texts for human evaluation. We carefully examined human-written and model-generated texts in the realm of news articles. To prevent any potential bias caused by language models compiled
C Detailed results of Informativeness Human Evaluation
The table 8 con- tains the statistical information of human-written texts and model-generated texts, which includes the number of words, number of entities, and number of events.
Detailed statistic information
memorizing specific information, dataset consisting of 20 news articles published after January 1, 2023, from reputable sources such BBC and CNN.
Model generation setting. We adopted the experimental setup outlined in (Xie et al., 2023), where our models were tasked with generating stories based on provided headlines. To encourage greater creativity in the generated texts, we utilized a temperature of 0.8 during the text generation process. Furthermore, we constrained
Win-rate between human and models. The figure 4 shows the win rate between humans and models as follows. Since there is bie-breaker option in the questionnaire, the win rate is likely be less than 100%.

---

D Case Study
We provide comprehensive case study comparing the outputs of GPT-b.5-turbo with and without the dynamic entity guide in Table 9. By incorporating the dynamic entity guide, the model generates mentions of entities such "Starbase Launch Site." This demonstrates the effectiveness of the entity guidance in influencing the generation process and improving the relevancy and informativeness of the generated text.
"Falcon 9" and

---

Input:Elon Musk sets low expectations before lirst SpaceX luunch of Starship. most powerful rocket ever built
Human:
Just a few months after NASA introduced the world to the most powertul rocket ever flown to orbir, Hlon Musk's s Space is prepared to set off its own creation which could pack oearly twiee the power of anything flown before. SpaceX's vehicle called Starship. is currenily sitting launch pad the company's facilinies
company targating liftoff 8 a.m. CT (9 a.m. ET) on Monday, alihough it has the ability to take ofl anyLime between 8 u.m. CT (9 a.m. ET) and 9:30 a.m. CT (10:30 a.m. ET). "I guess I'd like to just sel expoctations low." SpaceX CBO Blon Musk said during a Twitter "Spaces" event
southem Texas coastline.
subseribers Sunday evening. "If we get far enough away from launch pad before something goes wrong. then I think I would consider that to be a success. Just don't blow up the pad" He alded: "There's a good chance that it gets posiponed since we're 'e going to be pretly careful about this launch." SpaceX has livestrean of the Stunship launch here. Folks on the ground near SpaceX's faciliries South Texas can certainly catch an in-person plimpse. Locals are known line the surmonnding henches in South Padre Island to watch tests, and this launch sure drw spectators, SpaceX has regeatedly warned those in the area, however. stay away from "Keepout Zone" the areas directly surrounding the launch site that have been deemed close the rocket to be safe during lift olf. The "Keepoul Zone" includes the cousiline south of South Padre Island and stretches a fow miles inland. About this mission This will mark SpaceX's first attempe launch fully assembled Starship vehicle. huilding a years-long testing campaign. Musk has talked about Stursiip muking elaborate presentations about its design and purpose- - for hall a decade, and be frequently hurps on its potential for carrying cagoo and humans to Mars. Musk his even said that his sole purpose for founding SpaceX was develop vehicle like Starship could establish buman settlement Mars. Additionally. NASA already awarded SpaceX contracts and options worth several billions of dollars to use Starship ferry gevernment astronauts to the surface of the moon under the space agency's Artemis program. The inaugural light lest will not complete full oebil uround Earth, If successful, however, will travel about into altitids deemed to be outer space. Starship consists of two parts: the Super Hewvy booster, a gargantuan rocket that houses 55 ongines, and the Starship spacecraft, which sits stop the hoodter during launch and is designed misaion. The massive Super Heavy rucket booster will give the first blast of power at lifoll. Less than three minutes after takeolf, it's expected expend its fuel and soparate from the Starship spaceertt, leaving the boaster to be discarded in the occan. The Starship will use its own six engines, blazing more than minutes, to propel itself nearly orbital speeds. The vehicle will then complete a partial lap of the planet. reentering the Earth's is atmosphere neur Hawaii. It's expected to splash down off the coast about an hour and a half after liftol". Getting here Development Sturship
miles above Earli's surface,
break away after the bouster expends its fuel to finish the
been based at SpaceX's privately hold spaceport about 40 minutes outside Brownsville,, Texas, on US-Mexico horder. Tesring hegan years ago with hrief hhp tests" of early suaceeraft prototypes. The company began with hrief flights that lifted a few dosen feet off the ground before evolving high aliitud llggtts most of which resulted in drumatic explusions the company allerupted land them upright. One suborbital light test in May 2021. however, ended in success. Since then, SpaceX has also boen working to get its Super Henvy booster prepared for fight. The massive, 230 foot tall (69 meter tall) cylinder is packed with 33 of the company's i Raptor engines. Fully stacked, Sturship and Super Heavy stand about feet (120 meters) tall. SpaceX been wailing more tham a year to get FAA approval launch altempt The company, and federal regulatoes lisked with certifying SpaceX launches won't I pase risks to people pruperty in the area have faced significant pushibakk from the local community, including
this
surrounding the launch Aviation Administration, which licenses commercial rocket launches, announced Friday. April 14, that it granted the company's is request for uncrewed lighht test of the rocket of the Space lacilities in South Texas. "Afler comprehensive license evaluation process, the FAA determined Space met all snfery, environmental policy. payload, airspace integration and financial responsihilin requirements." said in statement. During a call with reporters last week, an FAA official, who declined to be named for publication, said that the agency has been oversexing SpaceX's compliance with the mitigating actions, some of which are the works, even as the company prepares launch.
environmental groups. But the Fedoral
agency
GPT.a.5-turbo
On April 16th, 2023, Elon Musk, the CEO of SpaceX, set low expectations before the first launch of the Sturshpp the most powerful rocket ever built. Starship, designed carry up to 100 passengers, is intended to transport humans and cargo the moon, Mars, and beyond. Despite the groundbreaking technology and ambitious goals of the project, Mask has warned that the first launch may not be successful According to uncertain. He acknowledged the high risk associated with space
Musk, the lirst launch will be experimental and the chances of rravel emphasized importaace of taking calculated risks achieve prograss. Musk revealed that the company has conducted namerous tests and simulations to minimize the risks, but there are still many unknown variables that could impact the launch. The Starship project has been development several years, SpaceX has invested billions of dollars the program. and cost-effective, which could revelutioniee the space industry. The success of the project would pave the way for future space explocation and colonization. Despite the potential henefits, Musk has maintained realistic approach and acknow space travel. The unnouncement of low expectations may disappoint some supporters, but it shows the company' is comminment to safety and responsible innovation. Space exploration is high-stakes endeavor, and SpaceX's cautious appreach retlects their dedication to responsible space travel.
rocket designed to reusable
challenges assoelated with
Text- Text-davinci-00:: i0nii-0::::: On April 16 2023, SpaceX CEO Elon Musk set low expectations for the first launch of the compuny's 's Sturship, the most powerful rocket over built. The project has been a work--in-rogress for years, and Musk cognizant of the fact that many of the rocketts is more show modesty,
components are untested. expectations appropriately." Musk said. "I wen't be surprised if it blows on the launchpud, or it's an spic fail. Musk was equally vocal about the potenrial of the meket, staring that if successful, the Starship could send people to Mars in five years." said Musk Starship is designed to be a fully reusable rocket, miking it mach more cost effective than current models. It is powered by three Raptor engines. capable of lifting payload of more than 100 tons into arbit. launch scheduled for carly June 2023 and will take place from the company's s imucc site in Boca Chica. Texas. will the first launch of the Starship, and the first time anyone has atrempted to launch such a pomerful racker The launch will he clocely watched by the accospace community, as well as hy the general public,
suggested
the success
launch was far from certain.
want set
he used bring hamans to Mars. "We" Illl
he able
may be the first step lowards a new eru of space exploration. It is also testament to ingenuity of the SpuceX team and Musk's commitment to pushing the boundarics of space exploration Regardless of the outcome of the launch, ond thing is for cortain: Musk has the bar for space exploration higher than ever before with the Sturship.
Elon
Table 7: The full texts of wwitten news and model generated news mentioned in Figure 1
model
# of entitiy
# of event
#o of word
# of sentence
average rank
Human
61.75
376.30
888.35
32,40
1.60
Text- D0xt-Davince-003
16.60
124.45
303.30
12.55
2.75
GPT-3.5-turo
14.05
126.90
254.60
10.35
3.25
I.AAMA-13R
40.95
159.45
460.95
21.05
4.00
GPT-3.5-
10.80
57.55
151.60
6.55
4.40
GPT-3-davinci
39.45
137.70
526.40
21.20
4.60
Table 8: Detailed statistic information of numan--wittee texts and model-generated texts

---

Haman U.
0350
5.007
8190
0217
0403
5002
0.17
GPT Stoto
0243
2-02
LLAM 138
02
E017
0233
0254
0433
230
G ve
Human
LLMM
GPt3
OPF 0-t-eel
Figure 4: Impact the number of selected knowledge
Imput:Elon Musk sets low expectations before lirst SpaceX luunch of Starship. most powerful rocket ever built
GPT-3.5-trrbo
On April 16th, 2023, Elon Musk, CEO of SpaceX, set low expectations before first launch of the Starship, most powerful rocket over built. The Starship designed to carry up to 100 pussengers, is intended to transport humans and cargo to the moon. Mars, and beyond. Despite the groundbresking uchnology umbitious gouls the project. Musk has warned that first luuch may not succesful. According Musk, the linst launch will be experimental and the clances of success are uncertain. He acknowledged the high risk associated with space travel emphasized importance of taking calculsted tests and simulations to minimize the risks, but there are still many unknown variahles that could impact the launch. The Starship project has boen in development for several and cost-effeative, which could revolutionize and colonization. Despise the potemial benefits, Musk has maintained a realistic approach and acknewledged the challenges associated with space travel. The announcement of low expectations may disappoint some supporters, it responsible innovation. Space exploration is a high-stakes endeavor, and SpaceX's 's cautious appreach retlects their dedication to responsible space travel.
achieve progress. Musk revealed that the company
conducted namerous
and SpaceX has invested billions of dollars into the program. The rocket designed to be reusable success the project would pave way future space explocation
space industry.
company's commitment safety
APPrA.5-turbo-gnllle entitly
"NASA" 'SpaceX" In surprising turn of events, Elon Musk, the visionary CEO of SpaceX, has taken the unprecedented step of setting low expectations ahead of the highly aniicipated inaugural launch of Starship. the most powerful rocket comstructed. Acknow ledging the complexities of space exploration and the inherent risks involved, Musk addressed both NASA and Space eathusiusts, emphusizing the need for cantions ootimism. With NASA's long-standing collshoration with SpaceX, this move showcases Musk's commirment to transparency safety in pursuit of ambitious space missions. 'SpaceX" "Starship" 'Taloon 9* upcoming launch of Starship marks significant milestone for SpaceX, as the company sims to demenstrate the capabilities of its revolutionary spacerraft. Designed to carry both crew and cargo, Starship poised supass the achievements of its prodecssoor, the Falcon 9, With towering heighr, sleak design, and propulsion power, Slarship has captured the imagination of space entbusiasts worldwide. This highly anticipated mission will serve as crucial test for the recke's performance, paving the way for furare interplanctary exploration and potentially revolutionizing "uuk"" Elon
travel we know it the future space exploration has materialized in form the Starship, which is heing assembled at Starhase Launch Site in Boca Chica, Toxas. Musk's decision to construct Starship in Texas reflects his dedication to pushing boundaries and laking on monumental challenges. expansive manufacturing site engincers and technicians meticulously working on the intricat: componentts of the spacoeraft. Musk's presence and hands-on involvement in the project have inspired the team, creating sense of shared purpose and determination they strive make Starship reality. "Starship" "Musk" "NASA" The upcoming luunc of Slandip not only represents a mnjor milestone for SpwceX but also signifies a significant partnership between Flon Musk's company and NASA. The collabortion between SpaceX and the renowned quace agency has heen instumenntll in the development testing Stanship, with
"Starship" "Starhase Launch
u of activity.
providing valuable expertise partnenships in advancing space exploration capabilities. As Musk and his team prepare for the historic launch, the support and guidance from NASA serve as testament the slaand commitment pushing the boundaries of space travel realizing ambitious goals set forth by hoth SpaceX and the agency.
resources. This joint effort highlights
importance of public private
Table 9: Case of generating texts with GPT-3.5-turb and GPT-3.5- --r3.--turbo+guided entity. We highlight guided entity in red and place it before each paragraph.


Reviews:
[{"review_id": "enQoO57xTV", "reviewer": "Reviewer_GVGV", "summary": "", "questions": "Table 1 offers significant insight that helps to motivate the paper. However, it would be beneficial if the authors could elaborate on the data used and the analysis performed. Providing more detailed information will help to clarify the basis of the motivation and enhance the overall understanding of the paper.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable questions and comments.\n### Answers to questions\nFollowing your suggestions, we will add additional detailed information and analysis in the revised version. The following table contains the statistical information of human-written texts and model-generated texts, which includes the number of words, number of entities, and number of events. \n\n| model            | # of entitiy | # of event | # of word | # of sentence | average rank |\n| ---------------- | ------------ | ---------- | --------- | ------------- | ------------ |\n| Human            | 61.75        | 376.30     | 888.35    | 32.40         | 1.60         |\n| Text-Davince-003 | 16.60        | 124.45     | 303.30    | 12.55         | 2.75         |\n| GPT-3.5-turbo    | 14.05        | 126.90     | 254.60    | 10.35         | 3.25         |\n| LLAMA-13B        | 40.95        | 159.45     | 460.95    | 21.05         | 4.00         |\n| GPT-3.5-curie    | 10.80        | 57.55      | 151.60    | 6.55          | 4.40         |\n| GPT-3-davinci    | 39.45        | 137.70     | 526.40    | 21.20         | 4.60         |\n\nWe will also give the win rate between humans and models as follows. Since there is a tie-breaker option in the questionnaire, the win rate is likely to be less than 100%. If needed, we would offer more detailed results of our questionnaire.Follow your suggestion, we will add these detailed information in the revised versions. \n\n| win-rate                | Human  | Text-Davince-003 | GPT-3.5-turbo | LLAMA-13B | GPT-3.5-curie | GPT-3-davinci |\n| ----------------------- | ------ | ---------------- | ------------- | --------- | ------------- | ------------- |\n| Human v.s. X            | /      | 80.00%           | 81.67%        | 81.67%    | 90.00%        | 96.67%        |\n| Text-Davince-003 v.s. X | 16.67% | /                | 51.67%        | 65.00%    | 76.67%        | 76.67%        |\n| GPT-3.5-turbo v.s. X    | 15.00% | 21.67%           | /             | 60.00%    | 66.67%        | 71.67%        |\n| LLAMA-13B v.s. X        | 10.00% | 30.00%           | 33.33%        | /         | 43.33%        | 53.33%        |\n| GPT-3.5-curie v.s. X    | 8.33%  | 16.67%           | 15.00%        | 55.00%    | /             | 61.67%        |\n| GPT-3-davinci v.s. X    | 1.67%  | 23.33%           | 25.00%        | 43.33%    | 30.00%        | /             |\n\n\n\n### Reply to reject reasons\n#### Reply to reject reason1:\nIn our opinion, informativeness means more details and background information in the paper. When we introduce such information, it's unavoidable to introduce additional entities. That is the reason for incorporating additional entities into informative text generation. As you pointed that the news article should maintain a strong correlation with its title. That is the principle that we design our local knowledge filter, which filters knowledge and entities that are correlated with the title.\n\n#### Reply to reject reason2:\nThe reasons for NER and entity link(EL) tools:\n\nFor English datasets, we choose Flair and BLINK as NER and EL tools for the following reasons:\n    1. Flair and BLINK are widely used in NLP research and application. Google Scholar citation of Flair is 789 and BLINK is 340. The Github star number of Flair is 12k and BLINK is 1.1k.\n    2. They have a promising performance on English news texts. We compare them with other NER and El tools such as SpaCy, NLTK, and tagme on a small subset of our datasets. They have a better performance.\n    3. BLINK offers model parameters and codes officially, enabling us parallelly process our data on our multiple servers.\n\nFor Chinese datasets, we choose SpaCy and Wikidata query as NER and EL tools for the following reasons:\n    1. NER and EL tools such as Flair, NLTK, BLINK, and tagme do not support Chinese texts.\n    2. SpaCy has good performance and is easy to use.\n   \nThe reasons for relation extraction(RE) tools: \n    The intuition of distant supervision is that any sentence that contains a pair of entities that participate in a known relation in KG is likely to express that relation in some way[1]. This method is simple, effective, and widely used in NLP research[2,3,4].  \n\nThe reasons for FiD models: \n    FiD converts knowledge to natural texts, which is widely used in knowledge-retrieval LM. This method enables us to parallelly train our model. That is the main reason for choosing FiD models.\n\nFollowing your suggestions, we will add above reasons in the appendix of the revised version.\n\n#### Reply to reject reason3:\nThe motivation of our paper is to solve the \"Over-concentrated Information\", which leads to a low informativeness of texts generated by pre-trained models. From this perspective, we use pre-trained models as our baselines, including PLMs like GPT2 and BART, and LLMs like GPT-3.5-turbo. Other work in this area mainly focuses on the off-prompt and incoherence problems. In the current version, we do not compare the proposed methods with those methods. Follow your suggestion, we will compare our method  with more recent models in the revised version..\n\n#### Reply to reject reason4:\nIterative generation is a common generation paradigm in the natural language generation tasks. As you point out, Iterative generation can increase additional costs. Fortunately, the average number of entities per sentence is quite small, especially 1.43 for PENS, 2.31 for CNN, and 1.40 for ChinaNews. Therefore, prediction entities for each sentence do not impose much additional costs. Thanks for your questions, we will add the costs into the revised version.\n\n#### Reply to reject reason5:\nHuman evaluation is the golden standard metric for open-ended text generation, especially for LLMs. So we report human evaluation results of experiments on PLMs and LLMs in Table 6. Follow your suggestion, we will add the comparison between small PTMs and LLMs in the main results (Table 4). Meanwhile we will show more the case study.\n\n### References\n[1]Mintz, Mike, et al. \"Distant supervision for relation extraction without labeled data.\" Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP. 2009.\n\n[2]Chen, Danqi, et al. \"Reading Wikipedia to Answer Open-Domain Questions.\" Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2017\n\n[3]Ji, Shaoxiong, et al. \"A survey on knowledge graphs: Representation, acquisition, and applications.\" IEEE transactions on neural networks and learning systems 33.2 (2021): 494-514.\n\n[4]Sun, Yu, et al. \"Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation.\" arXiv preprint arXiv:2107.02137 (2021).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed proplem (\"over-concentrated information\" ) is interesting and the paper provides some insight to address such an issue.", "reasons_to_reject": "1.While the proposed problem is indeed significant in the realm of open-ended text generation, I believe the authors could provide a more comprehensive explanation as to why additional entities should be incorporated into the text generation process. From my perspective, a news article should maintain a strong correlation with its title, rather than diverging into a multitude of topics.\n2.The pipeline incorporates several existing techniques and models such as NER, RE, and FiD. I would recommend that the authors provide a rationale for their selection of these specific models over other comparable models in the field.\n3.The baselines utilized, namely BART, GPT2, and KG-GPT, are somewhat outdated. I would suggest that the authors compare their pipeline with more recent models in the task, such as the top-performing models in the CNN dataset leaderboard.\n4.As I understand it, the pipeline employs an iterative process, wherein a single sentence is generated based on the context and predicted entities. I am curious as to whether this process could potentially compromise efficiency.\n5.The paper indicates that experiments were conducted on small PLMs and LLMs. However, there seems to be a lack of comparison in the main results (Table 4) and the case study. I would recommend that the authors address this issue for a more comprehensive analysis.", "questions_to_authors": "Table 1 offers significant insight that helps to motivate the paper. However, it would be beneficial if the authors could elaborate on the data used and the analysis performed. Providing more detailed information will help to clarify the basis of the motivation and enhance the overall understanding of the paper.", "paper_topic_and_main_contributions": "The paper introduces the concept of an \"over-concentrated information\" issue in open-ended text generation, suggesting that generation models may overly focus on the entities present in the input text, rather than generating a diverse range of entities. To tackle this issue, the authors have designed a comprehensive pipeline. This pipeline includes a local knowledge filter, a dynamic entity selector, and a FiD-based generator. The paper further elaborates on how this pipeline can enhance the performance of Language Learning Models (LLMs) such as ChatGPT.\nThe paper's contributions are twofold: Firstly, it identifies and addresses the \"over-concentrated information\" issue by proposing a novel framework. Secondly, it offers three automatically constructed datasets to evaluate the proposed problem.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HS0w3iNXlM", "reviewer": "Reviewer_yWwV", "summary": "", "questions": "Question A: The total document sizes of PENS and CNN in Table 2 are not the same as those in the original papers (Ao et al. 2021, Hermann et al. 2015). More concretely, Table 2 of this paper shows 113,218 documents in the PENS dataset, but in Section 3.1 of Ao et al. 2021' paper, the authors described, \"The PENS dataset contains 113,762 pieces of new articles ...\". In addition, Table 2 of this paper shows 92,465 documents in the CNN dataset, but the dataset contains 92,579 documents (= 90,266 + 1,220 + 1,093) based on the statistics in Table 1 of Hermann et al. 2015's paper. Why are there such differences?\n\nQuestion B: When does Algorithm 1 terminate? There is no termination condition in the while loop of Algorithm 1 (lines 8 to 14 in Algorithm 1). \n\nQuestion C: Why did the authors use the Wikipedia dumps archived on October, 4, 2022? Since the date is more recent than the dates in the ChinaCNN dataset, it could cover many of the entities in the dataset. Does that make it a fair experimental setting?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable questions and comments.\n### Answers to questions\n#### Answer to  question A\nThank you very much for your detailed question. The reasons that document sizes of PENS and CNN in Table 2 are not the same as those in the original papers are listed as follows:\n\nFor CNN dataset, there are 114 empty samples, which means the sample has empty news article content. For example, there is no news content in \"fe9b7daaab79553adaea3e9aa086510bca4b1fb3.story\". We remove such samples and get 92465 news articles. \n\nFor PENs dataset, there are some noisy samples , which is full of advertisement and web link. We removed the article if web link occupies over 50% content of the text.\n\nFollow your suggestion, we will add these details in Chapter 3 Datasets Construction.\n#### Answer to  question B\nIt has two termination conditions. While training PLMs, we use an empty sentence(<bos><eos>) at the end of the news. When the text generator outputs an empty sentence, we terminate the generation process. Besides, we have a maximum number of tokens of generated texts, which is set to 1024. When generated texts touch the limit, we terminate the generation process and discard the incomplete results. While prompting LLMs, termination condition is the maximum number of tokens, which is also set to 1024. We will elaborate the termination condition more clear in the revised version.\n#### Answer to  question C\nWikidata dumps archived on October 4, 2022, is the latest version when we start to annotate the datasets. Because there lack of Chinese entity link tools and models, we use the Wikidata query API service, which uses the current latest Wikidata dumps. Ideally, news should be linked with the Wikidata dump which has the most recent timestamp.  However, it is too costly, and choosing the current latest version is our balanced choice.\n\n### Reply to reject reasons\n#### Reply to reject reason1\nWe are sorry for these typos and we will check our article and correct them. \n#### Reply to reject reason2\n1. Our questionnaire result shows that human preference is more correlated with the number of knowledge-constrained entities, rather than the length or number of entities. This motivates us to make use of knowledge-constrained entities to enhance the informativeness of generated texts. We conduct human evaluation and the result indicates that our method indeed improves the informativeness of generated texts.\n2. Experiments with GPT-3.5-turbo indicate that our method works with LLMs. We chose GPT-3.5-turbo as the baseline mainly because its web version, known as ChatGPT, is popular and convenient to conduct pre-experiments. We will conduct more experiments on different LLMs in the revised version.\n#### Reply to reject reason3\n1. Knowledge graph is a kind of verified and widely-used structured knowledge base. There exist several general knowledge graphs such as Freebase[1], Wikidata[2] and domain specific such as  AgriKG (agricultural)[3], Acekg(academic)[4], MathGraph(education)[5], SemaTyP(drug discovery)[6]. In knowledge-intensieve text generation, it is rare that entities do not appear in the given input, like news headline or questions.\n2. Our method will degrade to the baseline when no triple is available. However, when no triples can be obtained, it is questionable whether informativeness is required in such a setting. \n\n### Reply to Missing References:\nThank you for providing papers that we may have missed. We will read them and cite them in our paper.\n\n### Reply toTypos Grammar Style And Presentation Improvements:\nThank you for your advice and we will correct them.\n\n### References\n[1] Bollacker, K., Evans, C., Paritosh, P., Sturge, T., & Taylor, J. (2008). Freebase: A collaboratively created graph database for structuring human knowledge, In Proceedings of the 2008 ACM sigmoid international conference on management of data. \n\n[2] Vrandečić, D., & Krötzsch, M. (2014). Wikidata: a free collaborative knowledgebase. Communications of the ACM, 57(10), 78-85. \n\n[3] Chen, Y., Kuang, J., Cheng, D., Zheng, J., Gao, M., & Zhou, A. (2019). Agrikg: An agricultural knowledge graph and its applications, In International conference on database systems for advanced applications. Springer. \n\n[4]Wang, R., Yan, Y., Wang, J., Jia, Y., Zhang, Y., Zhang, W., & Wang, X. (2018). Acekg: A large-scale knowledge graph for academic data mining, In Proceedings of the 27th ACM international conference on information and knowledge management. \n\n[5] Zhao, T., Huang, Y., Yang, S., Luo, Y., Feng, J., Wang, Y., Yuan, H., Pan, K., Li, K., Li, H. Et al. (2019). Mathgraph: A knowledge graph for automatically solving mathematical exercises, In an International conference on database systems for advanced applications. Springer. \n\n[6] Sang, S., Yang, Z., Wang, L., Liu, X., Lin, H., & Wang, J. (2018). Sematyp: A knowledge graph-based literature mining method for drug discovery. BMC bioinformatics, 19(1), 193.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed methods consistently outperformed the baselines on several evaluation metrics in open-ended news headline generation.\n\n- An attempt to give the dynamic entity selector output as additional inputs to the prompt may be beneficial to some readers.\n\n- The authors reported that the two modules, the local knowledge filter and the dynamic entity selector, are both effective for performance improvement.", "reasons_to_reject": "- The readability of this paper is low due to crucial typos (e.g., \"KG-GPT\" in Table 4 -> \"CK-GPT\") and unclear points (e.g., no termination condition in Algorithm 1).\n\n- Some evaluation metric and evaluation settings are favorable to the proposed method. For example, the proposed evaluation metric \"KC_entity\" uses the entities obtained from a knowledge graph, but the similar concepts are considered in the local knowledge filter and dynamic entity selector of the proposed model. In addition, the number of the triples used in the experiments were adjusted referring to the target evaluation datasets in Section 5.4. Furthermore, the choice of GPT-3.5 model in Section 4.2 is also unnatural. According to the preliminary results in Table 1, GPT-3.5-davinci is better than GPT-3.5-turbo with regard to both informativeness and #entity, but the authors chose GPT-3.5-turbo in Section 4.2 with no explanation.\n\n- The proposed method strongly depends on the triples obtained from a knowledge graph. This indicates that when no triple cannot be obtained from a knowledge graph, the proposed method does not maintain its generation quality. Since the evaluation task in this paper is news headline generation, where triples can be collected relatively easily from a knowledge graph, it is difficult to know how the performance degrades when no triple is available in the proposed method.", "questions_to_authors": "Question A: The total document sizes of PENS and CNN in Table 2 are not the same as those in the original papers (Ao et al. 2021, Hermann et al. 2015). More concretely, Table 2 of this paper shows 113,218 documents in the PENS dataset, but in Section 3.1 of Ao et al. 2021' paper, the authors described, \"The PENS dataset contains 113,762 pieces of new articles ...\". In addition, Table 2 of this paper shows 92,465 documents in the CNN dataset, but the dataset contains 92,579 documents (= 90,266 + 1,220 + 1,093) based on the statistics in Table 1 of Hermann et al. 2015's paper. Why are there such differences?\n\nQuestion B: When does Algorithm 1 terminate? There is no termination condition in the while loop of Algorithm 1 (lines 8 to 14 in Algorithm 1). \n\nQuestion C: Why did the authors use the Wikipedia dumps archived on October, 4, 2022? Since the date is more recent than the dates in the ChinaCNN dataset, it could cover many of the entities in the dataset. Does that make it a fair experimental setting?", "paper_topic_and_main_contributions": "This paper proposed a variant of guided text generation methods to solve the problem of generating text with excessive use of entities/events contained in an input prompt. The proposed method utilizes a knowledge graph to obtain the triples relevant to an input text, and the obtained triples are encoded to use them as additional inputs of the decoder. To use the reliable triples in the proposed method, the authors introduced the local knowledge filter and dynamic entity selector, both of which are BERT-based modules. In the training of the proposed method, the authors used small LLMs (e.g., BERT-base and BART-base) and fine-tuned them with the dataset of news headline generation. In the evaluation, the authors used three datasets, PENS, CNN and ChinaCNN (ChinaCNN data was created by the authors). In addition to typical generation setting using the proposed encoder-decoder architecture, the authors also evaluated the generation quality by modifying input prompts of GPT-3.5. More precisely, the authors giving the output of the dynamic entity selector as additional inputs to the prompt to investigate whether the entities obtained by the dynamic entity selector has a positive effect on the quality of generation. The experimental results show that the proposed method consistently outperformed the GPT-2/BART-based methods and other guided text generation method with respect to the perplexity, Dist-3, ROUGE-L, BERTScore and newly introduced informativeness-based evaluation metric (KC_entity). Human evaluations also support the superiority of the proposed method over GPT2/BART-based methods.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- Event Transition Planning for Open-ended Text Generation.\n  Qintong Li, Piji Li, Wei Bi, Zhaochun Ren, Yuxuan Lai and Lingpeng Kong.\n  Findings of the Association for Computational Linguistics: ACL 2022, pp.3412-3426.\n  https://aclanthology.org/2022.findings-acl.269.pdf\n\n- ChatGPT is not Enough: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling.\n  Linyao Yang, Hongyang Chen, Zhao Li, Xiao Ding, Xindong Wu.\n  https://arxiv.org/abs/2306.11489 \n\n- LMExplainer: a Knowledge-Enhanced Explainer for Language Models.\n  Zichen Chen, Ambuj K Singh, Misha Sra.\n  https://arxiv.org/abs/2303.16537", "Typos_Grammar_Style_and_Presentation_Improvements": "- The authors should right-align the numbers in Table 2 and Table 4 for readability.\n\n- Table 7: the caption said \"We highlight guided entity in red and place it before each paragraph.\" but there is no highlighted entities in this table."}}, {"review_id": "pNEER7xu7p", "reviewer": "Reviewer_GfBH", "summary": "", "questions": "In figure 1, what is the prompt or instructions? Is there any possibility that with better prompts, the generated texts from llm will be more informative?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your encouragement.\n\n### Answers to questions\nWe provide the prompts used in our case study(Figure 1) and questionnaires (Table 1). We use \"[the news title]\" to represent the content of a news title. As different models have different capabilities, we use several prompts.\n\nGPT-3.5-turbo is the API version of ChatGPT and it has a system prompt and user prompt. The system prompt for GPT-3.5-turbo is \"You are a helpful news writing assistant. You are required to write informative English news based on the given title.\" and the user prompt is \"The title is [the news title]\"\n\nText-Davinci-003 and GPT3.5-curie are finetuned with instruction, which means that they can follow instruction prompts. The prompt for Text-Davinci-003 and GPT3.5-curie is \"You are a helpful news writing assistant. You are required to write an informative English news article based on the given title. Here is the title: [the news title]. \"\n\nLLAMA-13B and GPT-3-davinci are just language models and they can only continue writing. The prompt for LLAMA-13B and GPT-3-davinci is \"Title:[the news title]\\n News content:\"\n\nWe will add these prompts into the revised version.\n\nThe better-designed prompts could lead to more informative texts since our method could be seen as a kind of knowledge-enhanced prompt. But our insight, which is \"Knowledge-constrained entities are helpful to generate more informative texts\", is more important and could be taken into consideration for prompt design.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper is well organized and written, the structure is logical, the ideas are presented coherently and clearly.\nThe method, infogen, is well-motivated, the experimental design is elaborate and the result is solid.\n", "reasons_to_reject": "This paper is worth accepting", "questions_to_authors": "In figure 1, what is the prompt or instructions? Is there any possibility that with better prompts, the generated texts from llm will be more informative?", "paper_topic_and_main_contributions": "The paper introduces a novel approach that leverages a knowledge graph to guide the text generation process, specifically targeting the issue of over-concentrated information in large language models. The combination of a local knowledge filter, dynamic knowledge selector, and knowledge-enhanced text generator provides a unique contribution to the field of open-ended text generation.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers cite the motivation and clarity of the experimental design as strengths of the paper.

There were some concerns about limitations of this work, e.g., whether the resources/models used in the pipeline approach would be effective in another domain, and whether the evaluation used competitive baselines. Discussion with the authors led two reviewers to slightly increase their scores, though excitement scores remained borderline.
Decision: Accept-Findings