Structure-aware Knowledge Graph-to-text Generation with Planning Selection and Similarity Distinction
Authors: Feng Zhao
Hongzhi Zou
Cheng Yan
Venue: EMNLP 2023 Main (2023)
Keywords: KG-to-text generation
Pre-trained language model
Planning Selection
Similarity Distinction
TL;DR: Structure-aware Knowledge Graph-to-Text Generation

Abstract: The knowledge graph-to-text (KG-to-text) generation task aims to synthesize coherent and engaging sentences that accurately convey the complex information derived from an input knowledge graph. One of the primary challenges in this task is bridging the gap between the diverse structures of the KG and the target text, while preserving the details of the input KG. To address this, we propose a novel approach that efficiently integrates graph structure-aware modules with pre-trained language models. Unlike conventional techniques, which only consider direct connections between first-order neighbors, our method delves deeper by incorporating Relative Distance Encoding as a bias within the graph structure-aware module. This enables our model to better capture the intricate topology information present in the KG. To further elevate the fidelity of the generated text, Planning Selection and Similarity Distinction are introduced. Our approach filters the most relevant linearized sequences by employing a planning scorer, while simultaneously distinguishing similar input KGs through contrastive learning techniques. Experiments on two datasets demonstrate the superiority of our model.

Structure-aware Knowledge Graph-to-Text Generation with Planning Selection and Similarity Distinction
Feng Zhao, Hongzhi Zou, Cheng Yan
Natural Language Processing and Knowledge Graph Lab, School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China {zhaof, hongzhizou, yancheng] @ @hust.edu.cn
Abstract
James Pain
archteet
The knowledge graph-to-tex. (KG-to-text) generation task aims to synthesize coherent and engaging sentences that accurately con- vey the complex information derived from an input knowledge graph. One of the pri- mary challenges in this task bridging the between the diverse structures of the KG and the target text, while preserving the de- tails of the input KG. To address this, propose a novel approach that efficiently in- tegrates graph structure-aware modules with pre-trained language models. Unlike conven- tional techniques, which only consider direct connections between first-order neighbors, our method delves deeper by incorporating Rela- tive Distance Encoding as a bias within the graph structure-awar module. This enables our model to better capture the intricate topol- ogy information present the KG. fur- ther elevate the fidelity of the generated text, Planning Selection and Similarity Distinction are introduced. Our approach filters the most relevant lincarized sequences by employing a planning scorer, while simultaneously dis- tinguishing similar input KGs through con- trastive learning techniques. Experiments on two datasets demonstrate the superiority our model.
Adare Manor
completion date
1862
building start date
1700
Description Text:
James Pain was the architect of the Adare Manor building
construction started
completed
Linearized Sequence:
<H>Adare Manor <R> architect [T] James Pain
<HoAAdre Manor buildingg start date (T] 1700
<H>Adare Manor <R> completion date (T] 1862
The KG and its corresponding text. The enti- ties text are marked color. The order triples in the linearized sequence control the accuracy of the generated text.
Figure 1:
challenges for the task are the different struc- tures of the input KG and output texts as well as the need to ensure generated description texts completely and accurately cover the KG informa- tion, including all entities and relations.
Benefitting from the powerful capabilities pre-trained language models (PLMs), methods based on PLMs (Ke et al., 2021) are currently achieving outstanding results KG-to-text gener- ation tasks. Since there is structural gap between the graph structure of KGs and the sequence in- put requirements demanded by PLMs, an impor- tant step to linearize the given KG into an in- put sequence. The results of many previous works have shown that the final generated texts are signif- icantly influenced by the triple orders of linearized sequences. Many existing works typically use pre- defined heuristic rules such as breadth-fin -frrst search (BFS) (Li al., 2021), or create content plan- ner using multistep prediction methods predict of the triple order (Zhao et al., 2020). Each step of these multistep prediction processes may have pre- diction bias, which results in the final sequence or-
1 Introduction
Knowledge graph (KG) is structured data repre- sentation form that contains rich knowledge infor- mation and is more convenient for processes such as information retrieval and reasoning. Although KGs facilitate computational processes, it is diffi- cult for humans to intuitively understand the con- tent in KGs, the proposed KG-to-text genera- tion task aims produce correct descriptive text for the input KG. KG-to-text has various applica- (Pal et al., 2019)
tions, like guestion-and-anseen and dialogue systems (Zhou et al., 2018). Figure shows a KG and its corresponding text. The main

---

der easily deviating significantly from the ground- truth order. Considering this problem, we train an efficient planning scorer to score the generated re- sults of all linearized sequences corresponding to KG, and pick the sequence with the highest score to be the final input, thus enabling the bias caused by multistep predictions to be avoided.
graph structure directly using graph neural net- works (GNNs) (Guo al., 2019; Zhao al., 2020; Ribeiro et al., 2020b; et al., 2021) or graph-tranfformers (Schmitt et al., 2020) and then decode into texts. DUALENC (Zhao et al., 2020) feeds the input KG into two GNN en- coders for order planning and sentence gener- ation. Graformer (Schmitt et al., 2020) intro- duces a model that combines relative position in- formation compute self-attention. Other ap- proaches (Wang al., 202 Liu al., 2022; Guo et al., 2020; Ribeiro et al., 2020a) first linearize KG into sequences and then feed them into the sequence-to-sequeenee (Seq2Seq) model for gener- ating desired texts. In this paper, we employ the Seq2Seq model with planning selector to control linearized sequence orders.
The linearized sequence may ignore the graph structure information in the corresponding KG, such as the connectivity of nodes and the direc- tion of edges. Some works (Colas et al., 2022: Ke et al., 2021) have improved PLMs by adding graph structure-aware modules to every layer of the transformer encoder. These graph-aware mod- ules only consider the directly connected entities in one triple, i.c., the first-order neighbors in the KG. During the specific text generation process, only the information within each triple is consid- ered, while the connections of the entities between triples are ignored. Therefore, to fully utilize the topological structure information of a KG, we con- sider the edges in the KG as nodes to compute the shortest path distance among nodes and add it to the weight computation as bias. Addition- ally, to enable better distinction of similar input KGs and similar entities in the same KG, we use contrastive learning to enable the transformer en- coder differentiate the input sequences at finer- grained level. We use two datasets to test our model: WebNLG and DART, and it outperforms the previous models. Our contributions are three- fold:
2.2 Sequence Order Generation
Existing works (Zhao al., 2020) have shown that the linearized order of the given triples has an effect on the generated text's quality. Previ- ous works mainly use graph traversal (Li et al.. 2021) or multistep prediction (Su et al., 2021; Liu et al., 2022; Zhao et al., 2020) methods for triple order generation. (Li et al., 2021) uses the relation-biased BFS (RBFS) strategy to traverse and lincarize KGs into sequences. (Zhao et al., 2020) uses the content planner to select one of the remaining unvisited triples each step until all triples have been visited. These graph traversal- based approaches are difficult to handle when fac- ing isolated points that are not connected to other points. Methods based on multistep prediction can be incorrect at each step of the chain decisions. Inspired by previous work (Kertkeidkachom and Takamura, 2020), use PLM based efficient planning scorer to control the order generation pro- cess.
We propose an efficient score control-based planning scorer to generate remarkably accurate linearized sequences for text generation.
Our approach involves capturing and distin- guishing the information of KG using fine- grained control methods, including relative distance-based structure-awars control and con- trast lcarning- sequence representation dis- tinction.
2.3 Pre-trained Language Model
PLMs such as T5 (Raffel et al., 2020) and BART (Lewis et al., 2020) have achieved superior performance on KG-to-text generation tasks. Ex- isting works for solving this task using PLMs fall into two main categories. One category does not change the internal structures of PLMs, but rather optimizes the fine-tuning process by introducing other training tasks. (Li et al., 2021) utilizes the KG reconstruction task and the entity copying task to fine-tune the BART. The other category adjusts the internal structures of PLMs by incorporating
Results from two datasets demonstrate the su- periority of our model compared to other meth- ods, and our approach has the potential signif- icantly improve the quality of text generation.
2 Related Work
2.1 KG-to-Text Generation
To capture the KG structural information, many re- cent works on KG-to-text generation encode the

---

new modules in encoder or decoder. (Ke al.. 2021; Colas al., 2022) incorporate the graph structure-aware module into the encoder of PLMs. However, they only consider information di- rectly connected first-order neighbors and ignore the connection information of indirectly connected neighbors. Inspired by (Schmitt et al., 2020), we aggregate the relative distance information into the graph structure-awar module, thus exploiting the information of indirectly connected neighbors.
3.1 Problem Formulation
The aim is to generate accurate text to describe the input KG. The input KG consists of some triples and G = {< h, r, > 1h, E E, r. E R), where & and R. are sets of entities and relations, respec- tively. Following (Ke al., 2021), linearize the input KG Giincar (w1, " Wm). where m is the number of tokens. The target is generate the text T (41,22,.) 12, ,b,,, which give an accurate and complete description of the infor- mation in the input KG.
2.4 Sequence Embedding with Contrastive Learning
3.2 Planning Scorer
The purpose of the planning scorer is to compute the scores of all linearized KG sequences to select the highest scoring sequence. Previous methods incorporated GCN-based representation of graphs, but this process does not mainly affect the lin- earization order of the triples and instead severely affects the training efficiency in the first stage. We therefore propose planning scorer with simple structure to further speed up the training efficiency of the model while obtaining the correct lincariza-
Due to the power of contrastive learning exhib- ited in image representation (Radford et al., 2021), many approaches (Gao et al., 2021; Wang et al., 2021a; Reimers and Gurevych, 2019) have ap- plied it to sentence embedding well. Sentence- BERT (Reimers and Gurevych, 2019) presents a modification of BERT, which employs the Siamese network to derive fixed-sized vectors for input sentences and employs a similarity mea- sure to find semantically similar sentences, Sim- CSE (Gao et al., 2021) presents the unsupervised methods that apply different hidden dropout masks and the supervised methods that additional de- scription datasets to generate positives and nega- tives. Similar to the above sentence embedding approaches. in the KG-to-text generation task we embed linearized sequences triples using con- trastive learning to distinguish similar KGs and similar entities.
tion sequences. The planning scorer is also di- vided into two steps: 1) Plan Generation and 2) Plan Evaluation.
3.2.1 Plan Generation
We generate all triple orders as needed by Plan generation. Each graph in the WebNLG has no more than seven triples, so can generate all pos- sible triple orders by exhaustive enumeration. Fig- ure shows KG with three triples and the six cor- responding kinds of linearized sequences, where <H>, <R>, and <T> identify the head entity, rela- tion, and tail entity. For KGs with the number of triples more than seven in the DART dataset, we then randomly sampled only 5,000 possible triplet orders as the result of the planning generation.
3 Approaches
Our method not only ensures that the generated text and the target text have the same information, but also enables the model capture both topolog- ical information and specific node content infor- mation of the graph for the KG-to-text generation task. To accomplish the first goal, we train plan- ning scorer that can select the best linearized se- quence by scoring all possible input KGs. For the second goal, we first add the graph structure-aware module to the BART encoder. Then, calculate the cosine similarity of the sequence embeddings from the last layer of the encoder to generate new loss. Figure/ 2 represents our model architecture and the following sections describe the contents these modules.
3.2.2 Plan Evaluation
Planning evaluation performs evaluations for each generated linearized sequence. The structure the Plan Generation consists of two modules: Lin- earization Representation and Plan Score.
Linearization Representation maps each to- ken in the input lincarized sequence into high- dimensional vector. As shown in Figure 2, we employ BART achieve the linealization rep- resentation of the input sequence. Getting the plan p. we first obtain the linearized sequence L = (w), W2, .. (w)) in which w1, W2, .. W.

---

1) Planning Selection
2) Text generation
James Pain was the architect of the Adare Manor building which construction started in 1700 and completed in 1862.
loss
Plain
Socre
Text generation
Planning Scorer
FNN
Decoder
Average Pool
Constractive
BART
BART
Graph-aware
Reference:
Unnariized Sequence:
Linearized Sequeence:
James Pain was the architect of the Adare Manor building which construction started 1700 and completed
Adare Manor <R> architect <T> James Pain
<H> Adare Manor <R> architect <T> James Pain
-H--Adare Manor <R> building start date <T> 1700
<H> Adare Manor <R> building start date <T> 1700 <H> Adare Manor <R>
<H- Adare Manor <Rt completion date <T> 17852
completion date <T> 17862
Figure 2: Overview of our model. During the planning selection phase, the planning scorer first generates a linearized sequence of KGs and then computes the resulting score. The BLEU values of the generated texts obtained using references inputs used as the labels to train the planning scorer. During the text generation phase, the highest scoring linearized sequences screened by the planning scorer are sequentially fed into the graph structure-aware encoder and the decoder to finish the text generation process.
Gisyyde Soe wnonk
During the training phase, the linearized se- quences are fed into BART to generate new texts. And the generated texts and the target texts are used to calculate the BLEU metrics the target scores y. The optimized loss function is follows:
(elt-Accre Nanor <it completinn dat <T> 1062
Adare Manar
Completen
Adoe Mdeee cto haiding ront diwe cfs 1003
Bulling zert dae
unseroed Sequrcak:
disiddrs complatoo date cts 1882
de Adurt Moror addead Th Aeno Nan
linedies Suppweet
cilsAdare Mavor dis tect Jamsi tan
Lraatie Sequences
hs can das <tt mo ritsacane Nater ahi completon dat 1990
Albott Mover co bu rikiing own dee (1) 1926
shokkane Keasr dio avtibent (IT lonns bar ciiskcsw Masor des completion
Licaiixs Saqunnees
adh-Acano Nator dh dack <Tss James Pan noopletoos
Lpten
Σ
Σ #ES PEPA)
y. D.pll
(3)
Figure 3: A KG with three relations and all of its corre-
sponding linearized sequences.
where S denotes the corpus and for sample s the corpus, P(s) denotes the set of generated order plans.
represent each token. Then we feed the lin- earized sequence L into BART and take the em- bedding of the decoder's last hidden layer X (Z1, 22, .n) as the representation result. Finally the Linearization Representation C is learned using mean pooling layer by the follow- ing equation:
During the testing phase, all possible planning sequences for each sample KG in the dataset are fed into the planning scorer to compute the scores, and the linearized sequences with the high- est scores are kept in the file for subsequent gener- ation tasks.
3.3 Graph Structure-Aware Model
C Z
The linearized sequences used as the inputs for text generation. We add the graph structure-aware module to each layer of the transformer encoder, and also use the relative distance as a bias when computing the self-attention weights.
where the linearized sequence has n tokens and x; denotes the embedding of the i-th token.
Plan Score is estimate the BLEU score based on the Lincarization Representation. Here we feed the Linearization Representation C' to a feedfor- ward neural network (FNN), and the output is the score:
Global Attention. Each layer in the trans- former encoder starts with Self-Attention mod- ule, which captures the global relationships be- tween all tokens of lincarized sequences. Assum- ing that the input linearized sequence is X, the
= CWENN +
(2)
where WENN is a linear matrix and 6 is the bias.

---

self-attention is computed as follows:
where gather() redistributes the m-dimensional node representation X onto the n-dimensional (4) token representation. Finally, after the encoder- decoder, for the input linearized sequence Ginears the text generation loss is formulated as follows:
X.W (
0, a
Vi Σ (x)(X,
(5)
where WV, W and W are learned parameters. d represents the dimension of word vectors. o() denotes the softmax function.
Lr = -- Σ ( ,y - 1; Gisnear), (11)
where P is the predicted probability of each token.
Graph Aware Attention. Global attention cap- tures the global relations between all tokens, but ignores the connectivity among entities and rela- tions in the KG. To further capture the connectiv- ity between entities and relations in the KG, in- cluding those not directly connected, add the structure-aware module that incorporates relative distances behind each Global Attention layer. First the mean pooling is used to generate the represen- tation XP R.mdd every component from the output V c Rundd of Global Attention:
3.4 Similarity Distinction
For cases of similar input KGs or similar entities in one KG, the generated texts are often confused and erroneous, Therefore we reduce the similar- ity of different sequence representations by con- trastive learning. First we use different sequences in the same batch as simple negative samples for the differentiation of similar input KGs and each linearized KG as its own positive sample. Taking the embeddings H from the encoder's last hidden layer as inputs, we first obtain the representation rj for each sequence using a mean pooling layer. We assume that all sequences within batch are simple negative samples of each other and then compute the contrastive loss by cosine similarity as follows:
x² poolinggg)))),
(6)
where m denotes the sum of the numbers of en- tities and relations. n is the number of tokens from the linearized sequence. Then we compute the representation X of each entity and relation by Graph Structure- Attention:
X X (X- W K
of
vd
+z(()). (7)
Le = - log
Σ E cmmlorrgttt
(12)
N o (XPWVs +g.
where P denotes the corpus and J/ denotes the set of training samples the batch. To distinguish entities in the same KG, we generate some new sequences as hard negative samples for each se- quence in batch. Specifically, we randomly re- place each entity in the linearized sequence by other entities with the 15% probability to obtain hard negative samples. All simple negative sam- ples in the same batch and the generated hard neg- ative samples will be used together in the compu- tation of the contrastive loss in Eq. (12).
X
(8)
where WQS, WKS WKC WVS and VVC are the weight matrices to be learned. E mm de- notes the adjacency matrix that records the connec- tivity among entities and relations, and the value q is either 1 or 0. > (R) E Rmxm denotes the relative distance encoding matrix. Following (Schmitt et al., 2020), we consider the relations of KG as graph's 's nodes and compute the path distance be- tween each pair of nodes to generate the relative distance matrix R:
At last, the total loss Leotal consists of the text generation loss LT and the contrastive learning loss Lo:
if 8(1, n²) = x and mi) = 20 if 8 ny) n(n,,
Rag- -8(ng, ne). 8(1i, ns))
Linnt Lr XLC.
(13)
if 185 m3) S 8(n;,
where X is the combination coefficient.
where 8(ns, ny) denotes the shortest length from n; to ny. Finally, the graph structure-aware rep-
Experiments
resentation X is added to the self-attentive repre- sentation V after residual layer:
4.1 Datasets
WebNLG (Gardent et al., 2017) is a frequently used dataset. A sample in the dataset contains
V = gather( (xater((*)=a + V,
(10)

---

Table 1: Results on WebNLG, #Param represents the number of training parameters for models with PLMs. Pre- task represents if models have been pre-trained additional tasks. and indicate that the results reprinted from (Ke et al., 2021) and (Colas et al., 2022). SOTA-NPT indicates the state-of-the-art (SOTA) model without any pre-training.
Model
#Param
Pre-task
BL.UU
ROUGE
METFOR
SOTA-NPT' (Shimorina and Gardent, 2018)
No
61.00
71.00
42.00
BART-hase" (Ribeiro et al., 2020a)
140M
No
64.55
75.13
46.51
T5-hasef (Riheiro et al., 2020a)
220M
No
64.42
74.77
46.58
KGPT (Chen et al., 2020)
177M
Yes
64.11
74.57
46.30
JointGT (BARTy (Ke et al., 2021)
160M
Yes
65,92
76.10
47.15
GAPt Mr + (Colas al., 2022)
153M
No
66,20
76.36
46.77
Ours
161M
No
66.53
76.54
47.39
one to seven triples corresponding to one to five texts. We randomly select one text at time from one sample as the final target text and replace the underscores with spaces. Following (Ke et al., 2021), we use the version 2.0 and the numbers of KG-text pairs in the
Table
Results on DART. Seven other models are compared with our model. indicates that the results reprinted from (Liu et al., 2022).
Model
BLEU
METEOR
MoverScore
BLEURT
Sey2Seq-Att
29.66
0.27
0.31
-0.13
are
TS-base
49.21
0.40
0.53
0.43
24,352/4,316/44222
TS-large
50.66
0.40
0.54
0,44
BART- base
47.11
0.38
0.51
0.37
DART (Nan et al., 2021) is an -pen-domain dataset with KGs extracted from Wikipedia tables incorporated from other datasets like Cleaned E2E (Novikova et al., 2017). The numbers of KG-text pairs in the training/valiidationnessngg 30,348/2,759/5,097.
BART-large
48.56
0.39
0.52
0.41
Jointri
54.24
0.44
0.59
S-OSC:
62.01
Ours
62.67
0.44
0.49
of both datasets, we select the best performing model.
4.2
Implementation Details
4.3 Main Results
For the Planning Scorer, we employ BART-base for Linearization Representation. In the FNN, we utilize hidden layers and the ReLU activation. The Planning Scorer is trained with the learning rate of 0,001, the batch size of 16, the epoch of 10 and Adam (Kingma and Ba, 2015) to optimize the parameters.
The WebNLG results displayed in Table 1 rep- resent that our model outperforms other models in overall performance including three metries, i.e., BLEU, METEOR, and ROUGE. Regarding the BLEU metric on WebNLG, our model outper- forms the SOTA Seq2Seq model without any pre- training by 5.53%. Compared with BART and T5, our models improve by 1.98% and 2.11%, respec- tively. Both JointGT and KGPT utilize additional pre-training tasks such as graph-text alignment and text or graph reconstruction. The BLEU im- provements of 2.42% over KGPT and 0.61 % over JointGT show that model performs well with- out additional pre-training tasks. Compared with GAP, which considers only directly connected first-order neighbors in the graph structure-awe aware module, the improvement of 0.33% on BLEU and 0.62% on METEOR shows that our model better captures the input graph's topological structure.
For text generation, the hyperparameters are set as follows: batch size: 100, epoch: 60, learning rate: 2c-5 and optimizer: Adam. The т in the contrastive learning is 0.05. The maximum input length of Linearized sequences is 256. The total loss is optimized by Eq. 13 and À is set to 0.5. When generating text in the inference time, the beam search size
Following previous work, the evaluation met- rics for experiments on WebNLG are ME- TEOR (Banerjee and Lavie, 2005), ROUGE- L (Lin, 2004) and BLEU-4 (Papineni al., 2002). For DART, besides BLEU and METEOR we use two additional metrics, MoverScore (Zhao al., 2019) and BLEURT (Sellam et al., 2020). With the BLEU metrics produced on the validation sets
The results on DART are displayed in Table 2. The large gaps between "Seq2Seq-AIt (Nan et al., 2021) and the other models on all metrics demon-

---

Table 3: WebNLG.
Ablation study for different modules on
Table 4: Results different methods. PS denotes Plan- ning Scorer. RBFS and RDFS (Li et al., 2021) denote the relation-biased BFS and the relation-biase DFS, respectively. RS denotes the random sorting order.
Model
BLEU
METEOR
ROUGE
foll model
66.53
47.39
76.54
w/o PS
47.20
76.27
Methods
BLEU
METEOR
ROUGE
w/o RDE
65.82
47.13
76.25
PS
66.53
47.39
76.54
w/o DK 66.17
47.41
76.29
RBFS
66.30
47.28
76.34
RDFS
47.04
76.02
58.22
46.17
73.09
strate the power of PLMs for KG-to-text genera- tion tasks. Compared to the method using only PLMs (e.g. T5-large), our model has an improve- ment of 12.01% on BLEU, 0.04% METEOR, 0.11% on MoverScore and 0.05% BLEURT. The 8.43% improvement achieved in the BLEU metric also shows that our model makes more effi- cient use of the graph structure. We observe im- provements of 0.66% on BLEU, 0.01% on ME- TEOR and 0.01 % on MoverScore, over S-OSC. To summarize, our model generates more fluent and accurate texts than the other models.
Table 5: BLEU scores for the different models on
DART with different graph sizes.
#Triples
Model 1-3
4-6
JointGT
75.46
56.14
47.36
w/o
73.37
57.50
full model
73.98
60.44
54.30
proach and the random sorting order WebNLG, Table 4 displays the results of different approaches to generate orders of triples. The large drop-off be- tween the random sorting order (RS) and the other methods on the three metries suggests that lin- carized orders of triples significantly impact the ef- fectiveness of generated texts. The planning scorer outperforms RBFS and RDFS in three metrics in- dicating the advantage of evaluating the linearized order of triples from a holistic perspective.
5 Analysis
5.1 Ablation Study
additional experiments on WebNLG, we evalu- ated the impact of the Planning Scorer, relative dis- tance encoding in the graph structure-awan mod- ule and the distinction of KGs on the performance of our model. Besides the full model, we also con- struct the following three variations: without Plan- ning Scorers (w/o PS) and just relying orders of the input triples, without relative distance coding in the graph structure-aware module (w/o RDE) and without distinction of KGs (w/o DK).
5.3 KG Size
To verify the effectiveness of the graph structure- aware module with relative distance encoding, we classify DART into "1-3", "4-6" and "27" ac- cording to the number of triples and compare the BLEU metrics for generating texts. Table dis- plays the BLEU metric results of JoinGGT, our model without the Planning Scorer (w/o PS), and our full model under different conditions. From the top two results, 3" case, Join outperforms our model without the Planning Scorer (w/o PS) by 2.09%, which indicates that for small KG inputs, relative dis- tance coding may present redundant information leading to poor results. With the increase of the triples' number, the results of JointGT are gradu- ally overtaken by "w/o PS", which indicates that the relative distance coding can obtain more infor- mation about the graph structures of larger graphs with more indirectly connected nodes. The bottom two results show that the role of Planning scorer becomes more important as the number of input triples increases.
The results in Table 3 demonstrate that all three modules impact the performance of our model on BLEU, METEOR and ROUGE metrics. By re- moving the planning scorer (w/o PS) and the rel- ative distance coding in the graph structure-aware module (w/o RDE), the BLEU metrics drop by 0.67% and 0.71%, respectively, which reflects the importance of the linearized orders and the infor- mation abtained from nodes that are not directly connected. Removing the distinction of KGs (w/o DK), the small reductions of BLEU and ROUGE metrics indicate that contrastive learning can im- prove the accuracy of generated texts to some ex- tent.
observe that in the "1-
5.2 Linearized Order of Triples
To confirm our planning scorer's effectiivensss, we compare it with the RBFS multistep prediction ap-

---

Table 6: Examples the generated texts and linearized KGs produced by JointGT and our model on WebNLG and DART, where bold represents entities in KGs.
ausociatic cratial
Buytum Alborrs
Kaaer Chist
Knowledge Graph
succiied musial
Law Couroo Auaeri
Allican Arhercane
start pear
Maryy Sarilcee
ETHHUNC Chice
25
Arcdków
Universal RRaconds
CARITA
Real
JoHt unet Linwer
The United States. where the capital Washington DC, includes the ethnic group Americans. Auburn located in the country and is purt of Lee County Alabama.
Musician Andrew White began in 2003 and associated with artists Marry Banilow and the Kaiser Chiefs. He was signed the defunct Universal records.
Reference
Lincarized
JointT
The musician Andrew White started his 2003 associated with musical anisst Marry Banilow. He was signed the defunct Records label but now plays for Kaiser Chiels.
Auburn, Alabama is Lee County. Alabama which Alahama in the United States where African Americans ethnic group capital Washington
Generated Text
Linearized KG
e-0-C-C--6
e- - C---------- @0-0
Andrew White started his career 2003 and ussociated with the musical artist
the capital Washington DC, has the ethnic group of African Americans. Auburn part of County which in Alabama the United States.
United States
Generated
Marry Banilow and Kaiser Chiefs. He was once signed now defunct Universal Records.
From the WebNLG example we can see that Join reverses the position of node 3 and node 5, resulting in incorrect inference results of "once" and "now" in the generated text. With the cor- rect linearization order, our model makes an effec- tive distinction between node 3 and node 4, which have the same relation. On DART, the long tity chains in the input KG lead many attributive clauses and unclear expressions in the text gener- ated by JointGT, The linearized order generated by our model is not exactly correct, but under the in- fluence of the graph structure-aware module with relative distance encoding and entity distinction, the generated text is able to correctly and concisely describe the input KG.
without hard samples
95
with hard samples
88
90
Batch Size
Figure 4: Entity coverage without hard negative sam- ples and with hard negative samples under different batch sizes.
5.4 Entity Coverage
We explore the effect of contrastive learning on dif- ferentiating similar KGs and similar entities by cal- culating the entity coverage in the generated texts over the entities of input KGs. Figure 4 reports the variation curves of entity coverage with and with- out hard negative samples under different batch sizes. It can be observed that the entity coverage increases with increasing batch size. The hard neg- ative samples are helpful to improve the entity cov- erage, but the effect becomes smaller as the batch size increases. We set the batch size to 100 be- cause the entity coverage changes very little after the batch size reaches 100.
Conclusion
The paper presents a two-step training model for KG-to-text generation including planning scorer and text generation. In the first step the plan- ning scorer first obtains the linearized KG order with the highest score. Then the second step uses graph structure module with relative dis- tance encoding and KG distinction by contrastive learning ensure that the model captures the topo- logical information and specific entity features in the input KG. Additional experiments and qualita- tive analyses indicate our model outperforms the existing KG-to-text approaches on two datasets.
5.5 Qualitative Analysis
Table shows examples of generated texts from our model and JointGT on WebNLG and DART.

---

Limitations
Claire Gardent, Anastasia Shimorina, Shashi Narayan. and Laura Brrez-Beltrachini. 2017. The WebNLG challenge: Generating text from RDF data. In Pro- ceedings of the 10th International Conference on Natural Language Generation, INLG 2017, Santi- ago de Compostela, Spain, September 4-7, 2017, pages 124-133.
Since the Planning Scorer needs to score all possi- ble linearized sequences of the input KG, the pro- cessing becomes less efficient as the number of the input triples increases. Therefore our model is not suitable for handling long text or paragraph gener- ation tasks. Because of the large amount of param- eters in PLMs, our model also consumes a large amount of GPU resourees when performing com- parative learning. Four Tesla V100 GPUs are used to train our model, and the maximum batch size that can be set is 312. Finally, our model can accu- rately describe entities and relations in the KG, but lacks reasoning power. For example, the triples (A.husband,B) and CB, as inputs, it can- not generate "A is the father of C". These above limitations are also the direction of our continuing research in the future.
Qipeng Guo, Zhijing Jin, Ning Dai, Xipeng Qiu, Xi- angyang Xue, David Wipf, and Zheng 7hang. 2020. P2: A Plan-and-Pretrain Approach for Knowledge Graph-to-Text Generation. In Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Weh (WebNL.++), pages 100-106, Dublin, Ireland (Virtual).
Zhijiang Guo, Yan Zhang, Zhiyang Teng, and Wei Lu. 2019. Densely Connected Graph Convolutional Net- works for Graph-to-Sequence Learning. Trans. As- soc. Comput. Linguistics, 7:297-312.
Pei Kc, Haozhe Ji, Yu Ran, Xin Cui, Liwei Wang, Lin- feng Song, Xiaoyan Zhu, and Minlie Huang. 2021. JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphs. Findings of the Association for Computational Lin- guistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pages 2526-2538.
Acknowledgements
This work was supported in part by National Nat- ural Science Foundation of China under Grants No.62072203.
Natthawut Kertkeidkachom and Hiroya Takamura. 2020. Text-to-Text Pre-Training Model with Plan Selection for RDF-to-Text Generation. In Proceed- ings of the 3rd International Workshop on Natu- ral Language Generation from the Semantic Web (WebNLG+), pages 159-166, Dublin, Ireland (Vir- tual).
References
Satanjeev Banerjee and Alon Lavie. 2005. METEOR:
An automatic metric for MT evaluation with im- proved correlation with human judgments. In Pro- ceedings of the Workshop on Intrinsic and Ex- trinsic Evaluation Measures for Machine Transla- tion and/or Summarization @ ACL 2005, Ann Arbor, Michigan, USA, June 29, 2005, pages 65-72.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings.
Wenhu Chen, Yu Su, Xifeng Yan, and William Yang Wang. 2020. KGPT: Knowledge-groundd pre-
training for data-to-text generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, On- line, November 16-20, 2020, pages 8635-8648
Mike Lewis, Yinhan Liu, Naman Goyal, Mar- jan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising soguence-to-Sequenee Pre- training for Natural Language Generation, Transla- tion, and Comprehension. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, ACI. 2020, Online, July 5-10, 2020, pages 7871-7880. Association for Computa- tional Linguistics.
Anthony Colas, Mehrdad Alvandipour, and Daisy Zhe Wang. 2022. GAP: A Graph-aware Language Model Framework for Knowledge Graph-to-Text Generation. In Proceedings of the 29th Inter- national Conference on Computational Linguistics. COLING 2022, Gyeongju. Republic of Korea, Octo- ber 12-17, 2022, pages 5755-5769.
Junyi Li, Tianyi Tang, Wayne Xin Zhao, Zhicheng Wei. Nicholas Jing Yuan, and Ji-Rong Wen. 2021. Few- shot Knowledge Craph-to-Tess Generation with Pre- trained Language Models. In Findings of the Associ- ation for Computational Linguistics: ACI/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pages 1558- 1568. Association for Computational Linguistics.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceediings the 2021 Conference on Empirical Methods in Natural Language Pro- cessing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 6894-6910.

---

Chin- Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using siamese BERT- Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLPP 2019, Hong Kong. China, November 3-7, 2019, pages 3980-3990.
Jin Liu, Chongfeng Fan, Fengyu Zhou, and Huijuan Xu. 2022. Syntax Controlled Knowledge Graph- to-Text Generation with Order and Semantic Con- sistency. In Findings of the Association for Com- putational Linguistics: NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 1278-1291. Association for Computational Linguistics.
Leonardo F.
R. Ribeiro, Martin Schmitt, Hinrich Schütze, and Tryna Gurevych. 2020a. Investigating Pretrained Language Models for Graph-to- -ext Gen- eration. CoRR, abs/2007. .0842.......
Linyong Nan, Dragomir R. Radev, Rui Zhang, Am- rit Rau, Abhinand Sivaprasad, Chiachun Hsich, Xi- angru Tang, Aadit Vyas, Neha Verma, Pranav Kr- ishna, Yangxiaokang Liu, Nadia Irwanto, Jessica Pan, Faiaz Rahman, Ahmad Zaidi, Mutethia Mu- tuma, Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Chern Tan, Xi Victoria Lin, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. 2021. DART: Open-domain structured data record to text genera- tion. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, NAACL-HILT 2021, Online, June 6-11, 2021, pages 432-447.
Leonardo F. R. Ribeiro, Yue Zhang. Claire Gardent, and Iryna Gurevych. 2020b. Modeling Global and Local Node Contexts for Text Generation from Knowledge Graphs. Trans. Assoc. Comput. Linguis- rics, 8:589-604.
Martin Schmitt, Leonardo F. R. Ribeiro, Philipp Dufter, Iryna Gurevych, and Hinrich Schütze. 2020. Mod- eling Graph Structure via Relative Position for Bet- ter Text Generation from Knowledge Graphs. CoRR, 092222........
Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. 2020. BLEURT:
Learning robust metrics for text generation. In Proceedings of the 58th Annual Meet- ing of Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7881- 7892.
Jekaterina Novikova, Ondrej Dusek, and Verena Rieser. 2017. The E2E dataset: New challenges for end-to- end generation. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, Saar- brücken, Germuny, August 15-17, 2017, pages 201- 206.
Anastasia Shimorina and Claire Gardent, 2018. Han- dling rare items in data-to-text generation. In Pro- ceedings of the 11th International Conference on Natural Language Generation, Tilburg University, the Netherlands, November 5-8, 2018, pages 360- 370.
Vaishali Pal, Manish Shrivastava, and Irshad Bhat. 2019. Answering naturally: Factoid to full length answer generation. In Proceedings of the 2nd Work- shop on New Frontiers in Summarization, pages 1-9, Hong Kong, China. Association for Computational Linguistics.
Yixuan Su, David Vandyke, Sihui Wang. Yimai Fang. and Nigel Collier. 2021. tlan-teen-Generaae:: Con- trolled Data-to-Texx Text Generation via Planning. In Findings of the Association for Computational Lin- guistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, pages 895-909. Association for Computational Linguis- tics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: A method for automatic eval- uation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, July 6-12, 2002, Philadelphia. PA, USA, pages 311-318
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural lan- guage supervision. In Proceedings of the 38th In- ternational Conference on Machine Learning, ICML. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 9748-8763
Dong Wang, Ning Ding, Piji Li, and Haitao Zheng. 2021a. CLINE: Contrastive learning with semantic negative examples for natural language understand- ing. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume I: Long Papers), Virtual Event, August 1-6, 2021, pages 2332-2342.
Qingyun Wang, Semih Yavuz, Xi Victoria Lin, Heng Ji. and Nazneen Fatema Rajani. 2021b. Stage- wise Fine-tuning for Graph-to-Ter Text Generation. In Proceedings of the ACI-IJCNIP 2021 Student Re- search Workshop, ACL 2021, Online, JUli 5-10, 2021, pages 16-22. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with Unified Text-to- Text Transformer. J. Mach, Learn, Res,, 21:140:1- 140:67.

---

Chao Zhao, Marilyn A. Walker, and Snigdha Chaturvedi. 2020. Bridging the Structural Gap Between Encoding and Decoding for Data-To-Text Generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, ACI. 2020, Online, July 5-10, 2020, pages 2481-2491. Association for Computational Linguis- tics.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Chris- tian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized em- beddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Pro- cessing, EMNLP-IJCNII 2019, Hong Kong, China, November 3-7, 2019. pages 563-578.
Hao Zhou, Tom Young. Minlie Huang, Haizhou Zhao. Jingfang Xu, and Xiaoyan 7hu. 2018. Com- monsense knowledge aware conversation generation with graph attention. Proceedings of the Twenty- Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stock- holm, Sweden, pages 4623 4629.


Reviews:
[{"review_id": "qmCsxPg0VP", "reviewer": "Reviewer_5CDR", "summary": "", "questions": "1. How do you ensure that the order of the  planning scorer predictions is appropriate for the latter\n2. What the mean of gather(g) in Eq. (10)?\n3. It is better to analysis the effect  of lamda in model performance. \n4. One target of KG-to-Text task is diversity. Do you consider this in your experiments?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Reviewer 5CDR\n\nThanks for the detailed and constructive comments. We address the questions and concerns in the following remarks.\n\n**1. There are some contradictions in Eq. (9). The first question is how do we initialize the $R_{ij}$? The second question is that whether $R_{ij}$ is equal to $R_{ji}$ in the relative distance matrix R.**\n\nThe relative distance matrix, denoted as $R$, serves as a bias in the computation of the graph structure-aware module, enabling the acquisition of information pertaining to nodes that lack direct connections. $R_{ij}$ is generated during the dataset preprocessing stage, based on the relative distances between nodes in the KG. Given the deterministic nature of the distances between nodes in the KG, the initial computation of $R_{ij}$ is non-iterative and conducted in its entirety. Notably, the relative distance calculation is predicated on a directed graph, with the minus sign indicating directionality. $R_{ij}$ takes the minimum of the distance from i to j and j to i. The minus represents the direction, so $R_{ij}$ and $R_{ji}$ are different.\n\n**2. The scale of batch size in Figure 4 seems incorrect.**\n\nTo distinguish those similar KGs and those similar entities, we employ a contrastive learning approach. Each linearized KG and itself serve as positive samples for each other. Different linearized KGs in the same batch serve as negative samples to differentiate similar KGs. In order to verify the effect of the batch size on model's effectiveness, we conducted experiments on four scales. To facilitate readability, we have transformed the experimental results in the figure into a tabular format, as shown in the table below. The table displays the entity coverage under varying batch sizes, distinguishing between results obtained without hard samples (w/o HS) and with hard samples (HS), respectively. The results show that the entity coverage increases with the increasing of the batch size. We have optimized Figure 4 in the latest version of the paper.\n\n| batch size |  w/o HS |  HS |\n| :----------: | :------------------------------------: | :---------------------------------: |\n| 50         | 90                                   | 91                                |\n| 100        | 94                                   | 95                                |\n| 150        | 94.4                                 | 95.1                              |\n| 200        | 94.5                                 | 95.2                              |\n\n**3. How do you ensure that the order of the planning scorer predictions is appropriate for the latter.**\n\nThe text generation model remains rooted in BART, with the inclusion of a structure-aware semantic aggregation module within the Encoder.  The incorporation of similar structures serves to uphold order prediction consistency. Notably, the structure-aware semantic aggregation module is dedicated to capturing connectivity features among nodes and does not directly influence order prediction.  Observing the experimental results in Table 3, it becomes evident that the introduction of the Planning Scorer yields improvements in the model's effectiveness.\n\n**4. What the mean of gather(g) in Eq. (10)?**\n\nThe gather() in Eq.10 redistributes the m-dimensional node representation X onto the n-dimensional token representation. Eq.10 represents that the m-dimensional node representation is expanded to an n-dimensional token representation, which is then computed in the residual layer to obtain the final representation of the token. We will add descriptions in the manuscript.\n\n**5. It is better to analysis the effect of lamda in model performance.**\n\nKeeping the batch size 100 and other parameters unchanged, we conducted experiments on the WebNLG dataset for different values of lamda. The results are tabulated below.\n\n| lamda |   0   |  0.1  |  0.3  |  0.5  |   1   |   2   |\n|-------|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|\n| **BLEU**  | 66.17 | 66.32 | 66.49 | 66.53 | 66.45 | 66.44 |\n\nThe results show that the BLEU value is maximum when the lamda size is 0.5. Whereas, the model is least effective when lamda is 0, i.e., with no distinction of KG (w/o DK), which indicates that the DK module is effective. Therefore, we set lamda to 0.5 in our previous experiments.\n\n**6. One target of KG-to-Text task is diversity. Do you consider this in your experiments?**\n\nYes, the results generated by our model exhibit greater diversity. The diversity of KG-to-text refers to the ability to generate more text with diverse content, which includes the ability to generate text that contains richer entities. We therefore demonstrate the diversity capability of our model by validating the increase in the entity coverage of the generated text. The improved entity coverage in Figure 4 indicates that our model can generate texts with more diverse entities. The purpose of incorporating contrastive learning in our model is to enhance the model's capability to differentiate the content of similar KGs, which leads to more precise and more diverse generated texts. Below are two examples of similar KGs in WebNLG and their corresponding output when using (Text1 and Text2) and not using (TextDK1 and TextDK2) the DK(distinction of KGs) module:\n\n \n\nKG1: < Malaysia, leaderName, Abu Zahar Ujang>, < Malaysia, ethnicGroup, Malaysian Malay>, < Malaysia, ethnicGroup, Malaysian Indian>\n\n \n\nReference1: Abu zahar ujang is a leader in Malaysia, where there are ethnic groups called Malaysian Malay and Malaysian Indian.\n\n \n\nKG2: < Malaysia, capital, Kuala Lumpur>, < Malaysia, ethnicGroup, Malaysian Malay>, < Malaysia, ethnicGroup, Malaysian Indian>\n\n \n\nReference2: Malaysian Indian and Malaysian malay are ethnic groups of Malaysia, where the capital is kuala Lumpur.\n\n \n\nText1: The leader of Malaysia is called jusuf kalla and Malaysian Malay are an ethnic group there.\n\n \n\nText2: Malaysians Malay and Malay Malay ethnic group are both ethnic groups and Kuala Lumpur is the capital of Malasia.\n\n \n\nTextDK1: Abu Zahar Ujang is the leader of Malaysia where Malaysian Malay and Malaysian Indian are ethnic groups.\n\n \n\nTextDK2: Malaysian Malay and Malaysian Indian are ethnic groups in Malaysia where Kuala Lumpur is the capital.\n\n \n\nBased on the results above, it can be observed that without the DK module, the output sometimes fails to capture crucial information and only generates duplicate content when dealing with similar KGs. For instance, Text1 only presents the common information shared between KG1 and KG2 while disregarding the triplet <Malaysia, leaderName, Abu Zahar Ujang>. Additionally, similar entities within a single KG, such as Malaysian Malay and Malaysian Indian in Text2, are not correctly distinguished. After adding the DK module, these problems are solved. Therefore, the contrastive learning approach improves the diversity of generated texts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The experiments demonstrate that the proposed model is slightly better than the baselines.", "reasons_to_reject": "1. This paper used the BART as generation model to train the Planning Scorer while the text generation is another different model (Graph Encoder-Decoder). It is difficult to ensure that the order of the former predictions is appropriate for the latter. There are inconsistency in this process.  \n2. There are some contraditions in Eq. (9). The first question is how do we initialize the R_ij? The second question is that whether R_ij is equal to R_ji in the relative distance matrix R. According to the Eq. (9), the computation process must be iterative until it is unchanged. If not, the node id will influence the final distance. Based on this, the distance of R_ij will be equal to R_ji since there is the minus sign for the second case in Eq. (9). This means that R_ij and R_ji are equal in R. It makes the complex computation in Eq. (9) meanless. \n3. Some key points are missed, eg. the lamda in Eq. (13). It should analysis the effect of the value of lamda through more experiment.\n4. The scale of batch size in Figure 4 seems incorrect.", "questions_to_authors": "1. How do you ensure that the order of the  planning scorer predictions is appropriate for the latter\n2. What the mean of gather(g) in Eq. (10)?\n3. It is better to analysis the effect  of lamda in model performance. \n4. One target of KG-to-Text task is diversity. Do you consider this in your experiments?", "paper_topic_and_main_contributions": "This paper focus on the KG-to-Text task. It proposes structure-aware KG-to-Text generation model with planning selection and similarity distinction. It designs Planning Scorer module to predict the sequence of knowledge triples. It integrates graph structure-aware representation in to KG representation in the graph structure-aware encoder. It also use contrastive loss to distinguish the similar KGs.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "m91L3ltA4s", "reviewer": "Reviewer_wGZN", "summary": "", "questions": "- Can the authors clarify what are the differences between their approach and Schmitt et al., 2020 in the graph encoding using biases? My interpretation is the the both methods (the method proposed in the paper and Schmitt et al., 2020 are the same in spirit) since they use biases to compute path distances between nodes in the graph.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Reviewer wGZN\n\nThank you for the detailed and constructive comments. We address the questions and concerns in the following remarks.\n\n**1. Human evaluation can provide a more accurate assessment of the model's performance. It can also help to identify areas where the model could be improved.**\n\nWe have added human evaluation to the latest version of the paper. We conducted the human evaluation on WebNLG to further evaluate the generated text. Following Chen et al., 2019[1], we adopt human evaluation criteria such as Factual correctness including Supp (counting the number of facts that co-exist in the KG and generated text) and Cont (counting the facts in the generated texts missing from or contradicting with KG), Language naturalness including NF (evaluating the accuracy and fluency of generated sentences). We randomly selected 100 knowledge graphs for human evaluation. Five native English speakers volunteered to score all the 100 knowledge graphs. The following are the results.\n\n| Methods                      | Supp. | Cont. | NF.  |\n| ---------------------------- | ----- | ----- | ---- |\n| Ground_truth                 | 3.94  | 0.13  | 4.85 |\n| JointGT (Ke et al., 2021[3]) | 3.36  | 0.26  | 4.04 |\n| Our model                    | 3.58  | 0.20  | 4.28 |\n\nThe results show that the generated texts of our model are more authentic and consistent with KG than JointGT.\n\n**2. Moreover, my current understanding is that the text generation model is not built on top of pretrained LM, which is surprising to me since all previous state-of-the-art approaches are based using some flavor of PLM.**\n\nCertainly, state-of-the-art models currently rely on the formidable capabilities of PLMs, and our generation model follows suit in this regard. However, there is a generation gap between the graph structure and the sequence structure required by the PLM in KG-to-text tasks. To address this disparity, we employ a BART model that incorporates graph structure-aware modules within the Encoder for text generation. As a result, the PLM-based model is utilized during both training stages, thereby ensures the consistency of order prediction.\n\n**3. Can the authors clarify what are the differences between their approach and Schmitt et al., 2020 in the graph encoding using biases?**\n\nFirstly, our relative distance coding is grounded in the node level, thereby taking into consideration the interconnectivity between nodes and prioritizing the preservation of graph structure-awareness in the generated content. This approach ensures comprehensive coverage and comprehension of KG content in the generated text. In contrast,  the approach proposed by Schmitt et al., 2020[2] encodes at token level, with a focus on the accuracy of the generated text.\n\nSecondly, we have streamlined the process of relative distance encoding, resulting in the elimination of the computation process between tokens within a node. \n\nFinally, we have incorporated relative distance coding into the graph structure-aware module. To the best of our knowledge, no work has been done this way. The results presented  in Table 3 demonstrate the effectiveness of our relative distance coding in improving the performance of the graph structure-aware model.\n\n \n\n[1] Zhiyu Chen, Harini Eavani, Wenhu Chen, Yinyin Liu, and William Y ang Wang. 2019. Few-shot nlg with pre-trained language model. arXiv preprint arXiv:1904.09521.\n\n[2] Martin Schmitt, Leonardo F. R. Ribeiro, Philipp Dufter, Iryna Gurevych, and Hinrich Schütze. 2020. Modeling Graph Structure via Relative Position for Better Text Generation from Knowledge Graphs. CoRR, abs/2006.09242.\n\n[3] Pei Ke, Haozhe Ji, Yu Ran, Xin Cui, Liwei Wang, Linfeng Song, Xiaoyan Zhu, and Minlie Huang. 2021. JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphs. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pages 2526-2538.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed model achieved strong performance on two KG-to-text datasets, as measured by automatic metrics. \n- The results indicate that the contrastive learning approach that uses negative entities is effective on entity coverage.", "reasons_to_reject": "- Since the reported numbers for the automatic metrics are close to previous approaches, it would be very valuable to execute human evaluation in order to validate the improvements that the model made. Automatic metrics are not always reliable, as they can be fooled by certain techniques. Human evaluation can provide a more accurate assessment of the model's performance. It can also help to identify areas where the model could be improved.\n\n- The proposed graph aware attention is very similar to the proposed approach by Schmitt et al. 2020 which reduces the novelty of the approach (please see questions for the authors). Moreover, my current understanding is that the text generation model is not built on top of pretrained LM, which is surprising to me since all previous state-of-the-art approaches are based using some flavor of PLM.", "questions_to_authors": "- Can the authors clarify what are the differences between their approach and Schmitt et al., 2020 in the graph encoding using biases? My interpretation is the the both methods (the method proposed in the paper and Schmitt et al., 2020 are the same in spirit) since they use biases to compute path distances between nodes in the graph.", "paper_topic_and_main_contributions": "This paper proposes an encoding mechanism for encoding the structure of the input graph for the task of graph(KG)-to-text generation. This approach encodes the distances between nodes in the graph using a relative encoding bias.\n\nBesides the graph encoding, the paper proposes a planning scorer with the goal of better linearizing the KG triples for generation and a contrastive learning loss which uses as negative examples linearized triples with replaced entities.\n\nThe method is evaluated on two KG-to-text benchmarks (Weblog and DART) and the presented results show strong performance (measured by automatic text generation metrics) compared to previous approaches. Ablation studies show that each proposed component (planning scores, relative distance encoding and contrastive learning) helps the final performance even though the gains are relatively small.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- Structural Adapters in Pretrained Language Models for AMR-to-Text Generation (Ribeiro et al., EMNLP 2021)\n- Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation (Moryossef et al., NAACL 2019)"}}, {"review_id": "IbwLDNcdvG", "reviewer": "Reviewer_uUx9", "summary": "", "questions": "- section 3.2.2 : For training the plan evaluator, do I understand correctly that both BARTs in Fig. 2 (both violet and green boxed ones) are kept static and that only the FNN is trained during that stage?\n- And does the violet BART in Fig. 2 receive the candidate linearized sequence as input to generate some text, or is it the reference? The figure and also the text (L.277) suggest the latter, but it doesn't make sense to me. The planning scorer is about predicting the BLEU score the generated text from BART would receive if the candidate linearized sequence was its input, no? So shouldn't the linearized sequence be the input for both of them? (obviously, the reference is needed to compute the \"gold\" BLEU score, but my confusion is about what is fed to the neural models)\n- L330: If q denotes the adjacency matrix, should it be $q\\in \\{0,1\\}^{m\\times m}$? I mean, is q any real value or only either 1 or 0, depending on whether there is a connection or not? If it can be something else than 1/0, is it a learned parameter?\n- L343: Could you explain the meaning of gather in equation 10? Is this to redistribute the m\\times d graph representations X onto the n\\times d token representations V?\n- L386: \"one to five texts\" Do you mean \"one to five sentences\"? As far as I know, WebNLG only has one reference text per input KG, no?\n- Given that WebNLG has very few disconnected nodes in the input KGs (as far as I know), why did you not evaluate on some other dataset like AGENDA? There, the positive effects of your relative distance encoding in the graph structure-aware module might have been even larger.\n- I don't quite understand the meaning of \"linearized KG\" in table 6. It shows the order of entities. Fig. 1, however, suggests that the linearized sequence is rather determined by the order of triples, i.e., entities need to be repeated if necessary. In table 6, it looks like every entity occurs only once in a linearized sequence. Could you clarify?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Reviewer uUx9\n\nThank you for the detailed and constructive comments. We address the questions and concerns in the following remarks. \n\n**1. section 3.2.2 : For training the plan evaluator, do I understand correctly that both BARTs in Fig. 2 (both violet and green boxed ones) are kept static and that only the FNN is trained during that stage?**\n\nIndeed, during the training stage of the planning scorer, we maintain the BART parameters as fixed and solely train the FNN component. This approach aims to produce a planning scorer based on BART that aligns consistently with the model employed for subsequent text generation.\n\n**2. And does the violet BART in Fig. 2 receive the candidate linearized sequence as input to generate some text, or is it the reference?**\n\nThank you for your careful review. In Figure 2, the candidate linearization sequence is inputted into both BART models. The violet BART model utilizes the reference during the computation of BLEU. We have made revisions to clarify the confusing aspect of Figure 2.\n\n**3. Is q any real value or only either 1 or 0, depending on whether there is a connection or not?**\n\nIndeed, q denotes the adjacency matrix, serving the purpose of discerning the interconnectivity among nodes. A value of 1 in q indicates a connection between nodes, while a value of 0 signifies no connection. Further elaboration on the particulars of this matrix will be provided in the manuscript.\n\n**4. Could you explain the meaning of gather in equation 10?**\n\nAs you mentioned, the gather() operation in equation 10 redistributes the m-dimensional node representation X onto the n-dimensional token representation. Equation 10 illustrates the expansion of the m-dimensional node representation to an n-dimensional token representation, which is subsequently processed in the residual layer to derive the final token representation.  To provide a comprehensive understanding of this process, we will add descriptions in the manuscript.\n\n**5. \"one to five texts\" Do you mean \"one to five sentences\"? As far as I know, WebNLG only has one reference text per input KG, no?**\n\nThe WebNLG dataset we used is from Ke et al., 2021[1] and may have version differences. Each of these knowledge graphs contains one to five references. We randomly select one of them as the target reference during preprocessing.\n\n**6. Given that WebNLG has very few disconnected nodes in the input KGs (as far as I know), why did you not evaluate on some other dataset like AGENDA?**\n\nThe entities within the AGENDA knowledge graph exhibit limited connectivity, resulting in the majority of computed relative distance encoding values approaching infinity. Consequently, the relative position encoding function fails to effectively operate. Additionally, when dealing with large knowledge graphs, the planning scorer encounters inefficiencies in enumerating all potential linearized sequences, thereby hindering the training process. Instead of using the AGENDA dataset, we conducted experiments on the DART dataset with a relatively large knowledge graph and achieved good results.\n\n**7. I don't quite understand the meaning of \"linearized KG\" in table 6.**\n\nThe linearized KG is determined based on the ordering of the triples, and the specific results of this linearization are presented in Figure 1. In Table 6, we employ abbreviations to succinctly represent the linearized KG, with the aim of visually demonstrating the ordered sequence of entities.\n\n\n\n[1] Pei Ke, Haozhe Ji, Yu Ran, Xin Cui, Liwei Wang, Linfeng Song, Xiaoyan Zhu, and Minlie Huang. 2021. JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphs. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pages 2526-2538.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- the paper is overall well written and clearly understandable\n- the presented model is well motivated and novel\n- combining the writing capabilities of pretrained LMs with graph-aware encoders is an important and exciting research direction\n- convincing experimental results, including an ablation study showing the benefits of the new model's different components\n- stand-alone evaluation of the input preprocessing model (planner), and a study on entity coverage of the generated texts", "reasons_to_reject": "No big reasons to be found. If you want to be picky, you could highlight the rather small effects shown in the ablation study, which puts a little doubt on the importance of the new modeling architecture. More experiments, e.g., averaging the results for different random seeds, might mitigate these concerns, showing the robustness and consistently improved performance of the new model in a more convincing manner but I would not see this as a major reason to reject.", "questions_to_authors": "- section 3.2.2 : For training the plan evaluator, do I understand correctly that both BARTs in Fig. 2 (both violet and green boxed ones) are kept static and that only the FNN is trained during that stage?\n- And does the violet BART in Fig. 2 receive the candidate linearized sequence as input to generate some text, or is it the reference? The figure and also the text (L.277) suggest the latter, but it doesn't make sense to me. The planning scorer is about predicting the BLEU score the generated text from BART would receive if the candidate linearized sequence was its input, no? So shouldn't the linearized sequence be the input for both of them? (obviously, the reference is needed to compute the \"gold\" BLEU score, but my confusion is about what is fed to the neural models)\n- L330: If q denotes the adjacency matrix, should it be $q\\in \\{0,1\\}^{m\\times m}$? I mean, is q any real value or only either 1 or 0, depending on whether there is a connection or not? If it can be something else than 1/0, is it a learned parameter?\n- L343: Could you explain the meaning of gather in equation 10? Is this to redistribute the m\\times d graph representations X onto the n\\times d token representations V?\n- L386: \"one to five texts\" Do you mean \"one to five sentences\"? As far as I know, WebNLG only has one reference text per input KG, no?\n- Given that WebNLG has very few disconnected nodes in the input KGs (as far as I know), why did you not evaluate on some other dataset like AGENDA? There, the positive effects of your relative distance encoding in the graph structure-aware module might have been even larger.\n- I don't quite understand the meaning of \"linearized KG\" in table 6. It shows the order of entities. Fig. 1, however, suggests that the linearized sequence is rather determined by the order of triples, i.e., entities need to be repeated if necessary. In table 6, it looks like every entity occurs only once in a linearized sequence. Could you clarify?", "paper_topic_and_main_contributions": "The paper presents a novel approach to KG-to-text generation that combines graph-structure-aware modules with pretrained language models. It contains a new learnable approach (called planner) to preprocess an input KG for feeding it into a pretrained language model.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Table 4 is missing the dataset (WebNLG) used for evaluation in the caption (would make it easier to read).\n- Previous studies using pretrained language models for KG-to-text generation identify hallucinations as a big problem. PLMs tend to rather parrot back facts from their pretraining texts than stay truthful to the input KG. It would be interesting to see how your improved model fares on the challenge of hallucinations. Especially, the impact of contrastive learning would be interesting to investigate."}}]

Author rebuttal:

Meta-review: The paper presents a new framework for knowledge graph to text generation (KG-to-text) that combines graph-structures modules with pretrained language models (PLMs). A planner is also proposed to fuse the input knowledge graph to the PLMs. Although concerns are raised about ablation study for contribution of different components and novelty. Overall, the paper is well written with strong empirical results. Incorporating graph-aware encoders with PMLs is an exciting direction and it can be beneficial to a broader community.
Decision: Accept-Main