SlowGuess/NAACL_2022_OCR huggingface dataset:
Jin Liu<sup>1</sup>, Chongfeng Fan<sup>1</sup>, Fengyu Zhou<sup>1</sup>, Huijuan Xu<sup>2</sup>
1 Shandong University
2 Pennsylvania State University
{202120638,202114753,zhoufengyu}  $@$  mail.sdu.edu.cn, hkx5063@psu.edu
The knowledge graph (KG) stores a large amount of structural knowledge, while it is not easy for direct human understanding. Knowledge graph-to-text (KG-to-text) generation aims to generate easy-to-understand sentences from the KG, and at the same time, maintains semantic consistency between generated sentences and the KG. Existing KG-to-text generation methods phrase this task as a sequence-to-sequence generation task with linearized KG as input and consider the consistency issue of the generated texts and KG through a simple selection between decoded sentence word and KG node word at each time step. However, the linearized KG order is commonly obtained through a heuristic search without data-driven optimization. In this paper, we optimize the knowledge description order prediction under the order supervision extracted from the caption and further enhance the consistency of the generated sentences and KG through syntactic and semantic regularization. We incorporate the Part-of-Speech (POS) syntactic tags to constrain the positions to copy words from the KG and employ a semantic context scoring function to evaluate the semantic fitness for each word in its local context when decoding each word in the generated sentence. Extensive experiments are conducted on two datasets, WebNLG and DART, and achieve state-of-the-art performances. Our code is now public available<sup>1</sup>.
Knowledge graphs (KGs) record the common sense knowledge in a structural way and have many potential applications, e.g., question answering (Saxena et al., 2020), recommendation system (Wang et al., 2021) and storytelling (Xu et al., 2021). One typical KG is shown in Figure 1, with circle nodes indicating entities, and the directional edges connecting the head node to the tail node and representing the relation among connected entities. This
Figure 1: One example KG from DART dataset (Nan et al., 2020). AWH is short for AWH Engineering College. Text a) is the ground-truth sentence, text b) is the POS sequence obtained by NLTK, and most words copied from the KG are nouns. The linearized order of KG should correlate with the information sequence expressed in the corresponding sentence (words indicated in bold in GT sentence), and each decoded word should fit in its local semantic context (e.g. the blue dashed lines).
structural representation in KG is easy for information storage while not convenient for human understanding. In this paper, we focus on the task of KG-to-text generation, which aims to describe an input KG with fluent language sentences in an easy-to-understand way. Compared to the traditional text generation task, KG-to-text generation poses the extra challenge of maintaining the word authenticity in the generated sentence given the input KG. With the word authenticity and sentence fluency in mind, existing KG-to-text generation methods (Ribeiro et al., 2020a; Koncel-Kedziorski et al., 2019; Gardent et al., 2017) phrase this task as a sequence-to-sequence generation task, where the KG is linearized as input sequence and decoded into sentences, and the word from the KG are selected to be inserted into the decoded sentence with predicted confidence at each time step. However, when the KG is linearized into a sequence, simple heuristic search-based algorithms are usually utilized, e.g., breadth-first search (BFS) or using other pre-defined rules for sorting (Li et al., 2021;
'https://github.com/LemonQC/KG2Text
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1278 - 1291 July 10-15, 2022 ©2022 Association for Computational Linguistics
Ribeiro et al., 2020a), without considering the word sequence information in the ground-truth sentences. The order inference of the KG is not tightly correlated to the word sequence information in the ground-truth sentences and is generated in a disjoint prestige. The decoded sentence is further conditioned on the linearized KG order which might incur cascaded errors (Cornia et al., 2019). To tackle this problem, we extract the order information from the ground-truth sentence and use this order information to directly supervise the KG order prediction with graph structural local context. In this case, our order prediction component for KG encodes the sentence sequence prior information and will benefit the sentence generation in the follow-up stage.
Also, most existing methods (Li et al., 2021; Koncel-Kedziorski et al., 2019) maintain the word authenticity by maximizing the copy probability of the tokens from the KG, while ignoring the syntactic correctness and semantic relevance. A simple observation from the example in Figure 1, you can find that the POS tags of most words copied from the KG are nouns. Motivated by this observation, we introduce a POS generator to guide the sentence generation process by applying the POS information as additional supervision at each time step and limit the position scope of the word selection from KG. Moreover, to further enhance semantic relevance of generated sentence, we consider the structural information and local semantic information of the KG by designing a semantic context scoring function with sliding windows of different sizes, and combine the semantic context score into the word selection process at each time step of the sentence generation.
In summary, we propose a Syntax controlled KG-to-text generation model with Order and Semantic Consistency, called S-OSC. The main contributions are summarized as follows:
- We propose a learning-based sorting network to obtain the optimal KG description order with graph structural context for more fluent caption generation.
- We enhance the authenticity of generated sentences to the KG through syntactic and semantic regularization. POS tag information is incorporated into the sentence modeling and helps to determine the word selection from KG, together with one additional semantic context scoring function.
- Extensive results on two benchmark datasets indicate that our proposed S-OSC model outperforms previous models and achieves new state-of-the-art performance.
2.1 KG-To-Text Generation
KG-to-text generation task has been a hot research topic since the first dataset WebNLG was proposed (Gardent et al., 2017). Recent works for solving this task have two main categories. One category is using graph neural networks (Marcheggiani and Perez-Beltrachini, 2018; Ribeiro et al., 2020b; Li et al., 2021; Cheng et al., 2020) or graph transformers (Koncel-Kedziorski et al., 2019) to directly capture the graph structure information and decode into sentences. E.g., the recent work (Ribeiro et al., 2020b) forms four different encoder architectures for combining local and global node contexts. ENTDESC (Cheng et al., 2020) introduces multi-graph structure to better aggregate knowledge information. The other category is first linearizing the KG (Ribeiro et al., 2020a; Yang et al., 2020; Gardent et al., 2017; Hoyle et al., 2020) and then formulating a sequence-to-sequence generation task with lineared KG nodes as input to generate sentences. E.g., Distiawan et al. (Distiawan et al., 2018) utilize a fixed tree traversal order to directly flatten the KG into a linearized representation. The structural information of the KG is not preserved when generating the KG order. In this paper, we follow the second general pipeline with local structure encoded in the order generation process.
2.2 Knowledge Graph Order Generation
A line of KG-to-text models (Ribeiro et al., 2020a; Braude et al., 2021) tries to generate sentences conditioned on the KG order, where the order generation is especially important as different KG description orders may result in various generated texts. Most previous works focus on graph traversal-based approaches (Flanigan et al., 2016; Gardent et al., 2017; Ke et al., 2021; Li et al., 2021) for KG order generation. (Li et al., 2021) proposes to use a relation-biased breadth first search (RBFS) strategy to linearize the KG. These previous graph traversal-based approaches are heuristic without considering the word sequence information in the ground-truth sentences. Inspired by previous work (Cornia et al., 2019) which generates image caption with optimal object description order, we extract the sequence
information from the ground-truth sentences as supervision and train one order prediction module to generate optimal order. Moreover, our order prediction considers the local graph structure in triplet.
2.3 Captioning with POS Tags
POS tags have been used in various text generation (e.g. image captioning and video captioning) to impose the syntactic constraint. In the neural text generation work (Yang and Wan, 2021), the authors propose to use POS guided softmax function as the linguistic prior information for modeling the posterior probabilities of next-POS and next-token, in order to increase text generation diversity. In the image caption, Bugliarello et al. (Bugliarello and Elliott, 2021) claim that incorporating POS tag information in the sentence generation process consistently improves the quality of the generated text. In video captioning, Hou et al. (Hou et al., 2019) propose to define the templates of POS tag sequences to represent the syntactic structure of the generated text. In KG-to-text generation task, we not only use the POS tags to ensure the syntactic correctness of the generated text, but also use the POS tags to constrain the positions to copy words from KGs.
2.4 Pre-Trained Language Models
Pre-trained language models (PLMs) on massive corpora, such as BERT (Devlin et al., 2018), BART (Lewis et al., 2019) and T5 (Raffel et al., 2019), have achieved superior performance in various natural language generation tasks, including KG-to-text generation task (Ribeiro et al., 2020a; Peters et al., 2019; Ke et al., 2021). Ribeiro et al. (Ribeiro et al., 2020a) leverage the generation ability of PLMs and use the linearized KG as input to generate texts. We also use the PLM in our method to guarantee the model generalization ability, and at the same time, design extra order prediction and context scoring components to maintain the semantic consistency.
3 Approaches
In this section, we first formulate the KG-to-text generation problem setting and then elaborate the proposed S-OSC in detail.
$$
\begin{array}{l} \text {G i v e n t h e p u t i n G K G , w h i c h i s c o m p o s e d o f} \\ \{(h _ {1}, r _ {1}, t _ {1}), \dots , (h _ {n}, r _ {n}, t _ {n}) | h _ {*}, t _ {*} \in \mathcal {E}, r _ {*} \in \end{array}
$$
$\mathcal{R}\}$ , where  $\mathcal{E}$  denotes the entity set and  $\mathcal{R}$  represents the relation set, the KG-to-text generation task aims to generate a fluent and reasonable text sequence  $\mathcal{T} = < t_1, t_2, \dots, t_k > (t_k \in \mathcal{V})$ , where  $\mathcal{V}$  denotes the vocabulary. In this paper, we follow the general pipeline (Ribeiro et al., 2020a; Ke et al., 2021; Ribeiro et al., 2021) of linearizing the input KG into sequence  $\mathcal{G}_{linear} = < g_1, g_2, \dots, g_m>$  consisting of  $m$  tokens, and then decoding the KG token sequence into sentences.
3.2 Our Proposed S-OSC Model
We propose S-OSC, illustrated in Figure 2, which consists of two main components: one learning-based sorting network for KG, and one copy or prediction selection module for decoding each word in the sentences. The sorting network generates the optimal description sequence for the input KG. Based on the KG order sequence, the sentence decoder generates each word with a certain probability predicted by the copy or prediction selection module to replace the decoded word with the word in the KG. Thus, the model can maintain the word authenticity in the generated sentence compared to the KG. A key innovation in our learning-based sorting network is to utilize the sequence information extracted from the ground-truth sentence to directly supervise the optimal sequence prediction instead of heuristic search in the KG without considering the description sequence prior. Our copy or prediction selection module for decoding each word in the sentences incorporates additional POS syntactic constraint and semantic context consistency scoring function evaluating the semantic fitness of each word in its sliding windows with various sizes. The details of each module in our S-OSC model are illustrated in the following sections.
3.2.1 Sorting Network
The description order of the KG will affect the content of the generated sentence. In a worse scenario, poor order may result in the loss of important information (see the example in Figure 7). To overcome the drawback of disjoint learning for KG order generation and sentence generation in the previous heuristic-based methods (Li et al., 2021; Ribeiro et al., 2020a; Yang et al., 2020), we propose a learning-based sorting network with the order supervision extracted from the ground-truth sentence. Notably, our sorting network is based on the features from the structural Triplet Encoder, where the
Figure 2: S-OSC model architecture. The pre-trained language model takes the KG as input and extracts head-relation-tail triplet structure features through Triplet Encoder. Then, these triplet features are fed into the Sorting Network to generate one optimal description order under the sequence supervision extracted from ground-truth sentence. Conditioned on the KG order, we further decode the sequence into sentences through Word Decoder and apply additional syntactic supervision through POS Generator. To maintain the word authenticity in the generated sentences, we further design one Copy or Prediction component incorporating the POS syntactic information and semantic context scoring with sliding windows to help determine when to copy tokens from KG, in addition to the word decoding probability from the Word-Encoder-Decoder and the binary classification probability for copy or prediction at each time step.
head-relation-tail triplet structure features are extracted through pre-trained KG embedding method TransH (Wang et al., 2014) and pre-trained language model BART (Lewis et al., 2019). Due to the variable length of the KG, we introduce a placeholder to pad it into fixed length  $N$ , which also denotes the number of possible position classes. The head-relation-tail triplet structure features  $F_{stru}$  are concatenated with the padding  $F_{pad}$  and fed through the Fully Connected (FC) layers with softmax classifier  $FC_{s}$  to obtain  $S_{matrix}$  and predict the sorting order  $S_{order}$ .
$$
\begin{array}{l} S _ {\text {m a t r i x}} = \mathrm {F C} _ {s} ([ F _ {\text {s t r u}}; F _ {\text {p a d}} ]) \\ \text {S} _ {\text {s t r u}} (\text {s t r u}) \end{array} \tag {1}
$$
$$
S _ {\text {o r d e r}} = \operatorname {a r g m a x} _ {\text {r o w}} \left(S _ {\text {m a t r i x}}\right)
$$
In this paper, we treat the order prediction task as a classification problem, where  $\mathbf{N}$  denotes the number of classes (the maximum of the triplets in KG). Thus, we measure the cross-entropy loss between the ground-truth order  $G_{order}$  and the sorting order  $S_{order}$ :
$$
L _ {s o r t} = - \sum_ {n = 0} ^ {N} \log \left(S _ {\text {m a t r i x}} ^ {n}\right) \cdot G _ {\text {o r d e r}} ^ {n} \tag {2}
$$
Figure 3: The architecture of the Word-Encoder-Decoder with a POS Generator.
3.2.2 Copy or Prediction Network
To maintain the authenticity of KG words in generated sentences, the model needs to selectively copy words from KG words instead of using the predicted words from Word Decoder. Besides directly predicting the copy probability from the hidden state (Koncel-Kedziorski et al., 2019; Li et al., 2021; See et al., 2017), the Copy or Prediction Network in our S-OSC model further enhances the syntactic and semantic consistency of selected words in the decoded sentence through the incorporation of POS generator (shown in Figure 3) and semantic context scoring (shown in Figure 4). Next, we will introduce each module in detail.
POS Syntactic Constraint Conditioned on the KG order  $G_{order}$ , we first linearize the KG by
Figure 4: The architecture of semantic context scoring module with sliding windows.
adding the tokens  $<$  Head  $>, <$  Relation  $>$ ,  $<$  Tail  $>$  to the corresponding position for each triplet and obtain the  $\mathcal{G}_{\text{linear}}$ . Then, the Word Encoder and the POS Generator take  $\mathcal{G}_{\text{linear}}$  as their inputs and output the word encoding  $\mathrm{WI} = \{w_i, i \in 1 \dots m\}$  and POS tag encoding  $\mathrm{PI} = \{p_i, i \in 1 \dots m\}$ , respectively. Then, the token encoding  $w_i$  and POS tag encoding  $p_i$  are combined in the fusion module to get the updated token encoding  $w_i$ .
$$
w _ {i} = \operatorname {L N} \left(F C ([ w _ {i}; p _ {i} ]) + w _ {i}\right), \tag {3}
$$
where LN denotes the layer normalization. The updated token encoding  $w_{i}$  after fusing is decoded into sentence  $\mathrm{WI}' = \{w_i', i \in 1 \dots k\}$  in Word Decoder.
POS generator is supervised through POS tags pre-extracted from the sentence. The loss function is formulated as:
$$
L _ {p o s} = - \sum_ {l = 1} ^ {M} \log \left(P _ {g e n} \left(p _ {l} \mid p _ {1}, \dots , p _ {l - 1}; G _ {\text {o r d e r}}\right)\right), \tag {4}
$$
where  $P_{gen}$  denotes the predicted probability from POS generator. Similarly, the objective of WordEncoder-Decoder is as follows:
$$
L _ {\text {t o k e n}} = - \sum_ {j = 1} ^ {K} \log \left(W _ {\text {g e n}} \left(w _ {j} ^ {\prime} \mid w _ {1} ^ {\prime}, \dots , w _ {j - 1} ^ {\prime}; \mathrm {W I} ^ {\prime}\right)\right), \tag {5}
$$
where  $W_{gen}$  denotes the predicted probability of each word token.
Semantic Context Scoring Besides the syntactic constraint for copied words, we also design one semantic context scoring component, illustrated in Figure 4, to evaluate the semantic consistency of copied or predicted words in the sliding windows. Sliding windows are generated for each word to provide the local context, e.g., the sliding window
size is set to 3 in Figure 4. Besides, padding is needed for the first several words when forming the sliding windows. Word features in the sliding window are contacted to get the context information  $F_{\text{context}}$ , and are fed into the FC layers to obtain the semantic score  $X_{\text{semantic}}$ .
$$
X _ {\text {s e m a n t i c}} = \sigma (\operatorname {F C} (F _ {\text {c o n t e x t}})), \tag {6}
$$
where  $\sigma$  denotes the sigmoid function.
Word Copy Probability Prediction With our newly introduced POS token embeddings  $v_{p_k}$  and the semantic context score  $X_{semantic}$ , the probability  $p_{copy}^k$  for copying words from KG is computed in Eq. 7, and is used in testing time for final selection between predicted words from Word Decoder and words in KG at each time step when generating sentences.
$$
\begin{array}{l} t _ {c o p y} ^ {k} = \sigma (W _ {1} v _ {w _ {k}} + W _ {2} v _ {p _ {k}} + W _ {3} s _ {k} + b _ {c o p y}), \\ p _ {c o p y} ^ {k} = \lambda \cdot X _ {s e m a n t i c} + (1 - \lambda) \cdot t _ {c o p y} ^ {k}, \tag {7} \\ \end{array}
$$
where  $W_{1}, W_{2}, W_{3}$  and  $b_{copy}$  are learnable parameters.  $v_{w_k}$  represents token embedding and  $s_k$  represents the last hidden state of Word-Decoder at each time step.  $\lambda$  is a trade-off coefficient and is set as 0.3.
The semantic context scoring module is jointly optimized with copy probability prediction and benefits the copy probability prediction. The copy or prediction loss function is defined as:
$$
\begin{array}{l} L _ {c o p y} = - \sum_ {k = 0} ^ {K} \left(y ^ {k} \cdot \log \left(p _ {c o p y} ^ {k}\right) + \left(1 - y ^ {k}\right) \right. \tag {8} \\ \cdot \log (1 - p _ {c o p y} ^ {k})), \\ \end{array}
$$
where  $y^{k}$  is the ground-truth 0-1 label indicating copying or predicting word at  $k - th$  time step, which is generated from KG and the ground-truth sentence (see more details in supplementary material).
Finally, the total training loss  $L_{total}$  in our S-OSC model is composed of four components: sorting loss  $L_{sort}$  (Eq. 2), POS generation loss  $L_{pos}$  (Eq. 4), word generation loss  $L_{token}$  (Eq. 5) and copy or prediction loss  $L_{copy}$  (Eq. 8).
$$
L _ {t o t a l} = L _ {t o k e n} + \lambda_ {1} L _ {p o s} + \lambda_ {2} L _ {s o r t} + \lambda_ {3} L _ {c o p y}, \tag {9}
$$
where  $\lambda_{1},\lambda_{2}$  and  $\lambda_{3}$  are the trade-off coefficients.
In this section, we report the comparison results with state-of-the-art methods and further analyze the performance of each component in our S-OSC model through ablation studies. We also evaluate our model performance through human evaluation and qualitative analysis.
In this paper, two benchmarks: WebNLG (Li et al., 2021) and DART (Nan et al., 2020) are utilized to evaluate the performance of our S-OSC model.
WebNLG WebNLG (Li et al., 2021) is a most widely used dataset in KG-to-text generation task. Each graph is extracted from DBPedia and consists of two to seven triplets. The train/val/test splits are 7362/1389/5427.
DART Compared to WebNLG, DART (Nan et al., 2020) is a larger open-domain dataset, where triples are composed of tree-structured ontology. The train/val/test splits are 30348/2759/5097.
Following the previous works (Ribeiro et al., 2020a,b; Li et al., 2021) on WebNLG dataset, we adopt four automatic language evaluation metrics for WebNLG dataset, i.e., BLEU-4 (Papineni et al., 2002),CIDEr (Vedantam et al., 2015), Chrf++ (Popovic, 2015) and ROUGE-L (Lin, 2004). Following previous works (Nan et al., 2020) on DART dataset, in addition to BLEU-4 (Papineni et al., 2002), we use four additional automatic evaluation metrics, i.e., METEOR (Banerjee and Lavie, 2005),MoverScore (Zhao et al., 2019), BERTScore (Zhang et al., 2019) and BLEURT (Sellam et al., 2020).
In the sorting network, the fixed KG order length  $N$  is set to 8 and 10 for WebNLG and DART, respectively. The POS generator operates on the POS sequences parsed from ground-truth sentences via NLTK, and trains from BART-Base pretrained model (Lewis et al., 2019). For Word-Encoder-Decoder, we follow the code in JointGT (Ke et al., 2021) and utilize Bart-Base with self-attention (Shaw et al., 2018). The beam search size for generating sentences in inference time is set to 5. We optimize all the parameters under the supervision of the total loss in Eq. 9 using the OpenAI AdamW optimizer. The loss weights  $\lambda_{1},\lambda_{2}$
Datasets	WEBNLG
Metrics	B-4	R-L	CIDEr	Chrf++
Li et al.† (Li et al., 2021)	57.10	75.20	4.20	75.00
Li et al. (Li et al., 2021)	61.88	75.74	6.03	79.10
GraphWriter (Koncel-Kedziorski et al., 2019)	45.84	60.62	3.14	55.53
CGE-LW (Ribeiro et al., 2020b)	48.60	62.52	3.85	58.66
T5-Base (Ribeiro et al., 2020a)	48.86	65.57	3.99	66.08
BART-Base (Ribeiro et al., 2020a)	49.81	63.10	3.45	67.65
T5-Large (Ribeiro et al., 2020a)	58.78	68.22	4.10	74.40
BART-Large (Ribeiro et al., 2020a)	52.49	65.61	3.50	72.00
JointGT† (Ke et al., 2021)	57.00	77.10	4.73	76.90
S-OSC(ours)	61.90	79.30	5.30	79.70
S-OSC(ours)-GT	63.10	80.00	5.40	81.00
Table 1: Results for different models on WebNLG dataset. B-4 and R-L are short for BLEU-4 and ROUGE-L, respectively. Bold and underline fonts represent the best and the second best performing results. "S-OSC-GT" indicate our "S-OSC" model with ground-truth order in inference (The same term is applied in the following).  $\dagger$  denotes the results of reproduction. Other cited results are from Li et al. (Li et al., 2021).
and  $\lambda_{3}$  in total training loss (Eq.9) are set to 0.7, 0.4, and 0.3, respectively.
4.4 Main Results on WebNLG and DART