SlowGuess/NAACL_2022_OCR huggingface dataset:
Junwei Yang $^{1,2}$ , Zequn Liu $^{2}$ , Ming Zhang $^{2*}$ , Sheng Wang $^{3*}$
$^{1}$ School of Electronics Engineering and Computer Science, Peking University, Beijing, China
$^{2}$ School of Computer Science, Peking University, Beijing, China
<sup>3</sup>Paul G. Allen School of Computer Science and Engineering, University of Washington, Seattle, WA
yjwtheonly, zequnliu, mzhangCs@pku.edu.cn
swang@cs.washington.edu
Biomedical pathways have been extensively used to characterize the mechanism of complex diseases. One essential step in biomedical pathway analysis is to curate the description of a pathway based on its graph structure and node features. Neural text generation could be a plausible technique to circumvent the tedious manual curation. In this paper, we propose a new dataset Pathway2Text, which contains 2,367 pairs of biomedical pathways and textual descriptions. All pathway graphs are experimentally derived or manually curated. All textual descriptions are written by domain experts. We form this problem as a Graph2Text task and propose a novel graph-based text generation approach kNN-Graph2Text, which explicitly exploited descriptions of similar graphs to generate new descriptions. We observed substantial improvement of our method on both Graph2Text and the reverse task of Text2Graph. We further illustrated how our dataset can be used as a novel benchmark for biomedical named entity recognition. Collectively, we envision our method will become an important benchmark for evaluating Graph2Text methods and advance biomedical research for complex diseases. $^{1}$
Many complex diseases, such as cancer and neurodegenerative disorders, are driven by reactions among a combination of genes and metabolites instead of one single gene (Manolio et al., 2009). These reactions, which are formally referred to as pathways (Kanehisa et al., 2017; DS et al., 2020;
Gillespie et al., 2022), are represented as a heterogeneous graph (Figure 1). Each node in this graph is a biomedical entity, such as gene, chemical or metabolite. Each edge is a specific biomedical reaction. Using natural language to describe this pathway graph is of great importance for scientific communication and further promotes applications in complex disease research (Whirl-Carrillo et al., 2012, 2021). To date, these descriptions are almost entirely curated manually by domain experts, thus substantially slowing down downstream biomedical applications (Naithani et al., 2019). Neural text generation has shown promising results in many applications (Bowman et al., 2016; Sutskever et al., 2014; Song et al., 2020; Brown et al., 2020; Raffel et al., 2020; Lewis et al., 2020). Among them, Graph-to-Text (Graph2Text) generation, such as AMR-to-Text (Song et al., 2018; Marcheggiani and Perez-Beltrachini, 2018; Fan and Gardent, 2020), and Knowledge-Graph-to-Text (Colas et al., 2021; Wang et al., 2021), is most similar to pathway description generation. Therefore, we hypothesize that neural text generation could also be a solution here. To fill in the gap, we first propose a novel biomedical pathway description dataset Pathway2Text, which contains 2,367 pairs of pathway and description. Each description is written by domain experts, describing the function and property of this pathway. In contrast to many other Graph2Text datasets (Banarescu et al., 2013; Colas et al., 2021) that use automatic approach to extract the graph from the text, pathways in our dataset are all experimentally measured or manually curated, presenting a high-quality structured data corresponding to the textual description. To the best of our knowledge, Pathway2Text is the first large-scale dataset studying the problem of biomedical pathway description generation.
One unique feature of our dataset is the rich textual information on each node in the graph. Specif
1Our dataset is available at https://zenodo.org/record/6510039#.Ym9F15NBz0o. Our code is available at https://github.com/yjwtheonly/Pathway2Text.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1441 - 1454
Figure 1: An example of a pathway and its description in our dataset. Each pathway is a heterogeneous graph containing different node types and edge types. Each node has three features: textual label, textual description and node type. For Graph2Text task, the input is the graph and the output is the graph description.
Label	Description	Type
Succinic acid	Succinic acid is a dicarboxylic acid. The anion, succinate, is a component of the citric acid cycle capable of donating electrons to the electron transfer chain. Succinate dehydrogenase (SDH) plays an important role in the mitochondria, being both part of the respiratory chain and the Krebs cycle ...	Chemical
Hypoxia-inducible factor 1-alpha	Functions as a master transcriptional regulator of the adaptive response to hypoxia. Under hypoxic conditions, activates the transcription of over 40 genes, including erythropoietin, glucose transporte-rs, glycolytic enzymes, vascular endothelial growth factor, HILPDA, and other genes whose ...	Macromolecule
...	...	...
Output: description
Succinic acid, or its anion succinate, can leave the mitochondria and can directly inhibit the prolyl 4-hydroxylase subunit alpha-3 protein, which then allows for additional activation of the hypoxia-inducible factor 1-alpha (HF-1α). The higher levels of HF-1α enhance the expression of genes, including those for interleukin-1 beta (IL-1β). Succinic acid is also necessary for the succinylation of proteins, leading to changes in their structure and function.
ically, each node is associated with a node type, a concise textual label and a detailed textual description. In contrast, many other Graph2Text datasets only have a short textual label or a fixed-size feature vector on each node (Belz et al., 2011; Banarescu et al., 2013; Gardent et al., 2017; Jin et al., 2020; Wang et al., 2021). We found that conventional graph neural network architectures were unable to fully exploited these rich node features, resulting in less accurate graph description generation. And the advantages of exploiting similar input data have been demonstrated in many related works (Baran et al., 2019; Khandelwal et al., 2020; Wang et al., 2022). We therefore propose kNN-Graph2Text, which explicitly incorporates descriptions of similar graphs into the definition generation process. In particular, our method first calculates a description-guided graph embedding and then finds similar graphs for a test graph based on these embeddings. After that, the new description is generated by jointly considering the description of neighbors and the graph structure using a multi-head attention framework (Vaswani et al., 2017).
We evaluated kNN-Graph2Text on our dataset and observed substantial improvement over conventional graph neural network architectures as well as methods that do not fully utilize the heterogeneous node features. We next demonstrated that our dataset can be used to study the reverse task of Text2Graph. In particular, we investigated how graph description can enhance the performance of link prediction and node classification, and ob
tained accuracy of 0.781 in link prediction and accuracy of 0.352 in node classification. Moreover, our dataset can be used as a novel benchmark for biomedical named entity recognition by extracting the ground truth entity types according to the annotated node types. Collectively, our dataset and our method present the first study in automatic biomedical pathway description generation. We envision Pathway2Text to be an important benchmark for general Graph2Text methods and facilitate downstream biomedical applications.
2 Pathway2Text Dataset
2.1 Data processing
Our dataset was synthesized from five biomedical databases: Reactome (Gillespie et al., 2022), KEGG (Kanehisa et al., 2017), Pathbank (DS et al., 2020), UniProt (Consortium, 2020) and ChEBI (Hastings et al., 2015). We collected biomedical pathways and their associated textual descriptions from Reactome, KEGG, and Pathbank, and aligned nodes in pathways to entities in UniProt and ChEBI for retrieving missing node descriptions. Specifically, the raw data was processed as follows:
Data format definition. We firstly modified SBGN (Le Novère et al., 2009), a standard export format for biology graphical notation, to organize multiple components in pathway. We followed most of the original definitions in SBGN (e.g., reaction representing format), but we (1) omitted nodes that are not involved in any reaction (e.g., Compartment nodes), (2) reconstructed each Complex node
(a container for other nodes) into a tree structure, by adding additional edges between nodes and their container, (3) merged different nodes referring to the same entity into a single node for each pathway, and (4) rewrote the entire dataset into more readable JSON file (cf. Appendix B for more intuitive explanation of our data format).
Format translation. Reactome and Pathbank already provide SBGN files, making it straightforward to adapt pathways in both database into our data format. But KEGG only provides KGML (Kanehisa et al., 2017) file, a specific representation of KEGG pathways. So we applied additional modifications to translate KGML file into our data format: we (1) used Process node to represent a reaction instead of directly adding edges between substrates and products, (2) treated a Group of proteins acting on the same reaction as a Complex node, and (3) adjusted node types to match our definition. We refer readers to Appendix A for an illustration for these operations.
Node description gathering. Neither SBGN nor KGML file contains detailed node descriptions. SBGN file provides node label (a short text for display), and KGML file provides a KEGG identifier for each node. (1) For Pathbank database, each pathway is also recorded in PWML (DS et al., 2020) format, which contains textual node labels and descriptions. We therefore used node labels given by SBGN file to retrieve node descriptions from PWML file. (2) For Reactome database, each pathway is also stored in BioPAX (Demir et al., 2010) format.  $49.2\%$  nodes in BioPAX file have long descriptions while most of the others are only linked to identifiers in external biological entity databases. Among these databases, the Function attribute in UniProt and the Definition attribute in ChEBI are appropriate to be utilized as complements to node descriptions. So we aligned each node in SBGN file to node in corresponding BioPAX file using node label. And then extracted node descriptions from the union of BioPAX file, UniProt and ChEBI. (3) For KEGG database, each KEGG identifier indicates particular information (stored in a TXT file) of a specific entity. We parsed this file to pick entity name, textual Comment and external database identifiers. We used entity names as node labels, used Comments as node descriptions for entities having this attribute  $(3.7\%)$ , and used identifiers of UniProt and ChEBI to retrieve node descriptions for others.
2.2 Dataset description
After excluding duplicate pathways and pathways that do not have textual description, we finally obtained 2,367 pairs of pathway and description. An example is shown in Figure 1. Each textual description is a few sentences describing functions and structures of the pathway. The textual description has on average  $129.5 \pm 101.4$  words and  $7.6 \pm 5.3$  sentences. Each pathway can be viewed as a heterogeneous graph that contains different types of edges and nodes. There are 7 edge types and 7 node types in the entire dataset, where each pathway has on average  $3.5 \pm 1.4$  edge types and  $4.5 \pm 1.4$  node types. Each node type (e.g., chemical) has a large number of specific classes (e.g., succinic acid). Each class is associated with a concise textual label and a detailed textual description. The average length of the textual description is 114.8 words. We refer to the class description as the node description and the pathway description as the graph description throughout the paper. Each pathway has on average  $61 \pm 52$  nodes and  $75 \pm 80$  edges. In summary, there are four data fields for each pathway description pair: graph description, graph structure, node description and node label.
Figure 2: Scatter plot showing the consistency between graph-based representation similarity and description-based representation similarity. Each dot is a pair of graphs.
To examine the feasibility of conducting Graph2Text and Text2Graph tasks using our dataset, we examined the consistency between graph similarity and description similarity (Figure 2). We used GAT (Velicković et al., 2018) to embed each graph into a dense representation. We also obtained a dense representation for each graph description using BioBERT (Lee et al., 2020). For every two graphs, we calculated one similarity score based on their graph-based representations and an
other similarity score based on their description-based representations. We observed a Pearson correlation 0.35 between these two similarity scores, reflecting a substantial consistency between these two similarity metrics. This indicates that graphs with similar structure tend to have similar textual descriptions, suggesting the possibility to generate textual description using the graph structure and vice versa.
We aim to generate the textual description for a given biomedical pathway graph and generate the biomedical pathway graph from a given textual description. Let  $\mathbf{D} = \{\mathbf{D}_{\mathcal{G}}, \mathbf{D}_{\mathcal{S}}\} = \{(G_i, S_i)\}_{i=1}^N \stackrel{\text{dist}}{\sim} \mathbb{P}(\mathcal{G}, \mathcal{S})$  be a dataset of paired pathway and its textual description. Each pathway is a directed graph  $G = (V, E, F)$ , where  $V$  represents the set of nodes,  $E \subseteq V \times V$  represents the set of edges, and  $F$  represents node features. Since each pathway is a heterogeneous graph, we refer to pathway as graph in this paper.
One unique property of the graphs in our dataset is the rich node features  $F = \{g, t, d\}$ . In particular, each node  $v$  is associated with three features  $\pmb{g}_v, t_v$ , and  $d_v$ .  $\pmb{g}_v \in \{0, 1\}^{n_c}$  is a one-hot vector representing the node type of  $v$ .  $\pmb{g}_v^i = 1$  if node  $v$  is type  $i$ .  $t_v \triangleq \left\langle t_v^1, t_v^2, \ldots, t_v^{|t_v|} \right\rangle$  is the textual label of node  $v$ .  $d_v \triangleq \left\langle d_v^1, d_v^2, \ldots, d_v^{|d_v|} \right\rangle$  is the textual description of node  $v$ .  $t_v^i \in \mathbb{C}$  and  $d_v^i \in \mathbb{C}$ , where  $\mathbb{C}$  is the vocabulary. In practice, the textual label is often a phrase and the textual definition is a few sentences. As a result,  $|d_v|$  is often much larger than  $|t_v|$ . Each edge is associated with an edge type  $r \in \mathbb{R}$ , where  $\mathbb{R}$  is the set of edge types in the dataset. Each graph description is a token sequence defined as  $S \triangleq \left\langle S^1, S^2, \ldots, S^{|S|} \right\rangle$ , where  $S^i \in \mathbb{C}$ .
We use an inductive learning framework in our experiment. The whole dataset  $\mathbf{D}$  is randomly divided into  $\mathbf{D}_{train} = \{(G_i, S_i)\}_{i=1}^{|\mathbf{D}_{train}|}$  and  $\mathbf{D}_{test} = \{(G_i, S_i)\}_{i=|D_{train}|+1}^N$ . For each task, we train our model on  $\mathbf{D}_{train}$  and evaluate its performance on  $\mathbf{D}_{test}$ . Graph  $G$  and textual description  $S$  are always observed for the training data. We define three tasks based on the unobserved information in the test data as follows:
Graph2Text. The input of this task is a graph  $G$ . All node features are observed on this graph. The output is the description text  $S$  for this graph.
Text2Graph link prediction. This task aims to
Figure 3: Flow chart of our two-step approach kNN-Graph2Text. In the first step, we learnt a representation for each graph by projecting graphs to descriptions. In the second step, we find similar graphs for a test graph and jointly use descriptions of similar graphs and node embeddings of the test graph to generate the final description.
predict missing links in a test graph. The inputs are graph description  $S$ , all node features  $F$  and a subset of edges  $\{e\}$  in the graph  $G$ . For a test edge  $e_{u,v} \in V \times V - \{e\}$ , our goal is to classify  $e_{u,v}$  into a specific edge type  $r \in \mathbb{R}$ .
Text2Graph node classification. This task aims to classify each test node into a specific node type in graph  $G$ . We split nodes in  $G$  into training nodes and test nodes. For training nodes, we observed all node features  $F$ , including textual label, textual description and node type, whereas none of these features is observed for the test node. We also observed the graph description  $S$  for  $G$ . Instead of predicting the node type, we aim at predicting the specific textual label, which is a more challenging task. We form this problem as a node classification task instead of textual generation.
4.1 Graph2Text
The overall framework of our method is shown in Figure 3. We propose a two-step approach. In the first step, we embed each graph into a dense representation through jointly considering its graph structure and node features. In the second step, we use the learnt graph embeddings to find similar graphs for each test graph and then leverage the description of these similar graphs to help the generation.
4.1.1 Description guided graph embedding
One unique property of our dataset is the rich textual features on each node. We hypothesize that unsupervised graph embedding methods might be unable to fully exploit these textual features. Therefore, we first use a supervised approach to obtain
graph embeddings. Since we don't have any class label for each graph, we treat the graph description as the pseudo label in the supervised learning framework to embed graphs.
In particular, we learn an encoder Enc that projects the graph  $G$  into a dense representation  $h_G$ , and then a decoder Dec that maps this representation into the textual description  $S$ . The decoder will be discarded in the second step, while the encoder will be used to obtain the representation of an input graph.
Our encoder could be any existing graph neural network architectures (Kipf and Welling, 2017; Velicković et al., 2018; Xu et al., 2019). We first use a pretrained language model BioBERT to encode the textual label  $t_v$  and the description  $d_v$  of each node  $v$  into a dense vector  $t_v$  and a dense vector  $d_v$ , and fuse them to get the initial node embedding for node  $v$ :
$$
\boldsymbol {h} _ {v} ^ {0} = \operatorname {R E L U} \left(\left[ \boldsymbol {t} _ {v} \right| \left| \boldsymbol {d} _ {v} \right] \mathbf {W}\right), \tag {1}
$$
where  $\mathbf{W}$  represents a trainable parameter matrix and  $||$  is the concatenation operation.
We then propagate this embedding on the graph using a chosen graph neural network architecture, which learns representation of node  $v$  through iteratively updating it with neighbors' information  $\pmb{h}_{\mathcal{N}(v)}^l$  as:
$$
\begin{array}{l} \boldsymbol {h} _ {\mathcal {N} (v)} ^ {l} = \operatorname {A G G} \left(\left\{\left(\boldsymbol {h} _ {u} ^ {l - 1}, e _ {u, v}\right) \mid u \in \mathcal {N} (v) \right\}\right), \tag {2} \\ \boldsymbol {h} _ {v} ^ {l} = \operatorname {U P D A T E} (\boldsymbol {h} _ {v} ^ {l - 1}, \boldsymbol {h} _ {\mathcal {N} (v)} ^ {l}), \\ \end{array}
$$
where  $\mathcal{N}_v$  denotes the set of neighbors for  $v$ . AGG and UPDATE are the aggregation and the update function of the specific graph neural network architecture. We studied the performance of using GIN, GCN and GAT as the neural network architecture in our experiments.
After  $L$  iterations, the final embedding  $\pmb{h}_v^L$  can be used to represent the local subgraph comprising node  $v$ 's  $L$ -hop neighbors. Next, for each node, we concatenate its node embeddings from all layers to fuse the information from different ranges of neighbors. We then calculate the graph-level representation by applying a READOUT function to the concatenated node embedding:
$$
\begin{array}{l} \boldsymbol {h} _ {v} = \left[ \boldsymbol {h} _ {v} ^ {1} \| \boldsymbol {h} _ {v} ^ {2} \| \dots \| \boldsymbol {h} _ {v} ^ {L} \right] \mathbf {W}, \\ \boldsymbol {h} _ {G} = \operatorname {R E A D O U T} \left(\left\{\boldsymbol {h} _ {v} \right\} _ {v \in V}\right). \tag {3} \\ \end{array}
$$
Our decoder is a Transformer based on the pretrained BioBERT. It generates textual description conditioned on  $h_G$ :
$$
P \left(\hat {S} ^ {i} \mid \boldsymbol {h} _ {G}\right) = \operatorname {D e c} \left(\boldsymbol {h} _ {G}, S ^ {1, \dots , i - 1}\right). \tag {4}
$$
Finally, the decoder Dec and the encoder Enc are trained jointly using the following loss function:
$$
\mathcal {L} _ {1} = - \frac {1}{| \mathbf {D} _ {\text {t r a i n}} |} \sum_ {(G, S) \in \mathbf {D} _ {\text {t r a i n}}} \sum_ {S ^ {i} \in S} \frac {\log P \left(S ^ {i} \mid \boldsymbol {h} _ {G}\right)}{| S |}. \tag {5}
$$
4.1.2 Exploiting descriptions of similar graphs in generation
The above encoder-decoder framework could already be used to generate the description for a given test graph. However, we observed that such generations were not of great quality in our experiment, partially due to the poor utilization of the node textual features. We thus propose to train a new decoder by leveraging the descriptions of similar graphs.
We first use  $\pmb{h}_{G_i}$  to find  $k$  similar graphs in the training data:
$$
\begin{array}{l} \operatorname {d i s} _ {i j} = \left\| h _ {G _ {i}} - h _ {G _ {j}} \right\| _ {\mathrm {F}} ^ {2}, \\ \bar {S} _ {i} = \left\| _ {G _ {j} \in k \mathrm {N N} \left(G _ {i}\right)} \left(S _ {j}\right), \right. \tag {6} \\ \end{array}
$$
where  $S_{j}$  is the description for  $k$  nearest graphs measured by  $\mathrm{dis}_{ij}$ . We then embed neighbor's description  $\bar{S}_i$  into a dense representation  $\bar{s}_i$  using BioBERT:
$$
\begin{array}{l} \langle \bar {\mathbf {s}} _ {i} ^ {j} \rangle = \operatorname {B i o B E R T} (\bar {S} _ {i}) \mathbf {W}, \tag {7} \\ \bar {s} _ {i} = \operatorname {M a x p o o l i n g} (\langle \bar {s} _ {i} ^ {j} \rangle). \\ \end{array}
$$
Next, we use multi-head attention framework to calculate a new dense representation  $\pmb{v}_s^a$  based on description embedding  $\bar{s}_i$  and  $\langle \bar{s}_i^j \rangle$ , and a new dense representation  $\pmb{v}_g^a$  based on graph embedding  $h_G$  and  $\{h_v\}$  as:
$$
\mathrm {s} ^ {a} (\boldsymbol {u}, \boldsymbol {v} _ {i}, V) = \frac {\exp \left(\mathcal {Q} ^ {a} (\boldsymbol {u}) ^ {\mathrm {T}} \mathcal {K} ^ {a} (\boldsymbol {v} _ {i})\right)}{\sum_ {\boldsymbol {v} _ {j} \in V} \exp \left(\mathcal {Q} ^ {a} (\boldsymbol {u}) ^ {\mathrm {T}} \mathcal {K} ^ {a} (\boldsymbol {v} _ {j})\right)},
$$
[ \text{Attention}^a(\boldsymbol{u}, V) = \text{LeakyReLU}(\sum_{\boldsymbol{v}_i \in V} s^a(\boldsymbol{u}, \boldsymbol{v}_i, V) \boldsymbol{v}_i), ]
$$
\begin{array}{l} \boldsymbol {v} _ {g} ^ {a} = \operatorname {A t t e n t i o n} ^ {a} \left(\boldsymbol {h} _ {G}, \left\{\boldsymbol {h} _ {v} \right\}\right), \\ \boldsymbol {v} _ {s} ^ {a} = \operatorname {A t t e n t i o n} ^ {a} \left(\bar {\boldsymbol {s}} _ {i}, \left\langle \bar {\boldsymbol {s}} _ {i} ^ {j} \right\rangle\right), \tag {8} \\ \end{array}
$$
where  $a \in \{1, \ldots, A\}$  indicates the attention head number.  $\mathcal{Q}^a$  is a projection function mapping a vector to the query space, which is defined as  $\mathcal{Q}^a(\boldsymbol{v}) = \tanh(\boldsymbol{v} \mathbf{Q}^a)$ , where  $\mathbf{Q}^a$  represents a trainable parameter matrix. Similarly, we use  $\mathcal{K}^a$  to map a vector to the key space.
Finally, we concatenate the new graph embedding  $\pmb{v}_g^a$  and new description embedding  $\pmb{v}_s^a$ , and use a pretrained Transformer as the decoder to gen
erate textual content:
$$
\mathbf {V} = \left[ \begin{array}{l l l l l} \boldsymbol {v} _ {g} ^ {1} | | & \dots & | | \boldsymbol {v} _ {g} ^ {A} | | & | \boldsymbol {v} _ {s} ^ {1} | | & \dots | | \boldsymbol {v} _ {s} ^ {A} |, \end{array} \right], \tag {9}
$$
$$
P (\hat {S} ^ {i} | \mathbf {V}) = \mathrm {D e c} (\mathbf {V}, S ^ {1, \dots , i - 1}).
$$