SlowGuess/NAACL_2022_OCR huggingface dataset:
Since we didn't use the position embedding in the input of the Transformer encoder, it implicitly performs cross attention between graph and description. The loss function is finally defined as:
$$
\mathcal {L} _ {2} = - \frac {1}{| \mathbf {D} _ {\text {t r a i n}} |} \sum_ {(D, S) \in \mathbf {D} _ {\text {t r a i n}}} \sum_ {S ^ {i} \in S} \frac {\log P (S ^ {i} | \mathbf {V})}{| S |}. \tag {10}
$$
4.2 Text2Graph
For Text2Graph, we studied link prediction and node classification.
4.2.1 Link prediction
To predict the edge type between node  $u$  and node  $v$  on graph  $G$ , we used the node embedding  $\pmb{h}_u$ , node embedding  $\pmb{h}_v$  and the graph description  $S$  as the input features. We first define the edge feature  $\pmb{w}_{u,v}$  and the graph description feature  $\langle \pmb{s}_i^j\rangle$  as:
$$
\begin{array}{l} \langle \boldsymbol {s} _ {i} ^ {j} \rangle = \operatorname {B i o B E R T} (S _ {i}) \mathbf {W}, \tag {11} \\ \boldsymbol {w} _ {u, v} = \left[ \boldsymbol {h} _ {u} \right| \left| \boldsymbol {h} _ {v} \right]. \\ \end{array}
$$
Then we use the same attention mechanism as in Equation. 8 to obtain a new embedding  $\pmb{h}$  from these two features and define the predicted distribution  $P(\hat{r}_{u,v}|e_{u,v})$  for edge type  $r$  as:
$$
\begin{array}{l} \boldsymbol {h} = \operatorname {A t t e n t i o n} \left(\boldsymbol {w} _ {u, v}, \langle \boldsymbol {s} _ {i} ^ {j} \rangle\right), (12) \\ P \left(\hat {r} _ {u, v} | S\right) = \operatorname {s o f t m a x} \left(\mathrm {M L P} \left(\left[ h _ {u} \right| \left| h _ {v} \right| \left| h \right]\right)\right). (12) \\ \end{array}
$$
Here, MLP is a multi-layer perceptron. The final training loss is defined as:
$$
\mathcal {L} _ {3} = - \frac {1}{| \mathbf {D} _ {\text {t r a i n}} |} \sum_ {(G, S) \in \mathbf {D} _ {\text {t r a i n}}} \sum_ {e _ {u, v}} \frac {P \left(r _ {u , v} \mid S\right)}{\left| \left\{e _ {u , v} \right\} \right|}. \tag {13}
$$
4.2.2 Node classification
To classify a test node  $v$ , we applied a similar attention mechanism on its node embedding  $\pmb{h}_v$  and graph description feature  $\langle \pmb{s}_i^j\rangle$  as:
$$
\begin{array}{l} \langle \boldsymbol {s} _ {i} ^ {j} \rangle = \operatorname {B i o B E R T} (S _ {i}) \mathbf {W}, \tag {14} \\ \boldsymbol {h} = \operatorname {A t t e n t i o n} \left(\boldsymbol {h} _ {v}, \left\langle s _ {i} ^ {\prime} \right\rangle\right). \\ \end{array}
$$
We then define the predicted label distribution and loss function accordingly as:
$$
\begin{array}{l} P \left(\hat {t} _ {v} | S\right) = \operatorname {s o f t m a x} (\operatorname {M L P} \left(\left[ h _ {v} \right| \mid \boldsymbol {h} ]\right)), \\ \mathcal {L} _ {4} = - \frac {1}{| \mathbf {D} _ {\text {t r a i n}} |} \sum_ {(G, S) \in \mathbf {D} _ {\text {t r a i n}}} \sum_ {v} \frac {P (t _ {v} | S)}{| \{v \} |}. \tag {15} \\ \end{array}
$$
We exclude any pathway that is a subgraph of another pathway in all experiments to avoid data leakage. For Graph2Text, we randomly split the graph description pairs into  $75\%$  training pairs and  $25\%$  test pairs. We used a fixed Transformer encoder in BioBERT and initialized the GNN with xavier initialization. We used a learning rate 5e-5. We found that this method performed better than using a fixed Transformer and warming GNN before the training. We used GAT (Velicković et al., 2018), GCN (Kipf and Welling, 2017) and GIN (Xu et al., 2019) as different graph encoders. The hidden state embedding dimension was set to 128 for GAT and 512 for others. The number of heads of GAT was set as 4. AGG and UPDATE functions were implemented according to the original papers. Global mean pooling was used as the READOUT function. Since Transformer can hardly generate more than 512 tokens, we calculated the loss functions and evaluated the generation only on the first 3 sentences, which have an average token length  $69 \pm 23$  (maximum token length is 471). However, the entire text was used as the input in all tasks through the attention mechanism, and we set the attention head number  $A = 128$ . We set  $k$  to 1 in the  $k\mathrm{NN}$  framework. We focused on the 1,173 pathway from Pathbank (DS et al., 2020) in our experiments.
For Text2Graph node classification, we randomly split the graph and description pairs into  $75\%$  training pairs and  $25\%$  test pairs. We sampled  $10\%$  nodes as the test node in each graph. In Text2Graph link prediction task, we varied the proportion of the test set (10%, 30%, 50%, 70%, 90%). We sampled  $40\%$  edges for each graph and the same number of edges from the complementary graph as the test edge. In link prediction and node classification, we only used GAT since it obtained the best performance in Graph2Text. We set the learning rate to 5e-4. We used Adam optimizer for all optimizations.
In Graph2Text task, we compared our methods to supervised graph neural network which jointly trains a graph neural network and a transformer. We denote them as GNN (des.), GNN (label), GNN (des. + label) and GNN(structure only) based on the node features used. In particular, GNN (des.) uses textual description as node feature. GNN (label) uses textual label as the node feature. GNN (des. + label) uses both textual label
Figure 4: Performance of our method on Graph2Text and Text2Graph link prediction. a, Bar plot comparing our method and baselines using different graph neural network architectures on Graph2Text. b, Scatter plot comparing the F1 score of using the graph structure to the F1 score of without using the graph structure. Each dot is one edge type. c, Scatter plot comparing the F1 score of using the graph description to the F1 score of without using the graph description. Each dot is one edge type.
Method	BLEU1	BLEU2	BLEU3	METEOR	NIST	ROUGE-L
GNN (structure only)	14.3	2.2	0.9	12.1	0.8	19.4
GNN (des.)	18.7	2.5	0.9	11.9	1.1	16.6
GNN (label)	21.4	4.2	1.3	13.2	1.2	17.1
GNN (des. + label)	27.1	11.9	10.8	20.5	1.9	23.9
kNN-Transformer	26.8	12.3	10.6	20.4	1.9	24.3
kNN-Graph2Text (Ours)	29.6	13.8	11.4	23.0	2.2	24.4
Table 1: Comparison on Graph2Text using different metrics.
and description as the node feature. We also compared to a kNN-Transformer model which trained a transformer using descriptions of similar graphs to the final description. Different GNN architectures are used to identify nearest neighbors in kNN based on the graph information.
5.2 Graph2Text
We sought to evaluate the performance of our method on the task of Graph2Text (Figure 4a, Table 1). Overall, we found that our method achieves the best performance on all metrics (0.296 BLEU-1 score, 0.230 METEOR, 2.2 NIST, and 0.244 ROGUE-L), demonstrating the effectiveness of jointly modeling graph structure, node description and node label. We first compared our method to graph neural network, which performed the first step of our framework and used concatenated node embeddings instead of single graph embedding as the input to Transformer. We observed substantial improvement over it on all three kinds of graph neural networks, indicating the importance of retraining using descriptions of similar graphs. We also observed that our method was better than kNN-Transformer, reflecting how our description-guided graph embeddings enhance the description generation.
To further understand the importance of each type of node feature, we evaluate the variants that only consider node description or node textual la
bel (Figure 4a). We found that the performance of both variants dropped substantially, demonstrating the importance of both node textual label and node description. We further observed that the improvement of our method was consistent when using other graph neural network architectures, including GIN and GCN, demonstrating the robustness of our method. When replacing GAT to a multi-layer perception that cannot model the graph structure, the BLEU score of our method dropped substantially from 0.296 to 0.187, again confirming the necessity of considering the graph structure in this task.
5.3 Text2Graph
We next investigated the performance on the task of Text2Graph. Here, we studied two classic graph prediction tasks: link prediction and node classification. We summarized the performance of link prediction in Figure 5a. We obtained an average of 0.781 accuracy score across 8 different edge types, demonstrating an accurate prediction of the graph structure using the graph description. We further examined the effect of using the graph description in Figure 4c and observed that all 8 edge types had better F1 score when the graph description was used. We observed the same improvement of using the graph description when evaluated using the accuracy. We also performed the ablation study for the graph structure and observed similar improvement in Figure 4b. These results collectively confirm that our method can generate the graph structure based on the graph description, offering biologists novel insights in pathway analysis.
We then studied the performance of node classification. We considered three most frequent node types in our dataset: macromolecule, multimer
Figure 5: Performance on Text2Graph link prediction, node classification and named entity recognition. a, Bar plot showing the ablation studies on using the graph description and using the graph structure on link prediction. b, Box plot showing the comparison between using the graph description and without using the graph description on node classification. c, Bar plot showing the performance of named entity recognition on chemical and protein on our dataset.
and chemical. For each node type, we formed the node classification task as a multi-class classification problem, where each test node is classified into a specific class defined by the textual label. We noticed that each node type has a large number of classes. Therefore, we first evaluated two naive baselines: random guess and majority vote. Random guess obtained 0.0009 average accuracy, while majority vote obtained 0.046 average accuracy, suggesting a challenging classification task. Our method obtained a desirable classification performance, which was substantially higher than the performance of the variant that does not consider the graph description (Figure 5b). The improvement of using graph description on both node classification and link prediction further confirm that our dataset could be a promising benchmark for Text2Graph task.
6 Application to Named Entity Recognition
Named entity recognition (NER) is essential in detecting chemicals, genes, and diseases from biomedical text (Leaman et al., 2016; Luo et al., 2018; Kim et al., 2019; Yoon et al., 2019), and further facilitating downstream bioNLP applications, such as relation extraction(Xing et al., 2020). A major bottleneck in NER is the lack of curated benchmarks since such curation often requires substantial domain expertise. Our dataset Path2wayText can be used as a novel curated benchmark for NER.
Specifically, we used the graph description as the sentences that one wants to perform NER. We then obtained the ground truth entity type of phrases in these sentences according to their curated node types in the graph. Since the graphs, including all
node types, are curated by domain experts, such node types can be used as the ground truth entity types for NER. Here, we focused on two most frequent entity types in our dataset: protein and chemical. We noticed that some phrases in the graph description sentences might also be a protein or chemical, even though they were not curated in the graph. We excluded such phrases in the evaluation in order to maintain the quality of our NER benchmark.
To this end, we obtained the graph-based curation of 8,779 protein entities and 1,621 chemical entities, offering a good complementary to existing biomedical NER datasets (Kim et al., 2003; Smith et al., 2008; Doğan et al., 2014; Krallinger et al., 2015; Li et al., 2016; Wei et al., 2018). To further investigate the performance of our novel NER datasets, we tested a few state-of-the-art biomedical NER methods, including BERN (Kim et al., 2019), CollaboNet (Yoon et al., 2019), Multi-BioNER (Wang et al., 2019), and NeuroNER (Dernoncourt et al., 2017). We observed that NeuroNER obtained the best performance on protein and Multi-BioNER achieved the best performance on Chemical (Figure 5c). Moreover, existing approaches only consider the graph description sentences when labelling entity types. In addition to graph description, our dataset also contains the corresponding graph structure, which has been shown to be critical in graph description generation in our experiments. Therefore, we hypothesize that graph structure might be also helpful in NER, and envision our dataset to be an important resource for benchmarking graph-based NER methods (Radford et al., 2015; Rijhwani et al., 2020; He et al., 2020; Nie et al., 2021).
Graph2Text, which aims at generating a textual description for a structured graph, has attracted attention in different applications. Existing Graph2Text datasets aims to generate text from RDF data (Gardent et al., 2017), knowledge graph (Koncel-Kedziorski et al., 2019; Jin et al., 2020; Cheng et al., 2020; Colas et al., 2021; Wang et al., 2021), street view map (Schumann and Riezler, 2021), Abstract Meaning Representation (AMR) (Banarescu et al., 2013; Marcheggiani and Perez-Beltrachini, 2018; Song et al., 2018; Ribeiro et al., 2019; Zhu et al., 2019; Hajdik et al., 2019; Damonte and Cohen, 2019; Mager et al., 2020; Zhang et al., 2020; Zhao et al., 2020; Fan and Gardent, 2020; Wang et al., 2020), terminology ontology (Liu et al., 2021) and graph-transduction grammars (Belz et al., 2011; Mille et al., 2019, 2020). Our dataset is the first Graph2Text dataset that focuses on biomedical pathway generation. In addition, our dataset has more complicated node features than many existing Graph2Text datasets, where each node in our dataset has a node type, a concise textual label and a detailed textual description.
Text2Graph can be viewed as an information extraction task, which aims at mining structured knowledge from free text. The datasets that are more relevant to our task could be generating a knowledge graph from long document (Kertkeid-kachorn and Ichise, 2017; Bosselut et al., 2019; Kannan et al., 2020; Wu et al., 2020). Many of these existing datasets use automatic annotation to extract the graph information from corpus (Kertkeidkachorn and Ichise, 2017; Bosselut et al., 2019), which might introduce bias from the extraction method. In contrast, graphs in our dataset are either experimentally derived or manually curated, presenting a high-quality complementary to existing Text2Graph datasets.
8 Conclusion and Future work
We have presented a novel dataset Pathway2Text for biomedical pathway description generation. Our dataset contains 2,367 pairs of curated pathway and its associated description. To generate description for biomedical pathways, we have proposed a kNN-Graph2Text approach, which utilizes neighbor's description to enhance the text generation. We have extensively evaluated our method and observed substantial improvement in comparison to conventional graph neural network archi
tectures. Furthermore, we have investigated the reverse task of Text2Graph and illustrated how our dataset can serve as a novel benchmark for biomedical NER.
In addition to Graph2Text, Text2Graph and NER, our dataset can also be used to investigate other important applications. For example, our dataset can be used as a relation extraction benchmark by regarding graph descriptions as sentences and graph edge types as the ground truth relation type. We can also use our dataset to study other graph-based tasks, such as generating node description given the graph structure and the graph description. Another interesting application is to identify the importance of each node in the graph, which has important applications in recommender system and social media. The order of mentions of each node in the graph description can be used to evaluate the node importance since the graph description often starts from the most important node.
From a methodological perspective, we plan to develop semi-supervised approaches to leverage many other biomedical pathways that currently do not have curated description. For example, we can train a Graph Transformer (Cai and Lam, 2020) on these unlabelled pathways and then finetune the model on pathways with graph description. We also want to explore other geometric embedding methods, such as hyperbolic embedding (Cvetkovski and Crovella, 2009) and spherical embedding (Meng et al., 2019, 2020), since biomedical pathways often form a hierarchical structure.
More importantly, our dataset could also open up new venues in biomedical research. Any computational biology tools that utilize biomedical pathways as features in their pipeline can exploit the graph description as additional features. For biomedical pathways that do not have the corresponding description, one can use the description generated by our kNN-Graph2Text as the feature. We envision this will substantially advance a wide range of biomedical research that involves pathway analysis, and our dataset will introduce other new text generation tools developed in the NLP community to broader audience in biomedicine.
This paper is partially supported by National Key Research and Development Program of China with Grant No. 2018AAA0101902 as well as the National Natural Science Foundation of China (NSFC Grant No. 62106008 and No. 62006004).
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffith, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse.
Yael Baran, Akhiad Bercovich, Arnau Sebe-Pedros, Yaniv Lubling, Amir Giladi, Elad Chomsky, Zohar Meir, Michael Hoichman, Aviezer Lifshitz, and Amos Tanay. 2019. Metacell: analysis of single-cellrna-seq data using k-nn graph partitions. Genome Biology, 20(1):206.
Anja Belz, Michael White, Dominic Espinosa, Eric Kow, Deirdre Hogan, and Amanda Stent. 2011. The first surface realisation shared task: Overview and evaluation results. In Proceedings of the 13th European Workshop on Natural Language Generation, pages 217-226.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: commonsense transformers for automatic knowledge graph construction. In ACL, pages 4762-4779.
Samuel Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. pages 10-21.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Deng Cai and Wai Lam. 2020. Graph transformer for graph-to-sequence learning. In AAAI, pages 7464-7471.
Liying Cheng, Dekun Wu, Lidong Bing, Yan Zhang, Zhanming Jie, Wei Lu, and Luo Si. 2020. ENT-DESC: Entity description generation by exploring knowledge graph. In EMNLP 2020, pages 1187-1197.
Anthony Colas, Ali Sadeghian, Yue Wang, and Daisy Zhe Wang. 2021. Eventnarrative: A largescale event-centric dataset for knowledge graph-to-text generation. CoRR, abs/2111.00276.
The UniProt Consortium. 2020. UniProt: the universal protein knowledgebase in 2021. *Nucleic Acids Research*, 49(D1):D480-D489.
Andrej Cvetkovski and Mark Crovella. 2009. Hyperbolic embedding and routing for dynamic graphs. In INFOCOM, pages 1647-1655.
Marco Damonte and Shay B. Cohen. 2019. Structural neural encoders for amr-to-text generation. In *NAACL-HLT* 2019, June 2-7, 2019, Volume 1, pages 3649-3658.
Emek Demir, Michael P. Cary, Suzanne Paley, Ken Fukuda, Christian Lemer, Imre Vastrik, Guanming Wu, Peter D'Eustachio, Carl Schaefer, Joanne Luciano, Frank Schacherer, Irma Martinez-Flores, Zhenjun Hu, Veronica Jimenez-Jacinto, Geeta Joshi-Tope, Kumaran Kandasamy, Alejandra C. Lopez-Fuentes, Huaiyu Mi, Elgar Pichler, Igor Rodchenkov, Andrea Splendiani, Sasha Tkachev, Jeremy Zucker, Gopal Gopinath, Harsha Rajasimha, Ranjani Ramakrishnan, Imran Shah, Mustafa Syed, Nadia Anwar, Ozgün Babur, Michael Blinov, Erik Brauner, Dan Corwin, Sylvia Donaldson, Frank Gibbons, Robert Goldberg, Peter Hornbeck, Augustin Luna, Peter Murray-Rust, Eric Neumann, Oliver Ruebenacker, Matthias Samwald, Martijn van Iersel, Sarala Wimalaratne, Keith Allen, Burk Braun, Michelle Whirl-Carrillo, Kei-Hoi Cheung, Kam Dahlquist, Andrew Finney, Marc Gillespie, Elizabeth Glass, Li Gong, Robin Haw, Michael Honig, Olivier Hubaut, David Kane, Shiva Krupa, Martina Kutmon, Julie Leonard, Debbie Marks, David Merberg, Victoria Petri, Alex Pico, Dean Ravenscroft, Liya Ren, Nigam Shah, Margot Sunshine, Rebecca Tang, Ryan Whaley, Stan Letovsky, Kenneth H. Buetow, Andrey Rzhetsky, Vincent Schachter, Bruno S. Sobral, Ugur Dogrusoz, Shannon McWeeney, Mirit Aladjem, Ewan Birney, Julio Collado-Vides, Susumu Goto, Michael Hucka, Nicolas Le Novere, Natalia Maltsev, Akhilesh Pandey, Paul Thomas, Edgar Wingender, Peter D. Karp, Chris Sander, and Gary D. Bader. 2010. The biopax community standard for pathway data sharing. Nature Biotechnology, 28(9):935-942.
Franck Dernoncourt, Ji Young Lee, and Peter Szolovits. 2017. NeuroNER: an easy-to-use program for named-entity recognition based on neural networks. Conference on Empirical Methods on Natural Language Processing (EMNLP).
Rezarta Islamaj Dogan, Robert Leaman, and Zhiyong Lu. 2014. NCBI disease corpus: a resource for disease name recognition and concept normalization. J Biomed Inform, 47:1-10.
Wishart DS, Li C, Marcu A, Badran H, Pon A, Budinski Z, Patron J, Lipton D, Cao X, Oler E, Li K, Paccoud M, Hong C, Guo AC, Chan C, Wei W, and Ramirez-Gaona M. 2020. Pathbank: a comprehensive pathway database for model organisms. In *Nucleic Acids Res*.