SlowGuess/NAACL_2021_OCR huggingface dataset:
$^3$ In case of a featureless graph, an identity matrix,  $\mathbf{I}$ , replaces  $\mathbf{X}_K$ .
Table 2: Link prediction results for SNOMED and three citation networks. Numbers for SNOMED are obtained from rerunning their code on the dataset. The rest of the results are reported from corresponding papers.
Model	SNOMED	Cora	Citeseer	PubMed
AUC	AP	AUC	AP	AUC	AP	AUC	AP
GAE (Kipf and Welling, 2016)	0.773	0.844	0.914	0.926	0.908	0.920	0.964	0.965
LoNGAE (Tran, 2018)	0.890	0.910	0.954	0.963	0.953	0.961	0.960	0.963
ARVGE (Pan et al., 2018)	0.805	0.864	0.924	0.926	0.924	0.930	0.968	0.971
SCAT (Zou and Lerman, 2019)	0.902	0.918	0.945	0.946	0.973	0.976	0.975	0.972
GIC (Mavromatis and Karypis, 2020)	-	-	0.935	0.933	0.970	0.968	0.937	0.935
EDGE (This work)	0.916	0.944	0.973	0.975	0.974	0.976	0.969	0.968
3.4 Model Discussions
We have proposed a general framework for graph enrichment and embedding by exploiting auxiliary knowledge sources. What we consider as a source of knowledge is a textual knowledge base that can provide additional information about the entities of the original knowledge graph. It is a secondary source of knowledge that supplies new sets of features outside of the existing feature space, which improves the quality of representations.
The proposed graph alignment approach can fully exploit augmented knowledge graph and thus improve the graph embeddings. Although  $a\mathcal{KG}$  is a supergraph of  $\mathcal{KG}$ , its connectivity pattern is different. With the help of our customized loss function for graph alignment, both graphs contribute in the quality of derived embeddings. We will also demonstrate the superiority of our joint embedding approach over the independent graph embedding approach (with only  $a\mathcal{KG}$ ) in the experiments, and we investigate which component of our model contributes more in the final performance in the ablation study in Subsection 4.4.
We design our experiments to investigate effectiveness of different components of EDGE as well as its overall performance. To this end, we aim to answer the following three questions<sup>4</sup>.
Q1 How well does EDGE perform compared to state-of-the-art in the task of link prediction? (Section 4.1)
Q2 How is the quality of embeddings generated by EDGE compared to similar methods? (Sections 4.2 and 4.3)
Q3 What is the contribution of each component (augmentation and alignment) in the overall performance? (Section 4.4)
4.1 Task 1: Link Prediction
To investigate Q1 we perform link prediction on four benchmark datasets, and compare the performance of our model with five relevant baselines. For this task we consider SNOMED and three citation networks. For SNOMED, similar to (Kartsaklis et al., 2018), we select 21K medical concepts from the original dataset. Each entity in SNOMED is a text description of a medical concept, e.g., Nonvenomous insect bite of hip without infection. According to the procedure explained in subsection 3.2, we construct an augmented knowledge graph,  $a\mathcal{KG}$ . Additionally, we consider three other datasets, namely Cora, Citeseer, and PubMed, which are citation networks consisting of 2,708, 3,312, and 19,717 papers, respectively. In all three datasets, a short text accompanies each node which is extracted from the title or abstract of the paper. For these networks, relation is defined as citation and the textual content of the nodes enables us to obtain  $a\mathcal{KG}$ . Cora and Citeseer datasets come with a set of default features. We defer the detailed description of datasets in the supplementary.
In this experiment, for each dataset, we train the model on  $85\%$  of the input graph. Other  $15\%$  of the data is split into  $5\%$  validation set and  $10\%$  as part of the test set (positive samples only). An additional set of edges are produced, equal to the number of positive samples, which does not exist in the graph, as negative samples. The union of positive and negative samples are used as the test set. In all baselines, we test the model on  $\mathcal{KG}$ . We obtain the following values for loss ratios after hyper-parameter tuning:  $\alpha = 0.001$ ,  $\beta = 10$ ,  $\gamma = 1$ . We discuss parameter tuning and explain the small value of  $\alpha$  in Section 4.5.
We provide comparison against VGAE (Kipf and Welling, 2016) and its adversarial variant ARVGE (Pan et al., 2018). Also we consider LoNGAE (Tran, 2018), SCAT (Zou and Lerman, 2019) and
4We plan to release our code upon publication.
Table 3: Node classification results in terms of accuracy for citation networks. TR stands for training ratio and un. and semi. are short for unsupervised and semi-supervised.
Model	Approach	Cora TR=0.5	Citeseer TR=0.03	PubMed TR=0.003
DeepWalk	un.	0.67	0.43	0.65
GCN	semi.	0.81	0.70	0.79
GAT	semi.	0.83	0.72	0.79
LoNGAE	semi.	0.78	0.71	0.79
MixHop	semi.	0.82	0.71	0.81
EDGE	un.	0.81	0.66	0.76
GIC (Mavromatis and Karypis, 2020) which are designed for link prediction task on graphs, hence they make strong baselines. Table 2 presents the Area Under the ROC Curve (AUC) and average precision (AP) scores for five baselines and our methods across all datasets. We observe that EDGE outperforms all baselines in three out of four datasets and produces comparable results for PubMed dataset.
4.2 Task 2: Node Classification on Citation Networks
To evaluate the quality of embeddings (Q2) we design a node classification task based on the final product of our model. For this task, we use Cora, CiteSeer and PubMed datasets, and follow the same procedure explained in 3.2 to generate  $a\mathcal{KG}$  and jointly map the two graphs into an embedding space. All the settings are identical to Task 1. To perform node classification, we use the final product of our model, which is a 160 dimensional vector per node. We train a linear SVM classifier and obtain the accuracy measure to compare the performance of our model with state-of-the-art methods. Training ratio varies across different datasets, and we consider several baselines to compare our results against.
We compare our approach with state-of-the-art semi-supervised models for node classification, including GCN (Kipf and Welling, 2017), GAT (Velicković et al., 2018), LoNGAE (Tran, 2018), and MixHop (Abu-El-Haija et al., 2019). These models are semi-supervised, thus they were exposed to node labels during training while our approach is completely unsupervised. We also include DeepWalk, an unsupervised approach, to have a more complete view for our comparison. Table 3 reveals that our model achieves reasonable performance compared with semi-supervised models in two out of three datasets. Since EDGE
(a) EDGE
(b) GAE
Figure 3: Pair-wise similarity comparison between GAE and EDGE.
(a) feature-based
Figure 4: Visualization of embedding vectors on Cora: a) with and b) without features to study the effect of features on quality of embeddings in node classification.
(b) featureless
is fully unsupervised, it is fair to declare that its performance is comparable as other methods are exposed to more information (i.e., node labels).
4.3 Embedding Effectiveness
Further, to measure the quality of embeddings produced by our model and compare it against the baseline, we visualize the similarity matrix of node embeddings for two scenarios on the Cora dataset: 1) GAE on  $\mathcal{KG}$ , and 2) EDGE on  $\mathcal{KG}$  and  $a\mathcal{KG}$ . The results are illustrated in Figure 3. In this heatmap, elements are pair-wise similarity values sorted by different labels (7 classes). We can observe that the block-diagonal structure learned by our approach is clearer than that of GAE, indicating enhanced separability between different classes.
Next, we examine our model in more details and study how different parameters affect its performance.
To investigate the effectiveness of different modules of our model (Q3), we consider two scenarios. First we use a single graph to train our model. Note that when we use a single graph, the graph alignment and locality preserving losses are discarded and our model is reduced to GAE. In single graph scenario we consider two versions of augmented graph,  $a\mathcal{KG}$  that was explained in subsection 3.2
(a) Cora  $\beta = 1, \gamma = 1$
(b) Cora  $\alpha = 1, \gamma = 1$
(c) Cora  $\alpha = 1, \beta = 1$
Figure 5: Effect of parameterization on link prediction performance
(d) Cora  $\beta = 10, \gamma = 1$
Table 4: Link prediction results for SNOMED dataset. In this table  $a\mathcal{KG}^*$  is the augmented knowledge graph generated using the method explained in (Kartsaklis et al., 2018)
Input	Model	AUC	AP
KG	GAE (Kipf and Welling, 2016)	0.77	0.84
aKG*	GAE (Kipf and Welling, 2016)	0.85	0.88
aKG	GAE (Kipf and Welling, 2016)	0.86	0.90
KG + aKG*	EDGE (This work)	0.90	0.93
KG + aKG	EDGE (This work)	0.91	0.94
and  $a\mathcal{K}\mathcal{G}^*$  that was created based on co-occurrence proposed by (Kartsaklis et al., 2018). In the second scenario, we use two graphs to jointly train EDGE, and we feed our model with  $\mathcal{KG} + a\mathcal{KG}^*$  and  $\mathcal{KG} + a\mathcal{KG}$  to show the effect of augmentation.
For link prediction we only consider SNOMED dataset which is the largest dataset, and as Table 4 presents we observe that our augmentation process is slightly more effective than co-occurrence based augmentation. More importantly, by comparing second two rows with first two rows we realize that alignment module improves the performance more than augmentation process which highlights the importance of our proposed joint learning method. Moreover, we repeat this exercise for node classification (see Table 5) which results in a similar trend across all datasets.
Finally, we plot the t-SNE visualization of embedding vectors of our model with and without features. Figure 4 clearly illustrates the distinction between quality of the clusters for the two approaches. This implies that knowledge graph text carries useful information. When the text is incorporated into the model, it can help improve the model performance.
4.5 Parameter Sensitivity
We evaluate the parameterization of EDGE, and specifically we examine how changes to hyper parameters of our loss function (i.e.,  $\alpha$ ,  $\beta$  and  $\gamma$ ) could affect the model performance in the task of link prediction on Cora dataset. In each analysis, we fix the values of two out of three parameters and study
Table 5: Node classification results in terms of accuracy for citation networks. TR stands for training ratio and  $a\mathcal{KG}^*$  is an augmented knowledge graph produced by the method proposed in (Kartsaklis et al., 2018)
Input	Model	Cora TR=0.5	Citeseer TR=0.03	PubMed TR=0.003
KG	GAE	0.62	0.51	0.60
aKG*	GAE	0.70	0.57	0.65
aKG	GAE	0.75	0.61	0.67
KG + aKG*	EDGE	0.80	0.64	0.73
KG + aKG	EDGE	0.81	0.66	0.76
the effect of the variation of the third parameter on evaluating AUC scores across 200 epochs. The detailed results are shown in Figure 5.
Figure 5a shows the effect of varying  $\alpha$ , when  $\beta = 1$  and  $\gamma = 1$  are fixed. We observe a somewhat consistent trend across performance for different values of  $\alpha$ . It is evident that decreasing  $\alpha$  improves the performance.  $\alpha$  is the coefficient of  $\mathcal{L}_T$  (see Equation 2). This examination suggests that the effect of this loss function is less significant, because we re-address it in the  $\mathcal{L}_N$  part of the loss function, where we consider the same graph ( $a\mathcal{KG}$ ) and try to optimize distance between its nodes but with more constraints.
Figure 5b illustrates the effect of varying  $\beta$ , while  $\alpha = 1$  and  $\gamma = 1$  are fixed. Tuning  $\beta$  results in more radical changes in the model performance, which is again consistent between the two datasets. Small values for  $\beta$  degrades performance remarkably, and we observe a much more improved AUC score for larger values of  $\beta$ . This implies the dominant effect of the joint loss function,  $\mathcal{L}_J$ , which is defined as the distance between corresponding entities of  $\mathcal{KG}$  and  $a\mathcal{KG}$ .
Next, we fix  $\alpha = 1$  and  $\beta = 1$  and tweak  $\gamma$  from 0.1 to 10. As Figure 5c reveals, the variation in performance is very small. Finally, as we obtained the best results when  $\beta = 10$ , we set  $\gamma = 1$  and once again tune  $\alpha$ . Figure 5d shows the results for this updated setting. These experiments confirm the insignificance of parameter  $\alpha$ . In practice, we obtained the best results by setting  $\alpha$  to 0.001.
Sparsity is a major challenge in KG embedding, and many studies failed to properly address this issue. We proposed EDGE, a novel framework to enrich KG and align the enriched version with the original one with the help of auxiliary text. Using external source of information introduces new sets of features that enhance the quality of embeddings. We applied our model on three citation networks and one large scale medical knowledge graph. Experimental results show that our approach outperforms existing graph embedding methods on link prediction and node classification.
This research is supported in part by the U.S. Army Research Office Award under Grant Number W911NF-21-1-0109, and a gift funding from Adobe Research.
Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. 2019. MixHop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In Proceedings of the 36th International Conference on Machine Learning, pages 21-29.
Soren Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. Dbpedia: A nucleus for a web of open data. In *The semantic web*, pages 722-735. Springer, Berlin, Heidelberg.
Olivier Bodenreider. 2004. The unified medical language system (uml5): integrating biomedical terminology. *Nucleic acids research*, 32(suppl_1):D267-D270.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Advances in neural information processing systems, pages 2787-2795.
Andrew Carlson, Justin Betteridge, Bryan Kisel, Burr Settles, Estevam R Hruschka, and Tom M Mitchell. 2010. Toward an architecture for never-ending language learning. In Twenty-Fourth AAAI Conference on Artificial Intelligence.
Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Database. Language, Speech, and Communication. MIT Press, Cambridge, MA.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
Xiaodong Jiang, Pengsheng Ji, and Sheng Li. 2019. Censnet: Convolution with edge-node switching in graph neural networks. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 2656-2662.
Xiaodong Jiang, Ronghang Zhu, Pengsheng Ji, and Sheng Li. 2020. Co-embedding of nodes and edges with graph neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence.
Dimitri Kartsaklis, Mohammad Taher Pilehvar, and Nigel Collier. 2018. Mapping text to knowledge graph entities using multi-sense lstms. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1959-1970.
Thomas N Kipf and Max Welling. 2016. Variational graph auto-encoders. In Bayesian Deep Learning Workshop (NIPS 2016).
Thomas N Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations.
Chaitanya Malaviya, Chandra Bhagavatula, Antoine Bosselut, and Yejin Choi. 2020. Commonsense knowledge base completion with structural and semantic context. In Proceedings of the 34th AAAI Conference on Artificial Intelligence.
Costas Mavromatis and George Karypis. 2020. Graph infoclust: Leveraging cluster-level node information for unsupervised graph representation learning. arXiv preprint arXiv:2009.06946.
Roberto Navigli and Simone Paolo Ponzetto. 2012. *BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network*. Artificial Intelligence, 193:217-250.
Shirui Pan, Ruiqi Hu, Guodong Long, Jing Jiang, Lina Yao, and Chengqi Zhang. 2018. Adversarily regularized graph autoencoder for graph embedding. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 2609-2615. AAAI Press.
Heng-Shiou Sheu and Sheng Li. 2020. Context-aware graph embedding for session-based news recommendation. In Fourteenth ACM Conference on Recommender Systems, pages 657-662.
Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William W Cohen. 2018. Open domain question answering using early fusion of knowledge bases and text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4231-4242.
Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2019. Rotate: Knowledge graph embedding by relational rotation in complex space. In International Conference on Learning Representations.
Phi Vu Tran. 2018. Learning to make predictions on graphs with autoencoders. In 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), pages 237-245. IEEE.
Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph attention networks. In International Conference on Learning Representations.
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge graph embedding by translating on hyperplanes. In Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, pages 1112-1119. AAAI Press.
Zhigang Wang and Juanzi Li. 2016. Text-enhanced representation learning for knowledge graph. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, pages 1293-1299. AAAI Press.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. 2020. A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems.
Yikun Xian, Zuohui Fu, Handong Zhao, Yingqiang Ge, Xu Chen, Qiaoying Huang, Shijie Geng, Zhou Qin, Gerard de Melo, S. Muthukrishnan, and Yongfeng Zhang. 2020. CAFE: coarse-to-fine neural symbolic reasoning for explainable recommendation. In The 29th ACM International Conference on Information and Knowledge Management (CIKM), pages 1645-1654. ACM.
Jun Yan, Mrigank Raman, Tianyu Zhang, Ryan A. Rossi, Handong Zhao, Sungchul Kim, Nedim Lipka, and Xiang Ren. 2020. Learning contextualized knowledge structures for commonsense reasoning. CoRR, abs/2010.12873.
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding entities and relations for learning and inference in knowledge bases. In International Conference on Learning Representations.
Wen Zhang, Bibek Paudel, Liang Wang, Jiaoyan Chen, Hai Zhu, Wei Zhang, Abraham Bernstein, and Huajun Chen. 2019. Iteratively learning embeddings and rules for knowledge graph reasoning. In The World Wide Web Conference, pages 2366-2377. ACM.
Dongmian Zou and Gilad Lerman. 2019. Encoding robust representation for graph generation. In 2019 International Joint Conference on Neural Networks (IJCNN), pages 1-9. IEEE.