SlowGuess/NAACL_2021_OCR huggingface dataset:
ICEWS (García-Durán et al., 2018) is collected from the integrated crisis early warning system<sup>3</sup> which was built to monitor and forecast national and internal crises. The datasets contain political events that connect entities (e.g., countries, presidents, intergovernmental organizations) to other entities via predicates (e.g., "make a visit", "sign formal agreement", etc.). ICES14 contains events in the year 2014, while the ICEWS05-15 contains
events occurring between 2005 and 2015. Temporal information is not used in our experiments.
Data statistics and the train/validation/test splits are summarized in Table 2.
We adopt four commonly used evaluation metrics including hit rate with given cut-off (HR@1, HR@3, HR@10) and mean reciprocal rank (MRR). HR measures the percentage of true triples of the ranked list. MRR is the average of the mean rank inverse which reflects the ranking quality. Evaluation is performed under the commonly used filtered setting (Bordes et al., 2013), which is more reasonable and stable compared to the unfiltered setting.
To demonstrate the advantage of our approach, we compare the proposed method with several representative knowledge graph embedding approaches including TransE (Bordes et al., 2013), DistMult (Yang et al., 2014), ComplEx (Trouillon et al., 2016), Simple (Kazemi and Poole, 2018), and QuatE (Zhang et al., 2019). For FB15k-237, the results of RotatE (Sun et al., 2019) and RGCN (Schlichtkrull et al., 2018) are also included.
We implement our model using pytorch (Paszke et al., 2019) and run it on TITAN XP GPUs. We adopt Adam optimizer to learn our model (Goodfellow et al., 2016) and the learning rate is set to 0.01 without further tuning. The embedding size  $d$  is set to 100 and the number of negative samples is fixed to 50. The batch size is selected from  $\{128, 512, 1024\}$ . The regularization rate is searched from  $\{0.0, 0.01, 0.1, 0.2, 0.3, 0.5\}$ . For the disentanglement algorithm, the number of components  $K$  is selected from  $\{2, 4, 5, 10\}$  ( $K$  should be divisible by  $d$ ); the number of routing iterations  $T$  is tuned amongst  $\{2, 3, 4, 5, 7, 10\}$ . The hyperparameters are determined by the validation set. Each experiment runs five times and the average is reported. For convenience of implementation, the maximum neighbor sizes are: 16 (FB15K-237), 4 (WikiData), 10 (ICEWS14), 16 (ICEWS05-15). We apply zero padding to entities that have fewer neighbors.
The test results on the four datasets are shown in Tables 3, 4 and 5. Evidently, we can make the
1For example, a word with five meanings is represented with five different entities in WordNet. 2https://www.wikidata.org/ 3http://www.icews.com/
Models	FB15k-237
MRR	HR@10	HR@3	HR@1
TransE	0.294	0.465	-	-
DistMult	0.241	0.419	0.263	0.155
ComplEx	0.247	0.428	0.275	0.158
SimpleE	0.229	0.379	0.252	0.153
R-GCN♥	0.249	0.417	0.264	0.151
RotatE★	0.297	0.480	0.328	0.205
QuatE◇	0.311	0.495	0.342	0.221
KR-DistMult	0.275	0.450	0.302	0.190
KR-SimpleE	0.273	0.438	0.298	0.190
KR-QuatE	0.322	0.507	0.356	0.228
KR-D vs. D	+14.1%	+7.4%	+14.8%	+22.6%
KR-S vs. S	+19.2%	+15.5%	+18.2%	+24.2%
KR-Q vs. Q	+3.5%	+2.4%	+4.1%	+3.2%
Table 3: Results on the FB15K-237 dataset. Best results are in bold. “D”, “S”, and “D” stand for DistMult, SimplE, and QuatE, respectively. “♥”: results from (Schlichtkrull et al., 2018). “☆”: results from (Sun et al., 2019). For fair comparison, adversarial negative sampling is not used. “◇”: results from (Zhang et al., 2019) (without N3 regularization and type constraints).
Models	WikiData
MRR	HR@10	HR@3	HR@1
TransE	0.164	0.288	0.162	0.101
DistMult	0.863	0.902	0.883	0.837
ComplEx	0.850	0.895	0.871	0.821
SimpleE	0.878	0.902	0.890	0.861
QuatE	0.792	0.852	0.823	0.752
KR-DistMult	0.888	0.911	0.898	0.872
KR-SimpleE	0.898	0.912	0.900	0.891
KR-QuatE	0.900	0.912	0.900	0.893
KR-D vs. D	+2.9%	+1.0%	+1.7%	+4.2%
KR-S vs. S	+2.3%	+1.1%	+1.1%	+3.6%
KR-Q vs. Q	+13.6%	+7.0%	+9.4%	+18.7%
Table 4: Results on WikiData. Best results are in bold. "D", "S", and "D" stand for DistMult, SimpE, and QuatE, respectively.
following observations: (1) Models with Knowledge Router outperform the counterparts without it by a large margin, confirming the effectiveness of KnowledgeRouter and assuring the benefits of learning disentangled representations. This clearly answers our RQ I; (2) On the four datasets, we observe a consistent enhancement of KnowledgeRouter on both traditional embedding models such as DistMult, SimplE, as well as hypercomplex number based model QuatE. This is expected as our KnowledgeRouter is model agnostic (RQ II) and can be integrated to canonical knowledge embedding models. (3) The model KR-QuatE is usually the best performer on all datasets, indicating the generalization capability of KnowledgeRouter in more complex embedding spaces.
On the FB15k-237 dataset, the model KR-QuatE achieves the best performance compared to the re
cent translational model RotatE and the semantic matching model QuatE. Models such as DistMult and Simple are also outperformed by KR-DistMult and KR-SimpleE. In addition, it is good to note that the performance of each of the three KR-models is much higher than the graph convolutional networks based model, R-GCN. This implies that simply/naively incorporating graph structures might not lead to good performance. Knowledge Router also operates at the graph level, moreover, the neighborhood information is effectively utilized for disentanglement.
Similar trends are also observed on WikiData. Interestingly, we find that the performance differences of the three KR-models are quite small on this dataset. We hypothesize that the performance on this dataset has already been quite high, making further improvement more difficult.
Among the baselines, SimpleE is the best performer. We notice that even though the pure QuantE does not show impressive performance, the Knowledge Router enhances its results and enables it to achieve the state-of-the-art performance.
On the two ICEWS datasets, disentanglement usually leads to a large performance boost. The average performance gains of Knowledge Router based models (KR-DistMult, KR-SimplE, KR-QuatE) are high, compared with the original models (DistMult, SimplE, and QuatE). We also observe that KR-QuatE outperforms other models significantly.
To conclude, our experimental evidence shows that disentangling the entities can indeed bring performance increase and the proposed Knowledge Router can effectively be integrated into different models.
To answer RQ III and gain further insights, we empirically analyze the important ingredients of the model via qualitative analysis and visualization.
4.6.1 Visualization of similarity scores
The attention mechanism is critical to achieving the final disentanglement. To show its efficacy, we visualize four examples of attention weights  $w_{i,k}$  in Figure 2. The color scale represents the strength of the attention weights. Each row represents a neighbor of the selected entity and each column represents a disentangled component. We observe a clear staggered pattern in the attention weights. For example, in the upper left figure, the neighbors
Models	ICEWS 14	ICEWS05-15
MRR	HR@10	HR@3	HR@1	MRR	HR@10	HR@3	HR@1
TransE*	0.280	0.637	-	0.094	0.294	0.663	-	0.090
DistMult*	0.439	0.672	-	0.323	0.456	0.691	-	0.337
SimplE	0.458	0.687	0.516	0.341	0.478	0.708	0.539	0.359
ComplEx	0.638	0.753	0.677	0.574	0.708	0.821	0.748	0.645
QuatE	0.656	0.733	0.673	0.615	0.723	0.817	0.754	0.671
KR-DistMult	0.544	0.740	0.608	0.439	0.611	0.789	0.662	0.519
KR-SimplE	0.588	0.753	0.642	0.498	0.639	0.803	0.689	0.553
KR-QuatE	0.688	0.753	0.692	0.643	0.797	0.853	0.812	0.767
KR-DistMult vs. DistMult	+23.9%	+10.1%	-	+11.6%	+33.9%	+14.2%	-	+54.0%
KR-SimplE vs. SimplE	+28.3%	+9.6%	+24.4%	+46.0%	+33.7%	+13.4%	+27.8%	+54.0%
KR-QuatE vs. QuatE	+4.9%	+2.7%	+2.8%	+4.6%	+10.2%	+4.4%	+7.7%	+14.3%
Table 5: Results on ICEWS14 and ICEWS05-15. Best results are in bold. “ $\star$ ”: results from (García-Durán et al., 2018). Note that the embedding size is 100 for all models.
Figure 2: Four examples of attention weights learned during the routing process. Te upper two examples are taken from WikiData  $(K = 2)$  and the lower two examples are taken from ICEWS14  $(K = 4)$ . Rows represent neighbors and columns represent disentangled components. Best viewed in color.
1, 2, 3 give higher weights to the second component while 0 gives a stronger weight to the first component. In other figures, the attention weights are also staggered among the disentangled components.
4.6.2 Case study
We randomly pick one entity (Michael Rensing, a German footballer) from the WikiData and show the learned weight between him and his neighborhood entities in Figure 3. We observe that FC Bayern Munich and Jan Kirchhoff (who is also a team member of the FC Bayern Munich club) contribute more on the first component of the representation of Michael Rensing, while Germany national under-18 football team and Germany national under-21 football team make larger contributions to the second component. Clearly, the first component captures the fact that Michael Rensing is a member of the FC Bayern Munich association football club and the second component reflects that he is also a
Figure 3: Case study on WikiData for the German footballer Michael Rensing.
(b)
Figure 4: (a) The impact of number of components  $K$  on ICEWS14. (b) The impact of number of neighborhood routing iteration  $T$  on ICEWS14.
Germany national football team member. This case justifies our assumption that entities are connected for different reasons and demonstrates that Knowledge Router is able to disentangle the underlying factors effectively.
4.6.3 Impact of size  $K$
We analyze the impact of  $K$ . Intuitively,  $K$  is difficult to choose since there is no prior information on how many components we should decompose each entity into. The test results with varying  $K$  on ICEWS14 of KR-QuatE are shown in Figure 4 (a).
As can be seen, using large  $K$  could result in a performance degradation. One possible reason is that there are not enough neighborhood entities to be divided into 20 groups. Empirically, we found that setting  $K$  to a small value around 2 to 5 can usually render reasonable results. A practical suggestion is that  $K$  should not exceed the average degree of the knowledge graph.
4.6.4 Impact of routing iteration  $T$
We study the influence of number of routing iterations. As shown in Figure 4 (b), the model performance is stable when using different iterations. The reason is that the Knowledge Router algorithm is not prone to saturation and has good convergence properties. In practice, we find that using a small number of iterations (e.g., 3) could lead to ideal enhancement without putting on much computation burden.
In this paper, we present Knowledge Router, an algorithm for learning disentangled entity representations in knowledge graphs. Our method is model agnostic and can be applied to many canonical knowledge graph embedding methods. Extensive experiments on four benchmarking datasets demonstrate that equipping popular embedding models with the proposed KnowledgeRouter can outperform a number of recent strong baselines. Via qualitative model analysis, we discover that KnowledgeRouter can effectively learns the hidden factors connecting entities, thus leading to disentanglement. We also showcase the impact of certain important hyper-parameters and give suggestions on hyperparameters tuning.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1828.
Christopher M Bishop. 2006. Pattern recognition and machine learning. Springer.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Advances in neural information processing systems, pages 2787-2795.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi.
2019. Comet: Commonsense transformers for automatic knowledge graph construction. arXiv preprint arXiv:1906.05317.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthew, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. 2018. Understanding disentangling in beta-vae. arXiv preprint arXiv:1804.03599.
Ines Chami, Adva Wolf, Da-Cheng Juan, Frederic Sala, Sujith Ravi, and Christopher Ré. 2020. Low-dimensional hyperbolic knowledge graph embeddings. arXiv preprint arXiv:2005.00545.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. 2016. InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pages 2172-2180.
Emily L Denton et al. 2017. Unsupervised learning of disentangled representations from video. In Advances in neural information processing systems, pages 4414-4423.
T Dettmers, P Minervini, P Stenetorp, and S Riedel. 2018. Convolutional 2d knowledge graph embeddings. In 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, volume 32, pages 1811-1818. AAI Publications.
Lisa Ehrlinger and Wolfram Wöß. 2016. Towards a definition of knowledge graphs. SEMANTiCS (Posters, Demos, SuCESS), 48:1-4.
Alberto García-Durán, Sebastijan Dumančić, and Mathias Niepert. 2018. Learning sequence encoders for temporal knowledge graph completion. arXiv preprint arXiv:1809.03202.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. 2016. Deep learning, volume 1. MIT press Cambridge.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680.
Irina Higgins, Loic Matthew, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 2016. beta-vae: Learning basic visual concepts with a constrained variational framework.
Geoffrey E Hinton, Alex Krizhevsky, and Sida D Wang. 2011. Transforming auto-encoders. In International conference on artificial neural networks, pages 44-51. Springer.
Sarthak Jain, Edward Banner, Jan-Willem van de Meent, Iain Marshall, and Byron C Wallace. 2018. Learning disentangled representations of texts with application to biomedical abstracts. In Proceedings
of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4683-4693.
Guoliang Ji, Kang Liu, Shizhu He, Jun Zhao, et al. 2017. Distant supervision for relation extraction with sentence-level attention and entity descriptions.
Seyed Mehran Kazemi and David Poole. 2018. Simple embedding for link prediction in knowledge graphs. In Advances in neural information processing systems, pages 4284-4295.
Diederik P Kingma and Max Welling. 2013. Autoencoding variational bayes. arXiv preprint arXiv:1312.6114.
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning entity and relation embeddings for knowledge graph completion. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pages 2181-2187.
Kun Liu, Shen Li, Daqi Zheng, Zhengdong Lu, Sheng Gao, and Si Li. 2019. A prism module for semantic disentanglement in name entity recognition. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5358-5362.
Jianxin Ma, Peng Cui, Kun Kuang, Xin Wang, and Wenwu Zhu. 2019a. Disentangled graph convolutional networks. In International Conference on Machine Learning, pages 4212-4221.
Jianxin Ma, Chang Zhou, Peng Cui, Hongxia Yang, and Wenwu Zhu. 2019b. Learning disentangled representations for recommendation. In Advances in Neural Information Processing Systems, pages 5711-5722.
Liqian Ma, Qianru Sun, Stamatios Georgoulis, Luc Van Gool, Bernt Schiele, and Mario Fritz. 2018. Disentangled person image generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 99-108.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in neural information processing systems, pages 8026-8037.
Pengda Qin, Weiran Xu, and William Yang Wang. 2018. Robust distant supervision relation extraction via deep reinforcement learning. arXiv preprint arXiv:1805.09927.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. 2017. Dynamic routing between capsules. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 3859-3869.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In European Semantic Web Conference, pages 593-607. Springer.
Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2019. Rotate: Knowledge graph embedding by relational rotation in complex space. arXiv preprint arXiv:1902.10197.
Joshua B Tenenbaum and William T Freeman. 2000. Separating style and content with bilinear models. Neural computation, 12(6):1247-1283.
Luan Tran, Xi Yin, and Xiaoming Liu. 2017. Disentangled representation learning gan for pose-invariant face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1415-1424.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. International Conference on Machine Learning (ICML).
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge graph embedding by translating on hyperplanes. In Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, pages 1112-1119.
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2014. Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv:1412.6575.
Shuai Zhang, Yi Tay, Wenqi Jiang, Da-cheng Juan, and Ce Zhang. 2021. Switch spaces: Learning product spaces with sparse gating. arXiv preprint arXiv:2102.08688.
Shuai Zhang, Yi Tay, Lina Yao, and Qi Liu. 2019. Quaternion knowledge graph embeddings. In Advances in Neural Information Processing Systems, pages 2735-2745.
Zhenyao Zhu, Ping Luo, Xiaogang Wang, and Xiaouu Tang. 2014. Multi-view perceptron: a deep model for learning face identity and view representations. In Advances in Neural Information Processing Systems, pages 217-225.