SlowGuess/NAACL_2021_OCR huggingface dataset:
graph embedding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2713-2722, Online. Association for Computational Linguistics.
Kristina Toutanova and Danqi Chen. 2015. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality, pages 57-66, Beijing, China. Association for Computational Linguistics.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML'16, page 2071-2080. JMLR.org.
Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, Nilesh Agrawal, and Partha Talukdar. 2020. Interacte: Improving convolution-based knowledge graph embeddings by increasing feature interactions. In Proceedings of the AAAI Conference on Artificial Intelligence.
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge graph embedding by translating on hyperplanes. In Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, July 27-31, 2014, Quebec City, Quebec, Canada, pages 1112-1119. AAAI Press.
Wentao Xu, Shun Zheng, Liang He, Bin Shao, Jian Yin, and Tie-Yan Liu. 2020. SEEK: Segmented embedding of knowledge graphs. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3888-3897, Online. Association for Computational Linguistics.
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding entities and relations for learning and inference in knowledge bases. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA.
Shuai Zhang, Yi Tay, Lina Yao, and Qi Liu. 2019. *Quaternion knowledge graph embeddings*. In *Proceedings of the 33rd Annual Conference on Neural Information Processing Systems*, pages 2731-2741, Vancouver, BC, Canada.
Mo Zhou, Tianyi Liu, Yan Li, Dachao Lin, Enlu Zhou, and Tuo Zhao. 2019. Toward understanding the importance of noise in training neural networks. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 7594-7602. PMLR.
Yushan Zhu, Wen Zhang, Hui Chen, Xu Cheng, Wei Zhang, and Huajun Chen. 2020. DistilE: Distiling knowledge graph embeddings for faster and cheaper reasoning.
A Gram-Schmidt Process of Tang et al. (2020)
The Gram-Schmidt process takes a set of tensor  $S = \{v_{1},\dots ,v_{k}\}$  for  $k\leq d_{s}$  and generates an orthogonal set  $S^{\prime} = \{u_{1},\dots ,u_{k}\}$  that spans the same  $k$ -dimensional subspace of  $\mathcal{R}^{d_s}$  as  $S$ , such that
$$
u _ {i} = \frac {t _ {i}}{\left\| t _ {i} \right\|} \text {s . t .} t _ {i} = v _ {k} - \sum_ {j = 1} ^ {k - 1} \frac {\langle v _ {k} , t _ {j} \rangle}{\langle t _ {j} , t _ {j} \rangle} t _ {j}, \tag {A.1}
$$
where  $t_1 = v_1$  and  $\langle v, t \rangle$  denotes the inner product of  $v$  and  $t$ . Its complexity is  $\mathcal{O}(d_s^3)$  and the parallelisation is not trivial.
B Bandwidth Comparison
To further ascertain the efficiency advantage of PROCRUSTES by ruling out factors such as numbers of all epochs, we pick four frameworks with strongest MRR performance and estimate their bandwidth during training, as illustrated in Fig. B.1.
Figure B.1: Comparison of bandwidth (number of processed samples per second). The upper and the lower are respectively for WN18RR and FB15k-237.
Figure C.2: 3D PCA visualisation of PROCRUSTES entity embeddings and pretrained word embeddings (top 40K vectors of English Wikipedia Embeddings at https://tinyurl.com/ft-wiki-vec). It is obvious that they have very distinct typologies.
We can see that although some baselines have been engineered for enhanced computational efficiency, e.g., by default RotH creates 24 threads for multiprocessing, on both datasets they still substantially underperform PROCIRSTES with huge margins in terms of bandwidth.
C Visualisation Comparisons
(a) PROCIRSTES entity embeddings (Fig. 4 with 3D axes and grids enabled)
(b) Word embeddings