SlowGuess/ACL_2020_OCR huggingface dataset:
Curvature estimate To estimate the curvature of a relation  $r$ , we restrict to the undirected graph  $G_{r}$  spanned by the edges labeled as  $r$ . Following (Gu et al., 2019), let  $\xi_{G_r}(a,b,c)$  be the curvature estimate of a triangle in  $G_{r}$  with vertices  $\{a,b,c\}$ ,
Figure 7: Histogram of embeddings norm learned with fixed and trainable curvatures for the hypernym relation in WN18RR.
which is given by:
$$
\begin{array}{l} \xi_ {G _ {r}} (a, b, c) = \frac {1}{2 d _ {G _ {r}} (a , m)} \left(d _ {G _ {r}} (a, m) ^ {2} \right. \\ + d _ {G _ {r}} (b, c) ^ {2} / 4 \\ - \left(d _ {G _ {r}} (a, b) ^ {2} + d _ {G _ {r}} (a, c) ^ {2}\right) / 2), \\ \end{array}
$$
where  $m$  is the midpoint of the shortest path connecting  $b$  to  $c$ . This estimate is positive for triangles in circles, negative for triangles in trees, and zero for triangles in lines. Moreover, for a triangle in a Riemannian manifold  $M$ ,  $\xi_{M}(a,b,c)$  estimates the sectional curvature of the plane on which the triangle lies (see (Gu et al., 2019) for more details). Let  $m_r$  be the total number of connected components in  $G_{r}$ . We sample 1000  $w_{i,r}$  triangles from each connected component  $c_{i,r}$  of  $G_{r}$  where  $w_{i,r} = \frac{N_{i,r}^3}{\sum_{i=1}^{mr} N_{i,r}^3}$ , and  $N_{i,r}$  is the number of nodes in the component  $c_{i,r}$ .  $\xi_{G_r}$  is the mean of the estimated curvatures of the sampled triangles. For the full graph, we take the weighted average of the relation curvatures  $\xi_{G_r}$  with respect to the weights  $\frac{\sum_{i=1}^{mr} N_{i,r}^3}{\sum_{r} \sum_{i=1}^{mr} N_{i,r}^3}$ .
Krackhardt hierarchy score For the directed graph  $G_{r}$  spanned by the relation  $r$ , we let  $R$  be the adjacency matrix ( $R_{i,j} = 1$  if there is an edge from node  $i$  to node  $j$  and 0 otherwise). Then:
$$
\mathrm {K h s} _ {G _ {r}} = \frac {\sum_ {i , j = 1} ^ {n} R _ {i , j} (1 - R _ {j , i})}{\sum_ {i , j = 1} ^ {n} R _ {i , j}}.
$$
See (Krackhardt, 1994) for more details. We note that for fully observed symmetric relations (each edge is in a two-edge loop),  $\mathrm{Khs}_{G_r} = 0$  while for anti-symmetric relations (no small loops),  $\mathrm{Khs}_{G_r} = 1$ .
Figure 8: The curvature estimate  $\xi_G$  and the Krackhardt hierarchy score  $\mathrm{Khs}_G$  for several simple graphs. The top-left graph is the most hierarchical, while the bottom-right graph is the least hierarchical.
Model	WN18RR	FB15k-237	YAGO3-10
MRR	H@10	MRR	H@10	MRR	H@10
Dihedral	.486	.557	.300	.496	.388	.573
ATTE	.490	.581	.351	.543	.575	.709
Table 6: Comparison of Dihedral and ATTE in high-dimensions.
A.3 Experimental details
For all our Euclidean and hyperbolic models, we conduct a hyperparameter search for the learning rate, optimizer (Adam (Kingma and Ba, 2015) or Adagrad (Duchi et al., 2011)), negative sample size and batch size. We train each model for 500 epochs and use early stopping after 100 epochs if the validation MRR stops increasing. We report the best hyperparameters for each dataset in Table 7.
A.4 Tangent space optimization
Optimization in hyperbolic space normally requires Riemannian Stochastic Gradient Descent (RSGD)
(Bonnabel, 2013), as was used in MuRP. RSGD is challenging in practice. Instead, we use tangent space optimization (Chami et al., 2019). We define all the ATTH parameters in the tangent space at the origin (our parameter space), optimize embeddings using standard Euclidean techniques, and use the exponential map to recover the hyperbolic parameters.
Note that tangent space optimization is an exact procedure, which does not incur losses in representational power. This is the case in hyperbolic space specifically because of a completeness property: there is always a global bijection between the tangent space and the manifold.
Concretely,  $\mathsf{ATTH}$  optimizes the entity and relationship embeddings  $(\mathbf{e}_v^E)_{v\in \mathcal{V}}$  and  $(\mathbf{r}_r^E)_{r\in \mathcal{R}}$ , which are mapped to the Poincaré ball with:
$$
\mathbf {e} _ {v} ^ {H} = \exp_ {\mathbf {0}} ^ {c _ {r}} \left(\mathbf {e} _ {v} ^ {E}\right) \text {a n d} \mathbf {r} _ {r} ^ {H} = \exp_ {\mathbf {0}} ^ {c _ {r}} \left(\mathbf {r} _ {r} ^ {E}\right), \tag {12}
$$
The trainable model parameters are then
6913
Dataset	embedding dimension	model	learning rate	optimizer	batch size	negative samples
WN18RR	32	REFE	0.001	Adam	100	250
ROTE	0.001	Adam	100	250
ATTE	0.001	Adam	100	250
REFH	0.0005	Adam	250	250
ROTH	0.0005	Adam	500	50
ATTH	0.0005	Adam	500	50
500	REFE	0.1	Adagrad	500	50
ROTE	0.001	Adam	100	500
ATTE	0.001	Adam	1000	50
REFH	0.05	Adagrad	500	50
ROTH	0.001	Adam	1000	50
ATTH	0.001	Adam	1000	50
FB15k-237	32	REFE	0.075	Adagrad	250	250
ROTE	0.05	Adagrad	500	50
ATTE	0.05	Adagrad	500	50
REFH	0.05	Adagrad	500	250
ROTH	0.1	Adagrad	100	50
ATTH	0.05	Adagrad	500	100
500	REFE	0.05	Adagrad	500	50
ROTE	0.05	Adagrad	100	50
ATTE	0.05	Adagrad	500	50
REFH	0.05	Adagrad	500	50
ROTH	0.05	Adagrad	1000	50
ATTH	0.05	Adagrad	500	50
YAGO3-10	32	REFE	0.005	Adam	2000	NA
ROTE	0.005	Adam	2000	NA
ATTE	0.005	Adam	2000	NA
REFH	0.005	Adam	1000	NA
ROTH	0.001	Adam	1000	NA
ATTH	0.001	Adam	1000	NA
500	REFE	0.005	Adam	4000	NA
ROTE	0.005	Adam	4000	NA
ATTE	0.005	Adam	2000	NA
REFH	0.001	Adam	1000	NA
ROTH	0.0005	Adam	1000	NA
ATTH	0.0005	Adam	1000	NA
Table 7: Best hyperparameters in low- and high-dimensional settings. NA negative samples indicates that the full cross-entropy loss is used, without negative sampling.
$\{(\Theta_r,\Phi_r,\mathbf{r}_r^E,\mathbf{a}_r,c_r)_{r\in \mathcal{R}},(\mathbf{e}_v^E,b_v)_{v\in \mathcal{V}}\}$ , which are all Euclidean parameters that can be learned using standard Euclidean optimization techniques.
A.5 Comparison to Dihedral
We compare the performance of Dihedral (Xu and Li, 2019) versus that of ATTE in Table 6. Both methods combine rotations and reflections, but our approach learns attention-based transformations, while Dihedral learns a single parameter to determine which transformation to use. ATTE significantly outperforms Dihedral on all datasets, suggesting that using attention-based representations is important in order to learn the right geometric transformation for each relation.
6914