SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Semantic consistency score. Let  $K$  be a logical constraint encoding some background knowledge over variables  $S, R$  and  $O$ . Given a test triple  $(s, r, o)$ , we first rank the possible completions to link prediction queries in the same way as for computing the MRR. Then, the semantic consistency score  $(\mathrm{Sem} @ k)$  [33] for some integer  $k > 0$  is computed as
$$
\operatorname {S e m} @ k = \frac {1}{2 k | \mathcal {G} _ {\text {t e s t}} |} \sum_ {(s, r, o) \in \mathcal {G} _ {\text {t e s t}}} \left(\sum_ {o ^ {\prime} \in \mathcal {A} _ {O} ^ {k} (s, r, o)} \mathbb {1} \{(s, r, o ^ {\prime}) \vDash K \} + \sum_ {s ^ {\prime} \in \mathcal {A} _ {S} ^ {k} (s, r, o)} \mathbb {1} \{(s ^ {\prime}, r, o) \vDash K \}\right)
$$
where  $\mathcal{G}_{\mathrm{test}}$  denotes the set of test triples,  $\mathcal{A}_O^k (s,r,o)$  (resp.  $\mathcal{A}_S^k (s,r,o)$ ) denotes the list of the top- $k$  candidate object (resp. subject) completions to the link prediction query  $(s,r,?)$  (resp.  $(?,r,o)$ ), and  $(s,r,o) \models K$  if and only if  $(s,r,o)$  satisfies  $K$ .
F.3 Empirical KTD Score
Let  $\mathcal{F} = \{x_i\}_{i=1}^m$ ,  $\mathcal{G} = \{y_j\}_{j=1}^n$  two sets of triples that are drawn i.i.d. from two distributions  $\mathbb{P}, \mathbb{Q}$  over triples. We compute the empirical KTD score with an unbiased estimator [28]  $\mathrm{KTD}_u(\mathcal{F}, \mathcal{G})$  as
$$
\frac {1}{m (m - 1)} \sum_ {i \neq j} ^ {m} k (\psi (x _ {i}), \psi (x _ {j})) + \frac {1}{n (n - 1)} \sum_ {i \neq j} ^ {n} k (\psi (y _ {i}), \psi (y _ {j})) - \frac {2}{m n} \sum_ {i = 1} ^ {m} \sum_ {j = 1} ^ {n} k (\psi (x _ {i}), \psi (y _ {j})).
$$
For each data set, we compute the empirical KTD score between  $n = 25$ , 000 triples sampled from GeKCs and  $m$  test triples. In case of  $m > n$ , we sample  $n$  triples randomly, uniformly and without replacement from the set of test triples. The time complexity of computing the KTD score is  $\mathcal{O}(nmh)$ , where  $h$  denotes the size of triple latent representations ( $h = 4000$  in our case, see Section 7.3). For efficiency reasons, we therefore follow Binkowski et al. [4] and randomly extract two batches of 1000 triples each from both the generated and the test triples sets and compute the empirical KTD score on them. We repeat this process 100 times and report the average and standard deviation in Table F.6.
F.4 Experimental Setting
Hyperparameters. All models are trained by gradient descent with either the PLL or the MLE objective (Eqs. (1) and (2)). We set the weights  $\omega_{s},\omega_{r},\omega_{o}$  of the PLL objective all to one, as to retrieve a classical pseudo-log-likelihood [70]. Note that Chen et al. [12] set  $\omega_{s},\omega_{o}$  to one and treat  $\omega_{r}$  as an additional hyperparameter instead that is opportunely tuned. The models are trained until the MRR computed on the validation set does not improve after three consecutive epochs. We fix the embedding size  $d = 1000$  for both CP and COMPLEX and use Adam [38] as optimiser with  $10^{-3}$  as learning rate. An exception is made for GeKCs obtained via non-negative restriction (Section 4.1), for which a learning rate of  $10^{-2}$  is needed, as we observed very slow convergence rates. We search for the batch size in  $\{5\cdot 10^{2},10^{3},2\cdot 10^{3},5\cdot 10^{3}\}$  based on the validation MRR. Finally, we perform 5 repetitions with different seeds and report the average MRR and two standard deviations in Table F.2.
Parameters initialisation. Following [40, 12], the parameters of CP and COMPLEX are initialised by sampling from a normal distribution  $\mathcal{N}(0,\sigma^2)$  with  $\sigma = 10^{-3}$ . Since the embedding values in  $\mathrm{CP}^{+}$  and  $\mathrm{COMPLEX}^{+}$  can be interpreted as parameters of categorical distributions over entities and predicates (see Appendix C.1), we initialise them by sampling from a Dirichlet distribution with concentration factors set to  $10^{3}$ . To allow unconstrained optimisation for  $\mathrm{CP}^{+}$  and  $\mathrm{COMPLEX}^{+}$ , we represent the embedding values by their logarithm and perform computations directly in log-space, i.e., summations and log-sum-exp operations instead of multiplications and summations, respectively. Moreover, the parameters that ensure the non-negativity of  $\mathrm{COMPLEX}^{+}$  (see Appendix C.2) are initialised by sampling from a normal distribution  $\mathcal{N}(0,\sigma^2)$  with  $\sigma = 10^{-2}$ . We initialise the parameters of  $\mathrm{CP}^{2}$  and  $\mathrm{COMPLEX}^{2}$  such that the logarithm of the scores are approximately normally distributed and centred in zero during the initial optimisation steps, since this applies for the scores given by CP and COMPLEX. Such initialisation therefore permits a fairer comparison. To do so, we initialise the embedding values by sampling from a log-normal distribution  $\mathcal{LN}(\mu ,\sigma^2)$ , where  $\mu = -\log (d) / 3 - \sigma^2 /2$  for CP and  $\mu = -\log (2d) / 3 - \sigma^2 /2$  for COMPLEX, both with  $\sigma = 10^{-3}$ . The mentioned values for  $\mu$  can be derived via Fenton-Wilkinson approximation [24]. Even though the parameters of GeKCs obtained via non-monotonic squaring are initialised to be non-negative, they are free of becoming negative during training (as we also confirm in practice).
Hardware. Experiments on the smaller knowledge graphs FB15K-237 and WN18RR were run on a single Nvidia GTX 1060 with 6 GiB of memory, while those on the larger ogbl-biokg and ogbl-wikikg2 were run on a single Nvidia RTX A6000 with 48 GiB of memory.
F.5 Additional Experimental Results
F.5.1 Link Prediction Results
In this section, we present the additional results regarding the link prediction experiments showed in Section 7.1 and analyse different metrics and learning settings.
Statistical tests and hits at  $k$ . Table F.2 shows the best test average MRRs (see Appendix F.2) with two standard deviation and average training time across 5 independent runs with different seeds. We highlight the best results in bold according to a one-sided Mann-Whitney U test with a confidence level of  $99\%$ . The showed results in terms of MRRs are also confirmed in Table F.3, which shows the best average Hits@  $k$  (see Appendix F.2) with  $k \in \{1, 3, 10\}$ .
Average log-likelihood. For the best GeKCs for link prediction showed in Table F.2, we report the average log-likelihood of test triples and two standard deviations (across 5 independent runs) in Table F.4. We again highlight the best results in bold, according to a one-sided Mann-Whitney U test.
Quickly distilling parameters. As discussed in Section 4.2, since learned KGE models mostly assign non-negative scores to triples (see Appendix D) we can initialise the parameters of GeKCs obtained by squaring with the parameters of already-learned KGE models, without losing much in terms of link prediction performances. Here, we test this hypothesis and fine-tune GeKCs initialised in this way by using either the PLL or MLE objectives (Eqs. (1) and (2)). To do so, we first collect the parameters of the best CP and COMPLEX found for link prediction (see Section 7.1). Then, we initialise GeKCs derived by squaring with these parameters and fine-tune them until the MRR computed on validation triples does not improve after three consecutive epochs. We employ Adam [38] as optimiser, and we search for the batch size in  $\{5\cdot 10^{2},10^{3},2\cdot 10^{3},5\cdot 10^{3}\}$  and learning rate in  $\{10^{-3},10^{-4}\}$ , as fine-tuning may require a lower learning rate than the one used in previous experiments (see Appendix F.4). Table F.5 shows the MRRs achieved by CP, COMPLEX and the corresponding GeKCs obtained via squaring that are initialised by distilling the parameters from the already-trained CP and COMPLEX. On FB15K-237 and WN18RR, distilling parameters induces a substantial improvement in terms of MRR with respect to  $\mathrm{CP}^2$  and COMPLEX whose parameters have been initialised randomly (see Appendix F.4). Furthermore, for COMPLEX and on WN18RR and ogbl-biokg we achieved similar MRRs with respect to COMPLEX without the need of fine-tuning.
Table F.2: GeKCs are competitive with their energy-based counterparts. Best test MRRs (and two standard deviations) of CP, COMPLEX and GeKCs trained with the PLL and MLE objectives (Eqs. (1) and (2)). In parentheses we show the average training time (in minutes).
Model	FB15k-237	WN18RR	ogbl-biokg
PLL	MLE	PLL	MLE	PLL	MLE
CP	0.310 ±0.001	(8)	0.105 ±0.007	(11)	0.831 ±0.001	(136)
CP+	0.237 ±0.003	(1) 0.230 ±0.003	(1) 0.027 ±0.002	(1) 0.026 ±0.001	(1) 0.496 ±0.013	(172) 0.501 ±0.010
CP2	0.315 ±0.003	(8) 0.282 ±0.004	(7) 0.104 ±0.001	(23) 0.091 ±0.004	(23) 0.848 ±0.001	(66) 0.829 ±0.001
COMPLEX	0.342 ±0.005	(36)	0.471 ±0.002	(16)	0.829 ±0.001	(180)
COMPLEX+	0.214 ±0.003	(10) 0.205 ±0.006	(5) 0.030 ±0.001	(6) 0.029 ±0.001	(3) 0.503 ±0.014	(245) 0.516 ±0.009
COMPLEX2	0.334 ±0.001	(10) 0.300 ±0.003	(16) 0.420 ±0.011	(37) 0.391 ±0.004	(19) 0.858 ±0.001	(71) 0.840 ±0.001
Table F.3: Hits@k results. Average test Hits@k for  $k \in  \{ 1,3,{10}\}$  of CP,COMPLEX and  ${\mathrm{{CP}}}^{2}$  and COMPLEX  ${}^{2}$  trained with the PLL or MLE objectives.
Model k =	FB15k-237	WN18RR	ogbl-biokg
PLL	MLE	PLL	MLE	PLL	MLE
1	3	10	1	3	10	1	3	10	1	3	10	1	3	10	1	3	10
CP	22.4	34.1	48.2	—	—	—	7.5	12.1	16.8	—	—	—	76.4	88.1	95.0	—	—	—
CP+	17.0	25.8	36.7	16.7	24.9	35.4	1.7	2.7	4.5	1.6	2.5	4.4	38.0	54.4	73.4	38.4	55.0	74.4
CP2	23.1	34.8	48.2	20.5	30.8	43.5	6.7	12.1	17.6	5.9	10.7	15.3	78.6	89.5	95.7	76.1	88.1	95.0
COMPLEX	25.2	37.5	52.5	—	—	—	43.3	48.6	54.6	—	—	—	76.1	87.9	95.0	—	—	—
COMPLEX+	15.7	23.1	31.7	15.0	22.1	30.4	1.5	2.7	4.5	1.6	2.5	4.4	38.8	55.1	74.1	40.0	56.7	75.9
COMPLEX2	24.5	36.9	51.1	21.6	33.0	46.7	36.0	45.6	52.4	34.5	42.3	46.9	80.0	90.1	95.8	77.5	88.8	95.4
Table F.4: Better distribution estimation with GeKCs obtained via squaring. Average log-likelihood of test triples achieved by baselines and GeKCs trained with the PLL or MLE objectives.
Model	FB15k-237	WN18RR	ogbl-biokg
Uniform	-24.638	-23.638	-26.829
NNMFAug	-19.270	-22.938	-17.562
PLL	MLE	PLL	MLE	PLL	MLE
CP+	-16.773 ±0.040	-16.592 ±0.059	-21.987 ±0.006	-22.103 ±0.010	-17.900 ±0.048	-17.416 ±0.049
CP2	-17.105 ±0.031	-15.982 ±0.028	-24.911 ±0.241	-26.352 ±0.077	-17.231 ±0.059	-16.533 ±0.013
COMPLEX+	-17.507 ±0.035	-17.592 ±0.039	-21.233 ±0.058	-21.432 ±0.008	-18.716 ±0.088	-17.749 ±0.019
COMPLEX2	-17.100 ±0.026	-15.744 ±0.041	-19.522 ±0.530	-19.739 ±0.214	-17.340 ±0.022	-16.518 ±0.003
Table F.5: Distilling parameters can improve performances. Test MRRs achieved by CP, COMPLEX and GeKCs obtained by squaring (Section 4.2). For  $\mathrm{CP}^2$  and  $\mathrm{COMPLEX}^2$  we report the best MRRs achieved by distilling the parameters from the already-learned CP and COMPLEX (denoted with  $\star$ ), and with  $\dagger$  we denote those results for which further fine-tuning with the PLL or MLE objectives did not bring better results. We underline results for which distilling parameters increased the MRR.
Model	FB15k-237	WN18RR	ogbl-biokg
PLL	MLE	PLL	MLE	PLL	MLE
CP	0.311	—	0.108	—	0.831	—
COMPLEX	0.344	—	0.470	—	0.829	—
CP2	0.317	0.285	0.103	0.089	0.849	0.830
CP2★	0.327	0.315	0.102	0.115	0.851	0.828 †
COMPLEX2	0.333	0.301	0.416	0.390	0.859	0.839
COMPLEX2★	0.342	0.340	0.462 †	0.463	0.859	0.828 †
F.5.2 Quality of Sampled Triples Results
In this section, we provide additional results regarding the evaluation of the quality of triples sampled by GeKCs (see Section 7.3). For these experiments, we search for the same hyperparameters as for the experiments on link prediction (see Appendix F.4), and train GeKCs until the average log-likelihood computed on validation triples does not improve after three consecutive epochs.
Table F.6 shows the mean empirical KTD score and one standard deviation (see Appendix F.3). In addition, we visualise triple embeddings of sampled and test triples in Fig. F.1 by leveraging t-SNE [69] as a method for visualising high-dimensional data. In particular, we apply the t-SNE method implemented in scikit-learn with perplexity 50 and number of iterations  $5 \cdot 10^{3}$ , while other parameters are fixed to their default value. As showed in Fig. F.1c, an empirical KTD score close to zero translates to an high clusters similarity between embeddings of sampled and test triples.
Table F.6: GeKCs trained by MLE generate new likely triples. Empirical KTD scores between test triples and triples generated by baselines and GeKCs trained with the PLL objective or by MLE (Eqs. (1) and (2)). Lower is better.
Model	FB15k-237	WN18RR	ogbl-biokg
Training set	0.055 ±0.007	0.260 ±0.013	0.029 ±0.010
Uniform	0.589 ±0.012	0.766 ±0.036	1.822 ±0.044
NNMFAug	0.414 ±0.014	0.607 ±0.028	0.518 ±0.035
PLL	MLE	PLL	MLE	PLL	MLE
CP+	0.404 ±0.016	0.433 ±0.015	0.633 ±0.033	0.578 ±0.029	0.966 ±0.040	0.738 ±0.030
CP2	0.253 ±0.014	0.070 ±0.007	0.768 ±0.036	0.768 ±0.036	0.039 ±0.009	0.017 ±0.013
COMPLEX+	0.336 ±0.016	0.323 ±0.015	0.456 ±0.018	0.478 ±0.019	0.175 ±0.019	0.097 ±0.013
COMPLEX2	0.326 ±0.016	0.102 ±0.010	0.338 ±0.020	0.278 ±0.017	0.104 ±0.010	0.034 ±0.007
(a) FB15k-237 KTD  $\coloneqq 0.102\pm 0.010$
(b) WN18RR KTD  $\coloneqq 0.278\pm 0.017$
(c) ogbl-biokg KTD  $\coloneqq 0.034\pm 0.007$
Figure F.1: Sampled triples are close to test triples. t-SNE [69] visualisations of the embeddings of test triples (in blue) and triples sampled by COMPLEX $^2$  (in orange). The distribution shift between training and test triples on WN18RR mentioned in Section 7.3 is further confirmed in Fig. F.1b, as it shows a region of test triples (at the bottom and in blue) that is not covered by many generated triples.
F.5.3 Calibration Diagrams
Existing works on studying the calibration of KGE models are based on interpreting each possible triple  $(s,r,o)$  as an independent Bernoulli random variable  $Y_{sro}$  whose likelihood is determined by the score function  $\phi$ , i.e.,  $\operatorname{Pr}(Y_{sro} = 1 \mid s,r,o) = \sigma(\phi(s,r,o))$  [61, 54, 79], where  $\sigma$  denotes the logistic function. While GeKCs encode a probability distribution over all possible triples, this does not impede us to reinterpret them to model the likelihood of each  $Y_{sro}$  by still considering scores in log-space as negated energies (see Section 2). Therefore, to evaluate the calibration of GeKCs encoding a non-negative score function  $\phi_{\mathrm{pc}}$  (see Section 4) we compute the probability of a triple  $(s,r,o)$  being true as  $p(Y_{sro} = 1 \mid s,r,o) \coloneqq \sigma(\log \phi_{\mathrm{pc}}(s,r,o))$ . However, the usage of the logistic
function might give misleading results in case of scores not being centred around zero on average. Therefore, we also report calibration diagrams (see paragraph below) where the  $p(Y_{sro} = 1 \mid s, r, o)$  is obtained via min-max normalisation of the scores given by KGE models (the logarithm of the scores given for GeKCs), where the minimum and maximum are computed on the training triples. Note that several ex-post (re-)calibration techniques are available [61, 79], but they should benefit GeKCs as they do with existing KGE models.
Setting and metrics. To plot calibration diagrams, we follow Socher et al. [59] and sample challenging negative triples, i.e., for each test triple  $(s,r,o)$  we sample an unobserved perturbed one  $(s,r,\widehat{o})$  by replacing the object with an entity that has appeared at least once with the predicate  $r$  in the training data. We then compute the empirical calibration error (ECE) [79] as ECE :=  $\frac{1}{b}\sum_{i=1}^{b}|p_j - f_j|$ , where  $b$  is the number of uniformly-chosen bins for the interval  $[0,1]$  of triple probabilities, and  $p_j, f_j$  are respectively the average probability and relative frequency of actually existing triples in the  $j$ -th bin. The lower the ECE score, the better calibrated are the predictions, as they are closer to the empirical frequency of triples that do exist in each bin. The calibration curves are plotted by considering the relative frequency of existing triples in each bin, and curves closer to the main diagonal indicate better calibrated predictions.
GeKCs are more calibrated out-of-the-box. Fig. F.2 (resp. Fig. F.3) show calibration diagrams for GeKCs derived from CP and COMPLEX trained with the MLE objective (Eq. (2)) (resp. PLL objective (Eq. (1))). In 19 cases over 24, GeKCs obtained via squaring (Section 4.2) achieve lower ECE scores and better calibrated curves than CP and COMPLEX. While GeKCs obtained via non-negative restriction (Section 4.1) are not well calibrated when using the logistic function, on ogbl-biokg [32] they are still better calibrated than CP and COMPLEX when probabilities are obtained via min-max normalisation. Furthermore, on WN18RR GeKCs achieved the highest ECE scores (corresponding to poorly-calibrated predictions), which could be explained by the distribution shift between training and test triples that was observed for this KG in Section 7.3 and further confirmed in Appendix F.5.2.
Figure F.2: Better calibrated predictions with  $\mathbf{CP}^2$ . Calibration diagrams of CP,  $\mathrm{CP^{+}}$  and  $\mathrm{CP}^2$  trained with either the PLL (Fig. F.2a) or MLE (Fig. F.2b) objectives. The probability of triples are obtained via the application of the logistic function (rows above) and min-max normalisation (rows below). See Appendix F.5.3 for details. The calibration curves for  $\mathrm{CP^{+}}$  where triple probabilities are obtained with the logistic function do not provide any meaningful information, as the logarithm of their scores are generally distributed over large negative values.
Figure F.3: Better calibrated predictions with COMPLEX $^2$ . Calibration diagrams of COMPLEX, COMPLEX $^+$  and COMPLEX $^2$  trained with either the PLL (Fig. F.3a) or MLE (Fig. F.3b) objectives. The probability of triples are obtained via the application of the logistic function (rows above) and min-max normalisation (rows below). See Appendix F.5.3 for details. The calibration curves for COMPLEX $^+$  where triple probabilities are obtained with the logistic function do not provide any meaningful information, as the logarithm of their scores are generally distributed over large negative values.