SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proposition A.20. For all  $t \geq 0$  and all knowledge graph  $G$ , let  $\mathsf{rw} \mid_2^{(0)}(G) \equiv \mathsf{rw} \mid_2^{+(0)}(G)$ , then the following statement holds:
1. There is a knowledge graph  $G$  and pair of nodes  $(u, v)$  and  $(u', v')$  such that  $\mathsf{rw} \mathsf{l}_2^{(t)}(G, u, v) = \mathsf{rw} \mathsf{l}_2^{(t)}(G, u', v')$  for all  $t \geq 0$  but  $\mathsf{rw} \mathsf{l}_2^{(1)}(G, u, v) \neq \mathsf{rw} \mathsf{l}_2^{(1)}(G, u', v')$ .
2. There is a knowledge graph  $G'$  and pair of nodes  $(u, v)$  and  $(u', v')$  such that  $\mathsf{rawl}_2^{+(t)}(G', u, v) = \mathsf{rawl}_2^{+(t)}(G', u', v')$  for all  $t \geq 0$  but  $\mathsf{rwll}_2^{(1)}(G', u, v) \neq \mathsf{rwll}_2^{(1)}(G', u', v')$ .
Proof. To show  $\mathsf{rw}|\_2(G)$  does not refine  $\mathsf{rawl}_2^+(G)$ , we consider a relational graph  $G = (V, E, R, c, \eta)$  such that  $V = \{u, u', v, v', x, x'\}$ ,  $E = \{r_1(v, x), r_2(v', x')\}$ , and  $R = \{r_1, r_2\}$ . We let the initial
Figure 5: Counterexample shown in Proposition A.20 to prove  $\mathsf{rw} \mathsf{l}_2^{(t)}(u, v) = \mathsf{rw} \mathsf{l}_2^{(t)}(u', v')$  for all  $t \geq 0$  but  $\mathsf{rawl}_2^{+(1)}(u, v) \neq \mathsf{rawl}_2^{+(1)}(u', v')$ .
labeling  $\eta$  for node pairs  $(u,v)$  and  $(u^{\prime},v^{\prime})$  to satisfy  $\mathsf{rwl}_2^{(0)}(u,v) = \mathsf{rwl}_2^{(0)}(u^{\prime},v^{\prime})$  and  $\mathsf{rwl}_2^{(0)}(u,v) = \mathsf{rwl}_2^{(0)}(u^{\prime},v^{\prime})$ . For such graph  $G$ , we consider node pair  $(u,v)$  and  $(u^{\prime},v^{\prime})$ . For  $\mathsf{rwl}_2^{(t)}$  where  $t\geq 0$ , we show by induction that  $\mathsf{rwl}_2^{(t)}(u,v) = \mathsf{rwl}_2^{(t)}(u^{\prime},v^{\prime})$ . The base case is trivial by assumption. The inductive step shows that by the inductive hypothesis,
$$
\begin{array}{l} \mathsf {r w l} _ {2} ^ {(t + 1)} (u, v) = \tau (\mathsf {r w l} _ {2} ^ {(t)} (u, v), \{\{\}, \{\{\} \}) \\ = \tau (\mathsf {r w l} _ {2} ^ {(t)} (u ^ {\prime}, v ^ {\prime}), \{\{\}, \{\{\} \}) \\ = \operatorname {r w l} _ {2} ^ {(t + 1)} \left(u ^ {\prime}, v ^ {\prime}\right) \\ \end{array}
$$
$$
\begin{array}{l} \left. \operatorname {r a w l} _ {2} ^ {+ (1)} (u, v) = \tau \left(\operatorname {r a w l} _ {2} ^ {+ (0)} (u, v), \{\{\left(\operatorname {r a w l} _ {2} ^ {+ (0)} (u, x), r _ {1} ^ {-}) \right] \}\right) \right. \\ \neq \operatorname {r a w l} _ {2} ^ {+ (1)} (u, v ^ {\prime}) = \tau (\operatorname {r a w l} _ {2} ^ {+ (0)} (u ^ {\prime}, v ^ {\prime}), \{\{\left( \right.\operatorname {r a w l} _ {2} ^ {+ (0)} (u ^ {\prime}, x ^ {\prime}), r _ {2} ^ {-}) \} \}) \\ \end{array}
$$
Finally, to show  $\mathsf{rawl}_2^+$  does not refine  $\mathsf{rw}l_2$ , we demonstrate the case for  $t \geq 0$ . Consider a relational graph  $G' = (V', E', R', c, \eta)$  such that  $V' = \{u, u', v, x\}$ ,  $E' = \{r(x, u')\}$ , and  $R' = \{r\}$ . We let the initial labeling  $\eta$  for node pairs  $(u, v)$  and  $(u', v)$  satisfy  $\mathsf{rw}l_2^{(0)}(u, v) = \mathsf{rw}l_2^{(0)}(u', v)$  and  $\mathsf{rawl}_2^{+(0)}(u, v) = \mathsf{rawl}_2^{+(0)}(u', v)$ . For such graph  $G'$ , we consider node pair  $(u, v)$  and  $(u', v)$ . For  $\mathsf{rawl}_2^{+(t)}$  where  $t \geq 0$ , and show by induction that  $\mathsf{rawl}_2^{+(t)}(u, v) = \mathsf{rawl}_2^{+(t)}(u', v)$ . The base case is trivial by assumption. The inductive step shows that by the inductive hypothesis,
$$
\begin{array}{l} \left. \operatorname {r a w l} _ {2} ^ {+ (t + 1)} (u, v) = \tau \left(\operatorname {r a w l} _ {2} ^ {+ (t)} (u, v), \{\{\}\}\right) \right. \\ = \tau \left(\operatorname {r a w l} _ {2} ^ {+ (t)} \left(u ^ {\prime}, v\right), \{\{\}\}\right) \\ = \operatorname {r a w l} _ {2} ^ {+ (t + 1)} \left(u ^ {\prime}, v\right) \\ \end{array}
$$
Note that the counter-example graph  $G'$  here is identical to the one in Proposition A.18.
B Runtime analysis
In this section, we present the asymptotic time complexity of R-MPNN, C-MPNN, and a labeling-trick-based relation prediction model GraIL [30] in Table 3, taken from Zhu et al. [40]. We also consider 2-RN, the neural architecture from Barceló et al. [4] corresponding to  $\mathsf{rwl}_2$  to showcase an example of higher-order GNNs. We present both the complexity of a single forward pass of the model as well as the amortized complexity of a single query to better reflect the practical use cases and the advantages stemming from parallelizability. To avoid confusion, we take CompGCN [32] as the example of R-MPNN, and basic C-MPNN, since specific architecture choices will impact the asymptotic complexity of the model.
Notation. Given a knowledge graph  $G = (V, E, R, c)$ , we have that  $|V|, |E|, |R|$  represents the size of vertices, edges, and relation types, respectively.  $d$  is the hidden dimension of the model, and  $T$  is the number of layers in the model.
Table 3: Model asymptotic runtime complexities. Observe that the amortized complexity of 2-RN becomes obselect by the fact that its forward pass being prohibitive in practice (quadratic in  $|V|$ ).
Model	Complexity of a forward pass	Amortized complexity of a query
R-MPNNs	O(T(|E|d + |V|d2))	O(T( |E|d / |R| |V|2 + d2 / |R| |V| + d))
C-MPNNs	O(T(|E|d + |V|d2))	O(T( |E|d / |V| + d2))
GraIL	O(T(|E|d2 + |V|d2))	O(T(|E|d2 + |V|d2))
2-RN	O(T(|V| |E|d + |V|2d2))	O(T( |E|d / |R| |V| + d2 / |R| + d))
Runtime of C-MPNNs. Let us note that C-MPNNs subsume NBFNets. Indeed, if we consider C-MPNNs without readout and set the history function as  $f(t) = 0$ , we obtain (a slight generalization of) NBFNets, as stated in the paper. In terms of the runtime, C-MPNNs and NBFNets are comparable: even though the readout component of C-MPNNs incurs a linear overhead, this is dominated by other factors. Thus, as shown in Lemma 2 of Zhu et al. [40], the total complexity for a forward pass is  $\mathcal{O}(T(|E|d + |V|d^2))$  to compute  $|V|$  queries at the same time, resulting an amortized complexity of  $\mathcal{O}(T(\frac{|E|d}{|V|} + d^2))$ . See Zhu et al. [40] for detailed discussion and proof.
Comparison with R-MPNNs. As directly carrying out relational message passing in R-MPNN for each triplet is costly, a common way to carry out link prediction task is to first compute all node representations and then use a binary decoder to perform link prediction. This would result in  $\mathcal{O}(T(|E|d + |V|d^2))$  for a complete forward pass. Consequently, we obtain  $|R||V|^2$  triplets, and since passing through binary decoder has a complexity of  $\mathcal{O}(d)$ , we have that the amortized complexity for each query is  $\mathcal{O}(T(\frac{|E|d}{|R||V|^2} +\frac{d^2}{|R||V|} +d))$ . Although the complexity of a forward pass for R-MPNN and C-MPNN are the same, we compute  $|R||V|^2$  queries with R-MPNN but only  $|V|$  queries with C-MPNN, resulting difference in amortized complexity of a single query. However, as discussed in Section 4, this comes at the cost of expressivity.
Comparison with the architectures using labeling trick. GraIL is an architecture using labeling trick and we focus on this to make the comparison concrete. In terms of runtime, architectures that rely on the labeling trick are typically slower, since they need to label both the source and the target node, as we state in Section 4. This yields worse runtime complexity for these models, and particularly for  $\text{GraIL}^4$ , with an amortized complexity of a query being the same as the one for a forward pass, that is,  $\mathcal{O}(T(|E|d^2 + |V|d^2))$ . In C-MPNNs, we only label the source node  $u$ , which allows parallel comparison for all queries in the form  $q(u, ?)$ : a single forward pass would compute all hidden representations of these queries. With the labeling trick,  $|V|$  forward passes need to be carried out as each time, both source  $u$  and target  $v$  need to be specified. See Zhu et al. [40] for detailed discussion and proof.
Comparison with higher-order GNNs. Regarding higher-order GNNs, we take 2-RN, the neural architecture from Barceló et al. [4] corresponding to  $\mathsf{rw}l_{2}$ . These higher-order models require  $\mathcal{O}((|V||E|d + |V|^2d^2))$  in each layer (assuming each message function takes  $\mathcal{O}(d)$  as before) to update all pairwise representations in a single forward pass, which is computationally prohibitive in practice. This makes the study of the amortized complexity obsolete, but we provide an analysis for the sake of completeness. Similar to R-MPNN, we need an additional query-specific unary decoder for the task of link prediction on knowledge graphs, resulting in  $\mathcal{O}(d)$  complexity overhead. The amortized complexity of a single query is thus  $\mathcal{O}(T(\frac{|E|d}{|R||V|} +\frac{d^2}{|R|} +d))$ .
C Experiments on inductive link prediction
C.1 Details of the experiments reported in Section 6
In this section, we report the details of the experiments reported in the body of this paper. Specifically, we report the performance of some baseline models (Table 4), dataset statistics (Table 5), hyperparameters (Table 6), and the number of trainable parameters for each model variation (Table 7).
Comparing to C-MPNN, the complexity for  $\mathrm{MSG}_r$  is different because GraIL utilizes RGCN as relational message passing model, which needs  $\mathcal{O}(d^2)$  for linear transformation in its  $\mathrm{MSG}_r$ .
Table 4: Inductive relation prediction results. We use basic C-MPNN architecture with AGG = sum, MSG =  $\mathsf{MSG}_r^1$ , and  $\mathrm{INIT} = \mathrm{INIT}^2$  with no readout component. All the baselines are taken from the respective works.
Model	WN18RR	FB15k-237
v1	v2	v3	v4	v1	v2	v3	v4
NeuralLP[35]	0.744	0.689	0.462	0.671	0.529	0.589	0.529	0.559
DRUM [25]	0.744	0.689	0.462	0.671	0.529	0.587	0.529	0.559
RuleN [20]	0.809	0.782	0.534	0.716	0.498	0.778	0.877	0.856
GraIL [30]	0.825	0.787	0.584	0.734	0.642	0.818	0.828	0.893
C-MPNN	0.932	0.896	0.900	0.881	0.794	0.906	0.947	0.933
Table 5: Dataset statistics for the inductive relation prediction experiments. #Query* is the number of queries used in the validation set. In the training set, all triplets are used as queries.
Dataset	#Relation	Train & Validation	Test
#Nodes	#Triplet	#Query*	#Nodes	#Triplet	#Query
WN18RR	v1	9	2,746	5,410	630	922	1,618	188
v2	10	6,954	15,262	1,838	2,757	4,011	441
v3	11	12,078	25,901	3,097	5,084	6,327	605
v4	9	3,861	7,940	934	7,084	12,334	1,429
FB15k-237	v1	180	1,594	4,245	489	1,093	1,993	205
v2	200	2,608	9,739	1,166	1,660	4,145	478
v3	215	3,668	17,986	2,194	2,501	7,406	865
v4	219	4,707	27,203	3,352	3,051	11,714	1,424
Table 6: Hyperparameters for inductive experiments with C-MPNN.
Hyperparameter	WN18RR	FB15k-237
GNN Layer	Depth(T)	6	6
Hidden Dimension	32	32
Decoder Layer	Depth	2	2
Hidden Dimension	64	64
Optimization	Optimizer	Adam	Adam
Learning Rate	5e-3	5e-3
Learning	Batch size	8	8
#Negative Samples	32	32
Epoch	20	20
Adversarial Temperature	1	1
Table 7: Number of trainable parameters used in inductive relation prediction experiments for C-MPNN architectures with  $\mathrm{INIT}^2$  initialization.
Model architectures	WN18RR	FB15k-237
AGG	MSGr	v1	v2	v3	v4	v1	v2	v3	v4
sum	MSG1r	132k	144k	157k	132k	2,310k	2,564k	2,755k	2,806k
sum	MSG2r	21k	22k	22k	21k	98k	107k	113k	115k
sum	MSG3r	128k	141k	153k	128k	2,240k	2,488k	2,673k	2,722k
PNA	MSG1r	199k	212k	225k	199k	2,377k	2,632k	2,823k	2,874k
PNA	MSG2r	89k	89k	90k	89k	165k	174k	181k	183k
PNA	MSG3r	196k	208k	221k	196k	2,308k	2,555k	2,740k	2,790k
All of the model variations minimize the negative log-likelihood of positive and negative facts. We follow the partial completeness assumption [10] by randomly corrupting the head entity or the tail entity to generate the negative samples. We parameterize the conditional probability of a fact  $q(u, v)$  by  $p(v \mid u, q) = \sigma(f(\boldsymbol{h}_{v|u, q}^{(T)}))$ , where  $\sigma$  is the sigmoid function and  $f$  is 2-layer MLP. Following RotatE [29], we adopt self-adversarial negative sampling by sampling negative triples from the following distribution with  $\alpha$  as the adversarial temperature:
$$
\mathcal {L} (v \mid u, q) = - \log p (v \mid u, q) - \sum_ {i = 1} ^ {k} w _ {i, \alpha} \log \left(1 - p \left(v _ {i} ^ {\prime} \mid u _ {i} ^ {\prime}, q\right)\right)
$$
where  $k$  is the number of negative samples for one positive sample and  $(u_i', q, v_i')$  is the  $i$ -th negative sample. Finally,  $w_i$  is the weight for the  $i$ -th negative sample, given by
$$
w _ {i, \alpha} := \operatorname {S o f t m a x} \left(\frac {\log \left(1 - p \left(v _ {i} ^ {\prime} \mid u _ {i} ^ {\prime} , q\right)\right)}{\alpha}\right).
$$
C.2 Experiments for evaluating the effect of initialization functions
Initialization functions (Q3). We argued that the initialization function  $\mathrm{INIT}(u,v,q)$  needs to satisfy the property of target node distinguishability to compute binary invariants. To validate the impact of different initialization regimes, we conduct a further experiment which is reported in Table 8, with the same experiment settings as in Section 6. In addition to the initialization functions  $\mathrm{INIT}^1$ ,  $\mathrm{INIT}^2$ ,  $\mathrm{INIT}^3$  defined in Section 4.1, we also experiment with a simple function  $\mathrm{INIT}^0$  which assigns 0 to all nodes. As expected, using  $\mathrm{INIT}^0$  initialization, results in a very sharp decrease in model performance in WN18RR, but less so in FB15k-237. Intuitively, the model suffers more in WN18RR since there are much fewer relations, and it is harder to distinguish node pairs without an initialization designed to achieve this. Perhaps one of the simplest functions satisfying target node distinguishability criteria is  $\mathrm{INIT}^1 = \mathbb{1}_{u = v} * \mathbf{1}$ , which pays no respect to the target query relation. Empirically,  $\mathrm{INIT}^1$  achieves strong results, showing that even the simplest function ensuring this property could boost the model performance. Interestingly, the performance of models using  $\mathrm{INIT}^1$  match or exceed models using  $\mathrm{INIT}^2$ , even though the latter additionally has a relation-specific learnable query vector. Note, however, that this shall not undermine the role of the learnable query relation: integrating the learnable query vector either in the initialization function or in the message computation function seems to suffice.
Table 8: Inductive relation prediction of C-MPNN using AGG = sum,  $\mathrm{MS}{\mathrm{G}}_{r} = {\mathrm{{MSG}}}_{r}^{1},f\left( t\right)  = t$  and different initialization methods.
Initialization
INIT(u,v,q)	WN18RR	FB15k-237
v1	v2	v3	v4	v1	v2	v3	v4
INIT0(u,v,q)	0.615	0.715	0.811	0.654	0.777	0.903	0.894	0.910
INIT1(u,v,q)	0.932	0.894	0.902	0.883	0.809	0.927	0.944	0.911
INIT2(u,v,q)	0.932	0.896	0.900	0.881	0.794	0.906	0.947	0.933
INIT3(u,v,q)	0.934	0.890	0.894	0.877	0.804	0.924	0.941	0.944
Random initialization (Q3). The idea of random node initialization is known to lead to more expressive models [26, 1]. Inspired by this, we can incorporate varying degrees of randomization to the initialization, satisfying the target node distinguishability property in expectation. The key advantage is that the resulting models are still inductive and achieve a nontrivial expressiveness gain over alternatives. In this case, the models with  $\mathrm{INIT}^3$  perform closer to the models with  $\mathrm{INIT}^2$ , but we do not see a particular advantage on these benchmarks.
D Experiments on transductive link prediction
We further conducted transductive link prediction experiments to empirically validate that C-MPNNs are more expressive than R-MPNNs via the abstraction given by  $\mathsf{rawl}_2$  and  $\mathsf{rw}l_{1}$ , respectively (Q4).
Table 9: Transductive knowledge graph completion task of C-MPNNs and R-MPNNs on WN18RR and FB15k-237. Here, C-MPNN-basic refers to basic C-MPNN model without readout.
Model class	Architectures	WN18RR	FB15k-237
MR	MRR	Hits@10	MR	MRR	Hits@10
R-MPNNs	RGCN	3069	0.367	0.405	210	0.205	0.387
CompGCN	3590	0.433	0.519	217	0.334	0.514
C-MPNNs	NeuralLP	-	0.435	0.566	-	0.240	0.362
DRUM	-	0.486	0.586	-	0.343	0.516
C-MPNN-basic	687	0.534	0.643	121	0.400	0.583
Table 10: Dataset statistics for transductive experiments on WN18RR and FB15k-237.
Dataset	#Nodes	#Relation	#Train	#Triplet #Valid	#Test
WN18RR	40,943	11	86,835	3,034	3,134
FB15k-237	14,541	237	272,115	17,535	20,466
Table 11: Hyperparameters for transductive experiments on FB15k-237 and WN18RR with C-MPNN.
Hyperparameter	WN18RR	FB15k-237
GNN Layer	Depth(T)	6	6
Hidden Dimension	32	32
Decoder Layer	Depth	2	2
Hidden Dimension	64	64
Optimization	Optimizer	Adam	Adam
Learning Rate	5e-3	5e-3
Learning	Batch size	8	8
#Negative Sample	32	32
Epoch	20	20
Adversarial Temperature	1	1
D.1 Experiments on WN18RR and FB15k-237
Datasets. We use two benchmark datasets for transductive link prediction experiments, namely WN18RR [31] and FB15k-237 [9], with the provided standardized train-test split. Similar to the inductive relation prediction experiments, we augment the fact  $r(u,v)$  with its inverse fact  $r^{-1}(v,u)$ . The detailed data statistics are shown in Table 10.
Implementation. For basic C-MPNN without readout (denoted as C-MPNN-basic in Table 9), all hyper-parameters used are reported in Table 11, and we adopt layer-normalization [2] and short-cut connection after each aggregation and before applying ReLU. We also discard the edges that directly connect query node pairs to prevent overfitting. For RGCN, we use two layers, each with a hidden dimension of 100. We consider the same basis-decomposition trick with a basis number equal to 30. In addition, we train the model with a learning rate of 0.1 and a dropout rate of 0.2, using Adam optimizer for 10,000 epochs. For CompGCN, we use three layers, each with hidden dimension 200, and adopt distmult scoring function as well as element-wise multiplication as a composition function. During the training process, we consider a learning rate of 0.001 and a dropout rate of 0.1, using Adam optimizer for 500 epochs. Additionally, since there are no node features for both of the datasets, we initialize node representations using learnable embeddings with Xavier initialization for RGCN and CompGCN. We take the results of NeuralLP from Yang et al. [35], and the results of DRUM from Sadeghian et al. [25], respectively. The best checkpoint for each model instance is selected based on its performance on the validation set. All experiments are performed on one NVIDIA V100 32GB GPU.
Evaluation. We consider filtered ranking protocol [6] with 1 negative sample per positive triplet, and report Mean Rank(MR), Mean Reciprocal Rank(MRR), and Hits@10 for each model. We also report averaged results of five independent runs for all experiments.
Results. From Table 9, it is evident that models under the class of C-MPNN consistently outperform R-MPNN models across both datasets and all evaluation metrics, with only one exception on FB15k-237 with NeuralLP. This aligns with our theoretical understanding, as C-MPNN models are inherently more expressive than R-MPNNs. In particular, the C-MPNN-basic model stands out by achieving the lowest MR and the highest values for both MRR and Hits@10, surpassing all other models by a significant margin, underscoring its efficiency and robustness in transductive knowledge graph completion tasks.
D.2 Experiments on biomedical datasets
To compare R-MPNNs and C-MPNNs on large-scale graphs, we carried out additional transductive knowledge graph completion experiments on biomedical datasets: Hetionet [15] and ogbl-biokg [16].
Datasets. Hetionet [15] is a large biomedical knowledge graph that integrates data from 29 different public databases, representing various biological connections and associations, and ogbl-biokg is a large-scale biomedical knowledge graph dataset, developed as a part of the Open Graph Benchmark (OGB) [16] suite. We have used the provided standardized train-test split for both datasets and similarly augmented every fact  $r(u,v)$  with its inverse relation  $r^{-1}(u,v)$ . We present the detailed dataset statistics in Table 13.
Implementation. We study the basic C-MPNN without readout, referred to as C-MPNN in the Table 12, the epoch number is set to 1 due to the datasets being larger and denser, leading to longer execution times for C-MPNN. All hyperparameters used for C-MPNN are reported in Table 14, and we adopt layer-normalization [2] and short-cut connection after each aggregation and before applying ReLU. We also discard the edges that directly connect query node pairs. For RGCN, we consider a batch size of 65536, a learning rate of 0.001, and 100 epochs for both datasets. On Hetionet, we use a hidden dimension size of 200, and a learning rate of 0.001 across 4 layers. On ogbl-biokg, we consider a dimension size of 500, and a dropout rate of 0.2 across 2 layers. For CompGCN on Hetionet, the parameters include a batch size of 65536, 100 epochs, a dimension size of 200, a learning rate of 0.01, and a single layer. We adopt distmult scoring function and element-wise multiplication as a composition function. On ogbl-biokg, CompGCN results cannot be reproduced to the best of our effort due to an out-of-memory (OOM) error. The best checkpoint for each model instance is selected based on its performance on the validation set. All experiments are performed on one NVIDIA V100 32GB GPU. In addition, since there are no node features for both of the datasets, we initialize node representations using learnable embeddings with Xavier initialization for RGCN and CompGCN.
Evaluation. We consider filtered ranking protocol [6], but restrict our ranking to entities of the same type. With each positive triplet, we use 32 number of negative samples and report the averaged results of Mean Reciprocal Rank(MRR), Hits@1, and Hits@10 over five independent runs for each model.
Results. From Table 12, it is evident that C-MPNN outperforms R-MPNN on both datasets by a large margin, despite the challenges posed by the large and dense nature of biomedical knowledge graphs. These results are reassuring as they have further consolidated our theory, showing that the expressivity gain of C-MPNNs compared to R-MPNNs has a significant impact on real-world biomedical datasets.
Table 12: Transductive experiments on biomedical knowledge graphs. We use basic C-MPNN architecture with  $\mathrm{AGG} = \mathrm{sum}$ ,  $\mathrm{MSG} = \mathrm{MSG}_r^1$ , and  $\mathrm{INIT} = \mathrm{INIT}^2$  with no readout component. OOM stands for out of memory.
Model	HETionet	ogbl-biokg
MRR	Hits@1	Hits@10	MRR	Hits@1	Hits@10
RGCN	0.120	0.067	0.228	0.636	0.511	0.884
CompGCN	0.152	0.083	0.292	OOM	OOM	OOM
C-MPNN	0.479	0.394	0.649	0.790	0.718	0.927
Table 13: Dataset statistics for transductive biomedical knowledge graph completion.
Dataset	#Nodes	#Node Type	#Relation	#Train	#Triplet #Valid	#Test
Hetionet	47,031	11	24	1,800,157	225,020	225,020
ogbl-biokg	93,773	5	51	4,762,677	162,886	162,870
Table 14: Hyperparameters for the transductive experiments on biomedical knowledge graphs with C-MPNN.
Hyperparameter	Hetionet	ogbl-biokg
GNN Layer	Depth(T)	4	6
Hidden Dimension	32	32
Decoder Layer	Depth	2	2
Hidden Dimension	32	32
Optimization	Optimizer	Adam	Adam
Learning Rate	2e-3	2e-4
Learning	Batch size	64	8
#Negative Sample	32	32
Epoch	1	1
Adversarial Temperature	1	0.5
E Evaluating the power of global readout on a synthetic dataset
We constructed a synthetic experiment to showcase the power of global readout (Q5).
Dataset. We proposed a synthetic dataset TRI-SQR, which consists of multiple pairs of knowledge graphs in form  $(G_{1},G_{2})$  s.t.  $G_{1} = (V_{1},E_{1},R,c_{1})$ ,  $G_{2} = (V_{2},E_{2},R,c_{2})$  where  $R = \{r_0,r_1,r_2\}$ . For each pair, we constructed as follows: