Towards Foundation Models for Knowledge Graph Reasoning
Authors: Mikhail Galkin
Xinyu Yuan
Hesham Mostafa
Jian Tang
Zhaocheng Zhu
Venue: ICLR 2024 poster (2024)
Keywords: graph neural networks
foundation model
knowledge graph
link prediction
knowledge graph reasoning
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: An approach for building foundation models for KG reasoning: pre-train on one (or several) graphs, run zero-shot inference and fine-tuning on any multi-relational graph

Abstract: Foundation models in language and vision have the ability to run inference on any textual and visual inputs thanks to the transferable representations such as a vocabulary of tokens in language. 
Knowledge graphs (KGs) have different entity and relation vocabularies that generally do not overlap.
The key challenge of designing foundation models on KGs is to learn such transferable representations that enable inference on any graph with arbitrary entity and relation vocabularies.
In this work, we make a step towards such foundation models and present ULTRA, an approach for learning universal and transferable graph representations. 
ULTRA builds relational representations as a function conditioned on their interactions.
Such a conditioning strategy allows a pre-trained ULTRA model to inductively generalize to any unseen KG with any relation vocabulary and to be fine-tuned on any graph.
Conducting link prediction experiments on 57 different KGs, we find that the zero-shot inductive inference performance of a single pre-trained ULTRA model on unseen graphs of various sizes is often on par or better than strong baselines trained on specific graphs. 
Fine-tuning further boosts the performance.

Published as a conference paper at ICLR 2024
TOWARDS FOUNDATION MODELS FOR KNOWLEDGE GRAPH REASONING
Mikhail Galkin' ; Xinyu Yuan Hesham Mostafa', Jian Tang2A. Zhaocheng Zhu Intel AI Lab, ?Mila University of Montréal "HEC Montréal & CIFAR Al Chair
ABSTRACT
Foundation models in language and vision have the ability inference any textual and visual inputs thanks to the transferable representations such as a vocabulary of tokens in language. Knowledge graphs (KGs) have different entity and relation vocabularies that generally do not overlap. The key challenge of de- signing foundation models on KGs to learn such transferable representations that enable inference on any graph with arbitrary entity and relation vocabularies. In this work, we make a step towards such foundation models and present ULTRA, approach for learning universal and transferable graph representations. UI.TRA builds relational representations as function conditioned on their interactions. Such conditioning strategy allows pre-trained ULTRA model to inductively generalize to any unseen KG with any relation vocabulary and to fine-tuned on any graph. Conducting link prediction experiments on 57 different KGs, we find that the zero-shol inductive inference performance of a single pre-trained ULTRA model unseen graphs of various sizes often on par better than strong base- lines trained on specific graphs. Fine-tuning further boosts the performance. The code is available: https: h/gthtbt.bt..///./// com/ Cerrraaheeaninngttte.
0.9
ULTRA fine-tuned
ULTRA 0-shot
Supervised SOTA
0.8
Freebase-derived
Wikideta-derived
NELL-derived
WordNet
Other
0.7
0.6
MRR
0.5
0.4
0.3
0.2
0.1
FB V2
V1
F8 V4
FB-100
FB-75
FB-25
FB-50
FB237 50
FB237 20
FB237_10 10
CoDEx-S
WDeinger
WK-75
CoDEx.
WK-25
ILPC-L
ILPC-S
WK- 100
WK-50
V1
NL y2
NL V3
NELL-895 -995
NL V4
NL-100
NL-50
NL-25
NL-75
NL-0
NEL TELL23K
WN V1
WN V2
WN V3
YAGO 310
DBP100K
Hetlonet
Aristo-v4
CNett 100K
Average
WN18RR
FB15k237
CaDExM
WIV
Figure I: Zero-shot and fine-tuned MRR (higher is better) of ULTRA pre-trained on three graphs (FB15k-237, WN18RR, CoDEE-Medium) On average, zero-shot performance is better than best
reported baselines trained on specific graphs (0.395 vs 0.344). More results in Figure 4 and Table
INTRODUCTION
Modern machine learning applications increasingly rely the pre-training and fine-tuming paradigm. In this paradigm,& a backbone model often trained on large datasets in a self-supervised fashion is commonly known a foundation model (FM) (Bommasani et al., 2021). After pre- training, FMs can be fine-tuned on smaller downstream tasks. In order to transfer to a broad set of downstream tasks, FMs leverage certain invariances pertaining to a domain of interest, e.g., large
*Correspondence: mi khai i memlinellccm comm

---

Published as a conference paper at ICLR 2024
language models like BERT (Devlin et al., 2019), GPT-4 (OpenAI, 2023), Llama-2 (Touvron al., 2023) operate on a fixed vocabulary of tokens; vision models operate on raw pixels (He et al., 2016; Radford et al., 2021) image patches (Dosovitskly et al., 2021); chemistry models (Ying et al., 2021; Zheng et al., 2023) learn a vocabulary of atoms from the periodic table.
Representation learning knowledge graphs (KGs), however, has not yet witnessed the benefits of transfer learning despite a wide range of downstream applications such as precision medicine (Chan- dak et al., 2023), materials science (Venugopal al., 2022; Statt et al., 2023), virtual assistants (Ilyas et al., 2022), or product graphs in e-commerce (Dong, 2018). The key problem is that different KGs typically have different entity and relation vocabularies. Classic transductive KG embedding mod- els (Ali et al., 2021) learn entity and relation embeddings tailored for each specific vocabulary and cannot generalize even to new nodes within the same graph. More recent efforts towards generaliza- tion across the vocabularies are known as inductive learning methods (Chen et al., 2023). Most of the inductive methods (Teru et al., 2020; Zhu et al., 2021; Galkin et al., 2022b; Zhang & Yao, 2022) generalize to new entities at inference time but require fixed relation vocabulary to learn entity representations as a function of the relations. Such inductive methods still cannot transfer to KGs with different set of relations, e.g., training on Freebase and inference Wikidata.
The main research goal of this work is finding the invariances transferable across graphs with ar- bitrary entity and relation vocabularies. Leveraging and learning such invariances would enable the pre-train and fine-ture paradigm of foundation models for KG reasoning where a single model trained on one graph (or several graphs) with one set of relations would be able to zero-shot transfer to any new, unseen graph with a completely different set of relations and relational patterns. Our approach to the problem is based on two key observations: (I) even if relations vary across the datasets, the interactions between the relations may be similar and transferable; (2) initial relation representations may be conditioned on this interaction bypassing the need for any input features. To this end, we propose ULTRA, method for unified, learnable, and transferable KG representations that leverages the invariance of the relational structure and employs relative relation representations on top of this structure for parameterizing any unseen relation. Given any multi-relational graph, ULTRA first constructs a graph of relations (where each node is a relation from the original graph) capturing their interactions. Applying graph neural network (GNN) with a labeling trick (Zhang et al., 2021) over the graph of relations, ULTRA obtains a unique relative representation of each relation. The relation representations can then be used by any inductive learning method for down- stream applications like KG completion. Since the method does not learn any graph-specifs entity or relation embeddings nor requires any input entity relation features, ULTRA enables zero-shot generalization to any other KG of any size and any relational vocabulary.
Experimentally, we show that ULTRA paired with the NBENet (Zhu et al., 2021) link predictor pre- trained on three KGs (FB15k-237, WNISRR, and CoDEX-M derived from Freebase, WordNet, and Wikidata, respectively) generalizes to 50+ different KGs with sizes of 1,000-120,000 nodes and 5K-1M edges. ULTRA demonstrates promising transfer learning capabilitiess where the zero-shot inference performance on those unseen graphs might exceed strong supervised baselines by up to 300%. The subsequent short fine-tuning of ULTRA often boosts the performance even more.
2 RELATED WORK
Inductive Link Prediction. In contrast to transductive methods that only support a fixed set of entities and relations during training, inductive methods (Chen et al., 2023) aim at generalizing to graphs with unseen nodes (with the same set of relations) or to both new entities and relations. The majority of existing inductive methods such as GralL (Teru et al., 2020), NBFNet (Zhu et al., 2021), NodePiece (Galkin et al., 2022b), RED-GNN (Zhang & Yao, 2022) can generalize to graphs only with new nodes, but not to new relation types since the node representations are constructed as a function of the fixed relational vocabulary.
First approaches that support unseen relations at inference resorted to meta-learning and few-shot learning (Chen et al., 2019; Zhang et al., 2020; Huang 2022). Meta-learning is computationally expensive and is hardly scalable to large graphs. Few-shot learning methods do not work on the whole new unseen inference graph but instead mine many support sets akin to subgraph sampling.
Both RMPI (Geng et al., 2023) and InGram (Lee et al., 2023) employ graphs of relations to general- ize to unseen domains. However, RMPI suffers from the same computational and scalability issues
2

---

Published as a conference paper at ICLR 2024
as subgraph sampling methods. InGram is more scalable but its featurization strategy relies on the discretization of node degrees that only transfers to graphs of a similar relational distribution and does not transfer to arbitrary KGs. Gao et al. (2023) introduce the notion of double equivariance, Le,, relation exchangeability in multi-relational graphs, as a general theoretical framework for in- ductive reasoning that transfers any relations inference. ISDEA (Gao et al., 2023) is the first approach to design doubly equivariant GNNs and MTDEA (Zhou et al., 2023) further extends the theory to partial equivariance, However, ISDEA and MTDEA are computationally expensive and cannot scale to graphs considered in this work. Similarly to RMPI, InGram, ISDEA, and MTDEA, ULTRA transfers to any unseen KG in the zero-shot fashion, but exhibits better generalization ca- pabilities, scales to graphs of millions of edges, and introduces only a marginal inference overhead (one-step pre-computation) to any inductive link predictor.
Text-based methods. A line of inductive link prediction methods like BLP (Daza et al., 2021), KEPLER (Wang et al., 2021), StATIK (Markowitz et al., 2022), RAILD (Gesese et al., 2022) rely textual descriptions of entities and relations and use language models to encode them. PRODIGY (Huang et al., 2023a) uses text features for few-shot node classification tasks. We deem this family of methods orthogonal to ULTRA and leverage only structural information encoded in the graph. Furthermore, the zero-shot inductive transfer an arbitrary KG studied in this work implies running inference on graphs from different domains that might need different language encoders, e.g. models trained on general English data are unlikely to transfer to graphs with descriptions in other languages or domain aanain- graphs.
assume the graphs do not have any input features
PRELIMINARIES
Knowledge Graph and Inductive Learning. Given a finite set of entities V (nodes), finite set of relations R (edge types), and a set of triples (edges) ε = Xxxx))), ), a knowledge graph G is tuple g (V, R,E), E). In the transductive setup, the graph at training time Grain = (Virain: Rirain; Eraain) and the graph at inference (validation or test) time Gmf = (Viy, Ring Emg) are the same, i.e., Graain = Gig- In the inductive setup, in the general case, the training and inference graphs are different, Grain o Ging- In the easier setup tackled by most of the literature, the relation set R. is fixed and shared between training and inference graphs, i.e., Grrgin = (Virain: R, Emaii)) and Gwy = (Vims R, Emg). The inference graph can be an extension of the training graph if Virain S Ving be separate disjoint graph (with the same set of relations) if Vinainn n Ving = O. In the hardest inductive case, both entities and relations sets are different, i.e., Virain n Van 0 and Reaain n Ring 0. In this work, tackle this harder inductive (also known as fully-inductiie) case with both new, unseen entities and relation types at inference time. Since the harder inductive case (with new relations inference) is strictly a superset of the easier inductive scenario (with the fixed relation set), any model capable of fully-inductive inference is by design applicable in casier inductive scenarios as well.
Problem Formulation. Each triple (h, t) E (V x R x V) denotes head entity h connected to a tail entity& / by relation The knowledge graph reasoning task answers queries (h, r, ?) or (2,r, It is common to rewrite the head-quen, (?, t) as (t, rt. ??) where r is the inverse relation of r. The set of target triples Epres is predicted based on the incomplete inference graph Giny which is a part of the unobservable complete graph Gre = (Ving: Riyg, Eng) where Eng = Eng U Epred-
r, t).
Link Prediction and Labeling Trick GNNs. Standard GNN encoders (Kipf & Welling, 2017; Veličković et al., 2018) including those for multi-relational graphs (Vashishth et al., 2020) underper- form in link prediction tasks due to neighborhood symmetries (automorphismss) that assign different (but automorphic) nodes the same features making them indistinguishable. To break those sym- metries, labeling tricks (Zhang et al., 2021) were introduced that assign each node a unique feature vector based on its structural properties. Most link predictors that use the labeling tricks (Teru et al., 2020; Zhang et al., 2021: Chamberlain et al., 2023) mine numerical features like Double Radius Node Labeling (Zhang & Chen, 2018) or Distance Encoding (Li et al., 2020). In contrast, multi- relational models like NBFNet (Zhu et al., 2021) leverage an indicator function INDICATOR( (h, v, r) and label (initialize) the head node h with the query vector that can be learned while other nodes are initialized with zeros. In other words, final node representations are conditioned on the query re- lation and NBFNet learns conditional node representations. Conditional representations were shown to be provably more expressive theoretically (Huang et al., 2023b) and practically effective (Zhu et al., 2022a; Galkin et al., 2022c) than standard unconditional GNN encoders.
3

---

Published as a conference paper at ICLR 2024
Eatily representations relative to Michaed Jackson
Eatily representatiins relative to Beatles
Relatioe representations relative geare
Relation representations relative
transer
Thriller
Nuthora
E2b petre
Wit
disco
rock
disco
Th
Beutics
12h
Michael
Quincy Jones
George Martin
Mislad Jackson
Quincy Junes
authored
collab
Donald
Robert Floyd
stulent
(4) Relative entity sepresentations transfer cutities (NBFNet, RED-GNN)
(b) Relative relation representations transfer
relations (ULTRA)
Figure 2: (a) relative entity representations used in inductive models generalize to new entities; (b) relative relation representations based on a graph of relations generalize to both new relations and entities. The graph of relations captures four fundamental interactions (t2h, h2h, l12r, h2h) independent from any graph-specifer relation vocabulary and whose representations can be learned.
METHOD
The key challenge of inductive inference with different entity and relation vocabularies is finding transferable invariances that would produce entity and relation representations conditioned on the new graph (as learning entity and relation embedding matrices from the training graph is useless and not transferable). Most inductive GNN methods that transfer to new entities (Zhu et al., 2021; Zhang & Yao, 2022) learn relative entity representations conditioned on the graph structure as shown in Fig. (a). For example, given a, b, are variable entities and a as a root node labeled with INDICATOR(). a structure a authored b
e might imply existence of the c. Learning such structure on training set with entities Michael Jackson
genre
c. A a collab, d genre
edge a gense
authored
Thriller genre disco scamlessly transfers to entities Beatles authord, Let It Be gous, rock inference time without learning entity embeddings thanks to the same relational structure and rel- ative entity representations. As training and inference relations are the same Rmain Ring, such approaches learn relation embedding matrices and use relations as invariants.
In ULTRA, we generalize KG reasoning to both new entities and relations (where Rroin + Ran) by leveraging graph of relations, i.e., graph where each node corresponds distinct relation type' in the original graph. While relations at inference time are different, their interactions remain the same and are captured by the graph of relations. For example, Fig. (b), tail node of the authored relation is also a head node of the genre relation. Hence, authored and genre nodes are connected by the tail-to-head edge in the relation graph. Similarly, authored and collab share the same head node in the entity graph and thus are connected with the head-to-hed edge in the relation graph. Overall, we distinguish four such core, fundamental relation-to-relatte to-ralaioon interactions?: tail- tail-to-head (12h), head-to-heat (h2h). head-to-tail (h21), and tail-to-tail (121). Albeit relations in the inference graph in Fig. 2 (b) are different, their graph of relations and relation interactions resemble that of the training graph. Hence, we could leverage the invariance of the relational structure and four fundamental relations to obtain relational representations of the unseen inference graph. As a typical KG reasoning task (h, ?) is conditioned on a query relation q, it is possible to build representations of all relations relative to the query g by using a labeling trick on top of the graph of relations. Such relative relation representations do not need any input features and naturally generalize any multi-relational graph.
Practically (Fig. 3), given query (h, q, ?) over graph G, ULTRA employs three-step algorithm that we describe the following subsections. (1) Lift the original graph 9 to the graph of relations G. Section 4.1: (2) Obtain relative relation representations R2 G.) conditioned on the query relation in the relation graph G, Section 4.2; (3) Using the relation representations R. as starting relation features, run inductive link prediction on the original graph G Section 4.3.
also add inverse relations as nodes to relation graph. Other strategies for capturing relation-to-relatin interactions might exist heside those four types and we leave their exploration for future work.
4

---

Published as a conference paper at ICLR 2024
Knovicdkg:Gng Query
Leamn Reielive Redation Represerrctionee
Refetive Entity Regassetotionss
Tuiller
Tuucker
great
INDUGAIOOs.N,.C
ENDICATON,(A, GNN (0
1
GNN, (b1 Sr Ryach HT
Miclad
autloood
Miched
lorees
[cres
Incluttive Link prediction using relation
Quay::
(Micheel Jackson, geare, ?)
Conditional relation representaionns genre
representaaionn condittione on genre
Figure 3: Given a query (h, q, ?) on graph G, ULTRA (1) builds a graph relations G, with four interactions Ryand (Sec. 4.1); (2) builds relation representations R, conditioned on the query relation q and G, (Sec. 4.2); (3) runs any inductive link predictor on G using representations Rg (Sec. 4.3).
4.1 RELATION GRAPHI CONSTRUCTION
Given a graph 9 = (V, (,,R,E) we first apply the lifting function G, LIFT(g) to build graph of relations G. = (R, Ramd; E where each node distinct relation type* in G. Edges & (R x Rfund x R. in the relation graph Gr denote interactions between relations in the original graph G, and we distinguish four such fundamental relation interactions Rjmnd tail-to-head (t2h) edges, head-to-head (h2h) edges, head-to-tail (h2t) edges, and tail-to-tail (t2t) edges. The full adjacency tensor of the relation graph is A, E IR R x4 Each of the four adjacency matrices can be efficiently obtained with one sparse matrix multiplication (Appendix B).
4.2 CONDITIONAL RELATION REPRESENTATIONS
Given query (h, q, ?) and relation graph Gr, then obtain d-dimensional node representations Rg R R of Gr (corresponding to all edge types R in the original graph 9) conditioned on the query relation q. Practically, we implement conditioning by applying a labeling trick to initialize the node q in G, through the INDICATOR, function and employ message passing GNN over G
h INDICATOR,
lo=g 14, U
h+1 1/o
UPDATE ( AGGREGATE (MESSAGE(H',, T) E (N, E Rymnd)
The indicator function is implemented as INDICATOR, (v, 1u=g 1dd that simply puts a vector of ones a node corresponding to the query relation q, and zeros otherwise. Following Huang et al. (2023b), we found that all-ones labeling with 1d generalizes better to unseen graphs of various sizes than learnable vector. The GNN architecture (denoted as GNN, it operates on the relation graph Gr) follows NBFNet (Zhu et al., 2021) with non-parametric DistMult (Yang et al., 2015) message function and sum aggregation. The only learnable parameters each layer are embeddings of four fundamental interactions Rjund E Ra linear layer for the UPDATE function, and an optional layer normalizaiton. Note that our general setup (Section 3) assumes no given input entity or relation features, of any multi-relational graph.
parameterization strategy can be used to obtain relational representations
To up. each unique relation E R in the query has its matrix of conditional relation representations R.g E RRR|dd used by the entity-level reasoner for downstream applications.
4.3 ENTITY-LEVEL LINK PREDICTION
Given a query (h, q. ?) over a graph & and conditional relation representations R. from the previous step, it is now possible to adapt any off- inductive link predictor that only needs relational features (Zhu et al., 2021; Zhang & Yao, 2022; Zhu et al., 2023; Zhang et al., 2023) to balance between performance and scalability. We modify another instance NBFNet (GNN, as it operates on the entity level) to account for separate relation representations per query:
0 INDICATOR, (u, ④) 1uze # Rgggl E
h. +++1 UPDATE h AGOREGATE MESSAGE( (r))[w (N(()),
[R] nodes after adding inverse relations the original graph.
5

---

Published as a conference paper at ICLR 2024
That is, we first initialize the head node h with the query vector & from R. whereas other nodes initialized with zeros. Each t-th GNN layer applies a non non-linear function g (-) to transform original relation representations to layer-specific relation representations edge features are taken for the MESSAGE function. g()) is implemented as a 2-layer MLP with ReLU. Similar GNN, in Section 4.2, function. After message passing, the final MLP s : R + IK maps the node states to logits p(h, v) denoting the score of a node be a tail of the initial query (h, q, ?).
R' 9 (R,) from which the
sum aggregation and linear layer for the UPDATE
Training. ULTRA can be trained on any multi-relational graph or mixture of graphs thanks to the inductive and conditional relational representations. Following the standard practices in the literature (Sun et al., 2019; Zhu et al., 2021), ULTRA is trained by minimizing the binary cross entropy loss over positive and negative triplets
f = - logg(u;
log(1 p(14, 9, v())
where (u, g, v) is a positive triple in the graph and {(u,, q, of) are negative samples obtained by corrupting either the head tail of the positive sample.
5 EXPERIMENTS
To evaluate the qualities of ULTRA as a foundation model for KG reasoning, we explore the fol- lowing questions: (1) Is pre-trained ULTRA able inductively generalize to unseen KGs in the zero-shot manner? (2) Are there any benefits from fine-tuning ULTRA on a specific dataset? (3) How does a single pre-trained ULTRA model compare to models trained from scratch on each target dataset? (4) Do more graphs in the pre-training mix correspond to better performance?
5.1 SETUP AND DATASETS
Datasets. We conduct broad evaluation on 57 different KGs with reported, non-saturated results on the KG completion task. The datasets can be categorized into three groups:
Transductive datasets (16 graphs) with the fixed of entities and relations at training and inference time (Guain = Guy): FB 15k-237 (Toutanova & Chen, 2015), WNISRR (Dettmers et al., 2018), YAGO3-10 (Mahdisoltani CoDEx (Small, Medium, and Large) (Safavi & Koutra, 2020), WDsinger, NELL23K, FB155237(10), FK152227()0), FB FK15k237(50) (Lv al., 2020), AristoV4 (Chen et al., 2021), DBpedia 100k 100k (Ding et al., 2018), ConceptNet100k (Malaviya et al., 2020), Het- ionet (Himmelstein et al., 2017)
al., 2014), NELL-995 (Xiong et al., 2017),
Inductive entity (e) datasets (18 graphs) with new entities inference time but with the fixed set of relations (Vrmin * Ving. Rumin = Rgn)): 12 datasets from GraIL (Teru et al., 2020), 4 graphs from INDIGO (Liu al., 2021; Hamaguchi et al., 2017), and 2 ILPC 2022 datasets (Small and Large) (Galkin et al., 2022a).
Inductive entity and relation (e, r) datasets (23 graphs) where both entities and relations
inference are new (Venain + Vinf, Ranan * Rey):: : 13 graphs from INGRAM (Lee et al., 2023) and 10 graphs from MTDEA (Zhou et al., 2023).
In practice however, a pre-trained ULTRA operates in the inductive (e,r) mode on all datasets (apart from those in the training mixture) as their sets of entities, relations, and relational structures are different from the training set. The dataset sizes vary from 1k to 120k entities and 1k-2M edges in the inference graph. We provide more details on the datasets in Appendix A.
Pretraining and Fine-tuning. ULTRA is pre-trained on the mixture of 3 standard KGs (WNI8RR, CoDEx-Medium, FB 15k237) to capture the variety of possible relational structures and sparsities in respective relational graphs G,.. ULTRA relatively small (177k parameters in total, with 60k parameters in GNN, and 1 17k parameters in GNN.) and is trained for 200,000 steps with batch size of 64 with Adam W optimizer on 2 A100 (40 GB) GPUs. All finetuning experiments were done on single RTX 3090 GPU. More details on hyperparameters and training are in Appendix C.

---

Published as a conference paper at ICLR 2024
ULT Brwekiredd
ULTRA Doint
Superiieed BO
0.8
0.8
0.
Morgoo
0.6
04
Messton
Mrchhoath
MTSHI
FENELL
NIT nosh
MTZ
MTSat
MT2
Avg.
NT
FBNELL
MF200
MTbart
NT20g
BMh
BM.3k
Moatm
FENELL
Mi2z0l
Aug.
MITI
Figure 4: ULTRA performance on 14 inductive datasets from MTDEA (Zhou et al., 2023) and INDIGO (Liu et al., 2021) for 8 of which only an approximate metric Hits @ (50 negs) is available (center). We also report full MRR (left) and Hits@ 10 (right) computed on the entire entity sets demonstrating that @its@ (50 negs) overestimates the real performance.
Table 1: Zero-shot and fine-tuned performance of ULTRA compared the published supervised SOTA on 51 datasets (as in Fig. I and Fig. 4). The zero-shot ULTRA outperforms supervised base- lines on average and on inductive datasets. Fine-tuning improves the performance even further. We report pre-training performance to the fine-tuned version. More detailed results are in Appendix D.
Inductive (c) (c,r)
Transductive (13 graphs)
Total Avg (40 graphs)
Pretraining
Inductive (c) 4 (c.r) (8 graphs)
Model
(27 graphs)
(3 graphs)
MRR
I@@ 10
MRR
I1@10
MRR
I@@10 10
MRR
@@@10 ||
Hits 10 (50 negs)
Supervised SOTA
0.342
0.482
0.348
0.494
0.344
0.486
0.439
0.585
0,731
ULTRA 0-shot
0.435
0.603
0.312
0,458
0.395
0.556
0.859
ULTRA fine-tuned
0.443
0.615
0.379
0.543
0.422
0.592
0.407
0.568
0.896
Evaluation Protocol. We report Mean Reciprocal Rank (MRR) and Hits @ 10 @H@ 10) as the main performance metrics evaluated against the full entity set of the inference graph. For each triple, we report the results of predicting both heads and tails. Only in three datasets from Lv et al. (2020) we report tail-only metries similar to the baselines. In the zero-shot inference scenario, we run pre-trained model on the inference graph and test set of triples. In the fine-tuning case, we further train the model on the training split of each dataset retaining the checkpoint of the best validation set MRR. We run zero-shot inference experiments once as the results are deterministic, and report an average of runs for each fine-tuning run on each dataset.
Baselines. On each graph, we compare ULTRA against the reported state-of-the-ar model (we list SOTA for all 57 graphs in Appendix A). To date, all of the reported SOTA models are trained end-to- end specifically on each target dataset. Due to the computational complexity of baselines, the only existing results on 4 MTDEA datasets (Zhou et al., 2023) and 4 INDIGO datasets (Liu et al., 2021) report Hits @ 10 against 50 randomly chosen negatives. We compare ULTRA against those baselines using this @its@ 10 (50 negs) metric as well as report the full performance the whole entity sets.
5.2
MAIN RESULTS: ZERO-SHIT INFERENCE AND FINE-TUNING OF ULTRA
The main experiment reports how ULTRA pre-trained on 3 graphs inductively generalizes to 54 other graphs both in the zero-shot (0-shot) and fine-tuned cases. Fig. 1 compares ULTRA with supervised SOTA baselines on 43 graphs that report MRR on the full entity set. Fig. presents the comparison on the rest 14 graphs including 8 graphs for which the baselines report Hits @ 10 (50 negs). The aggregated results on 51 graphs with available baseline results are presented in Table and the complete evaluation on 57 graphs grouped into three families according to Section 5.1 is in Table 2. Full per-dataset results with standard deviations can found in Appendix D.
On average, ULTRA outperforms the baselines even in the 0-shot inference scenario both in MRR and Hits 10. The largest gains are achieved on smaller inductive graphs, e.g., on FB-25 and FB- 50 0-shot ULTRA yields almost 3x better performance (291% and 289%, respectively). During pre- pre-training, ULTRA does not reach the baseline performance (0.407 vs 0.439 average MRR) and link that with the lower (-shot inference results larger transductive graphs. However, fine-
7

---

Published as a conference paper at ICLR 2024
Table 2: Zero-shot and fine-tuned ULTRA results on the complete set of 57 graphs grouped by the dataset category. Fine-tuning especially helps on larger transductive datasets and boosts the total average MRR by 10%. Additionally, we report (train e2e) the average performance of dataset- specific ULTRA models trained from scratch on each graph. More detailed results are in Appendix D.
Inductive e, (23 graphs)
Inductive e (18 graphs)
Transductive (13 graphs)
Total Avg
Pretraining (3 graphs)
Model
(54 graphs)
MRR
H@ 10
MRR
H@ 10
MRR
H@10
MRR
H@
MRR
H@10
ULTRA (train e2e)
0.392
0.552
0.402
0.559
0.384
0.545
0.393
0.552
0.403
0.562
ULTRA 0-shot
0.345
0,513
0.431
0.566
0.312
0,458
0.366
0.518
ULTRA fine-tuned
0.397
0,556
0.442
0.582
0.379
0.543
0.408
0.562
0.407
0.568
ULTRA
ULTRA0-ssot
0.8
0.6
02
0.0
Metafam
MT4 health
VAN VI
VAN V2
Mr3inda
VAN Vi
NL V2
YAG0810
NL Va
FB V2
FB V1
PB
PB Vg
NELL 195
FONELL
NI
NL: 100
FB-100
DBP 100K
HM indgo
NL-50
WDeinger
WMN va
NL-28 20-2
FB-75
Hestonel
FB-25
MTT MTTinnaltt
NL-75
CaDEsM
FB156237
Anaaoa
GaDENA
FB-50
MTT 11-20x
NL-0
WK-25
MT2 an
ILPC-
ILPC-S
MT4
NEWL zol
rB2077 50
FEZ37 20
FB227.
WIc Toc-T00
WI-30
MT22org
HM 1k
HM Sk
HM 6k
Avg.
Figure 5: Comparison of zero-shot and fine-tuned ULTRA per-dataset performance against training
a model from scratch on each dataset (Train e2e). Zero-shot performance of a single pre-trained model is on par with training from scratch while fine-tuning yields overall best results.
tuning ULTRA effectively bridges this gap and surpasses the baselines. We hypothesize that in larger transductive graphs fine-tuning helps to adapt different graph sizes (training graphs have 15-40k nodes while larger inference ones grow up to 123k nodes).
Following the sample efficiency and fast convergence of NBFNet (Zhu et al., 2021), find that 1000-2000 steps are enough for fine-tuning ULTRA. In some cases (see Appendix D) fine-tuning brings marginal improvements marginal negative effects. Averaged across 54 graphs (Table 2), fine-tuned ULTRA A brings further 10% relative improvement over the zero-shot version.
5.3 ABLATION STUDY
We performed several experiments to better understand the pre-training quality of ULTRA and mea- sure the impact of conditional relation representations on the performance.
Positive transfer from pre-training. We first study how single pre-trained UI.TRA model com- pares to training instances of the same model separately on each graph end-to-end. For that, for each of 57 graphs, we train 3 ULTRA instances of the same configuration and different random seeds until convergence and report the averaged results in Table 2 with per-dataset comparison in Fig. 5. We find that, on average, single pre-trained ULTRA model in the zero-shot regime performs al- most on par with the trained separate models, lags behind those on larger transductive graphs and exhibits better performance on inductive datasets. Fine-tuning a pre-trained ULTRA shows overall the best performance and requires significantly less computational resources than training model from scratch on every target graph.
Number of graphs in the pre-training mix. We then study how inductive inference performance depends on the training mixture. While the main ULTRA model was trained on the mixture of three graphs, here train more models varying the amount of KGs in the training set from a sin- gle FB15k237 to a combination of 8 transductive KGs (more details in Appendix C). For the fair comparison, we evaluate pre- pre-trained models in the zero-shot regime only inductive datasets (41 graphs overall). The results are presented in Fig. where we observe the saturation of performance having more than three graphs in the mixture. We hypothesize that getting higher inference perfor-
8

---

Published as a conference paper at ICLR 2024
Table 3: Ablation study: pre-training and zero-shot inference results of the main ULTRA, ULTRA
without edge types in the relation graph (no etypes), ULTRA without edge types and with InGram- like (Lee et al., 2023) unconditional GNN over relation graph where nodes are initialized with all ones (ones) with Glorot initialization (random). Averaged results over categories of datasets.
Inductive e, (23 graphs)
Inductive e (18 graphs)
Transductive
Total Avg (54 graphs)
Pretraining graphs)
Model
(13 graphs)
MRR
H@ 10
MRR
H@10
MRR
H@
MRR
H@10
MRR
H@ 10
ULTRA
0.345
0,513
0.431
0.566
0.312
0.458
0.366
0,518
0.407
0.568
etypes in rel. graph
0.292
0,466
0.389
0.539
0.258
0.409
0.316
0.477
0.357
0.517
etypes,
0.187
0.328
0.262
0.430
0.135
0.257
0.199
0.345
0.263
0.424
uncond. GNN (ones)
no ctypes,
0.177
0.309
0.250
0.417
0.138
0.255
0.192
0.332
0.266
0.433
uncond. GNN (random)
mance is tied up with model capacity, scale, and optimization. We leave that study along with more principled approached to selecting a pre- pre-training mix for future work.
Conditional unconditional relation graph encoding, To measure the impact of the graph of relations and conditional relation representations, same mixture of three graphs varying several components: (1) exclude four fundamental relation interactions (h2h, h2t, t2h, (2) from the relation graph making it homogeneous and single- relational; (2) homogeneous relation graph with an uncondi- tional GNN encoder following the R-GATv2 architecture from the previous SOTA approach, InGram (Lee et al., 2023). The un- conditional GNN needs input node features and we probed two strategies: Glorot initialization used in Lee et al. (2023) and ini- tializing all nodes with vector of ones 1d
pre-train three more models the
MRR
0.40
0.35
datasets
innd (e)
ind (e.r)
both
Hits110
0.55
The results are presented Table 3 and indicate that ablated models struggle to reach the same pre-training performance and exhibit poor sero generalization performance across all groups of graphs, e.g. up to 48% relative MRR drop (0.192 0.366) on the model with a homogeneous relation graph and ran- domly initialized node states with the unconditional R-GATv2 encoder. We therefore posit that conditional representations (both on relation and entity levels) crucial for transferable representations for link prediction tasks that often require pairwise representations to break neighborhood symmetries.
0.50
2
6
graphs pretraining
Figure 6: Averaged O-shot per- formance on inductive datasets and # graphs in pre-training.
6 DISCUSSION AND FUTURE WORK
Limitations and Future Work. Albeit ULTRA demonstrates promising capabilities as a foundation model for KG reasoning in the zero-shot and fine-tuning regimes, there several limitations and open questions. First, pre-training on more graphs does not often correspond to better inference performance. We hypothesize the reason might be in the overall small model size (177k parameters) and limited model capacity, i.e. with increasing the diversity of training data the model size should increase as well. On the other hand, our preliminary experiments did not show significant improve- ments of scaling the parameter count beyond 200k. We hypothesize it might be an issue of input normalization and model optimization. We plan to address those open questions the future work.
Conclusion. We presented ULTRA, an approach to learn universal and transferable graph represen- tations that can serve as one of the methods towards building foundation models for KG reasoning. ULTRAA enables training and inference on any multi-relations graph without any input features lever- aging the invariance the relational structure and conditional relation representations. Experimen- tally, single pre-trained ULTRA model outperforms state-of-the-art tailored supervised baselines on 50+ graphs of 1k 1-kk nodes even in the zero-shot regime by average 15%. Fine-tuning ULTRA is safple-efficient and improves the average performance by further 10%. We hope that ULTRA contributes to the search for inductive and transferable representations where a single pre-trained model can inductively generalize to any graph and perform a variety of downstream tasks.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
Foundation models can be run on tasks and datasets that were originally not envisioned by authors. Due to the ubiquitous nature of graph data, foundation graph models might be used for malicious activities like searching for patterns anonymized data. On the other, more positive side, foundation models reduce the computational burden and carbon footprint of training many non-transferable graph-specific models. Having a single model with zero-shot transfer capabilities to any graph renders tailored graph-specific models unnecessary. and fine-tuning costs are still lower than training any model from scratch.
REPRODUCIRILITY STATEMENT
The list of datasets and evaluation protocol are presented in Section 5.1. More comments and de- tails on the dataset statistics are available in Appendix A. All hyperparameters can be found in Appendix C, full MRR and Hits 10 results with standard deviations are in Appendix D. The source code available in the supplementary materials.
ACKNOWLEDGMENT
This project is supported by Intel-Mila partnership program, the Natural Sciences and Engineering Research Council (NSERC) Discovery Grant, the Canada CIFAR Al Chair Program, collaboration grants between Microsoft Research and Mila, Samsung Electronies Co., Ltd., Amazon Faculty Re- search Award, Tencent AI Lab Rhino-Birr Gift Fund and NRC Collaborative R&D Project (AI4D- CORE-06). This project was also partially funded by IVADO Fundamental Research Project grant PRF-2019-3583 1197777.777. The computation resource of this project is supported by Mila*, Calcul Québec" and the Digital Research Alliance of Canada
REFERENCES
Mehdi Ali, Max Berrendorf, Charles Tapley Hoyt, Laurent Vermue, Mikhail Galkin, Sahand Shar- ifzadeh, Asja Fischer, Volker Tresp. and Jens Lehmann. Bringing light into the dark: A large-scale evaluation of knowledge graph embedding models under unified framework. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021. doi: 10. 30.10///AAM1.2221
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Hen- derson, John Hewitt, Daniel E. Ho. Jenny Hong. Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovee, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir P. Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel OIr, Isabel Papadim- itriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R'c. Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam. Andy Shih, Krishna Parasuram Srini- vasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang. Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kait- lyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. ArXiv, 2021. URL hwtpstttt///... http///r......// http://....sss.ffrfo...oooo. haw.forddedu//eee edw/assetee peeportppd.
'nttps: quebec/
"https://ww, cul www.uueeeccca.aaacccccooo ca/
"https://all pattps://allaaneee http///alaacccccn.cmo
10

---

Published as a conference paper at ICLR 2024
Benjamin Paul Chamberlain, Sergey Shirobokov, Emanuele Rossi, Fabrizio Frasca, Thomas Markovich, Nils Yannick Hammerla, Michael M. Bronstein, and Max Hansmire. Graph neu- ral networks for link prediction with subgraph sketching. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https: /ttps////ppeeeeeeeee net. forum? id= m1 ogEOAoZUU.
Payal Chandak, Kexin Huang, and Marinka Zitnik. Building knowledge graph to enable precision medicine, Nature Scientific Data, 2023. doi: http:://bb.oor///000008844111170 https: wwww///wwww.... nature com/
URL
Mingyang Chen, Wen Zhang, Wei Zhang, Qiang Chen, and Huajun Chen. Meta relational learning for few-shot link prediction in knowledge graphs. In EMNLP, pp. 4217-4226, 2019.
Mingyang Chen, Wen Zhang, Yuxia Geng, Zezhong Xu, Jeff Z Pan, and Huajun Chen. Generalizing to unseen elements: survey on knowledge extrapolation for knowledge graphs. arXiv preprint arXiv:2302.00185, 2023.
Yihong Chen, Pasquale Minervini, Sebastian Riedel, and Pontus Stenetorp. Relation prediction as an auxiliary training objective for improving multi-relational graph representations. 3rd Con- ference Automated Knowledge Base Construction, 2021. URL
net/fo ftr.m/frrmm? 3ustmidddaaaaa7--222
Yihong Chen, Pushkar Mishra, Luca Franceschi, Pasquale Minervini, Pontus Stenetorp, and Se- bastian Riedel. Refactor GNNs: Revisiting factorisation-based models from a message-passing perspective. In Advances in Neural Information Processing Systems, 2022. URL https: //openzeview..o////////// // wttp:/netettooottttaaoaaoaaaa 1tt.mmmmmdd111111111111
Daniel Daza, Michael Cochez, and Paul Groth. Inductive entity representations from text via link prediction. In Proceedings of the Web Conference 2021, pp. 798-808, 2021.
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. Convolutional 2d knowledge graph embeddings. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pp. 4171-4186 Association for Computational Linguistics, 2019. URL https:tpp::... / accl antho ogy org/or/g///o.oo /N1 9- 123..
Francesco Di Giovanni, Lorenzo Giusti, Federico Barbero, Giulia Luise, Pietro Lio, and Michael M. Bronstein. On over-squashing in message passing neural networks: The impact width, depth, and topology. In Proceedings of the 40th International Conference Machine Learn- ing. pp. 7865-7885. PMLR, 2023. URL https: httpc:///ccceeddeeeeeeeee mlr. ptw.s/e222222. di-giovanni23a.hee
Boyang Ding, Quan Wang, Bin Wang, and Li Guo. Improving knowledge graph embedding using simple constraints. In Proceedings of the 56th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 110-121. Association for Computational Linguis- tics, 2018 doi: 10.186631111118-0111 URL https: http:///aaaaaaaaaaaeeoaaooooooaa ogy. wogg 3rg/P18-1011 1011.
Xin Luna Dong. Challenges and innovations in building product knowledge graph. In Proceedings of the 24th ACM SIGKDD International conference on knowledge discovery & data mining, pp. 1869-2869, 2018
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL https: / //openzeview.
cNTTy.
Mikhail Galkin. Max Berrendorf, and Charles Tapley Hoyt. An Open Challenge for Inductive Link Prediction on Knowledge Graphs. arXiv preprint a23.v:2203.015200 2022a. URL http://
arxiv.co oeriiv.org/bbee abs /20...///23333222

---

Published as a conference paper at ICLR 2024
Mikhail Galkin, Etienne Denis, Jiapeng Wu, and William L. Hamilton. Nodepiece: Composi- tional and parameter-efficien representations of large knowledge graphs. In International Con- ference Learning Representations, 2022b. URL https: wwwp//pepeeeeeeeeeeeeeeeeeeeeeee net/ forum?
i.d=xMJWKKJFF44
Mikhail Galkin, Zhaocheng Zhu, Hongyu Ren, and Jian Tang. Inductive logical query answering in knowledge graphs. In Advances Neural Information Processing Systems, 2022c. URL
fttt::://openevviie ew. net wtw.eeiwwwweeeeett/eeeeeeeo
Jianfei Gao, Yangze Zhou, and Bruno Ribeiro. Double permutation equivariance for knowledge graph completion. arXiv preprint arX1v:2302.01313, 2023.
Yuxia Geng, Jiaoyan Chen, Jeff Pan, Mingyang Chen, Song Jiang, Wen Zhang, and Huajun Chen. Relational message passing for fully inductive knowledge graph completion. In 2023 IEEE 39th International Conference on Data Engineering (ICDE), pp. 1221-1233. IEEE, 2023.
Genet Asefa Gesese, Harald Sack, and Mehwish Alam. Raild: Towards leveraging relation features for inductive link prediction in knowledge graphs. In Proceedings of the 11th International Joint Conference on Knowledge Graphs, pp. 82-90, 2022.
Jia Guo and Stanley Kok. BiQUE: Biquaternionic embeddings of knowledge graphs. Proceedings
of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 8338-8351. Association for Computational Linguistics, 2021. doi: 10. 20.18863//1/2D0 16653///2022..mm//// wew.022..mmppmmm....m 657. URL
attps://aaee https://aclanttol..o hwwphhhhooooyorgg/2222/2. wttp///1.......... wwt.2mm111pp1....mmmmmmmmm11:1mm caammp-mminne 657.
Takuo Hamaguchi, Hidekazu Oiwa, Masashi Shimbo, and Yuji Matsumoto. Knowledge transfer for
tie-off-noweedge- base entities A graph neural network approach. In Proceedings of the Twenty- Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pp. 1802 1808, 2017. doi: 2.24963/jjcai.2217//55 URL https: www.oooooooroooee 24963 ijcaii . 2017 2017/250.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 770-778, 2016.
Tao He, Ming Liu, Yixin Cao, Zekun Wang, Zihao Zheng, Zheng Chu, and Bing Qin. Exploring & exploiting high-order graph structure for sparse knowledge graph completion. arXiv preprint arXiv:2306. / 17034, 2023.
Daniel Scott Himmelstein, Antoine Lizee, Christine Hessler, Leo Brueggeman, Sabrina Chen, Dexter Hadley, Ari Green, Pouya Khankhanian, and Sergio E Baranzini. Systematic integration of biomedical knowledge prioritizes drugs for repurposing. Elife, 6:e26726. 2017.
Qian Huang, Hongyu Ren, and Jure Leskovec. Few-shot relational reasoning via connection subgraph pretraining. In Advances in Neural Information Processing Systems, 2022. URL https: openreview net
Qian Huang, Hongyu Ren, Peng Chen, Gregor Kržmanc, Daniel Zeng, Percy Liang, and Jure Leskovec. PRODIGY: Enabling in-context learning over graphs. In Thirty-seventh Conference on Neural Information Processing Systems, 2023a. URL https: /tttepp:://pppeeeeeee http///peeweeeeeee//eeee net forum?
id=pLwYhNNnoR.
Xingyue Huang, Miguel Romero Orth, ismail Îlkan Ceylan, and Pablo Barceló. A theory of link prediction via relational weisfeiler-leman. In Advances in Neural Information Processing Systems, 2023b. URL wwt.t:ttp/..........m tetttss://ppereeee /tpp//peeeeeeeeeeeeeeeeeeeeeeeee n / forum? i 1
Ihab F Ilyas, Theodoros Rekatsinas, Vishnu Konda, Jeffrey Pound, Xiaoguang Qi, and Mohamed Soliman. Saga: A platform for continuous construction and serving of knowledge scale. In Proceedings of the 2022 International Conference on Management of Data, pp. 2259-2272, 2022.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017. URL https:
//openeeview.. net. forum/ iiw.mmmyyi.dsuu
12

---

Published as a conference paper at ICLR 2024
Jaejun Lee, Chanyoung Chung, and Joyce Jiyoung Whang. InGram: Inductive knowledge graph embedding via relation graphs. In Proceedings of the 40th International Conference on Ma- chine Learning, volume 202, pp. 11779--18899 PMLR, 23-29 Jul 2023. URL https:// wwwps/e222222 htt.hn.........///..... ee23c. html.
proceedi .gg. mlr....
Pan Li, Yanbang Wang, Hongwei Wang, and Jure Leskovec. Distance encoding: Design provably more powerful neural networks for graph representation learning. In Advances in Neural Infor- mation Processing Systems, volume 33. pp. 4463-4478, 2020.
Shuwen Liu, Bernardo Grau, Ian Horrocks, and Egor Kostylev. Indigo: Gnn-based in- ductive knowledge graph completion using pair-wise encoding. In Advances in Neu- Curran Associates, Inc., ral Information Processing Systems, volume 34, pp. cc/paper_...p.. fi welles/ppppe tes/paper/20211 2021. URL
neurips. ctw.aps.cccppeee Papeeee-aappppppppp..
file 2tl1://66600SS3ccc11222222222223
Xin Lv, Xu Han, Lei Hou, Juanzi Li, Zhiyuan Liu, Wei Zhang, Yichi Zhang, Hao Kong, and Suhui Wu. Dynamic anticipation and completion for multi- reasoning over sparse knowledge graph. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5694-55997 5703. Association for Computational Linguistics, 2020. doi: 10. 10.18653/v1/ 2020. 4020.emnlp-mane mainnnn-maii.... 459. URL https: / ht//ccaantttt www.laanlnloooooooocoooo org/ org/202000002.....0 2020. em-np-maaa...mmmmmm in 459.
Farzaneh Mahdisoltani, Joanna Biega, and Fabian Suchanek. Yago3: A knowledge base from mul- tilingual wikipedias. In 7th biennial conference on innovative data systems research. CIDR Con- ference, 2014.
Chaitanya Malaviya, Chandra Bhagavatula, Antoine Bosselut, and Yejin Choi. Commonsense knowledge base completion with structural and semantic context. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 2925-2933, 2020.
Elan Markowitz, Keshav Balasubramanian, Mehrnoosh Mirtaheri, Murali Annavaram, Aram Gal- styan, and Greg Ver Steeg. StATIK: Structure and text for inductive knowledge graph com- pletion. In Findings of the Association for Computational Linguistics: NAACL 2022, pp. 604- 615, 2022. doi: 10. 10.18653/v1/2022. 122....iwi......ndiinnngg.g. naacl.46. URL httpp::/accaaattoooogg.... 2022. findings naw.ss-nnaa.444...
OpenAl. Gpt-4 technical report. arXiv preprint arXiv: 2303:00:000007744 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Proceed- ings of the 38th International Conference on Machine Learning, Proceedings of Machine Learn ing Research, pp. 8748-8763, PMLR, 18-24 Jul 2021. URL pttpppeiing...nnncee//e mmrr. press/ v 1139/ rad.ord2lfa..a..aaaaaoaaa html.
Tara Safavi and Danai Koutra. CoDEx: A Comprehensive Knowledge Graph Completion Bench- mark. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro- cessing (EMNLP), pp. 8328-8350, Online, 2020. Association for Computational Linguistics. doi: 10.18653/w1/22200 1/660///12220...mmpmaaa.... 2020. caammp-mmaii 669.
URL https:/ http::/wwwwwww// www.wwwwwww....... ort/ org/anthoooggy
Michael J. Statt, Brian A. Rohr, Dan Guevarra, Ja Nya Breeden, Santosh K. Suram, and John M. Gregoire. The materials experiment knowledge graph. Digital Discovery, 2:909-9111 2023. doi: 10.1039//D3DD00006B.. URL http: //tx doi org 10. 21539//3300077555
Zhiqing Sun, Zhi-Hony Deng, Jian- Yun Nie, and Jian Tang. Rotate: Knowledge graph embedding by relational rotation in complex space. In International Conference on Learning Representations, 2019. URL https::////. wpw.reviee.e.eeee//eeee. net forum? wtwrrrmmmmiikiiee Hd=
Komal Teru, Etienne Denis, and Will Hamilton. Inductive relation prediction by subgraph reasoning. In International Conference on Machine Learning, pp. 9448-9457. PMLR, 2020.
Kristina Toutanova and Danqi Chen. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd workshop on continuous vector space models and their compositionality pp. 57-66, 2015.
13

---

Published as a conference paper at ICLR 2024
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, and Partha Talukdar. Composition- multi- relational graph convolutional networks. In International Conference on Learning Representa- fions, 2020. URL htttps://opeereee/e
feet/forumiddByy 1d_tBd
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
Vineeth Venugopal, Sumit Pai, and Elsa Olivetti. Matkg: science entities, relations, and link prediction through graph representation learning. arXiv preprint 22t.::2221.1134.00 2022.
The largest knowledge graph materials
Xiaozhi Wang, Tianyu Gao. Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. Kepler: A unified model for knowledge embedding and pre-trained language representation. Transactions of the Association for Computational Linguistics, 9:176-194 2021.
Wenhan Xiong, Thien Hoang, and William Yang Wang. DeepPath: A reinforcement learning method for knowledge graph reasoning. In Proceedingy the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 564-573. Association for Computational Linguis- tics, 2017. URL https: /ac.aataaaaattooo...c..a.aaaalaa otg/
Bishan Yang. Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. International Conference on Learning Representations, 2015.
Gilad Yehudai, Ethan Fetaya, Eli A. Meirom, Gal Chechik, and Haggai Maron. From local struc- tures to size generalization in graph neural networks. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, volume 139, pp. 11975-11986. PMLR, 2021.
Chengxuan Ying. Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie- Yan Liu. Do transformers really perform badly for graph representation? In Thirty-Fifth Conference on Neural Information Processing Systems, 2021. URL net / Eorum? 1d=::ddwwwww0FFi
Chuxu Zhang, Huaxiu Yao, Chao Huang, Meng Jiang, Zhenhui Li, and Nitesh V Chawla. Few-shot knowledge graph completion. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 3041-3048, 2020.
Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. Advances in neural information processing systems, 31, 2018
Muhan Zhang, Pan Li, Yinglong Xia, Kai Wang, and Long Jin. Labeling trick: A theory of using graph neural networks for molti-node representation learning. In Advances in Neural Information Processing Systems, pp. 9061-9073, 2021.
Yongqi Zhang and Quanming Yao. Knowledge graph reasoning with relational digraph. In Proceed- ings of the ACM Web Conference 2022, pp. 912-924, 2022.
14

---

Published as a conference paper at ICLR 2024
Yongqi Zhang, Zhanke Zhou, Quanming Yao, Xiaowen Chu, and Bo Han. Adaprop: Learning adaptive propagation for graph neural network based knowledge graph reasoning. In KDD, 2023.
Shuxin Zheng, Jiyan He, Chang Liu, Yu Shi, Ziheng Lu, Weitao Feng, Fusong Ju, Jiaxi Wang, Jianwei Zhu, Yaosen Min, He Zhang, Shidi Tang, Hongxia Hao, Peiran Jin, Chi Chen, Frank Noé, Haiguang Liu, and Tie- Liu. Towards predicting equilibrium distributions for molecular systems with deep learning. arXiv preprint arXiv:2306. 05445, 2023.
Jincheng Zhou, Beatrice Bevilacqua, and Bruno Ribeiro. An ood multi- task perspective for link prediction with new relation types and nodes. arXiv preprint urXiv:2307.06044 2023.
Yangze Zhou, Gitta Kutyniok, and Bruno Ribeiro. Ood link prediction generalization capabilities of message- saassagg--ssinnggg gnns in larger test graphs. In Advances in Neural Information Processing Systems, 2022.
Zhaocheng Zhu, Zuobai Zhang, Jouis-Pascal Xhonneux, and Jian Tang. Neural bellman-ford net- works: A general graph neural network framework for link prediction. Advances in Neural Infor- mation Processing Systems, 01129466-24999 2021.
Zhaocheng Zhu, Mikhail Galkin, Zuobai Zhang, and Jian Tang. Neural-symbolle models for logical queries knowledge graphs. In International Conference Machine Learning. ICML 2022.
PMLR, 2022a.
Zhaocheng Zhu, Chence Shi, Zuobai Zhang, Shengchao Liu, Minghao Xu, Xinyu Yuan, Yangtian Zhang, Junkun Chen, Huiyu Cai, Jiarui Lu, al. Torchdrug: powerful and flexible machine learning platform for drug discovery. arXiv preprint arXiv:2202.08320. 2022b.
Zhaocheng Zhu, Xinyu Yuan, Mikhail Galkin, Sophie Xhonneux, Ming Zhang, Maxime Gazeau, and Jian Tang. * scalable path-based reasoning approach for knowledge graphs. In Advances in Neural Information Processing Systems, 2023.
15

---

Published as a conference paper at ICLR 2024
A DATASETS
We conduct evaluation on openly available KGs of various sizes and three groups, i.e., tranduc- tive, inductive with new entities, and inductive with both new entities and relations at inference time. The statistics for 16 transductive datasets are presented in Table 4, 18 inductive entity datasets in Table 5, and 23 inductive entity and relation datasets in Table 6. For each dataset, we also list currently published state-of-the-art model that, at the moment, are all trained specifically on each larget graph. Performance of those SOTA models is aggregated as Supervised SOTA in the results reported in the tables and figures. We omit smaller datasets (Kinships, UMLS, Countries, Family) with saturated performance non-representative.
For the inductive datasets HM 1k, HM 3k, and HM 5k used in Hamaguchi et al. (2017) and Liu et al. (2021), we report the performance of predicting both heads and tails (noted as b-/k, b-3K, b-5k in Liu et al. (2021)) and compare against the respective baselines. Some inductive datasets (MT2, MT3, MT4) from MTDEA (Zhou et al., 2023) do not have reported entity-only KG completion per- formance. For Hetionet, we used the splits available in TorchDrug (Zhu et al., 2022b) and compare with the baseline RotatE reported by TorchDrug.
B SPARSE MMS FOR RELATION GRAPH
The graph of relations G, can be efficiently computed from the original multi-relational graph G with sparse matrix multiplications (spmm). Four spmm operations correspond to the four fundamental relation types (h2t, h2h, 12h, 121] Rjund-
Given the original graph with [V] nodes and R relation types, its adjacency matrix is A e RIV [ [
Vl, For clarity, A can be rewritten with heads H. and tails T as E R R T From A we first build two sparse matrices E,, E [[|[x[R] and E, || that capture head-relation and tail-relation pairs, respectively. Computing interactions between relations is then equivalent to one spmm operation between relevant adjacencies:
Ab24 = spmm E7,, Ex) E R
Aç2: spmm E E:) R
A spmm ETT E,) R|
A spmm E Ex) E [R[R/|R]
A, [A2as A A Aral E R
For each of the four sparse matrices, the respective edge index is extracted from all non-zero values (or, similarly, by setting all non-zero values in the sparse matrix to ones). The final adjacency tensor of the graph of relations A, and corresponding graph of relations G, with four fundamental edge types can be obtained by stacking all four adjacencies ([:, denotes stacking).
C HYPERPARAMETERS
Main results. The hyperparameters for the pre-trained ULTRA model reported in Section 5.2 in- cluding Table 1, Table 2, Figure 1, and Figure 4 are presented in Table 7. Both GNNs over the relation graph G, and main graph G are 6-layer GNNs with hidden dimension of 64, DistMult message function, and sum aggregation roughly following the NBFNet setup. Each layer of the GNN, (inductive link predictor over the main entity graph) features a 2-layer MLP as function g()) that transforms conditional relation representations into layer-specific relation representations. The model trained on the mixture FB15k237, WN18RR, and CoDEx-Medine steps with batch size of 64 with AdamW optimizer and learning rate of 0.0005. Each batch contains only one graph and training samples from this graph. The sampling probability of the graph in the mixture is proportional to the number of edges in this training graph.
graphs for 200,000
16

---

Published as a conference paper at ICLR 2024
Table 4: Transductive datasets (16) used in the experiments. Train, Valid, Test denote triples in the respective set. Task denotes the prediction task: h/t is predicting both heads and tails, tails is only predicting tails. SOTA points to the best reported result.
Dutaset
Reference
Entiies
Rels
Train
Valid
Test
SOTA
CoDEx Small
Safavi & Koutra (2020)
2034
42
32888
1827
1828
b/t
ComplEx RP (Chen al. 2021)
WiDsinger
(2010)
10282
135
16142
2163
2203
LR-GCN (Ho 2023)
FB15K237_10
el al (2020)
11512
237
27211
15624
18150
tails
DacKGR (Lv al.
FBI5k217, 20
Ly et al (2020)
13166
237
54423
16963
19776
talls
DucKGR (L> et 2020)
FB15K237
et (2020)
14149
136057
17449
20324
DacKGR
2020)
FBI5K237
Toutanova & Chen (2015)
14541
237
272115
17535
20466
h/t
NBFNet (Zhu al., 2021)
CoDEX Medium
Safavi & Koutea (2020)
17050
51
185584
10310
10311
b/t
ComplEx RP (Chen et al., 2021)
NELL23K
Lv (2020)
25445
4952
LR-GCN al.. 2023)
WN18RK
Denmers et (2018)
40944
11
86835
3034
3134
h/
NBFNet (Zhu al., 2021)
AristoV4
Chen al. (2021)
449-9
1605
242567
20000
20000
b/t
ComplEx RP (Chen et al. 2021)
Hetioner
Himmelincin (2017)
451.58
2025177
112510
112510
RotatE et 2019)
NEL11995 1995
Xiong et al. (2017)
74536
2001
149678
543
2818
h/t
RED-GNN (Zhang & Yan, 2022)
CoDEx Large
Safavi Koutra (2020)
77951
551193
30622
30622
ComplEx RP (Chen et al., 2021)
ConcepeNer1
Malowiga et al. (2020)
78334
34
100000
1200
1200
BIQUE (Guo Kok, 2021)
0BpedialDOk YAGO310
Ding al. (2018)
99604
477
597572
50000
50000
ComplEx-NNE+ALE (Ding ct al., 2018) NBFNet (Zhu al., 2021)
Mahdisoluani (2014)
123182
37
1079040
5000
5000
Table 5: Inductive entity (e) datasets (18) used in the experiments. Triples denote the number of edges of the graph given at training, validation, test. Valid and Test denote triples to be predicted in the validation and test sets in the respective validation and test graph.
Dataset
Trinnin Graph
Validation Gruph
Test Graph
SOIA
Patities
Triples
Emiries
Triples
Valid
Entrties
Triples
Test
FB vl (Temettll, 2030)
180
1594
-2455
1594
4245
489
1093
1993
411
Net (2hu ct sl., 2023)
FB al. 2120)
200
2638
97.39
2608
9739
1166
1660
4145
947
NBFNet (2hu at. 2021)
al.
3668
3668
7406
1731
NBFNet (2hu st.l. 2021)
FB v1 al.
219
4707
27203
4707
27203
3352
3051
11714
2810
Net (2h et 2023)
WN <] (Tern 2020)
2746
5410
2746
5410
630
922
1618
373
NBFNet (Zhu 2t21.
WN v2 (Tore al. 2020)
10
6954
15262
6954
15762
1838
2757
4011
852
NBFNct (7hu st.,, 2021)
(Tera at.l. 2020)
12078
25901
12078
2590]
6327
NBFNet (2hu st. 2021)
WN (Torm 2020)
3861
7940
3861
7940
934
7084
12334
2823
Net (2h: et al. 2023)
NELI vl (Temueral, 2000
14
3103
4687
3103
4687
414
225
K33
201
RED-GNN (7haeg & Yan. 2002)
NELL al. 2020)
88
2564
8219
2564
8219
922
2086
45%6
935
RED-GNN (Zhsag & Yau, 202)
NELL al.
142
4647
16393
4647
16393
1851
3506
8048
1620
RED GNN (Zaag & Yau, 2002)
NELL al
2092
7546
2092
7073
1447
RED GNN (Zhang & Yau, 2022)
Imaal all.
48
10230
78616
6653
20960
2908
6653
20960
2902
NodePiece (Galkin et al.. 2022a)
ILPC Larlin et 2002a)
65
46626
202446
29246
77044
10179
29246
77044
10184
NodePicue (Galkin all. 2022a)
HM 1k (Hamaguchi et all, 2017)
36237
93364
36311
93364
1771
9899
18638
476
R-GCN (Lin a al. 2021)
Ollamagschi al. 2017)
32118
71097
32250
71097
38285
1319
Indigo (Li etal. Indigo (Lis etal.. 2021)
HM (llamagacti et al. 2017)
28601
S7601
28744
S7601
900
23792
48425
2124
IndigoBM (I.i et al., 2021)
220
12721
121601
12797
121601
14121
14775
250195
14904
Grall. (1 et al., 2001)
Table 6: Inductive entity and relation (e, r) datasets (23) used in the experiments. Triples denote the number of edges of the graph given at training, validation, or test. Valid and Test denote triples to be predicted in the validation and test sets in the respective validation and test graph.
Dutaset
Triinin Grah
Validation Graph
Test Craph
SOTA
Enthies
Rels
Triples
Entities
Reis
Triples
Valid
Entities
Rels
Triples
Test
FR-25 ctod 2003)
5190
163
91571
4097
216
17147
S716
4097
216
17147
5716
Infiram (L al. 2023)
FB ol. 2025)
5190
153
85375
4165
205
11636
3879
4445
205
11636
3879
InGram (Lee 2023)
FB-75 2023)
62809
2792
186
2792
186
3106
Inkiram 2023)
67509
2624
7634
2129
Infiram
(Lee 2013)
41873
3228
3228
1131
InGram (Lee al., 2023)
WK-50 ct. 2023)
52481
9328
3225
Inčiram
al., 2023)
WK-75 ct. 2015)
6853
28741
2722
65
3430
1143
2732
3430
1144
InGram (Lee et at., 2023)
WK. 100 (Lec at. 2023)
9784
57
45875
12135
37
13487
4496
12136
37
13487
4496
Inkiram (Lee ct al., 2025)
NL-O 2023)
1814
134
7796
2026
112
2287
763
202h
112
2287
763
InGram (Lee et al.. 2023)
NL-25 (laces 2023)
4396
106
17578
2146
120
2230
743
2146
1200
2230
744
Infiram (Laz al., 2023)
(1.50 25035
4396
106
17578
2335
119
2576
859
2335
119
2576
859
InGram (Lee 20223
XL-75
2023)
2607
11058
1578
116
1818
1578
116
607
InGiram 2023)
NL-100 (1.0000/aa
1709
793
Infiram
2023)
Metufam al.
13821
1316
NBFNet (2ho 202))
FENELL /Zhou st 2023)
10275
4636
100
10275
4752
185
597
NBFNet (2h al., 2023)
Wiki MT1 Ias (2hos all, 2015)
10800
17178
10000
10
17178
1908
10000
16526
1834
NBFNe (2hoo 202))
Wiki MIT health /2hoa al. 2023)
10800
14371
10000
7
14371
1556
10000
14110
1566
NBFNet /2ou et 2023)
Wiki MT2 org (2hou et al., 2023)
10800
10
23233
10800
10
23235
2581
10000
11
21976
2441
N/A
Wiki MT2 sci (Zhon at. 2023)
10000
16
16471
10000
16
16471
1830
10000
16
14852
1650
NIA
Wiki M13 art (Zhoo et al., 2023)
10630
45
27262
18830
45
27262
3026
10000
45
28023
3113
NIA
Wiki MT3
infra (Phus stall, 20035)
10630
24
21990
10800
24
31090
2443
10000
27
21646
2405
NIA
Wiki MT (Zhos
10000
12576
10000
42
12576
1397
10000
12516
1288
NIA
MT1 healih (2hon at. 2013))
15539
10000
15539
10000
15337
Table 8 reports training durations for fine- tine-tuning the pre- trained UL.TRA and training models from scratch on each dataset (for the ablation study in Figure 5
Fine-tuning and training from scratch.
17

---

Published as a conference paper at ICLR 2024
Table ULTRA hyperparameters for pre-training. GNN, denotes GNN over the graph of relations Gr, GNN, is a GNN over the original entity graph g.
Hyperparameter
ULTRA pre-training
# layers
6
hidden dim
64
GNN,
message
DistMult sum
aggeregation
# layers hidden dim message
6 64
GNN
DistMult
aggregation g(-)
sum 2 2-layer MLP
optimizer learning rate training steps adv temperature # negatives batch size
AdamW
0,0005
200,000
Learning
1
128
64
Training graph mixture
FB15K237, WNISRR, CoDEx Medium
and Section 5.3). In fine-tuning, if the number of fine-tuning epochs k is more than one, the best checkpoint (out of k) evaluated on the validation set the respective graph. Each fine-tuning repeated 5 times with different random seeds, each model trained from scratch was trained 3 times with different random seeds.
Ablation: graphs in the training mixture. For the ablation experiments reported in Figure 6, Table 9 describes the mixtures of graphs used in the pre-trained models. The mixtures of 5 and more graphs include large graphs of 100k+ entities each, so we reduced the amount of training steps to complete training within 3 days (6 GPU-days in total as each model was trained on 2 A A100 GPUs).
D FULL RESULTS
The full, per-dataset results of MRR and Hits 10 of the zero-shot inference of the pre-trained ULTRA model, the fine-tuned model, and best reported supervised SOTA baselines are presented in Table 10 and Table 11. The zero-shot results are deterministic whereas for fine-tuning performance we report average 5 different seeds with standard deviations.
Table 10 corresponds to Figure and contains results on 43 graphs where published SOTA baselines are available, that is, on 3 pre-training graphs, on 14 inductive entity (e) graphs, on 13 inductive entity and relation (e, r) graphs, and 13 transductive graphs. Table 11 contains results 16 graphs for which published SOTA exists only partially, that is, in terms of the Hits@ 10 (50 neg) metric computed against 50 randomly chosen negatives. We show that this metric greatly overestimates the real performance and encourage further works to report full MRR and Hits@k @k metrics computed against the whole entity set.
The results in Table on 57 graphs (Section 5.2) are aggregated from Table 10 and Table 11.
Commenting the performance of pre-trained ULTRA model larger transductive graphs, attribute the performance difference to the following factors:
Training data mixture and OOD generalization: the model reported in Table I was trained on 3 medium-sized KGs (15k 40k nodes, 80k 270k edges) while the biggest gaps are on larger graphs with many more nodes and edges (up to 120k nodes and 1M edges for YAGO 310), many more relation types (1600+ in Aristo ceptNet100k with 100k edges over 78k nodes). Size generalization issues are common for GNNs found in Yehudai et al. (2021): )) Zhou al. (2022). However, if we take the UL-
very sparse (as in Con-
18

---

Published as a conference paper at ICLR 2024
Table Hyperparameters for fine-tuning ULTRA and training from scratch in the format epochs, steps per epoch), e.g., (I, full) means one full epoch over the training set of the respective graph while (1, 1000) means epoch of 1000 steps over the training set.
Datasets
UlTRA fine-tuning
ULTRA train from scratch
Batch size
FB VI-V4
(1, full)
(10, full)
WN V1-V4
(1, full)
(10, full)
64
NELL V1-V4
(3, full)
(10, full)
HM 1k-5k, IndigoBM
100)
(10, 1000)
64
ILPC Small
(3, full)
(10,ful]
64
ILPC Large
(1, 1000)
(10, 1000)
16
FB 25-100
(3, full)
(10. full)
64
WK 25-100
(3, full)
full)
64
NL 0-100
(3, full)
(10, full)
64
MT1-MT4
(3, full)
(10, full)
Metafam, FBNELL
(3, full)
(10, full)
64
WDsinger
(3, full)
(10, 1000)
NELL23K
(3, full)
(10, 1000)
64
FB237_ 10
(1, full)
(10, 1000)
FB237_22 20
(1, full)
(10, 1000)
64
FB237_50
(1, 1000)
(10, 1000)
64
CoDEx-S
(1, 4000)
(10, 1000)
64
CoDEx-L
(1, 2000)
(10, 1000)
16
NELL-995
(1, full)
(10, 1000)
16
YAGO 310
2000)
2000)
16
DBpedial00k
(1, 1000)
(10, 1000)
16
AristoV4
(1, 2000)
(10. 1000)
16
ConceptNet100k
2000)
(10, 1000)
16
Hetionet
(1, 4000)
(10, 1000)
16
WN18RE
(1, full)
(10, 1000)
FB 15k237
(1, full)
(10, 1000)
64
CoDEx-M
(1, 4000)
(10, 1000)
Table 9: Graphs in different pre-training mixtures in Figure 6.
2
4
6
FB15k237
V
WNISRR
V
V
V
V
CoDEX-M
V
V
V
NELL995
V
V
V
YAGO 310
V
V
ConceptNet100k
DBpedial001 100k
Aristo
Batch size
32
16
64
16
16
16
16
# steps
200,000
400,000
200,000
400,000
200,000
200,000
200,000
TRA checkpoint pre-trained on graphs (Table 9) and run evaluation on all 16 transductive graphs, then the average performance is better than supervised SOTA models, i.e., 0.377 MRR/ / 0.537 @its@ 10 of ULTRA against 0.371 MRR 005 Hits 10 of the baselines.
Transductive models have the privilege of memorizing target data distributions into entityielationn-peccffe vectors with overall many millions of parameters, e.g., 80M pa- rameters for supervised SOTA BiQUE ConceptNet100k. This performance, however, comes with the absence of transferability across KGs. In contrast, all pre-trained ULTRA checkpoints rather small (about 170k parameters) but generalize any KG. We
19

---

Published as a conference paper at ICLR 2024
knowledge the scaling behavior in the Section 6 and consider it a very promising avenue for future work. In particular, scaling laws for GNNs and common graph learning tasks (like link prediction) are not derived yet connection between GNNs size, dataset size, graph topology, and expected performance. Generally, there is consensus in the graph learning community whether deep or wide (non-geomen moonggcometrie) GNNs bring immediate benefits mostly due to the rising issues of over- smoothing and oversquashing (some initial results were recently presented in Di Giovanni et al. (2023)). In our experiments, we observe that the diversity of graphs in the pre-training mixture plays an important role as well. Therefore, we believe that a brute-force increase of the model size is unlikely to bring benefits unless paired with more diverse training data and more intricate mechanisms for capturing relational interactions.
can only hypothesize whether there is any
E ON ADDING MORE FEATURES
Some graphs might have specific node and edge features such as numerical attributes and text de- scriptions. Often, KG features are heterogeneous, e.g. graph from the life sciences domain would contain biomedical features that might not overlap with geographical features in other graphs, and would require different feature encoders. In the text domain, not all KGs have text features readily available as we mentioned in Section 2. In this work we focus on the structural representations and feature-less graphs this can be applied to any KG with or without features.
Nevertheless, there is some evidence (Chen et al., 2022) that concatenating encoded text features (where available) to structural GNN features is likely further boost the performance in inductive tasks. We consider dataset-specific features complementary to UI.TRA representations and hypoth- esize that such additional features might be particularly useful the fine-tuning stages. This is an intriguing direction for the future work.
F COMPUTATIONAL COMPLEXITY
The time complexity of ULTRA is upper-bounded by the entity-level GNN, (because the GNN, on the graph of relations has negligible overhead the number of nodes in this graph is the same as number of unique relation types [R], and [R] << [V/, that is, the number of relation types is usually orders of magnitude smaller than the number of nodes). In our case, the main entity-level GNN, is NBFNet, so we mainly refer to the Appendix C of Zhu et al. (2021) for all necessary derivations.
The time complexity for single layer is generally linear in the number of edges O(E + |V ()) With T layers, the overall complexity of a single forward pass is ((((E)d+[V]dd)) but T is usually a small constant (6 layers) so the complexity is essentially linear to the number of edges. However, due to the sparsity of GNNs, they are usually bounded by memory. The memory complexity of the basic NBFNet implementation is O d) and linear to the number of edges, but thanks to the ef- ficient kernelized implementation of the relational message passing (already provided by NBFNet), the memory complexity is reduced to O(T]Vld) and is lincar in the number of nodes. Moreover, the complexity can be further reduced when applying more scal calable and optimized versions of entity- level GNNs such as AdaProp (Zhang et al., 2023) or A Net (Zhu et al., 2023).
20

---

Published as a conference paper at ICLR 2024
Table 10: Full results (MRR, @its@ 10) of ULTRA in the zero-shot inference and fine-tuning regimes on 43 graphs compared to the best reported Supervised SOTA. The numbers correspond to Figure 1.
ULTRA O-shot
ULTRA fine-tuned
Supervised SOTA
Dataset
MRR
Hits 10
MRR
Hits 10
MRR
Hits @ 10
pre-training datasets
WN18RR
0.480
0.614
0.551
0.666
FB 15k237 CoDEx Medium
0.368
0.564
0.415
0,599
0.372
0.525
0.352
0.49
inductive (e) datasets
WN VI
0.648
0.768
0,685 0.003
0.793
0.741
0,826
WN V2
0.663
0.765
0.679 = 0.002
0.779 0.003
0.704
0.798
WN V3
0.376
0.476
0,411
0.546
0.452
0.568
WN V4
0.611
0.705
0.614 0.003
0.720 0.001
0.661
0.743
FB VI
0.498
0.656
0.509
0.670
0.457
0.589
FB V2
0,512
0.700
0,524 0.003
0.710 0.004
0.510
0,672
FB V3
0.491
0.654
0,504 0.001
0.663 0.003
0.476
0.637
FB V4
0.486
0.677
0.496 0.001
0.684 0.001
0.466
0.645
NELL VI
0.785
0.913
0.757 = 0.022
0.878
0.035
0.637
0.866
NELL V2
0.526
0.707
0.575
0.004
0.761
0.419
0.601
NELL V3
0.515
0.702
0,563
0.004
0.755 0.006
0.436
0.594
NELL V4
0.479
0.712
0.469 0.020
0.733 0.011
0.363
0,556
ILPC Small
0.302
0.443
0.303 a 0.001
0.453
0.002
0.130
0,251
ILPC Large
0.290
0.424
0.308 0.002
0.431 0.00
0.070
0.146
inductive (e, r) datasets
FB-100
0.449
0.642
0.444 20000
0.643
0.00
0.223
0.371
FB-75
0.403
0.604
0.400
0.598 00004
0.189
0.325
FB-50
0.338
0.543
0.334 0.002
0.538 0.004
0.117
0.218
FB-25
0.388
0.640
0.383
0.635
0.133
0.271
WK-100
0.164
0,286
0.168 0.005
0,286 0.003
0.107
0.169
WK-75
0.365
0.537
0.380 0.001
0,530 0.009
0.247
0.362
WK-50
0.166
0.324
0.140 00000
0,280 0.012
0.068
0.135
WK-25
0.316
0.532
0.321 = 0.005
0.535 d0007
0.186
0.309
NL-100
0.471
0.651
0.458
0012
0,684
0011
0.309
0.506
NL-75
0.368
0.547
0.374
0.007
0.570
0.005
0.261
0.464
NL-50
0.407
0.570
0.418 0.005
0.595 0.005
0.281
0,453
NL-25
0.395
0,569
0.407 a 0.009
0.596
0.012
0,334
0,501
NL-0
0.342
0.523
0.329 2011
0.551 00012
0.269
0.431
transductive datasets
CoDEx Small
0.472
0.667
0.490 =000
0.686
00003
0.473
0.663
CoDEx Large
0.338
0.469
0.343 #0002
0.478
#0002
0.345
0.473
NELL-995
0.406
0.543
0,509
0013
0.660
0.066
0.543
0.651
YAGO 310
0.451
0.615
0.557 0.009
0.710
0.000
0.563
0.708
WDsinger
0.382
0,498
0.417 0.002
0.526 0.002
0.393
0,500
NELL23K
0.239
0.408
0,268 & 0.001
0.450 0.001
0.253
0.419
FB15k237 10
0.248
0.398
0.254 0.001
().11
0.001
0.219
0.337
FB15k237 20
0.272
0.436
0.274 + 0.001
0.445 0.002
0.247
0.391
FB15k237_55
0.324
0.526
0.325 0.002
0.528 0.002
0.293
0,458
DBpedia 100k
0.398
0.576
0.436 a 0.008
0.603 0.066
0,306
0.418
AristoV4
0.182
0.282
0.343 0.066
0,496 0.044
0.311
0.447
ConceptNet100k
0.082
0.162
0.310
= 0.004
0.529 00007
0,320
0,553
Hetionet
0.257
0.379
0.399 0.005
0.538 0.004
0.257
0.403
21

---

Published as a conference paper at ICLR 2024
Table 11: Full results (MRR, Hits@ 10) of ULTRA in the zero-shot inference and fine-tuning regimes on 14 graphs where Supervised SOTA reports estimate Hits@ 10 (50 negs) metric (where avail- able). The numbers correspond Figure 4.
ULTRA 0-shot
ULTRA fine-tuned
Supervised SOTA
Dataset
Hits@
Hits (50 neg)
Hits (50 neg)
MRR
@its@10
MRR
Hits@ 10
(50 neg)
inductive (c) datasets
HM 1k
0.059
0.092
0.796
0.042 n002
0.100
-0007
0.839 4001s
0.625
HM 3k
0.037
0.077
0.717
0.030 100e
0.090 =0oss
0,717
4001o
0.375
HM 5k
0,034
0,071
0.694
0.025 0001 0.068
20003
0.657
20016
0.399
IndigoBM
0.440
0.648
0.995
0.432
0.639
=0002
0.995
0.788
inductive (e, r) datasets
MI1
0.224
0.305
0.731
0.330 0006
0.459
-0056
0.994
0000
0.855
MTI health
0.298
0.374
0.951
0.380 2000
0.467
=0.06
0.982
10.00
0.858
MT2 org
0.095
0.159
0.778
0.104
u000
0.170
00001
0.855
+0012
MT2 sci
0.258
0.354
0.787
0.311
uumo
0,451
2042
0,982
+0000
MT3 art
0.259
0.402
0.883
0.306 2.003
0.473
20003
0,958
10.00
MT3 infra
0.619
0.755
0.985
0.657 2ums
0,807
00007
0,996
20000
MT4 sci
0.274
0,449
0.937
0.303 2007
0.478
00003
0,973
MT4 health
0.624
0.737
0.955
0.704
0.785
0.02
0.974
Metafam
0.238
0.644
1.0
0.997
1.0
1.0
1.0
FBNELL
0.485
0.652
0.989
0.481
0.661
0.987
0.95
22


Reviews:
[{"review_id": "QgA77Yx79o", "reviewer": "Reviewer_ebFz", "summary": "The key limitation of designing the foundation models for dealing with the Knowledge Graphs (KGs) is that the KGs have different entities and relations that generally do not overlap. To address this issue, this paper proposes ULTRA, which positively transfers the information of source KG to unseen KG. It constructs relation representations based on the interactions between the relations by introducing the graph of relations. The proposed approach has shown good performance on various tasks.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- From their experiments, the proposed methods have shown good performance on various tasks.\n- Research topics about the foundational models on graph-structured datasets is really interesting and important.\n- The paper is well written and easy to follow.", "weaknesses": "- The authors first pretrain the ULTRA model with the mixture of 3 standard KGs and then fine the model for the downstream task. But, the other supervised SOTA model only uses dataset of the downstream tasks without employing the pre-training datasets. If the supervised SOTA models are designed to deal with transductive settings, they may show worse performance on the downstream tasks. However, if the SOTA models are the models for the inductive setting, I think they may be possible to be pretrained like the ULTRA. So, could you measure the performance of the \"pretrained\" SOTA models on the inductive if possible?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ueZCPmCi8S", "reviewer": "Reviewer_c3Sg", "summary": "Paper claims to propose a foundation model, named Ultra, for knowledge graph representation learning. The proposed model can handle full inductive graphs in which new entities and relations may appear in the test set. To do so, the authors propose to lift the graph to a one with relations as the nodes and design 4 different edge types (head2head, head2tail, tail2head, tail2tail). The relational representations are then learnt using message passing on this graph. The learnt relation embeddings are then used in the original graph to perform inductive link prediction. For the experiments, the authors pre-train their method on 3 KGs and further evaluate in a zero-shot setting and also by fine-tuning the downstream tasks.", "questions": "- For weakness point 2: Any reason why this was not done by the authors?\n- For weakness point 3: How would this be addressed in future works for the model?\n-  For weakness point 4: Could the authors comment on why this would be the case and how would the model be improved to handle the transductive setting?\n- For weakness point 5: Any reason why the results on this dataset are not good?\n- Considering KGs are a rich source of textual/semantic data along with graph/structured data and the current model does not use this rich source of context information, how can we extend Ultra to incorporate the KG ontology? \n- Considering weaknesses 2,4,5 the claim of the foundation model seems a bit broad as of now and at best the model could be said to be a good inductive learner.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper proposes a transductive model that works in settings of new relations and entity nodes.\n- The method obtains good zero-shot pretraining results.", "weaknesses": "- The authors have not explicitly stated the computational complexity of the method. From the paper, it seems that the forward pass is run on the entire relational graph to obtain relation representations. This is then used to initialize the node embedding from the query triple and the process is repeated for every triple. Thus it seems that the entire graph is being used for link prediction every triple making the computational complexity O(E^2). This seems limiting for large graphs that have not been explored in the paper (such as wikidata-5m etc.).\n- From Table 2 we can see that finetuning over the pre-trained models helps the results significantly over the 0-shot setting. Also, the fine-tuning steps are too large to claim few shot results. This weakens the claim of the \"foundation model\" for KGs. A fair comparison would be to show the pretraining results for other inductive and transductive methods as well in addition to the SOTA comparison.\n- Another limitation is that of scale. Since the current model has fewer parameters, this would limit learning over larger pretraining datasets as can be seen in Figure 6 and also reported by the authors.\n- SoTA results for transductive models are better than the pre-trained Ultra model in many datasets. Thus the Ultra model seems to work well for the inductive setting rather than transductive. Thus the claim of the \"foundation model\" seems broader in scope.\n- We see that in the metafam dataset, the pretraining results are poor but on finetuning the results are improved drastically. This shows that the method works well in cases where the relational patterns of the downstream datasets are similar to the pre-trained one but when the data distribution changes the results suffer. Moreover, due to limited capacity, the model may not be able to handle such cases by increasing the pretraining datasets calling for downstream finetuning. Thus domain adaptation is not a problem which can be easily overcome by scaling the current model and this further weakens the claim of a \"foundation model\" for KGs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MRrxCuaYHL", "reviewer": "Reviewer_HLbG", "summary": "This work aims to build a foundation model for knowledge graph reasoning tasks, where the authors explore the setting of generalization to any edges and nodes, including unseen, of any multi-relational knowledge graphs without using node and edge features. To this end, the authors first construct a view of a relation-centric graph from an original graph where edges become nodes of this new relation graph, and then, based on this view, the authors represent the relation (node) relative to and conditioned on the query relation. Then, based on this relative relation representation, the authors use existing inductive link prediction methods to perform knowledge graph reasoning. The authors conduct link prediction experiments on various knowledge graphs considering both inductive and transductive settings, and show that the proposed method, namely ULTRA, outperforms other SOTA baselines sometimes without further fine-tuning on target knowledge graphs (i.e., zero-shot).", "questions": "* I would like to suggest emphasizing the performance differences between inductive and transductive setups when explaining Table 1. The proposed method w/ 0-shot settings are strong on inductive graphs; meanwhile, previous methods are superior to it on transductive graphs, which are worthwhile to discuss.\n* It may be beneficial to show the results of the ULTRA fine-tuned on the knowledge graphs used for pre-training the ULTRA. I am wondering if there are further performance improvements when further fine-tuning the model on the data used for pre-training.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* This work studies the very important, challenging, and practical setups of building a foundation model for knowledge graph reasoning, which aims to be generalizable to any other knowledge graphs involving unseen nodes and unseen edges, without leveraging features of nodes and edges. \n* The proposed method works well with different knowledge graphs, on zero-shot transfer learning setups without further fine-tuning on target knowledge graphs, and further shows the boosted performance with task-specific further fine-tuning on them, on most experiment setups.\n* This paper is very well-written and easy to follow.", "weaknesses": "* I would like to note that I don't see any major weakness, and below is the minor.\n* In Section 4.2, the explanation about the indicator function with variables $u$ and $v$ is a bit unclear to me. Could you elaborate more on the process and result of the indicator function according to those two variables, perhaps with visuals?\n* Text-based methods (e.g., LM-based methods) can be generalizable to any knowledge graphs including unseen nodes and unseen edges, as long as their nodes and edges are represented with texts. In this vein, I think one potential direction for building a foundation model for knowledge graph-related tasks might be to use the LMs, and the authors may highlight this point more and potentially make comparisons between the proposed approach and text-based methods. I don't think this should be the critical weakness of this paper since text-based methods are limited to knowledge graphs with textual features; meanwhile, given the framing of this work (\"Towards Foundation Models for Knowledge Graph Reasoning\"), this point should be carefully explained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oVta8PijA2", "reviewer": "Reviewer_sna7", "summary": "This paper presents ULTRA, a single model that can be directly used/finetuned for link prediction over different knowledge graphs. The key is to model the transferrable relationships between different relations across knowledge graphs. Specifically, a NBFNet is used to learn relative relation representations and generate relation embeddings, which is then fed into another NBFNet to perform link predictions. Extensive experiments are performed over many knowledge graph to demonstrate the performance of this model.", "questions": "- What are u and v in h_{u|v} in section 4.2?\n- Why are supervised SOTA baselines only reported for some datasets in Figure 4?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper is well written and easy to follow\n- The core method around the relative relationships between relations is clever and interesting.\n- The experiments demonstrate the gains of the method. It is especially impressive to see the competitive zero-shot performance of ULTRA over different knowledge graphs.", "weaknesses": "- The proposed method relies entirely on knowledge graph structure and does not consider using node embedding such as textual features of the knowledge graphs. In reality, text embedding of nodes and edges could be a better transferrable embedding. Such transferability has already been demonstrated by PRODIGY (https://arxiv.org/abs/2305.12600) and should be addressed.\n- The model does not scale well as the authors already pointed out.\n- The zero shot and fine-tuning performances are worse or on-par with the per dataset model performance, rendering pretraining not effective performance-wise. \n- Some notations are a bit hard to understand. See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)