Fast-and-Frugal Text-Graph Transformers are Effective Link Predictors
Authors: Andrei Catalin Coman, Christos Theodoropoulos, Marie-Francine Moens, James Henderson
Accepted at: acl25

Fast-and-Frugal Text-Graph Transformers are Effective Link Predictors

Andrei C. Coman
Idiap Research Institute, EPFL
andrei.coman@idiap.ch

Marie-Francine Moens
KU Leuven
sien.moens@kuleuven.be

Christos Theodoropoulos
KU Leuven
christos.theodoropoulos@kuleuven.be

James Henderson
Idiap Research Institute
james.henderson@idiap.ch

Abstract
We propose Fast-and-Frugal Text-Graph (FnF-TG) Transformers, a Transformer-based framework that unifies textual and structural information for inductive link prediction in text-attributed knowledge graphs. We demonstrate that, by effectively encoding ego-graphs (1-hop neighbourhoods), we can reduce the reliance on resource-intensive textual encoders. This makes the model both fast at training and inference time, as well as frugal in terms of cost. We perform a comprehensive evaluation on three popular datasets and show that FnF-TG can achieve superior performance compared to previous state-of-the-art methods. We also extend inductive learning to a fully inductive setting, where relations don’t rely on transductive (fixed) representations, as in previous work, but are a function of their textual description. Additionally, we introduce new variants of existing datasets, specifically designed to test the performance of models on unseen relations at inference time, thus offering a new test-bench for fully inductive link prediction.

1 Introduction
Knowledge graphs (KGs) represent complex information as a structured collection of entities and their relations. They are a fundamental component of various applications, including information extraction (Mintz et al., 2009; Bosselut et al., 2019; Theodoropoulos et al., 2021) and retrieval (Dalton et al., 2014; Gupta et al., 2019), question answering (Saxena et al., 2022; Yu et al., 2022; Coman et al., 2023), reasoning (Zhang et al., 2020; Jiang et al., 2022; Niu et al., 2022), fact-aware language modelling (Logan et al., 2019; Yang et al., 2023), and many others (Fensel et al., 2020; Schneider et al., 2022). Text-attributed KGs extend KGs by associating each entity and relation with a corresponding textual description, which provide a richer representation of the knowledge encoded in the graph. In particular, the text associated with an entity may provide a description of its relationships to other entities. This combination of explicit structural and implicit textual information makes modelling text-attributed KGs particularly challenging.

Initial attempts to model KGs focused on their graph nature, typically addressing a transductive setting (Bordes et al., 2013; Nickel et al., 2015; Wang et al., 2017). These models could only make predictions for entities observed during training and only considered the structural information of the KG, ignoring any textual information.

To overcome this limitation, later work focused on using the textual descriptions in KGs to address an inductive setting (Xie et al., 2016; Shi and Weninger, 2018; Wang et al., 2021b), meaning that predictions can be made even for entities not observed during training, using entity representations computed based on their textual descriptions.

Combining information from textual descriptions and graph structures has proven crucial (Schlichtkrull et al., 2017). An entity’s ego-graph, which represents it’s 1-hop neighbourhood, provides valuable context that can help disambiguate its role and distinguish it from similar entities. While there has been progress in leveraging ego-graphs, we believe that there is significant room for more effective approaches.

Modelling text-attributed KGs in an inductive setting poses several challenges, particularly when it comes to effectively integrating textual and structural information in embeddings. Transformers (Vaswani et al., 2017) have shown remarkable success at modelling unstructured (text) data (Devlin et al., 2019; Raffel et al., 2019; Brown et al., 2020; Touvron et al., 2023). While their ability to model structured (graph) data is less evident, the inherent graph processing abilities of a Transformer’s self-attention mechanism make it a natural fit for modelling graph structures (Henderson et al., 2023). We leverage recent advances in using Transformers for graph encoding (Mohammadshahi and Hender-
son, 2020, 2021; Miculicich and Henderson, 2022; Coman et al., 2024), and propose Fast-and-Frugal Text-Graph (FnF-TG) Transformers, which unify textual and structural information in a framework based solely on Transformers.

Another challenge is that text encoders are resource-intensive, especially when the textual descriptions of both entities and their ego-graph neighbours need to be encoded, leading to considerably increased training and inference time (Markowitz et al., 2022). This cost can be reduced by using smaller text encoders, but they can be considerably less effective. We demonstrate that we can reduce the dependence on large text encoders with a more effectively encoding of ego-graphs, using our FnF-TG Transformers and their more appropriate inductive biases. This makes the overall framework both fast in terms of time, as well as frugal in terms of cost.

A third challenge is that previous models fail to leverage the textual descriptions of relation labels. They still assume a fixed (transductive) inventory of relations, meaning that they cannot handle relations which they did not see during training and thus are not fully inductive. We propose an extension of this method to address the challenge of being fully inductive, by computing a relation embedding from the text describing that relation. This embedding serves as both the relation representation for link prediction, analogous to the transductive case, and also as the relation label which is input to the self-attention mechanism of the FnF-TG’s graph encoder component.

We showcase the effectiveness of our proposed model on three popular datasets for inductive link prediction in text-attributed KGs from the experimental setting of Daza et al. (2021) and Wang et al. (2021b), namely WN18RR_{IND}, FB15k-237_{IND}, and Wikidata-5M_{IND}. We show that it improves over the state-of-the-art in all cases.

Additionally, we introduce new variants of existing datasets which are specifically designed to evaluate the performance of models on relations which are unseen until test time, thus offering a new test-bench for fully inductive link prediction.

Contributions:

1. We propose a KG embedding model which leverages the intrinsic graph processing capabilities of Transformers to effectively capture the information in both the KG’s textual descriptions and the KG’s graph structure.
2. We demonstrate that Fast-and-Frugal Text-Graph (FnF-TG\(^1\)) Transformers achieve superior performance compared to previous state-of-the-art results on three popular datasets, even with small and efficient text encoders.
3. We extend inductive KG learning to a fully inductive setting, where both entity and relation representations are computed as functions of their textual descriptions.
4. We introduce a new test-bench for fully inductive link prediction by modifying existing datasets to specifically test models’ performance on unseen relations.

2 Related Work

Transductive Link Prediction  In this setting, link prediction aims to identify missing links within a fixed and fully observable graph where all entities and their other connections are known during training. Typically, it involves learning embeddings within a geometric space, as demonstrated by models like RESCAL (Nickel et al., 2011), NTN (Socher et al., 2013), TransE (Bordes et al., 2013), DistMult (Yang et al., 2014), ComplEx (Trouillon et al., 2016), TorusE (Ebisu and Ichise, 2018), RotatE (Sun et al., 2019), and SimplE (Kazemi and Poole, 2018). Additionally, there are approaches that incorporate convolutional layers such as R-GCN (Schlichtkrull et al., 2017), ConvE (Dettmers et al., 2018), HypER (Balazevic et al., 2018), and ConvR (Jiang et al., 2019). Moreover, recent advances have seen the integration of Transformers in models such as CoKE (Wang et al., 2019b) and HittER (Chen et al., 2021).

Inductive Link Prediction  In this setting, link prediction involves predicting missing links in a dynamic graph where only partial information is available during training. Much work has explored leveraging limited relational knowledge between novel entities and those already present in the training graph (Bhowmik and de Melo, 2020; Wang et al., 2020). Examples include LAN (Wang et al., 2019a), IndTransE (Dai et al., 2021), OpenWorld (Shah et al., 2019), GraIL (Teru et al., 2020), NBFNet (Zhu et al., 2021), NodePiece (Galkin et al., 2021), and BERTRL (Zha et al., 2021). Moreover, approaches such as DKRL (Xie et al., 2016), Commonsense (Malaviya et al., 2020), KG-BERT (Yao et al., 2019), KEPLER (Wang et al., 2021b),

\footnotetext{1https://github.com/fidiap/fnf-tg}
BLP (Daza et al., 2021), StAR (Wang et al., 2021a), SimKGC (Wang et al., 2022), StATIK (Markowitz et al., 2022), iHT (Chen et al., 2023), and KnowC (Yang et al., 2024) use language models to encode entities based on their textual descriptions. Among these, StATIK (Markowitz et al., 2022) stands out as it combines both a language model and a graph encoder, specifically employing a Message Passing Neural Network (MPNN) (Gilmer et al., 2017) to create entity embeddings. This makes StATIK particularly relevant to our approach and we will use it as the state-of-the-art method of reference and compare our proposed method against it to demonstrate its effectiveness.

Transformers and Graphs  Graph Transformers (GTs) represent a significant evolution in graph input methods within the Transformer architecture (Henderson et al., 2023). Early work such as G2GT (Mohammadshahi and Henderson, 2020, 2021; Miculicich and Henderson, 2022) laid the foundation by incorporating explicit graphs into Transformer’s latent attention graph. Later work introduced RoFormer (Su et al., 2021), which uses a rotation matrix to encode absolute positions, and Graphormer (Ying et al., 2021), which uses node centrality encoding and soft attention biases. Other models, like SSAN (Xu et al., 2021), JointGT (Ke et al., 2021), TableFormer (Yang et al., 2022), and GADePo (Coman et al., 2024), have applied GTs to various tasks such as document-level relation extraction, knowledge-to-text generation, table-based question answering, and graph-aware declarative pooling.

We continue to advance graph input methods and show that GTs, when combined with an effective inductive bias in the input and the latent attention graph, achieve superior performance compared to the previous state-of-the-art.

3  Background

3.1 Inductive Representation Learning
A text-attributed knowledge graph can be defined as \( \mathcal{G} = (\mathcal{E}, \mathcal{R}, \mathcal{T}, \mathcal{D}) \) where \( \mathcal{E} \) represents the set of entities, \( \mathcal{R} \) denotes the set of relation labels, \( \mathcal{T} \) consists of the set of relation triples \( (h, r, t) \in \mathcal{E} \times \mathcal{R} \times \mathcal{E} \), and \( \mathcal{D} \) contains the textual descriptions associated with entities and relation labels. In each triple, \( h \) and \( t \) represent the head and tail entities, respectively, which are connected by a directional relation \( r \). Inductive link prediction involves completing missing triples in the graph by leveraging the textual descriptions associated with the entities and relation labels. If the textual description of an entity mentions the target relation, then this resembles an open relation extraction task (Banko et al., 2007), and if not then it resembles a knowledge graph completion task (Lin et al., 2015), with a continuum of difficulties in between. Our goal is to learn an embedder that maps these descriptions and the partial graph to a representation space where the missing triples can be inferred. Specifically, given a training graph \( \mathcal{G}_{train} = (\mathcal{E}_{train}, \mathcal{R}, \mathcal{T}_{train}, \mathcal{D}_{train}) \), where \( \mathcal{E}_{train} \subset \mathcal{E} \), \( \mathcal{D}_{train} \subset \mathcal{D} \) and \( \mathcal{T}_{train} \subset \mathcal{T} \) only includes triples involving entities in \( \mathcal{E}_{train} \), the goal is to infer the missing triples in \( \mathcal{T} \setminus \mathcal{T}_{train} \). During evaluation, for a given query triple \( \mathcal{T}_i = (h, r, t) \), the model is tasked with performing head or tail prediction on the graph \( \mathcal{G} \setminus \mathcal{T}_i \). This involves two types of queries: tail prediction, where the query is of the form \( (h, r, \hat{e}) \) and head prediction, where the query is of the form \( (\hat{e}, r, t) \). In both cases, the model must rank all possible candidate entities \( \hat{e} \in \hat{\mathcal{E}} \) to identify the correct entity \( \hat{e}_t \) or \( \hat{e}_h \) and place it at the top of the ranked list.

3.2 Structural Objective and Loss Function
We adopt the margin-based ranking loss from Bordes et al. (2013) as our optimisation criterion. We construct two sets of triples: a set of true triples \( T \) and a set of negative triples \( T' \), where a negative triple consists of a corrupted version of a true triple with either the head or tail entity replaced by a random entity (target excluded) from the training minibatch. We define the structural objective function \( f \) using the TransE (Bordes et al., 2013) model, which represents each triple \( (h, r, t) \) as:

\[
f_{TransE}(h, r, t) = -||h + r - t||_1
\]

where \( h, r, \) and \( t \) are the vector representations of the head entity, relation, and tail entity, respectively. The loss function is then defined as:

\[
Loss = \sum_{(t, t') \in (T \times T')} max(0, 1 - f(t) + f(t'))
\]

where \( f(t) \) and \( f(t') \) are the scores assigned to the true triple \( t \) and the negative triple \( t' \), respectively.

3.3 Evaluation and Metrics
Evaluation Scenarios  We assess our models in two inductive scenarios, following Bordes et al. (2013). In the first setting, called dynamic evaluation, new entities may appear in the head or tail positions, and the candidates set is defined as \( \hat{\mathcal{E}} = \mathcal{E}_{train} \cup \mathcal{E}_{eval} \). In the second setting, called
transfer evaluation, both head and tail entities are new and unseen during training, and the candidates set is defined as \( \hat{\mathcal{E}} = \mathcal{E}_{eval} \), where \( \mathcal{E}_{eval} \) is disjoint from the training set of entities \( \mathcal{E}_{train} \).

Metrics For each evaluation triple, we create two types of queries: \((h, r, \hat{e})\) for predicting tails and \((\hat{e}, r, t)\) for predicting heads, where \( \hat{e} \in \hat{\mathcal{E}} \) represents all possible candidate entities, as described in Subsection 3.1. We rank candidate triples by their scores and evaluate the ranking of the correct triple. We report Mean Reciprocal Rank (MRR) and Hits@k (H@k) with \( k \in \{1, 3, 10\} \) averaged across head and tail prediction tasks. We adopt the filtered setting as in Bordes et al. (2013), removing valid triples from the set of negative candidate triples when ranking candidate targets.

3.4 Proposed Architecture

Figure 1 shows the overall architecture of our proposed model. There are three main components: Knowledge Graph (KG), Text Transformer Encoder (TT) and Graph Transformer Encoder (GT).

Knowledge Graph The text-attributed KG component contains a set of triples of type \((h_{KG}, r_{KG}, t_{KG})\), along with their corresponding textual descriptions. For each head \( h_{KG} \) and tail \( t_{KG} \), we also extract their ego-graphs (1-hop neighbourhood), denoted as \( E(h_{KG}) \) and \( E(t_{KG}) \), respectively. Then we encode each of these nodes with the text encoder, discussed below. Specifically, for each centre entity, we encode its textual description, as well as encoding the textual descriptions of its neighbouring entities and the relations that connect the centre entity to its neighbours.

Text Transformer Encoder The textual descriptions from the KG module are passed to the Text Transformer Encoder (TT), which produces vector representations \( x_{TT} \) for each entity and relation textual description \( x_{KG} \) in the ego-graph. More formally, we apply the following function:

\[
x_{TT} = \sigma(\text{BERT}_{\text{SIZE}}(x_{KG})[\text{CLS}] W_0) W_1
\]

where \( W_0, W_1 \in \mathbb{R}^{d \times d} \) are two linear projection matrices and \( \sigma \) is the SiLU (Elfwing et al., 2017) activation function. We employ BERT (Devlin et al., 2019) as our encoder and use the [CLS] vector representation output by the encoder as the embedding. BERT_{SIZE} indicates that we employ different sizes of this model released by Turc et al. (2019).

When encoding candidate entities \( \hat{e} \in \hat{\mathcal{E}} \), we simply pass the text associated with the entity through the TT component. The same is true for any neighbouring entities required by the graph encoder (discussed below). In contrast, when encoding queries \((h, r, \cdot)\) and \((\cdot, r, t)\) we condition \( h \) and \( t \) on the relation type \( r \). Similarly to STAR (Wang et al., 2021a) and STATIK (Markowitz et al., 2022), for tail prediction queries \((h, r, \cdot)\) we concatenate the text associated with \( h_{KG} \) and \( r_{KG} \), resulting in \([h|r]_{KG}\). For head prediction queries \((\cdot, r, t)\), we create an inverse version of the relation text by prepending its textual description with the text "inverse of", denoted as \( r_{KG}^{-1} \). We then concatenate the text associated with \( t_{KG} \) and \( r_{KG}^{-1} \), resulting in \([t|r^{-1}]_{KG}\).

Graph Transformer Encoder The outputs of the TT component, together with the ego-graphs \( E(h_{TT}) \) and \( E(t_{TT}) \) are then input to the Graph Transformer Encoder (GT). The input embedding for each node of the graph is the vector output by the TT encoder for that entity, as described above. In addition, we add learnable segment embeddings to each node input, indicated as \( s_{\text{CENTRE}} \) and \( s_{\text{NEIGHBOUR}} \), to disambiguate between the centre and neighbour nodes in the ego-graph. These
embeddings indicate to the model which input nodes will be used subsequently as the embedding representation of the ego-graph.

To encode the graph relations, we follow Mohammadshahi and Henderson (2020, 2021); Miculich and Henderson (2022); Coman et al. (2024) in leveraging the intrinsic graph processing capabilities of the Transformer model by incorporating graph relations as relation embeddings input to the self-attention function. But unlike in that previous work, our relation embeddings are computed from the text associated with the relation, rather than coming from a fixed set of relations. For every pair of input nodes \( i, j \), the pre-softmax attention score \( e_{ij} \in \mathbb{R} \) is computed from both the respective node embeddings \( x_i, x_j \in \mathbb{R}^d \), and the embedding of the relation \( r_{ij} \) between the \( i \)-th and \( j \)-th nodes, as:
\[
e_{ij} = \frac{x_i W_Q \operatorname{diag}(1 + \mathrm{LN}(r_{ij}) W_R) (x_j W_K)^T}{\sqrt{d}}
\]
where \( W_Q, W_K \in \mathbb{R}^{d \times d} \) represent the query and key matrices, respectively, \( r_{ij} \) represents the relation embedding output by the TT module when it encodes the text associated with the relation between the \( i \)-th and \( j \)-th nodes, and \( W_R \in \mathbb{R}^{d \times d} \) is the relation matrix. Thus, \( \mathrm{LN}(r_{ij}) W_R \) is the embedding of the relation between \( i \) and \( j \), where LN stands for the *LayerNorm* operation. Finally, \( \operatorname{diag}(1 + \ldots) \) maps this vector into a diagonal matrix plus the identity matrix.

When encoding candidate entities and queries in the GT, it is crucial to ensure that no information regarding the target triple \( (h, r, t) \) leaks into the \( E(\hat{e}) \), \( E(h_{TT}) \), or \( E(t_{TT}) \) ego-graphs. This precaution prevents the model from learning trivial solutions or biases from leaked information.

3.5 Datasets and Setting

Daza et al. (2021) introduced the WN18RR$_{\text{IND}}$ and FB15k-237$_{\text{IND}}$ inductive variants of the well-known WN18RR$_{\text{TRA}}$ (Dettmers et al., 2018) and FB15k-237$_{\text{TRA}}$ (Toutanova and Chen, 2015) transductive KGs. The inductive setting simulates a *dynamic* scenario where new entities and triples are dynamically added to the graph. The training graph is constructed as \( G_{train} = \{(h, r, t) \in \mathcal{T}_{train} : h, t \in \mathcal{E}_{train}\} \). The validation and test graphs are constructed by incrementally adding entities and triples, such that \( G_{val} = \{(h, r, t) \in \mathcal{T}_{val} : h, t \in \mathcal{E}_{train} \cup \mathcal{E}_{val}\} \) and \( G_{test} = \{(h, r, t) \in \mathcal{T}_{test} : h, t \in \mathcal{E}_{train} \cup \mathcal{E}_{val} \cup \mathcal{E}_{test}\} \).

In contrast, the Wikidata5M$_{\text{IND}}$ KGs curated by Wang et al. (2021b) provide a *transfer* learning scenario in which the evaluation graphs are constructed such that the validation and test entity and triple sets, \( \mathcal{E}_{val} \) and \( \mathcal{E}_{test} \), and \( \mathcal{T}_{val} \) and \( \mathcal{T}_{test} \) are disjoint from the training entity and triple set \( \mathcal{E}_{train} \) and \( \mathcal{T}_{train} \). The validation and test graphs are constructed as \( G_{val} = \{(h, r, t) \in \mathcal{T}_{val} : h, t \in \mathcal{E}_{val}\} \) and \( G_{test} = \{(h, r, t) \in \mathcal{T}_{test} : h, t \in \mathcal{E}_{test}\} \). Because the graphs \( G_{val} \) and \( G_{test} \) do not include the entities from \( G_{train} \), they are much smaller graphs (see Appendix Table 6), which poses challenges for generalisation with graph-aware models, as will be discussed further below. We evaluate our model’s ability to generalise to entirely new entities and triples in this setting.

We conduct our experiments in the above-mentioned settings of Daza et al. (2021) and Wang et al. (2021b), where textual information extraction is an integral part. Our method is directly comparable to DKRL (Xie et al., 2016), BLP (Daza et al., 2021), KEPLER (Wang et al., 2021b), StAR (Wang et al., 2021a), and the state-of-the-art method, StATIK (Markowitz et al., 2022) which employ the same textual encoder, structural objective, and loss function. Similar to StATIK, our work aims to jointly model the text and the structure of knowledge graphs, including extracting information about KG links from the text. This sets us apart from the setting of Teru et al. (2020), which uses different KGs splits and is employed in GraIL (Teru et al., 2020), NBFNet (Zhu et al., 2021), and NodePiece (Galkin et al., 2021), that solely focus on using the structure of the graph without incorporating any textual information extraction component.

3.6 Controlled Experimental Setup

When comparing the performance of different models on link prediction tasks, it is crucial to establish a fair and consistent baseline. Our experiments in Table 2 highlight the importance of carefully setting this baseline, as various factors can greatly influence the results.

Specifically, we demonstrate that the computational budget, which determines training hyperparameters, can have a substantial impact on model performance. Starting with the baseline model BLP$_{\text{BERTBASE}}$ (Daza et al., 2021), we introduce improvements such as using inductive relations, increasing the number of negative triples to match the batch size (negatives batch tying), increasing the embedding dimension from 128 to 768, doubling the batch size from 64 to 128, and modifying the
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">WN18RR<sub>IND</sub></th>
    <th colspan="4">FB15k-237<sub>IND</sub></th>
  </tr>
  <tr>
    <th>MRR</th>
    <th>H@1</th>
    <th>H@3</th>
    <th>H@10</th>
    <th>MRR</th>
    <th>H@1</th>
    <th>H@3</th>
    <th>H@10</th>
  </tr>
  <tr>
    <td colspan="9"><i>Text-Only Models</i></td>
  </tr>
  <tr>
    <td>DKRL*<sub>BERT<sub>BASE</sub></sub></td>
    <td>0.139</td>
    <td>0.048</td>
    <td>0.169</td>
    <td>0.320</td>
    <td>0.144</td>
    <td>0.084</td>
    <td>0.151</td>
    <td>0.263</td>
  </tr>
  <tr>
    <td>BOW*<sub>BERT<sub>BASE</sub></sub></td>
    <td>0.180</td>
    <td>0.045</td>
    <td>0.244</td>
    <td>0.450</td>
    <td>0.173</td>
    <td>0.103</td>
    <td>0.184</td>
    <td>0.316</td>
  </tr>
  <tr>
    <td>BLP*<sub>BERT<sub>BASE</sub></sub></td>
    <td>0.285</td>
    <td>0.135</td>
    <td>0.361</td>
    <td>0.580</td>
    <td>0.195</td>
    <td>0.113</td>
    <td>0.213</td>
    <td>0.363</td>
  </tr>
  <tr>
    <td>FnF-T<sub>BERT<sub>BASE</sub></sub></td>
    <td><b>0.373</b></td>
    <td><b>0.238</b></td>
    <td><b>0.442</b></td>
    <td><b>0.647</b></td>
    <td><b>0.266</b></td>
    <td><b>0.174</b></td>
    <td><b>0.297</b></td>
    <td><b>0.453</b></td>
  </tr>
  <tr>
    <td>FnF-T<sub>BERT<sub>MEDIUM</sub></sub></td>
    <td>0.342</td>
    <td>0.213</td>
    <td>0.405</td>
    <td>0.603</td>
    <td>0.253</td>
    <td>0.164</td>
    <td>0.281</td>
    <td>0.431</td>
  </tr>
  <tr>
    <td>FnF-T<sub>BERT<sub>SMALL</sub></sub></td>
    <td>0.320</td>
    <td>0.197</td>
    <td>0.379</td>
    <td>0.572</td>
    <td>0.239</td>
    <td>0.152</td>
    <td>0.265</td>
    <td>0.411</td>
  </tr>
  <tr>
    <td>FnF-T<sub>BERT<sub>MINI</sub></sub></td>
    <td>0.268</td>
    <td>0.156</td>
    <td>0.318</td>
    <td>0.498</td>
    <td>0.204</td>
    <td>0.128</td>
    <td>0.223</td>
    <td>0.354</td>
  </tr>
  <tr>
    <td>FnF-T<sub>BERT<sub>TINY</sub></sub></td>
    <td>0.193</td>
    <td>0.098</td>
    <td>0.230</td>
    <td>0.385</td>
    <td>0.164</td>
    <td>0.100</td>
    <td>0.176</td>
    <td>0.289</td>
  </tr>
  <tr>
    <td colspan="9"><i>Structure-Informed Models</i></td>
  </tr>
  <tr>
    <td>StAR<sub>BERT<sub>BASE</sub></sub></td>
    <td>0.321</td>
    <td>0.192</td>
    <td>0.381</td>
    <td>0.576</td>
    <td>0.163</td>
    <td>0.092</td>
    <td>0.176</td>
    <td>0.309</td>
  </tr>
  <tr>
    <td>StATIK<sub>BERT<sub>BASE</sub></sub></td>
    <td>0.516</td>
    <td>0.425</td>
    <td>0.558</td>
    <td>0.690</td>
    <td>0.224</td>
    <td>0.143</td>
    <td>0.248</td>
    <td>0.381</td>
  </tr>
  <tr>
    <td>FnF-TG<sub>BERT<sub>BASE</sub></sub></td>
    <td>0.732</td>
    <td>0.652</td>
    <td>0.785</td>
    <td><b>0.875</b></td>
    <td>0.316</td>
    <td>0.214</td>
    <td>0.350</td>
    <td><b>0.524</b></td>
  </tr>
  <tr>
    <td>FnF-TG<sub>BERT<sub>MEDIUM</sub></sub></td>
    <td><b>0.737</b></td>
    <td><b>0.661</b></td>
    <td><b>0.789</b></td>
    <td>0.873</td>
    <td>0.314</td>
    <td>0.214</td>
    <td>0.353</td>
    <td>0.515</td>
  </tr>
  <tr>
    <td>FnF-TG<sub>BERT<sub>SMALL</sub></sub></td>
    <td>0.727</td>
    <td>0.648</td>
    <td>0.781</td>
    <td>0.867</td>
    <td><b>0.316</b></td>
    <td><b>0.216</b></td>
    <td><b>0.354</b></td>
    <td>0.518</td>
  </tr>
  <tr>
    <td>FnF-TG<sub>BERT<sub>MINI</sub></sub></td>
    <td>0.713</td>
    <td>0.632</td>
    <td>0.768</td>
    <td>0.857</td>
    <td>0.302</td>
    <td>0.204</td>
    <td>0.337</td>
    <td>0.502</td>
  </tr>
  <tr>
    <td>FnF-TG<sub>BERT<sub>TINY</sub></sub></td>
    <td>0.638</td>
    <td>0.543</td>
    <td>0.700</td>
    <td>0.808</td>
    <td>0.288</td>
    <td>0.195</td>
    <td>0.318</td>
    <td>0.475</td>
  </tr>
</table>

Table 1: WN18RR<sub>IND</sub> and FB15k-237<sub>IND</sub> test set results. *Daza et al. (2021); *Markowitz et al. (2022).

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">WN18RR<sub>IND</sub></th>
    <th colspan="2">FB15k-237<sub>IND</sub></th>
  </tr>
  <tr>
    <th>MRR</th>
    <th></th>
    <th>MRR</th>
    <th></th>
  </tr>
  <tr>
    <td>BLP<sub>BERT<sub>BASE</sub></sub></td>
    <td>0.285</td>
    <td></td>
    <td>0.195</td>
    <td></td>
  </tr>
  <tr>
    <td>BLP*<sub>BERT<sub>BASE</sub></sub></td>
    <td>0.280</td>
    <td></td>
    <td>0.205</td>
    <td></td>
  </tr>
  <tr>
    <td>+ inductive relations</td>
    <td>0.281</td>
    <td></td>
    <td>0.219</td>
    <td></td>
  </tr>
  <tr>
    <td>+ negatives batch tying</td>
    <td>0.300</td>
    <td></td>
    <td>0.221</td>
    <td></td>
  </tr>
  <tr>
    <td>+ bigger embedding size</td>
    <td>0.339</td>
    <td></td>
    <td>0.254</td>
    <td></td>
  </tr>
  <tr>
    <td>+ bigger batch size</td>
    <td>0.366</td>
    <td></td>
    <td>0.260</td>
    <td></td>
  </tr>
  <tr>
    <td>+ better sampling method</td>
    <td>0.373</td>
    <td></td>
    <td>0.266</td>
    <td></td>
  </tr>
  <tr>
    <td>FnF-T<sub>BERT<sub>BASE</sub></sub> (ours)</td>
    <td><b>0.373</b></td>
    <td></td>
    <td><b>0.266</b></td>
    <td></td>
  </tr>
</table>

Table 2: WN18RR<sub>IND</sub> and FB15k-237<sub>IND</sub> test set results with cumulative additions over the baseline model BLP<sub>BERT<sub>BASE</sub></sub> (Daza et al., 2021) that lead to our improved baseline model FnF-T<sub>BERT<sub>BASE</sub></sub>. BLP*<sub>BERT<sub>BASE</sub></sub> indicates our reimplemention of BLP<sub>BERT<sub>BASE</sub></sub>.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">Text-Only Models</th>
  </tr>
  <tr>
    <th>MRR</th>
    <th>H@1</th>
    <th>H@3</th>
    <th>H@10</th>
  </tr>
  <tr>
    <td>KEPLER*<sub>BERT<sub>BASE</sub></sub></td>
    <td>0.402</td>
    <td>0.222</td>
    <td>0.514</td>
    <td>0.730</td>
  </tr>
  <tr>
    <td>BLP*<sub>BERT<sub>BASE</sub></sub></td>
    <td>0.478</td>
    <td>0.241</td>
    <td>0.660</td>
    <td>0.871</td>
  </tr>
  <tr>
    <td>FnF-T<sub>BERT<sub>BASE</sub></sub></td>
    <td><b>0.597</b></td>
    <td><b>0.427</b></td>
    <td><b>0.722</b></td>
    <td><b>0.896</b></td>
  </tr>
  <tr>
    <td>FnF-T<sub>BERT<sub>MEDIUM</sub></sub></td>
    <td>0.588</td>
    <td>0.418</td>
    <td>0.712</td>
    <td>0.890</td>
  </tr>
  <tr>
    <td>FnF-T<sub>BERT<sub>SMALL</sub></sub></td>
    <td>0.588</td>
    <td>0.417</td>
    <td>0.714</td>
    <td>0.889</td>
  </tr>
  <tr>
    <td>FnF-T<sub>BERT<sub>MINI</sub></sub></td>
    <td>0.562</td>
    <td>0.391</td>
    <td>0.683</td>
    <td>0.870</td>
  </tr>
  <tr>
    <td>FnF-T<sub>BERT<sub>TINY</sub></sub></td>
    <td>0.526</td>
    <td>0.348</td>
    <td>0.649</td>
    <td>0.849</td>
  </tr>
  <tr>
    <td colspan="5"><i>Structure-Informed Models</i></td>
  </tr>
  <tr>
    <td>StATIK<sub>BERT<sub>BASE</sub></sub></td>
    <td>0.770</td>
    <td><b>0.765</b></td>
    <td>0.771</td>
    <td>0.779</td>
  </tr>
  <tr>
    <td>FnF-TG<sub>BERT<sub>BASE</sub></sub></td>
    <td><b>0.799</b></td>
    <td>0.741</td>
    <td><b>0.833</b></td>
    <td><b>0.911</b></td>
  </tr>
  <tr>
    <td>FnF-TG<sub>BERT<sub>MEDIUM</sub></sub></td>
    <td>0.785</td>
    <td>0.727</td>
    <td>0.817</td>
    <td>0.900</td>
  </tr>
  <tr>
    <td>FnF-TG<sub>BERT<sub>SMALL</sub></sub></td>
    <td>0.781</td>
    <td>0.721</td>
    <td>0.816</td>
    <td>0.898</td>
  </tr>
  <tr>
    <td>FnF-TG<sub>BERT<sub>MINI</sub></sub></td>
    <td>0.779</td>
    <td>0.719</td>
    <td>0.814</td>
    <td>0.894</td>
  </tr>
  <tr>
    <td>FnF-TG<sub>BERT<sub>TINY</sub></sub></td>
    <td>0.761</td>
    <td>0.697</td>
    <td>0.799</td>
    <td>0.883</td>
  </tr>
</table>

Table 3: Wikidata5M<sub>IND</sub> test set results. *Wang et al. (2021b); *Daza et al. (2021); *Markowitz et al. (2022).

negative sampling strategy to two-sided reflexive, where both head and tail entities are considered as potential negatives. These cumulative improvements lead to the development of a new text-only model baseline, FnF-T<sub>BERT<sub>BASE</sub></sub>, which shows substantial improvements on both the WN18RR<sub>IND</sub> and FB15k-237<sub>IND</sub> datasets.

To ensure a fair comparison, we fixed our computational budget to a constant in this paper, using a consumer-grade GPU (NVIDIA RTX3090 24GB). This allows for a consistent and reproducible experimental setup, enabling a more accurate assessment of performance. For more details, see Appendix A.

3.7 Inductive Link Prediction Results

As shown in the top half of Table 1, for both the WN18RR<sub>IND</sub> and the FB15k-237<sub>IND</sub> datasets, our inductive relation embeddings and the enhanced controlled experimental setup result in improved text-only models. These models rely heavily on having powerful text encoders, as shown by the degradation in performance when using smaller versions of BERT as the text encoder.

The addition of our graph encoder to the model (bottom half of Table 1) leads to a substantial increase in link prediction accuracy over the text-only model. We also see that our TG (text-graph) encoder results in substantially better accuracy than the previous state-of-the-art model, StATIK. Interestingly, this more effective use of graph context also has a big impact on the model’s dependence on powerful text encoders. Reducing the size of the text encoder (BERT<sub>BASE</sub> > BERT<sub>MEDIUM</sub> > BERT<sub>SMALL</sub> > BERT<sub>MINI</sub> > BERT<sub>TINY</sub>) does re-
sult in some degradation of accuracy, but the differences are much smaller than in the text-only case. Even with a BERT_{TINY} text encoder, the graph-aware model performs better than the text-only model with a BERT_{BASE} encoder. This shows that the inductive bias of explicit graph relations can be an effective alternative to extracting the same information from text with a powerful text encoder.

This pattern of results is repeated in the transfer case, shown in Table 3. Here, the training set is much larger, but the graph in the test set is relatively small with each entity having fewer neighbours (see Appendix Table 6). This reduces the advantage gained from adding an effective graph encoder and the margin of our models’ improvement over the text-only models, and over the previous state-of-the-art model, STATIK.\footnote{STATIK has a surprisingly high H@1 score, almost identical to its H@3, H@10 and MRR scores. It is not clear why this is the case. Regardless, our model’s MRR, H@3, and H@10 scores are better than STATIK. MRR is the primary evaluation measure since it summarises the entire ranking.} But we still see the same pattern where the size of the text encoder has less effect on accuracy for the graph-aware model.

3.8 Ablation Study

Table 4 presents results from our ablation studies, showing the impact of removing various design features from our graph-aware model on its accuracy.

<table>
  <tr>
    <th>Model</th>
    <th colspan="2">MRR</th>
  </tr>
  <tr>
    <th></th>
    <th>WN18RR_{IND}</th>
    <th>FB15k-237_{IND}</th>
  </tr>
  <tr>
    <td>FnF-TG_{BERT_{MEDIUM}} | SMALL</td>
    <td>0.737</td>
    <td>0.316</td>
  </tr>
  <tr>
    <td>\( r_{ij} \)</td>
    <td>0.733</td>
    <td>0.306</td>
  </tr>
  <tr>
    <td>\( s_{[CENTRE]}, s_{[NEIGHBOUR]} \)</td>
    <td>0.677</td>
    <td>0.298</td>
  </tr>
  <tr>
    <td>\( E(h_{TT}), E(t_{TT}) \)</td>
    <td>0.480</td>
    <td>0.251</td>
  </tr>
  <tr>
    <td>\( [h|r]_{KG}, [t|r^{-1}]_{KG} \)</td>
    <td>0.342</td>
    <td>0.239</td>
  </tr>
</table>

Table 4: Ablation studies on the WN18RR_{IND} and FB15k-237_{IND} test sets using the top FnF-TG model. Each row indicates the performance after cumulatively removing a specific feature.

Removing the \( r_{ij} \) relation embeddings in the pre-softmax attention score leads to a decline in model performance, with a more substantial drop observed on the FB15K-237_{IND} dataset compared to the WN18RR_{IND} dataset. Note that with this modification the model still knows that there is some relation to the neighbours, but does not know its label. Removing the learnable segment embeddings \( s_{[CENTRE]} \) and \( s_{[NEIGHBOUR]} \) then removes this unlabelled graph structure, which considerably impacts the model’s performance. Eliminating the ego-graph neighbours altogether results in an even more substantial performance drop. Despite this, the model remains competitive as a text-only model compared to the BLP baseline, owing to its ability to leverage relation conditioning features to represent candidate relations. Finally, removing the relation conditioning \([h|r]_{KG}\) and \([t|r^{-1}]_{KG}\), results in a further notable decrease in performance. Without relation conditioning, the model loses its ability to anticipate the query relation, severely impacting its accuracy.

3.9 Efficient Text Encoders

Being able to reduce the size of the text encoder with minimal degradation in accuracy is important because the text encoder is a substantial part of the training cost. In Figure 2 we plot the relative reduction in accuracy against the relative reduction in training time as we reduce the size of the text encoder, for the WN18RR_{IND} and the FB15k-237_{IND} datasets. We see that reducing the encoder size by a factor of four reduces the training time by a factor of three for WN18RR_{IND} (and nearly two for FB15k-237_{IND}) with very little reduction in accuracy.

![Line plots showing relative MRR and latency vs. text encoder size for WN18RR and FB15k-237 datasets](page_1012_1342_377_282.png)

Figure 2: Accuracy and training time plotted as a function of text encoder size, relative to the largest text encoder with the highest accuracy, shown as (1.0, 1.0).
<table>
  <tr>
    <th rowspan="2">Training</th>
    <th rowspan="2">Evaluation</th>
    <th colspan="4">WN18RR</th>
    <th colspan="4">FB15k-237</th>
    <th colspan="4">Wikidata5M</th>
  </tr>
  <tr>
    <th>MRR</th>
    <th>H@1</th>
    <th>H@3</th>
    <th>H@10</th>
    <th>MRR</th>
    <th>H@1</th>
    <th>H@3</th>
    <th>H@10</th>
    <th>MRR</th>
    <th>H@1</th>
    <th>H@3</th>
    <th>H@10</th>
  </tr>
  <tr>
    <td colspan="14">FnF-T<sub>BERT_TINY</sub></td>
  </tr>
  <tr>
    <td>IND</td><td>IND</td><td>0.193</td><td>0.098</td><td>0.230</td><td>0.385</td><td>0.164</td><td>0.100</td><td>0.176</td><td>0.289</td><td>0.526</td><td>0.348</td><td>0.649</td><td>0.849</td>
  </tr>
  <tr>
    <td>FIR</td><td>IND</td><td>0.169</td><td>0.080</td><td>0.198</td><td>0.346</td><td>0.143</td><td>0.087</td><td>0.153</td><td>0.253</td><td>0.478</td><td>0.306</td><td>0.591</td><td>0.793</td>
  </tr>
  <tr>
    <td>IND</td><td>IND \ FIR</td><td>0.307</td><td>0.210</td><td>0.353</td><td>0.495</td><td>0.171</td><td>0.102</td><td>0.185</td><td>0.308</td><td>0.594</td><td>0.451</td><td>0.697</td><td>0.849</td>
  </tr>
  <tr>
    <td>FIR</td><td>IND \ FIR</td><td>0.064</td><td>0.012</td><td>0.082</td><td>0.159</td><td>0.024</td><td>0.007</td><td>0.026</td><td>0.052</td><td>0.219</td><td>0.054</td><td>0.334</td><td>0.499</td>
  </tr>
  <tr>
    <td colspan="14">FnF-T<sub>G_BERT_TINY</sub></td>
  </tr>
  <tr>
    <td>IND</td><td>IND</td><td>0.638</td><td>0.543</td><td>0.700</td><td>0.808</td><td>0.288</td><td>0.195</td><td>0.318</td><td>0.475</td><td>0.761</td><td>0.697</td><td>0.799</td><td>0.883</td>
  </tr>
  <tr>
    <td>FIR</td><td>IND</td><td>0.573</td><td>0.480</td><td>0.629</td><td>0.738</td><td>0.249</td><td>0.165</td><td>0.274</td><td>0.418</td><td>0.711</td><td>0.644</td><td>0.749</td><td>0.837</td>
  </tr>
  <tr>
    <td>IND</td><td>IND \ FIR</td><td>0.585</td><td>0.483</td><td>0.652</td><td>0.769</td><td>0.282</td><td>0.176</td><td>0.311</td><td>0.514</td><td>0.726</td><td>0.646</td><td>0.781</td><td>0.867</td>
  </tr>
  <tr>
    <td>FIR</td><td>IND \ FIR</td><td>0.108</td><td>0.028</td><td>0.147</td><td>0.242</td><td>0.050</td><td>0.023</td><td>0.051</td><td>0.099</td><td>0.401</td><td>0.287</td><td>0.480</td><td>0.589</td>
  </tr>
  <tr>
    <td colspan="14">Random baseline</td>
  </tr>
  <tr>
    <td>--</td><td>--</td><td>0.0003</td><td>--</td><td>--</td><td>--</td><td>0.0007</td><td>--</td><td>--</td><td>--</td><td>0.0013</td><td>--</td><td>--</td><td>--</td>
  </tr>
</table>

Table 5: Fully inductive link prediction results.

3.10 Fully Inductive Link Prediction Results
The experimental setting of Daza et al. (2021) and Wang et al. (2021b) do not support evaluation on unseen relations. One distinctive advantage of our model is that it is not restricted to a fixed set of relation labels learned during training. Although we do show that conditioning on relation texts improves accuracy even on seen relations (see Table 2), it is important to evaluate our model in a fully inductive setting, where relations are also unseen, in addition to entities.

To this end, we propose a new experimental setting for a fully inductive relations (FIR) evaluation, by converting the WN18RR<sub>IND</sub>, FB15k-237<sub>IND</sub>, and Wikidata-5M<sub>IND</sub> evaluations to their respective FIR versions. More specifically, we focus on the long tail of relations and remove the least frequent relation labels until 10% of edges have been removed from the training graph \( G_{train} \). We then train a new set of models on this new version so they have not seen the removed relation labels, and evaluate them on both the full set of test relations (IND) and specifically on the relations for the unseen labels (IND \ FIR). Given that all the previous models (DKRL, BLP, KEPLER, StAR, and StATIK) are inductive in entities but transductive in relations, none of them can make informed predictions in this setting, so we compare to a random baseline which computes the expected MRR for random rankings of candidate entities \( \mathcal{E} \) as follows:

\[
\mathbb{E}[\text{MRR}_{\text{random}}] = \frac{1}{|\mathcal{E}|} \sum_{i=1}^{|\mathcal{E}|} \frac{1}{i}
\]

The results in Table 5 show the performance of our model on this new setting. Our approach shows promising results as it outperforms the random baseline by a significant margin. However, the performance drops considerably when training on FIR and evaluating on IND \ FIR, indicating that the model struggles with unseen relations. Notably, the results on the Wikidata-5M dataset are considerably better than those obtained on the WN18RR and FB15k-237 datasets, probably due to having relations with more descriptive texts. Nevertheless, these results highlight the need for further research in developing models that can effectively generalize to unseen relations.

4 Conclusion

We presented a new Transformers-based approach to link prediction in text-attributed knowledge graphs that combines textual descriptions and graph structure in a fully inductive setting. Our Fast-and-Frugal Text-Graph (FnF-TG) Transformers outperform previous state-of-the-art models on three popular datasets, showcasing the importance of capturing rich structured information about entities and their relations. Our approach achieves superior performance while maintaining efficiency and scalability, making it a promising solution for large-scale knowledge graph applications. Moreover, our ablation studies provide insights into the key factors contributing to its effectiveness, demonstrating the value of each component in our model. Additionally, we proposed a new evaluation setting for fully inductive link prediction, where relations are also inductive, and demonstrated the potential of our approach in this setting.
Limitations

While our approach has achieved promising results, there are opportunities for further improvement.

One area for exploration is optimising the scalability of our Graph Transformer Encoder component (see Figure 1), which currently requires computing fully quadratic attention over the entire ego-graph of a given entity. In fact it could still require considerable resources if the number of nodes in the ego-graph is scaled to the order of thousands, hundreds of thousands, or even millions.

Our work demonstrates that effectively capturing even local neighbourhood information is both non-trivial and under-explored and that it can significantly enhance performance. Indeed, our simplification to a 1-hop neighbourhood (ego-graph) was a careful decision to balance effectiveness and complexity. This approach not only allows for a fair comparison with the current state-of-the-art method, StATIK (Markowitz et al., 2022), but also mitigates the exponential increase in computational complexity (see Appendix Subsection A.2) associated with larger neighbourhoods. While this predefined 1-hop neighbourhood provides a solid starting point, there is room to explore better alternatives. For instance, investigating multi-hop neighbourhoods or adaptive neighbourhood definitions could uncover more nuanced insights from the graph structure, potentially leading to even better results.

By building upon our framework, future work could refine these aspects, ultimately enhancing the effectiveness and versatility of our approach.

Ethics Statement

We do not anticipate any ethical concerns related to our work, as it primarily presents an alternative approach to a previously proposed method. Our main contribution lies in introducing a new approach for link prediction. In our experiments, we use the same datasets and pretrained models as previous research, all of which are publicly available. However, it is important to acknowledge that these datasets and models may still require further examination for potential fairness issues and the knowledge they encapsulate.

Acknowledgements

We extend our special gratitude to the Swiss National Science Foundation (SNSF) and Research Foundation – Flanders (FWO) for funding this work under grants 200021E_189458 and G094020N.

References

Ivana Balazevic, Carl Allen, and Timothy M. Hospedales. 2018. Hypernetwork knowledge graph embeddings. In International Conference on Artificial Neural Networks.

Michele Banko, Michael J. Cafarella, Stephen Soderland, Matthew Broadhead, and Oren Etzioni. 2007. Open information extraction from the web. In CACM.

Rajarshi Bhowmik and Gerard de Melo. 2020. Explainable link prediction for emerging entities in knowledge graphs. In The Semantic Web–ISWC 2020: 19th International Semantic Web Conference, Athens, Greece, November 2–6, 2020. Proceedings, Part I 19, pages 39–55. Springer.

Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc.

Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense transformers for automatic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762–4779, Florence, Italy. Association for Computational Linguistics.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.

Sanxing Chen, Hao Cheng, Xiaodong Liu, Jian Jiao, Yangfeng Ji, and Jianfeng Gao. 2023. Pre-training transformers for knowledge graph completion. ArXiv, abs/2303.15682.

Sanxing Chen, Xiaodong Liu, Jianfeng Gao, Jian Jiao, Ruofei Zhang, and Yangfeng Ji. 2021. HitTER: Hierarchical transformers for knowledge graph embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10395–10407, Online and Punta Cana,
Dominican Republic. Association for Computational Linguistics.

Andrei Coman, Gianni Barlacchi, and Adrià de Gispert. 2023. Strong and efficient baselines for open domain conversational question answering. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6305–6314, Singapore. Association for Computational Linguistics.

Andrei Coman, Christos Theodoropoulos, Marie-Francine Moens, and James Henderson. 2024. GADePo: Graph-assisted declarative pooling transformers for document-level relation extraction. In Proceedings of the 3rd Workshop on Knowledge Augmented Methods for NLP, pages 1–14, Bangkok, Thailand. Association for Computational Linguistics.

Damai Dai, Hua Zheng, Fuli Luo, Pengcheng Yang, Tianyu Liu, Zhifang Sui, and Baobao Chang. 2021. Inductively representing out-of-knowledge-graph entities by optimal estimation under translational assumptions. In Proceedings of the 6th Workshop on Representation Learning for NLP (Repl4NLP-2021), pages 83–89, Online. Association for Computational Linguistics.

Jeffrey Dalton, Laura Dietz, and James Allan. 2014. Entity query feature expansion using knowledge base links. In Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR ’14, page 365–374, New York, NY, USA. Association for Computing Machinery.

Daniel Daza, Michael Cochez, and Paul Groth. 2021. Inductive entity representations from text via link prediction. In Proceedings of the Web Conference 2021, WWW ’21, page 798–808, New York, NY, USA. Association for Computing Machinery.

Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018. Convolutional 2d knowledge graph embeddings. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI’18/IAAI’18/EAAI’18. AAAI Press.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Takuma Ebisu and Ryutaro Ichise. 2018. Toruse: knowledge graph embedding on a lie group. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI’18/IAAI’18/EAAI’18. AAAI Press.

Stefan Elfwing, Eiji Uchibe, and Kenji Doya. 2017. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks : the official journal of the International Neural Network Society, 107:3–11.

William Falcon and The PyTorch Lightning team. 2019. PyTorch Lightning.

Dieter A. Fensel, Umutcan Simsek, Kevin Angele, Elwin Huaman, Elias Kärle, Oleksandra Panasiuk, Ioan Toma, Jürgen Umbrich, and Alexander Wahler. 2020. Knowledge graphs: Methodology, tools and selected use cases. Knowledge Graphs.

Mikhail Galkin, Jiapeng Wu, E. Denis, and William L. Hamilton. 2021. Nodepiece: Compositional and parameter-efficient representations of large knowledge graphs. ArXiv, abs/2106.12144.

Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. 2017. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 1263–1272. JMLR.org.

Swarnil Gupta, Sreyash Kenkre, and Partha Talukdar. 2019. CaRe: Open knowledge graph embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 378–388, Hong Kong, China. Association for Computational Linguistics.

James Henderson, Alireza Mohammadshahi, Andrei Coman, and Lesly Miculicich. 2023. Transformers as graph-to-graph models. In Proceedings of the Big Picture Workshop, pages 93–107, Singapore. Association for Computational Linguistics.

Jinhao Jiang, Kun Zhou, Xin Zhao, and Ji-Rong Wen. 2022. Unikgqa: Unified retrieval and reasoning for solving multi-hop question answering over knowledge graph. In The Eleventh International Conference on Learning Representations.

Xiaotian Jiang, Quan Wang, and Bin Wang. 2019. Adaptive convolution for multi-relational learning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 978–987, Minneapolis, Minnesota. Association for Computational Linguistics.

Seyed Mehran Kazemi and David Poole. 2018. Simple embedding for link prediction in knowledge graphs. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
Pei Ke, Haozhe Ji, Yu Ran, Xin Cui, Liwei Wang, Lin-feng Song, Xiaoyan Zhu, and Minlie Huang. 2021. JointGT: Graph-text joint representation learning for text generation from knowledge graphs. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2526–2538, Online. Association for Computational Linguistics.

Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning entity and relation embeddings for knowledge graph completion. Proceedings of the AAAI Conference on Artificial Intelligence, 29(1).

Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2020. On the variance of the adaptive learning rate and beyond. In International Conference on Learning Representations.

Robert Logan, Nelson F. Liu, Matthew E. Peters, Matt Gardner, and Sameer Singh. 2019. Barack’s wife hillary: Using knowledge graphs for fact-aware language modeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5962–5971, Florence, Italy. Association for Computational Linguistics.

Chaitanya Malaviya, Chandra Bhagavatula, Antoine Bosselut, and Yejin Choi. 2020. Commonsense knowledge base completion with structural and semantic context. Proceedings of the 34th AAAI Conference on Artificial Intelligence.

Elan Markowitz, Keshav Balasubramanian, Mehrnoosh Mirtaheri, Murali Annavaram, Aram Galstyan, and Greg Ver Steeg. 2022. StATIK: Structure and text for inductive knowledge graph completion. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 604–615, Seattle, United States. Association for Computational Linguistics.

Lesly Miculicich and James Henderson. 2022. Graph refinement for coreference resolution. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2732–2742, Dublin, Ireland. Association for Computational Linguistics.

Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 1003–1011, Suntec, Singapore. Association for Computational Linguistics.

Alireza Mohammadshahi and James Henderson. 2020. Graph-to-graph transformer for transition-based dependency parsing. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3278–3289, Online. Association for Computational Linguistics.

Alireza Mohammadshahi and James Henderson. 2021. Recursive non-autoregressive graph-to-graph transformer for dependency parsing with iterative refinement. Transactions of the Association for Computational Linguistics, 9:120–138.

Maximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. 2015. A review of relational machine learning for knowledge graphs. Proceedings of the IEEE, 104(1):11–33.

Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. 2011. A three-way model for collective learning on multi-relational data. In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML’11, page 809–816, Madison, WI, USA. Omnipress.

Guanglin Niu, Bo Li, Yongfei Zhang, and Shiliang Pu. 2022. CAKE: A scalable commonsense-aware framework for multi-view knowledge graph completion. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2867–2877, Dublin, Ireland. Association for Computational Linguistics.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc.

Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67.

Apoorv Saxena, Adrian Kochsiek, and Rainer Gemulla. 2022. Sequence-to-sequence knowledge graph completion and question answering. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2814–2828, Dublin, Ireland. Association for Computational Linguistics.

M. Schlichtkrull, Thomas Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2017. Modeling relational data with graph convolutional networks. In Extended Semantic Web Conference.

Phillip Schneider, Tim Schopf, Juraj Vladika, Mikhail Galkin, Elena Simperl, and Florian Matthes. 2022. A decade of knowledge graphs in natural language processing: A survey. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 601–614,
Online only. Association for Computational Linguistics.

Haseeb Shah, Johannes Villmow, Adrian Ulges, Ulrich Schwanecke, and Faisal Shafait. 2019. An open-world extension to knowledge graph completion models. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3044–3051.

Noam M. Shazeer. 2020. Glu variants improve transformer. ArXiv, abs/2002.05202.

Baoxiu Shi and Tim Weninger. 2018. Open-world knowledge graph completion. In Proceedings of the AAAI conference on artificial intelligence, volume 32.

Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. 2013. Reasoning with neural tensor networks for knowledge base completion. In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc.

Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. 2021. Roformer: Enhanced transformer with rotary position embedding. ArXiv, abs/2104.09864.

Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2019. Rotate: Knowledge graph embedding by relational rotation in complex space. arXiv preprint arXiv:1902.10197.

Komal K. Teru, Etienne G. Denis, and William L. Hamilton. 2020. Inductive relation prediction by subgraph reasoning. In Proceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org.

Christos Theodoropoulos, James Henderson, Andrei Catalin Coman, and Marie-Francine Moens. 2021. Imposing relation structure in language-model embeddings using contrastive learning. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 337–348, Online. Association for Computational Linguistics.

Kristina Toutanova and Danqi Chen. 2015. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality, pages 57–66, Beijing, China. Association for Computational Linguistics.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13971.

Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In International conference on machine learning, pages 2071–2080. PMLR.

Iulia Turec, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: On the importance of pre-training compact models. arXiv: Computation and Language.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.

Bo Wang, Tao Shen, Guodong Long, Tianyi Zhou, Ying Wang, and Yi Chang. 2021a. Structure-augmented text representation learning for efficient knowledge graph completion. In Proceedings of the Web Conference 2021, WWW ’21, page 1737–1748, New York, NY, USA. Association for Computing Machinery.

Hongwei Wang, Hongyu Ren, and Jure Leskovec. 2020. Relational message passing for knowledge graph completion. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining.

Liang Wang, Wei Zhao, Zhuoyu Wei, and Jingming Liu. 2022. SimKGC: Simple contrastive knowledge graph completion with pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4281–4294, Dublin, Ireland. Association for Computational Linguistics.

Peifeng Wang, Jialong Han, Chenliang Li, and Rong Pan. 2019a. Logic attention based neighborhood aggregation for inductive knowledge graph embedding. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI’19/IAAI’19/EAAI’19. AAAI Press.

Quan Wang, Pingping Huang, Haifeng Wang, Songtai Dai, Wenbin Jiang, Jing Liu, Yajuan Lyu, Yong Zhu, and Hua Wu. 2019b. Coke: Contextualized knowledge graph embedding. arXiv preprint arXiv:1911.02168.

Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. 2017. Knowledge graph embedding: A survey of approaches and applications. IEEE Transactions on Knowledge and Data Engineering, 29(12):2724–2743.

Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021b. KEPLER: A unified model for knowledge embedding and pre-trained language representation. Transactions of the Association for Computational Linguistics, 9:176–194.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame,
Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics.

Ruobing Xie, Zhiyuan Liu, Jia Jia, Huanbo Luan, and Maosong Sun. 2016. Representation learning of knowledge graphs with entity descriptions. In AAAI Conference on Artificial Intelligence.

Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Liu. 2020. On layer normalization in the transformer architecture. ArXiv, abs/2002.04745.

Benfeng Xu, Quan Wang, Yajuan Lyu, Yong Zhu, and Zhendong Mao. 2021. Entity structure within and throughout: Modeling mention dependencies for document-level relation extraction. In AAAI Conference on Artificial Intelligence.

Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2014. Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv:1412.6575.

Guangqian Yang, Yi Liu, Lei Zhang, Licheng Zhang, Hongtao Xie, and Zhendong Mao. 2024. Knowledge context modeling with pre-trained language models for contrastive knowledge graph completion. In Findings of the Association for Computational Linguistics: ACL-2024, pages 8619–8630, Bangkok, Thailand. Association for Computational Linguistics.

Jingfeng Yang, Aditya Gupta, Shyam Upadhyay, Luheng He, Rahul Goel, and Shachi Paul. 2022. TableFormer: Robust transformer modeling for table-text encoding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 528–537, Dublin, Ireland. Association for Computational Linguistics.

Lin F. Yang, Hongyang Chen, Zhao Li, Xiao Ding, and Xindong Wu. 2023. Give us the facts: Enhancing large language models with knowledge graphs for fact-aware language modeling. IEEE Transactions on Knowledge and Data Engineering, 36:3091–3110.

Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. Kg-bert: Bert for knowledge graph completion. ArXiv, abs/1909.03193.

Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. 2021. Do transformers really perform bad for graph representation? In Neural Information Processing Systems.

Donghan Yu, Chenguang Zhu, Yuwei Fang, Wenhao Yu, Shuhang Wang, Yichong Xu, Xiang Ren, Yiming Yang, and Michael Zeng. 2022. KG-FiD: Infusing knowledge graph in fusion-in-decoder for open-domain question answering. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4961–4974, Dublin, Ireland. Association for Computational Linguistics.

Hanwen Zha, Zhiyu Chen, and Xifeng Yan. 2021. Inductive relation prediction by bert. ArXiv, abs/2103.07102.

Houyu Zhang, Zhenghao Liu, Chenyan Xiong, and Zhiyuan Liu. 2020. Grounded conversation generation as guided traverses in commonsense knowledge graphs. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2031–2043, Online. Association for Computational Linguistics.

Zhaocheng Zhu, Zuobai Zhang, Louis-Pascal Xhonneux, and Jian Tang. 2021. Neural bellman-ford networks: A general graph neural network framework for link prediction. In Neural Information Processing Systems.

A Appendix

A.1 Datasets statistics

Table 6 provides the statistics of the datasets used in our experiments. \( \mathcal{E} \) represents the set of entities, \( \mathcal{R} \) denotes the set of relation labels, \( \mathcal{T} \) consists of the set of relation triples \( (h, r, t) \in \mathcal{E} \times \mathcal{R} \times \mathcal{E} \), and \( E(e) \) shows the mean and standard deviation (\( \mu_{\sigma} \)) of the number of neighbours in an entity’s ego-graph.

A.2 Complexity of FnF-TG

Our method exhibits identical computational complexity to STATik (Markowitz et al., 2022), with \( O(N + Q) \) complexity, where \( N \) denotes the number of nodes in the graph and \( Q \) represents the number of queries \( (h, r, \hat{e}) \) and \( (\hat{e}, r, t) \).

A.3 Training and Implementation Details

We provide details on the training and implementation of our models on three datasets: WN18RRIND, FB15k-237IND, and Wikidata5MIND.

Seeds and Epochs We run our experiments with five different seeds (73, 21, 37, 3, 7) for WN18RRIND and FB15k-237IND, and two seeds (73, 21) for Wikidata5MIND due to its large scale (see Table 6). We train our models for 40 epochs on WN18RRIND and FB15k-237IND, and 5 epochs on Wikidata5MIND, following previous works (Daza et al., 2021; Markowitz et al., 2022).
<table>
  <tr>
    <th>Dataset</th>
    <th>\( \mathcal{R} \)</th>
    <th>\( \mathcal{E}_{train} \)</th>
    <th>\( T_{train} \)</th>
    <th>\( E(e)_{train} \)</th>
    <th>\( \mathcal{E}_{val} \)</th>
    <th>\( T_{val} \)</th>
    <th>\( E(e)_{val} \)</th>
    <th>\( \mathcal{E}_{test} \)</th>
    <th>\( T_{test} \)</th>
    <th>\( E(e)_{test} \)</th>
  </tr>
  <tr>
    <td>WN18RR<sub>IND</sub></td>
    <td>11</td>
    <td>32,755</td>
    <td>69,585</td>
    <td>2,123.15</td>
    <td>4,094</td>
    <td>11,381</td>
    <td>1,171.33</td>
    <td>4,456</td>
    <td>12,037</td>
    <td>1,181.35</td>
  </tr>
  <tr>
    <td>FB15K-237<sub>IND</sub></td>
    <td>237</td>
    <td>11,633</td>
    <td>215,082</td>
    <td>18,492.91</td>
    <td>1,454</td>
    <td>42,164</td>
    <td>4,701.63</td>
    <td>2,416</td>
    <td>52,870</td>
    <td>4,971.29</td>
  </tr>
  <tr>
    <td>Wikidata-5M<sub>IND</sub></td>
    <td>822</td>
    <td>4,579,609</td>
    <td>20,496,514</td>
    <td>4,484.41</td>
    <td>7,374</td>
    <td>6,699</td>
    <td>0.910.78</td>
    <td>7,475</td>
    <td>6,894</td>
    <td>0.920.81</td>
  </tr>
</table>

Table 6: WN18RR<sub>IND</sub>, FB15K-237<sub>IND</sub>, and Wikidata-5M<sub>IND</sub> datasets statistics. \( \mathcal{E} \) represents the set of entities, \( \mathcal{R} \) denotes the set of relation labels, \( \mathcal{T} \) consists of the set of relation triples \( (h, r, t) \in \mathcal{E} \times \mathcal{R} \times \mathcal{E} \), and \( E(e) \) shows the mean and standard deviation (\( \mu_{\sigma} \)) of the number of neighbours in an entity’s ego-graph.

Hyperparameters  We set the number of sampled neighbors per entity based on the dataset statistics (Table 6): 10 for WN18RR<sub>IND</sub>, 40 for FB15k-237<sub>IND</sub>, and 1 for Wikidata5M<sub>IND</sub>. We use 24 words of text for each \( x_{KG} \) in WN18RR<sub>IND</sub> and FB15k-237<sub>IND</sub>, and 64 words for Wikidata5M<sub>IND</sub>.

Graph Transformer Encoder  We implement the Graph Transformer Encoder layer using a pre-LayerNorm Transformer (Xiong et al., 2020) with a SwiGLU-type pointwise feed-forward network (Shazeer, 2020). We use a single GT layer, as multiple layers did not improve performance while increasing latency.

Optimisation  We set the learning rate to \( 1e^{-5} \) for a batch size of 32 and scale it proportionally with the batch size following a power-of-2 rule to fit the GPU budget. We use RAdam (Liu et al., 2020) as our optimiser and a cosine learning rate decay throughout the training process.

Libraries  We develop our models using PyTorch (Paszke et al., 2019), Lightning (Falcon and The PyTorch Lightning team, 2019), and Hugging Face’s Transformers (Wolf et al., 2020) libraries.

A.4 Computational Budget

We fix our computational budget to a constant consumer-grade GPU (NVIDIA RTX3090 24GB) as stated in Subsection 3.6 and report the GPU budget per run for each dataset on FnF-TG<sub>BERT_BASE</sub> relative to the largest text encoders. The GPU budget per run is 4 GPU/h for WN18RR<sub>IND</sub>, 6 GPU/h for FB15k-237<sub>IND</sub>, and 40 GPU/h for Wikidata5M<sub>IND</sub>.

Reviewer paper summaries: The paper proposed efficient and effective architecture for inductive relation prediction in text attributed knowledge graph. 

The model architecture can be summarized a text encoder connected by a graph encoder. 

TransE and contrastive loss are employed to train the model. 

The experiments were done on three popular dataset `WN18RR`, `FB15k-237`, and `Wikidata5M` using an inductive evaluation configuration. The experimental results showed that the proposed model out-performed previous state of the art model.

This paper presents the Fast-and-Frugal Text-Graph Transformer, an approach that addresses link prediction in text-attributed knowledge graphs. The authors propose a framework that integrates textual and structural information to improve link prediction while maintaining efficiency in terms of computational resources. This method reduces reliance on large, resource-intensive text encoders by effectively encoding ego-graphs (1-hop neighbourhoods). Experimental results demonstrate the effectiveness of the proposed method. It achieves superior results compared to some state-of-the-art models.
Summary of strengths: 1. The paper is well written, and easy to follow.

2. The architecture is clean, efficient and effective. 

3. The model performance is good compare with previous baselines. 

4. The ablation study is comprehensive.

- The paper proposes a novel approach that combines both textual and structural features.

- The proposed method outperforms existing approaches on several datasets.
Summary of weaknesses: 1. The idea of Text Encoder + Graph Encoder is not novel. According to survey [1], a lot of previous works such as  [2], [3], [4], [5] and [6] have employed idea or go beyond the idea (such as nested text encoder and graph encoder) and all of them achieved strong performance.  The architecture proposed in the paper list are very likely to be adapted into your setting, which use ego-graph to encode single node. 

2. You may also adapt more dataset from the following paper to demonstrate the architecture you designed. 

[1] Large Language Models on Graphs: A Comprehensive Survey
[2] GNN-LM: Language Modeling based on Global Contexts via GNN.
[3] TextGNN: Improving Text Encoder via Graph Neural Network in Sponsored Search.
[4] SimTeG: A Frustratingly Simple Approach Improves Textual Graph Learning
[5] Patton: Language Model Pretraining on Text-Rich Networks
[6] GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph

- The concept of fully inductive learning was already introduced in [1] and [2]. However, the authors repeat the construction of fully inductive datasets without referencing or discussing these prior works.

- In Section 3.1, the paper only defines inductive representation learning but does not define fully inductive representation learning.

- The font size in some figures is too small to read clearly.

- There is some ambiguity regarding the relationship between E_{val}, E_{test}, and E_{eval}. This should be clarified.

- The paper cites and discusses too few recent works, such as the latest inductive methods [3], ULTRA [4], TRIX [5], and KGICL [6].

[1] INGRAM: Inductive Knowledge Graph Embedding via Relation Graphs, ICLM 2023

[2] Relational Message Passing for Fully Inductive Knowledge Graph Completion, ICDE 2023

[3] A*Net: A Scalable Path-based Reasoning Approach for Knowledge Graphs, NeurIPS 2023

[4] Towards Foundation Models for Knowledge Graph Reasoning, ICLR 2024

[5] TRIX: A More Expressive Model for Zero-shot Domain Transfer in Knowledge Graphs, LoG 2024

[6] Prompt-Based Knowledge Graph Foundation Model for Universal In-Context Reasoning, NeurIPS 2024
Comments, suggestions and typos: Correspond to weakness 1, you may further improve the novelty of your paper and boost the performance, or demonstrate that `with more advanced  architecture, the improvement in performance is marginal`. 
Correspond to weakness 2, you may employ more dataset to make your architecture more convincing.

Please see "Summary Of Weaknesses".


--- after rebuttal ---

Thanks for your response. I increased the Overall Assessment to 2.5.
Overall assessment scores: 3

2.5