Graph Language Models (2024)

Abstract: While Language Models (LMs) are the workhorses of NLP, their interplay with structured knowledge graphs (KGs) is still actively researched. Current methods for encoding such graphs typically either (i) linearize them for embedding with LMs – which underutilize structural information, or (ii) use Graph Neural Networks (GNNs) to preserve the graph structure – but GNNs cannot represent text features as well as pretrained LMs. In our work we introduce a novel LM type, the Graph Language Model (GLM), that integrates the strengths of both approaches and mitigates their weaknesses. The GLM parameters are initialized from a pretrained LM to enhance understanding of individual graph concepts and triplets. Simultaneously, we design the GLM’s architecture to incorporate graph biases, thereby promoting effective knowledge distribution within the graph. This enables GLMs to process graphs, texts, and interleaved inputs of both. Empirical evaluations on relation classification tasks show that GLM embeddings surpass both LMand GNN-based baselines in supervised and zeroshot setting, demonstrating their versatility.1

heading: 1 Introduction
text: Knowledge Graphs (KGs) are essential for organizing vast data, to facilitate information retrieval, or revealing hidden insights for decisionmaking (Plenz et al., 2024). KGs excel in explicitly representing manifold relationships, so with an expanding wealth of information they become crucial tools in the digital age.
Many KGs consist of knowledge triplets, where nodes are entities and edges represent relationships holding between them. Each triplet represents a fact in pseudo-natural language, e.g., (Thailand; Capital; Bangkok) in DBpedia (Auer et al., 2007). Despite the (usual) simplicity of each individual
1https://github.com/Heidelberg-NLP/ GraphLanguageModels
triplet, complex structures emerge in KGs. We refer to such KGs as Graphs of Triplets (GoTs).
To use GoTs effectively, we need meaningful encodings of their components. A natural choice is leveraging LMs, as they can capture the semantics of textually encoded entities, relations or entire triplets. But LMs are not prepared to capture graphstructured information and cannot model complex interactions in a GoT. To alleviate this problem, one can leverage graph NNs (GNNs). But GNNs are not well suited to capture meanings associated with text, and hence often LMs are used to convert nodes (and possibly edges) to language-based semantic embeddings. But in such settings, semantic encoding leveraged from LMs and structural reasoning performed by GNNs are separated and are driven by distinct underlying principles. We expect this to limit model performance if both textual and structural information are important for a task.
In this work we introduce a Graph Language Model (GLM) that resolves this tension through early fusion of textual and structural information. Most LMs today are transformers. Since transformers operate on sets, Positional Encoding (PE) is used to inform them about the inherent sequential ordering of linguistic inputs. In our GLM formulation, we modify PE and self-attention to convert LMs (i.e., sequence transformers) to graph transformers that natively operate on graphs, while preserving their LM capabilities. Usually, a new architecture requires pretraining from scratch, which is
4477
extremely costly. By adopting some non-invasive changes in the LM’s self-attention module, we transform the LM to a Graph Transformer (GT) – while maintaining compatibility with its pretrained LM parameters. When encoding a graph, LM-like attention patterns process linearly organized textual information of individual triplets, while GT-like attention patterns aggregate information along the graph structure. Hence, the GLM inherits text understanding of triplets from the LM, while its GT architecture allows it to directly perform structural reasoning, without additional GNN layers.
Importantly, for text sequences – which can be seen as a special type of graph – the GLM is identical to the original LM. This allows the GLM to process interleaved inputs of text and GoT jointly, handling both modalities in a single framework.
Our main contributions are: (i) We propose Graph Language Models (GLMs) and a theoretical framework to construct them. GLMs are graph transformers, which enables graph reasoning. Simultaneously, they inherit and exploit LM weights, enabling them to represent and contextualize triplets in a GoT. Further, by encoding texts and graph components alike, it can naturally take graph and text data as interleaved inputs. (ii) Experiments on relation classification in ConceptNet subgraphs show that GLMs outperform LM- and GNN-based methods for encoding GoTs – even when the inherited LM parameters are not updated during GLM training. (iii) KG population experiments on Wikidata subgraphs and corresponding Wikipedia abstracts show that GLMs can reason over interleaved inputs of GoTs and text – again, outperforming strong LM-based methods.

heading: 2 Related Work
text: LMs One way to augment LMs with knowledge from KGs (Pan et al., 2024) is to formulate pretraining objectives that operate on a KG. E.g., LMs can be trained to generate parts of a KG, encouraging the LM to store KG content in its parameters. Typically, single triplets are used for pretraining (Bosselut et al., 2019; Wang et al., 2021; Hwang et al., 2021; West et al., 2023). In such cases, the graph structure is not a target of pretraining. Some works generate larger substructures, such as paths or linearized subgraphs (Wang et al., 2020a; Schmitt et al., 2020; Huguet Cabot and Navigli, 2021). In either case, the LM needs to memorize the KG, as it will not be part of the input during inference.
Another approach is to provide the linearized KG as part of the input during inference. This is common for KG-to-text generation (Schmitt et al., 2020; Ribeiro et al., 2021; Li et al., 2021), where models learn to take the linearized (typically small-sized) graphs as input. A recent trend is retrieval augmented generation, where relevant parts of a knowledge base, or KG, are retrieved, linearized and provided as part of a prompt (Gao et al., 2024).2
In both options the graph must be linearized to fit the input or output of a sequence-to-sequence LM. Hence, no graph priors can be enforced – instead, the LM has to learn the graph structure implicitly. By contrast, GLMs model a graph as a true graph and have inductive graph priors instilled in their architecture. This prepares a GLM for more proficient graph reasoning, compared to a LM approach.
GNNs LMs excel at representing single triplets, but struggle with structural reasoning. To alleviate this problem, LMs can be combined with GNNs. Many approaches get node and edge features from LMs and aggregate this information in the graph with GNNs (Lin et al., 2019; Malaviya et al., 2020; Zhao et al., 2023). Zhang et al. (2022); Yasunaga et al. (2022) train models consisting of a LM and a GNN that encode interleaved text and graph inputs jointly. They also use a LM to obtain node features.
While some approaches jointly train for textual understanding and graph reasoning, none offer a unified method. By contrast, our GLM formulation seamlessly integrates both in a holistic framework for embedding language and KGs.
Graph Transformers GTs, a special type of GNN (Bronstein et al., 2021), gain popularity in NLP and beyond (Min et al., 2022; Müller et al., 2023). E.g., Koncel-Kedziorski et al. (2019) and Wang et al. (2020b) train GTs to generate text from KGs and AMRs, respectively. Most relevant to our work is Schmitt et al. (2021), who use GTs for KGto-text generation. Similar to us, they employ PE matrices, but train their model from scratch, which limits its applicability: while their model trained on WebNLG (Gardent et al., 2017) has a vocabulary size of 2,100, initializing a GLM from T5, equips it with T5’s full vocabulary of 32,128 tokens.
Concurrently, and independently from our work, Li et al. (2024) also convert a LM to a graph transformer. They focus on data-to-text generation,
2Cf. www.llamaindex.ai and www.langchain.com
where they unify table, key-value and KG structures in a unified graph format, and apply structureenhanced pre-training to support data-to-text generation with their structure-enhanced transformer model. They apply attention maps similar to ours to better capture the graph-structured input, which the pre-trained model rewrites into natural language. Contrary to their work, we do not resort to structureenhanced pre-training – which is restricted in resources – but instead assess the GLMs’ innate capabilities. We showcase the versatility of the inherited LM parameters in conjunction with our graph transformer architecture, by applying them to challenging reasoning tasks, where the model needs to reason over complementary inputs from text and graphs, and where it needs to infer information not present in the input, unlike data-to-text generation. Moreover, we demonstrate that our architectural changes are highly compatible with the original LM weights, via linear probing experiments, where the GLM outperforms conventional LM and Graph Transformer models.

heading: 3 Preliminary: Graph Transformers (GT)
text: This section briefly introduces graph transformers, focusing on architectural choices relevant for our work. We also discuss some general properties of GNNs that motivate our design choices in §4.
The attention in self-attention can be written as
softmax ( QKT√
d +BP +M
) V, (1)
where Q, K and V are the query, key and value matrices, and d is the query and key dimension. The BP and M matrices can be used for positional encoding and masking. Setting BP = M = 0 yields the standard formulation (Vaswani et al., 2017).
Positional Encoding The self-attention mechanism of transformer models is permutation invariant, i.e., it doesn’t have any notion of the order of its input elements. Thus, positional Encoding (PE) is used to inform LMs of the ordering of tokens in a text (Dufter et al., 2022). Most approaches employ either absolute PE, where absolute token positions are encoded (Vaswani et al., 2017; Gehring et al., 2017) or relative PE, which encodes the relative position between pairs of tokens (Shaw et al., 2018; Raffel et al., 2020; Su et al., 2021; Press et al., 2022). Absolute PE is typically combined with the input sequence and hence, the PE does not need to be encoded in self-attention (BP = 0). For relative
PE, BP encodes a bias depending on the relative distances between pairs of tokens – for example, by learning one scalar for each possible distance:
BP = f(P ), (2)
where P is a matrix of relative distances and f(·) an elementwise function.
Similarly, GTs use PEs to encode the structure of the input, and hence, their PE has to encode a graph structure, as opposed to a sequence. This can again be done with absolute or relative PEs. However, defining an “absolute position” of a node or edge in a graph is not straightforward. While many methods exist, they are not directly compatible with the usual (absolute) “counting position” known from sequence encoding in LMs. In this work we thus focus on relative PE. Given a directed acyclic path in a graph, we can define the (signed) distance between any pair of nodes along a path simply as the number of hops between the nodes. The sign can be set by the direction of the path. Thus, by finding a consistent set of such paths in §4, we obtain relative distances and hence the graph’s PE.
Masked Attention In a vanilla transformer, selfattention is computed for all possible pairs of tokens in the input. By contrast, nodes typically only attend to adjacent nodes in GNNs. Therefore, information between more distant nodes has to be propagated across multiple GNN layers. For graphs, such sparse message passing approaches are sometimes preferred, as in most graphs the neighborhood size increases exponentially with increasing radius, which can cause loss of information due to over-smoothing (Chen et al., 2020). Thus, in GTs it can be beneficial to introduce graph priors, for example by restricting self-attention to local neighborhoods. This can be realized by setting elements of M to 0 for pairs of tokens that should be connected, and to −∞ otherwise.
On the other hand, it has been shown that a global view of the graph can enable efficient, longranged information flow (Alon and Yahav, 2021; Ribeiro et al., 2020). We will therefore present two model variants in §4 – a local and a global GLM.

heading: 4 Graph Language Model
text: GLM vs. GT We aim to design an architecture that can efficiently and jointly reason over text and graph-structured data. GTs can offer desired graph priors, but they lack language understanding.
One intuitive approach to bridge this gap is to pretrain a GT from scratch (Schmitt et al., 2021). But pretraining is costly and the necessary data bound to be scarce. We thus take a different avenue. We hypothesize that for reasoning over GoTs a model needs language understanding capabilities similar to those used for reasoning over text. Intuitively this should be the case, since (i) GoTs are designed to be understandable by humans and (ii) literate people can “read” and understand GoTs.
By initializing a GT with parameters from a compatible LM, we obtain our Graph Language Model (GLM). The GT architecture introduces graph priors, while parameter initialization from the LM gives it language understanding capabilities. In the following we explain the necessary modifications to the input graph and to the model, to make this work. The general idea is that (verbalized) triplets should resemble natural language as much as possible to enable LM weights to capture them, while graph reasoning should work via message passing.
Graph preprocessing A LM tokenizer converts text into a sequence of tokens from the LM vocabulary. Similarly, we process GoTs, such that the GLM can process the graphs “as a LM would do” (cf. Fig. 2). To achieve this, we first convert the GoT to its so-called Levi graph (Schmitt et al., 2021), i.e., we replace each edge with a node that contains the relation name as text feature, and connect the new node to the head and tail of the original edge via unlabeled edges, preserving the direction of the original edge. Next, we tokenize each node and split each node into multiple nodes, such that every new node corresponds to a single token. New edges connect adjacent nodes, again preserving the
original direction. This yields the extended Levi graph (see Fig. 2b). In this representation, each triplet is represented as a sequence of tokens – just as it would be for a standard LM.3
Positional Encodings As discussed in §3, we prefer PEs that encode the relative position between pairs of tokens, determined by their signed distance. We can directly adopt this method to encode the relative position between pairs of tokens occurring within the same triplet – by simply considering the triplet as a piece of text, and counting the token distance in this text. Note that a single token can occur
3Note that the token sequence of the converted GoT is not necessarily perfectly identical to the token sequence that corresponds to the input triplets. We tokenize each node in the Levi graph individually, to ensure consistent tokenization of concepts shared by multiple triplets. This removes whitespace between concepts and edges, which impacts tokenization. We leave investigation of the impact of this effect to future work.
in multiple triplets, leading to, e.g., multiple “lefthand side neighbors” (cf. animal in Fig. 2b and 3). While this does not occur in ordinary sequential text, it does not impose a problem for relative PE.
Yet, the approach above breaks when tokens do not belong to the same triplet. To determine the distance between such pairs of tokens, previous work considered, e.g., the length of the shortest path between them (Schmitt et al., 2021). However, this results in PEs that do not come natural to a LM, since a triplet would appear in reversed order, if it is traversed in “the wrong direction” in the shortest path.4 We therefore omit structure-informed PE between tokens that do not belong to the same triplet and instead propose two GLM variants: a local (ℓGLM) and a global (gGLM) one.
Local and global GLM Fig. 3 shows the relative token position matrix P for the graph in Fig. 2b. In the ℓGLM the self-attention mechanism is restricted to tokens from the same triplet. This means that attention to any token located beyond the local triplet is set to 0 – and hence does not require PE. Still, in such configurations, messages can propagate through the graph across multiple layers, since tokens belonging to a concept can be shared by multiple triplets. This is analogous to standard message passing in GNNs, where non-adjacent nodes have no direct connection, but can still share information via message passing. For example, the representation of dog is contextualized by the triplets black poodle is a dog and dog is a animal after the first ℓGLM layer. Hence in the second layer, when animal attends to dog, the animal embedding gets impacted by black poodle, even though there is no direct connection from animal to black poodle.
However, it has been shown that a global view can have benefits (Ribeiro et al., 2020). Hence, we also formalize the gGLM, as an alternative where self-attention can connect any node to every other node. For this setting we need to assign a PE to any pair of tokens, including those that do not occur within the same triplet. For these pairs we introduce a new graph-to-graph (G2G) relative position. LMs don’t have learned parameters for G2G connections, so we initialize the parameters with the corresponding parameters of a relative position of +∞. In a LM a relative position of +∞ means that the respective tokens occur somewhere “far” away in a remote text passage. LMs learn a
4For example, cat would see the graph as the following sequence: cat is a animal a is dog a is poodle black.
proximity bias during pretraining, i.e., they tend to have higher attention scores between tokens that are close to each other in the text. This means that tokens with a high relative distance tend to have low attention scores. For our gGLM this corresponds to a graph bias where distant nodes are less important, but are still accessible.5 Note that unlike in the ℓGLM, this bias is not part of the architecture. It originates from the pretrained parameters, meaning that the gGLM can learn to attend to distant tokens.
Along with P and M , the GLM takes a sequence of all tokens in the extended Levi graph as input. For this, we technically need to “linearize” the graph. However, the order of tokens in the resulting sequence does not matter: relative positions in P are determined by distances in the graph, not in the sequence. Permuting the input sequence simply means that rows and columns of P and M need to be permuted accordingly, but the resulting token embeddings remain unchanged. See example matrices for P and M for ℓGLM and gGLM in §A.
Being transformers, GLMs have the same computational complexity as their respective LM. For sparse graphs the ℓGLM could make use of sparse matrix multiplication, making it more efficient than a corresponding LM or gGLM. However, for our experiments this was not necessary.
Joint graph and text encoding If we use normal matrices for P and M , the GLM is identical to its underlying LM. Hence, GLMs can be applied to texts and – more interestingly – interleaved inputs of text and graph. In this joint setting, P and M each consists of four sub-matrices that correspond to self-attention between tokens from (i) graph-tograph, (ii) text-to-text, (iii) text-to-graph and (iv) graph-to-text. Graph-to-graph sub-matrices are formatted as described above for ℓGLM and gGLM, respectively. Text-to-text sub-matrices are standard matrices from conventional sequence transformers. We introduce new T2G and G2T relative positions for text-to-graph, and graph-to-text connections, respectively. With this, the model can learn interaction strength between the two modalities. Similar to G2G in gGLM, we initialize T2G and G2T parameters from +∞. See example matrices in §A. Uni- and Bidirectional LMs If a LM’s self-attention is unidirectional, information can only propagate along the direction of arrows in Fig. 2b for
5Preliminary experiments showed that initializing G2G parameters from +∞ outperforms random initialization, which outperforms initialization from 0.
the ℓGLM. This means that, e.g., the representation of the node black poodle is independent of the rest of the graph. We could augment the graph with inverse relations to enable bidirectional information flow with unidirectional LMs, but in this work, we restrict our analysis to bidirectional models.
T5 We use T5 (Raffel et al., 2020) – a bidirectional encoder with unidirectional decoder – as base LM to instantiate GLMs. In T5, relative distances in P group into so-called buckets, and each bucket maps to one learned positional bias in BP for each head. Positional biases are shared across layers. The decoder is not needed to encode graphs, but can be used to generate sequences, such as text or linearized graphs in future work.

heading: 5 Experiments
text: We assess the GLMs’ capabilities for embedding GoTs in two experiments on relation (label) classification, i.e., classifying which relation belongs to a given head and tail entity. One experiment uses ConceptNet (CN; Speer et al., 2017) subgraphs that we construct to enable analysis of the impact of structural graph properties. In a second experiment on Wikidata (Vrandečić and Krötzsch, 2014) subgraphs and associated Wikipedia abstracts we test GLMs on interleaved inputs of text and graph.

heading: 5.1 Representing and reasoning over Graphs
text: We construct a balanced dataset of English CN subgraphs consisting of 13,600 train, 1,700 dev and 1,700 test instances with 17 distinct relations as labels. We replace the relation to be predicted with <extra_id_0>, T5’s first mask token.
To investigate the impact of varying graph complexities, we experiment with different graph sizes denoted by their radius r. We ensure that small graphs are strict subgraphs of larger graphs, such that potential performance gains in larger graphs must stem from additional long-ranged context.
To evaluate model effectiveness when longranged connections are crucial, we mask complete subgraphs around the relation to be predicted. The size of a masked subgraph is m, where m = 0 means no mask, m = 1 masks neighboring concepts, m = 2 masks neighboring concepts and the next relations, etc. We replace each masked concept and relation with a different mask token. Construction details and statistics are shown in §B.1.1.

heading: 5.1.1 Experimental setup
text: The input to our model is a CN subgraph. The relation to be predicted is replaced with <extra_id_0>. The GLM encodes the graphs as in §4, producing an embedding for each token. A linear classification head gives the final prediction from the mask’s embedding. We verbalize unmasked relations using static templates (Plenz et al., 2023), shown in §B, Table 4.
In a finetuning setting we train the GLM and the classification head jointly. However, since the GLM is initialized from a LM, we hypothesize that it should produce meaningful embeddings, even without any training. To test this hypothesis, we train only the classification head, i.e., we only train a linear probe. In this setting, the GLM was never trained on any graph data, similar to a zero-shot setting. The linear probe only extracts linear features and hence, can only achieve high performance if the GLM embeddings show expressive features.
We report mean accuracy across 5 different runs. See §B.1.2 for hyperparameters. Unless stated otherwise, we use T5-small to allow many baselines.

heading: 5.1.2 Baselines
text: We compare to several baselines inspired by related work. For all baselines we utilize the T5 encoder as underlying LM. This allows us to focus on the architectural design of different model types.
LM For LM-based approaches we linearize the input graphs to a sequence, by concatenating the verbalized triplets. There are structured ways to linearize graphs, but such graph traversals generally require the graph to be directed and acyclic – which makes them inapplicable to linearizing GoTs. Instead, we order the triplets either randomly (T5 set) or alphabetically (T5 list). For T5 set, triplets are shuffled randomly in every training epoch such that the model can learn to generalize to unseen orderings. The concatenated triplets are passed to the T5 encoder, and the embedding of <extra_id_0> is presented to the classification head.
GNN For GNN baselines we encode each node of the original graph (cf. Fig. 2a) with the T5 encoder, and train a GNN using these static embeddings. After the final layer, the GNN returns 17 logits for each node. As final logits, we take the mean logit of the two nodes adjacent to the relation to be predicted. We experiment with different variants as GNN layers: GCN (Kipf and Welling, 2017) and GAT (Veličković et al., 2018). Since GNNs do not
come with pretrained weights, we only apply them in finetuning, when training all parameters.
Graph transformer Finally we compare GLMs to models with the same architecture, but random weight initialization (normal graph transformers). This allows us to assess the impact of weight initialization from a LM with two further baselines: ℓGT and gGT. We only consider GTs with finetuning.

heading: 5.1.3 Results
text: Linear probing Tab. 1 shows the relation label prediction accuracy for linear probing, i.e., when training only the classification head. Our first observation is that gGLM is consistently the best, outperforming ℓGLM and the LM baselines. For a radius of r = 1 we have exactly one triplet, which has almost the same representation in the GLM and LM approaches. The only difference is that the LM baselines have an end-of-sentence token, which the GLM does not have. Surprisingly, not having the end-of-sentence token seems to be an advantage with linear probing, but we will see later that this changes when updating model weights.
For r ≥ 3, LM baselines show decreasing performance with increasing radii. By contrast, both ℓGLM and gGLM show increasing performances with increasing radii. This indicates that GLMs can utilize the additional context. But LM baselines don’t have any inbuilt methods to grasp distances in the graph, which could cause them to fail at distinguishing relevant from less relevant information.
The performance gap between gGLM and LM models tends to increase for larger m, i.e., when larger sub-structures are masked. However, the ℓGLM underperforms for large m, highlighting the advantage of the global view in gGLM when longranged connections are necessary.
The overall high performance of GLMs confirms our assumption that GLMs are compatible with LM weights, even without any training. Increasing performance with increasing radii further shows that GLMs have good inductive graph biases. When long-range connections are relevant, the representations learned by gGLM outperform the locally constrained ℓGLM – which showcases the strength of the global view that the gGLM is able to take.
Finetuning Tab. 1 shows results when training all parameters. In this setting, models can adjust to the task and learn to reason over graphs through parameter updates. In addition, GLMs can tune parameters to better match the novel input structure.
The GLM and LM variants are consistently better than GNN and GT methods, which indicates that linguistic understanding is potentially more important than graph reasoning for this task. Models outperform their linear probing scores, which shows that finetuning is, as expected, beneficial.
Overall, the GLMs perform best, while GTs perform the worst. The only difference between the two model groups is weight initialization – the GLMs are initialized from T5, while the GTs are randomly initialized. Further, we observe that for r ≥ 1 and m = 0 the local GT (ℓGT) significantly outperforms its global counterpart gGT. For the GLM the global version is on par, or even better than the local one. This shows the effectiveness of T5’s attention mechanism: thanks to its weight initialization, gGLM attends to relevant tokens even in large context windows, while gGT suffers from potentially distracting long-ranged information.
For m = 0 the differences between GLM and LM approaches are small, with a slight trend for GLMs to outperform LMs on large graphs, and vice versa for small graphs. However, when graph rea-
soning is more important due to masking (m ≥ 1), then GLMs consistently and significantly outperform all other baselines. This indicates that LMs can learn to do simple graph reasoning through parameter updates, but underperform in more complex graph reasoning tasks where either graphs are larger, or long-ranged connections are required.
For m ≥ 1, the gGLM outperforms ℓGLM due to its global connections. In contrast to the linear probing setting, the ℓGLM outperforms other baselines for all non-zero levels of masking. This indicates that ℓGLM can learn to use long-ranged information during training, if the task requires it.
Impact of model size To investigate the effect of model size, we train the most promising approaches (GLM and LM) in 3 different sizes. Tab. 6 in §B.1.3 shows that overall larger models perform better. Surprisingly, the base models sometimes outperform the larger models for settings that require more graph reasoning, i.e., larger m. However, these differences are small and non-significant. In most cases, gGLM large or base are the best model.

heading: 5.2 Jointly representing Graph and Text
text: We now investigate GLM capabilities to process interleaved inputs of text and graph in a KG population setup, i.e., extending a KG with new relation instances. Subtask 1 performs text-guided relation classification where some relations may be inferrable from the text, while others may exploit graph knowledge to make predictions. In Subtask 2, models classify the source of a predicted relation, i.e., whether it can be inferred from the text, or whether it requires (additional) graph knowledge.
We construct our data from Huguet Cabot and Navigli (2021), who offer a corpus of Wikipedia abstracts that are linked to Wikidata via entity linking. Their focus is relation extraction, so they filter the graphs using NLI, such that all triplets are entailed by the text. We augment the entailed triplets with further triplets from Wikidata that are not entailed by the text. For a given text, subgraph, head and tail entity, models will jointly predict the relation and the source. We adopt the 220 most common relations in our train graphs and a “no-relation” label. For source labels we have 3 classes: entailed by the text, not entailed and no-relation. No-relation is the correct label iff the relation is also no-relation. §B.2.1 shows statistics and construction details.

heading: 5.2.1 Experimental setup and baselines
text: Unlike §5.1.1, models now receive text and graph data as input. We train two distinct classification heads on the mask’s embedding for relation and source classification. While the mask is part of the graph, its embedding depends on both modalities. The final loss is the sum of the relation classification and the source prediction loss, weighted by 0.9 and 0.1. We use T5-large, but otherwise baselines are as in §5.1.2. §B.2.2 shows the training details.

heading: 5.2.2 Results
text: Fig. 4 and Tab. 8 show test set performance for a) relation and b) source classification, at different training stages. gGLM performs the best overall, followed by ℓGLM. LM baselines are competitive, but lag behind at early stages and for source prediction. Again, GT baselines perform poorly, showcasing the advantage of weight initialization in GLM – even with large-scale training data. For all models, training plateaus beyond ∼ 500k seen instances (cf. Fig. 8 in §B.2.3), so we stop training at this cut-off.
Tab. 2 gives results for ablating different input modalities to GLMs. Since source prediction always requires text input, we test relation classifi-
cation w/o source prediction. Ablating the text or graph lowers performance by similar amounts, indicating that GLMs utilize both modalities. Training curves in Fig. 10 reveal that first, the model almost exclusively utilizes text data, but quickly learns to make use of the graph. For textually entailed triplets, text is more impactful than the graph, and vice versa for other triplets (cf. Tab. 9). Ablating graphs lowers source prediction by ∼4.5 points, which shows that GLMs benefit from graph information even for predominantly text oriented tasks.
The results show that GLMs can efficiently reason over interleaved inputs of graph and text, especially with limited training data. This makes GLMs a promising new model type for knowledge-intense NLP tasks, such as KG population or Q&A.

heading: 6 Conclusion
text: We present the Graph Language Model (GLM) – a graph transformer initialized with weights from a LM. It excels at graph reasoning, while simultaneously encoding textual triplets in the graph as LMs do, thereby bridging the gap between LMs and GNNs. GLMs can natively reason over joint inputs from texts and graphs, leveraging and enhancing each modality. Experiments show the GLM’s advantage over LM and GNN based baselines, even in a linear probing setting. In particular, GLMs greatly outperform graph transformers. This highlights the need for pretrained LM weights, even for graph reasoning. We therefore advocate GLMs as a valuable tool for advancing research in embedding and leveraging knowledge graphs for NLP tasks.

heading: Limitations
text: While GLMs are designed as general purpose tools for knowledge-intense NLP tasks, our evaluation is limited to English knowledge graphs. However, we explore various types of knowledge graphs (commonsense and factual) and tasks (relation classification, text-guided relation classification, and source prediction), broadening our empirical assessment. Confirming GLMs improved text and graph rea-
soning skills for different languages, domains and tasks is left for future work.
Our GLM framework supports instantiation from any LM with relative positional encoding, including rotary positional encoding. Comprehensive comparisons to determine the most suitable models for the GLM framework remain for future investigation. Nonetheless, bidirectional LMs are expected to perform best in the novel framework, because unidirectional LMs necessitate additional inverse relations, as discussed in §4.

heading: Ethical considerations
text: We do not foresee immediate ethical concerns for our research, as we rely on well-established datasets. However, even established datasets can contain undesirable biases which our method could potentially spread and amplify.
Looking ahead, our focus lies in enriching knowledge graph integration within language models, with the aim of enhancing factuality and mitigating hallucination. This advancement is expected to bolster the reliability and controllability of LMs, leading to positive societal impacts. Furthermore, LMs relying on knowledge graphs may facilitate easier maintenance, potentially reducing the need for frequent retraining of deployed models, thereby promoting sustainability in NLP practices.

heading: Acknowledgements
text: We want to thank Letit,ia Pârcălăbescu for providing feedback on our manuscript.
This work was funded by DFG, the German Research Foundation, within the project “ACCEPT: Perspectivized Argument Knowledge Graphs for Deliberation”, as part of the priority program “RATIO: Robust Argumentation Machines” (SPP1999).

heading: A Model
text: Fig. 5 shows the matrices P and M for ℓGLM and gGLM. Fig. 6 shows the same matrices for joint encoding of text and graph data.

heading: B Experiments

heading: B.1 ConceptNet

heading: B.1.1 Dataset
text: We experiment on randomly selected subgraphs from the largest connected component of the English part of CN version 5.7 (Speer et al., 2017), which consists of 125,661 concepts and 1,025,802 triplets. We select 17 distinct relation label classes (cf. Tab. 4), ensuring sufficient frequency and semantic dissimilarity. For each class, we randomly sample 1,000 triplets, allowing only cases where exactly one triplet connects the head and tail entities, to reduce label ambiguity. These 1,000 instances are split into train (800), dev (100), and test (100). This creates a balanced dataset of 13,600 train, 1,700 dev, and 1,700 test instances. To predict relation labels, we replace them with <extra_id_0>, T5’s first mask token. For our experiments, we replace CN (unmasked) relations with more natural verbalizations. Tab. 4 shows the static verbalization for each relation.
During graph construction we control the graph size, parameterized by the radius r. We start with a radius of r = 1, when we consider only the two concepts (head and tail) in the target triplet. To create a larger graph context, we randomly select 4 adjacent triplets – 2 for the head, and 2 for the tail entity of the original triplet. A graph with radius r = 2 is formed by the subgraph spanned by all entities used in these 5 triplets. For r = 3 we again randomly select 2 triplets for each of the outer (up to) 4 entities, yielding (up to) 13 triplets. To avoid accidentally adding more short-ranged information, we restrict the new triplets to triplets that actually extend the radius of the graph. This enables us to control graph size and complexity, while still enabling sufficient diversity in the graph structure. Further, the graphs are created such that graphs for smaller radii are strict subgraphs of graphs with larger radii. This ensures that performance changes with increasing radii are due to long-ranged connections, and not due to potentially different shortranged information. Tab. 3 shows structural properties of CN subgraphs, depending on their radius.
When masking subgraphs, we mask complete subgraphs of a certain size around the target to be predicted. The size of the masked subgraph is denoted by m, where m = 0 means no masking, m = 1 masks neighboring concepts, m = 2 masks neighboring concepts and the next relations, and so
on. Formally, m denotes the radius of the masked graph in Levi representation, which should not be confused with the extended Levi graph, nor the normal graph representation. We replace each concept and relation in the masked subgraph with a different mask token. This in principle enables LM baselines to internally reconstruct the graph.
B.1.2 Experimental setup and baselines Tab. 5 shows our hyperparameters. For the GNNs, we tested different numbers of layers (2, 3, 4, 5), hidden channel dimensions (32, 64, 128), and nonlinearities (ReLU, leaky ReLU) in preliminary experiments.

heading: B.1.3 Results
text: Tab. 6 shows performance on CN for different modelsizes.

heading: B.2 Wikidata and Wikipedia

heading: B.2.1 Dataset
text: Huguet Cabot and Navigli (2021) propose a largescale corpus of aligned Wikipedia abstracts and Wikidata (Vrandečić and Krötzsch, 2014) triplets. They first extract Wikidata entities from the abstract, and then link these entities with triplets in Wikidata. They are interested in triplets that are entailed by the text, so they use a NLI model to filter out all other triplets. They publicly released the extracted entities and the filtered triplets.
For our purpose, we are interested in aligned graphs and texts, but triplets in the graph do not necessarily have to be entailed by the text. Hence, we find all triplets between the extracted entities using the Wikidata Query Service.6 From Huguet Cabot and Navigli (2021) we know which triplets in our graphs are entailed by the text.
Similar to Huguet Cabot and Navigli (2021) we consider the 220 most common relations in the train split as our relation labels. Additionally, we add a “no-relation” label, yielding 221 relation classes.
For 10% of the graphs we randomly add a new triplet between previously unconnected head and tail entity, and the mask token as relation. For these graphs “no-relation” is the correct relation label. For the other 90% graphs we replace a random existing relation with the mask token, while making sure that (i) the existing relation is in our 220 labels and that (ii) there is no other triplet connecting the respective head and tail entities. We remove
6https://query.wikidata.org, accessed in Jan. 2024.
Model r 1 2 3 4 5 4 4 4 4 4 m 0 0 0 0 0 1 2 3 4 5
instances where no suitable triplet is available. This yields a dataset with 2,449,582 train, 135,828 val and 135,923 test instances.
Fig. 7 shows the label distributions for relation and source for train and test. Out of the 221 relations, only 195 and 194 relations occur in the train and test set, respectively. All relations in the test set also occur in the train set.
Tab. 7 shows graph statistics. Compared to CN subgraphs (c.f. Tab. 3) the graphs are relatively small, matching the size of r = 2. On CN we found that LMs can perform well on such small graphs, so we expect that the performance gap between GLMs and LM baselines on Wikidata would be larger if Wikidata subgraphs were larger.
B.2.2 Experimental setup and baselines
For these experiments we omit GNNs as a baseline, since they can’t natively process texts.
The other models all compute an embedding of the mask token, and then two separate classification heads produce predictions for the relations (221 classes) and the source (3 classes). For each prediction, we compute the Cross Entropy Loss. The final loss is the weighted sum of these losses, weighted by 0.9 and 0.1 respectively. The relation classification has a higher weight since it has
many more classes and hence, is potentially more difficult. This means that model parameters are optimized for both objectives jointly, while only the linear classification heads can specialize on their respective task.
The dataset is unbalanced (c.f. Fig 7), so report macro F1 scores instead of accuracy. This means that models only achieve high scores if they perform well on all classes, including minority classes.
We assume that classifying one out of 221 relations requires fine grained text understanding, so we initialize models from T5-large instead of T5small. To reduce computational load, we only train one model per setting. Further, we enable efficient batching by restricting inputs to a maximum of 512 tokens. This truncates 2.8% of train instances for GLMs and 5.1% for LM baselines due to their less efficient graph encoding.
Hyperparameters are identical to Tab. 5, except that (i) we reduce batch size to 8, (ii) train for at most 1 epoch and (iii) don’t use early stopping.

heading: B.2.3 Results
text: Fig. 8 shows the training curve when training for an entire epoch, i.e., 2,499,582. We observe that performances plateau beyond ∼ 0.2 epochs, so we stop training after 524,288 instances in our other experiments.
Tab. 8 shows concrete numbers for the models in Figures 4 and 8.
Fig. 9 shows confusion matrices for source prediction.
Fig. 10 shows the test performance in relation classification of ablated models during different training steps. Table 9 shows relation classification scores for (i) triplets entailed by text and for (ii)
other triplets.

heading: C Usage of AI assistants
text: We use GitHub Copilot (https://github.com/ features/copilot) for speeding up programming, and ChatGPT 3.5 (https://chat.openai. com) to aid with reformulations. The content of this work is our own, and not inspired by AI assistants.

heading: Future Work
text: Confirming GLMs improved text and graph rea-
soning skills for different languages, domains and tasks is left for future work. Our GLM framework supports instantiation from any LM with relative positional encoding, including rotary positional encoding. Comprehensive comparisons to determine the most suitable models for the GLM framework remain for future investigation. Nonetheless, bidirectional LMs are expected to perform best in the novel framework, because unidirectional LMs necessitate additional inverse relations, as discussed in §4