ConGraT: Self-Supervised Contrastive Pretraining for Joint Graph and Text Embeddings (2024)

Abstract: Learning on text-attributed graphs (TAGs), in which nodes are associated with one or more texts, has been the subject of much recent work. However, most approaches tend to make strong assumptions about the downstream task of interest, are reliant on hand-labeled data, or fail to equally balance the importance of both text and graph representations. In this work, we propose Contrastive Graph-Text pretraining (ConGraT), a general, self-supervised approach for jointly learning separate representations of texts and nodes in a TAG. Our method trains a language model (LM) and a graph neural network (GNN) to align their representations in a common latent space using a batch-wise contrastive learning objective inspired by CLIP. We further propose an extension to the CLIP objective that leverages graph structure to incorporate information about inter-node similarity. Extensive experiments demonstrate that ConGraT outperforms baselines on various downstream tasks, including node and text category classification, link prediction, and language modeling. Finally, we present an application of our method to community detection in social graphs, which enables finding more textually grounded communities, rather than purely graph-based ones.

heading: 1 Introduction
text: Recent advances in multimodal representation learning have shown the benefits of simultaneously modeling language with other modalities, which allows for more efficient training and improved downstream performance of both sets of learned representations. These benefits have been especially clear in text/vision or text/audio applications, which often see large improvements in predictive performance or generative modeling ability (Radford et al., 2021; Li et al., 2022; Mu et al., 2022;
* Corresponding author.
Elizalde et al., 2023). In this work, we address another modality that frequently co-occurs with text: network- or graph-structured data.
We consider in particular the scenario of a textattributed graph (TAG); that is, a graph over entities (i.e., nodes) associated with one or more texts. Such graphs occur frequently in the real world; examples include social media graphs of users and their posts, link graphs over web pages and their content, and citation networks of articles or authors and the texts of academic articles. In this setting, rather than modeling each source of data separately, graph information may be used to improve performance on language tasks and text information may be leveraged for graph tasks such as link prediction or node classification.
Prior work has approached the problem of combining these two modalities in several ways. Some approaches have used textual data to inform or supervise training of graph neural networks (GNNs) (Yang et al., 2015; Zhang et al., 2017; Liu et al., 2018; Zhang and Zhang, 2020), but these methods do not produce graph-informed text representations. This is more parameter-efficient for graphonly tasks, but means that separate modeling is needed to solve text-based tasks while leveraging graph data. Other works have considered the converse case of employing a TAG structure to fine-tune pretrained language models (PLMs) (Cohan et al., 2020; Yasunaga et al., 2022; Ostendorff et al., 2022). Although these approaches allow for the extraction of graph-informed text embeddings, they have the opposite limitation to the above of not learning node representations. While there have been attempts to learn joint representations of nodes and texts, they all have certain limitations, such as requiring a supervised objective and labeled data (Li and Goldwasser, 2019; Chandra et al., 2020), freezing either the text or graph embeddings/encoders (Gourru et al., 2020; Karpov and Kartashev, 2022), or relying on the particu-
19
lar structure of one application (Li et al., 2017). Several recent works have leveraged joint training of PLMs with GNNs to integrate both text and graph information for representation learning in each modality (Yang et al., 2021; Chien et al., 2022; Zhao et al., 2023). These methods, however, make specific modeling assumptions based on the tasks that they aim to solve, employ complex training procedures that alternately optimize the PLM and GNN modules, or need human-annotated knowledge distillation, which in general go against the goal of self-supervised learning.
In this work, we propose ConGraT (Contrastive Graph-Text pretraining), a general approach to self-supervised joint graph-text learning based on a batch-wise contrastive learning objective inspired by CLIP (Radford et al., 2021). The idea is to have separate encoders for texts and graph nodes (more specifically, a PLM and a GNN, respectively) that are trained to align their representations within a common latent space, as shown in Figure 1. Taking advantage of the fact that graphs have greater structure than images, we propose an extension to the CLIP objective that incorporates information about plausible “next guesses” based on graph similarity. Our objective also admits an interpretation as a continuous relaxation of the contrastive CLIP objective over each node’s two-hop neighborhood.
ConGraT provides flexibility in the choice of text and graph encoders and does not make assumptions on the structure of the TAG or any downstream task. As illustrated in our experiments, it is also inductive (Hamilton et al., 2017), with the encoders being able to generalize to previously unseen graphs as well as previously unseen texts. Experiments on various datasets show that ConGraT models consistently outperform strong baselines on various downstream tasks such as node and text category classification and link prediction. Additionally, we
analyze how joint training affects language modeling performance, finding that ConGraT also results in improvements on this task on all datasets.
The contributions of this work are threefold: 1) We propose ConGraT, a general self-supervised pretraining method for jointly learning graph node and text representations on TAGs, such as citation, link, or social graphs. 2) We demonstrate that our joint pretraining method improves performance over strong unimodal and cross-modal baselines on various downstream tasks. 3) We release our code and datasets, including in particular a version of the Pubmed (Sen et al., 2008) graph learning dataset fully rebuilt from ground-truth Pubmed APIs, which includes the text of titles and abstracts as well as network data.

heading: 2 Related Work

heading: 2.1 Text-Augmented GNNs
text: Text data can be incorporated into the learning of GNN representations in various ways. For example, Yang et al. (2015) extend the DeepWalk algorithm to incorporate text features into node representations. Liu et al. (2018) develop a seq2seq framework which learns node embeddings with inputs based on texts associated with the nodes. Tu et al. (2017) use a selective attention mechanism to generate text-informed node embeddings for particular social contexts. Zhang et al. (2017) leverage kernel methods to construct node representations from user profile information in a way that incorporates network structure. Other methods include extracting graphs from entity co-occurrence in texts and modeling them (Zhang and Zhang, 2020; Waller and Anderson, 2021). However, these approaches are limited in that, while they learn to represent nodes, they do not also learn graph-informed text representations.

heading: 2.2 Graph-Augmented PLMs
text: Another line of work uses information from graph structures to inform finetuning or further training of PLMs. SPECTER (Cohan et al., 2020) contrastively finetunes a language model by augmenting it with a measure of inter-node relatedness, with positive and negative examples for a triplet loss selected according to citation graph edges. LinkBERT (Yasunaga et al., 2022) uses a graph structure to assemble training samples for a masked language model, pairing anchor texts with texts from contiguous, linked, or random documents,
and uses an auxiliary document relation prediction (DRP) objective. SciNCL (Ostendorff et al., 2022) relaxes a discrete citation graph into a continuous domain with nearest-neighbor sampling. SocialBERT (Karpov and Kartashev, 2022) and LMSOC (Kulkarni et al., 2021) condition or augment the inputs to PLMs with frozen node representations that the model can attend over. The models these methods learn produce text embeddings for documents, but do not also generate text-informed node representations.

heading: 2.3 Joint Learning of PLMs and GNNs on TAGs
text: More recently, representation learning on TAGs that jointly leverages graph and text information has been growing in popularity. Prefix tuning (Li and Liang, 2021) is a lightweight way of learning node-specific linguistic information and generates dense node representations in the process; however, it takes no advantage of the graph structure over the nodes. For fixed text and graph encoders, one can learn mappings from their separate embedding spaces to a common one, such as by canonical correlation analysis (Gupta and Varma, 2017). Other methods jointly train text and graph encoders using an externally supervised objective (Chandra et al., 2020) or tailored for certain tasks (Li and Goldwasser, 2019; Gourru et al., 2020). However, these methods all address specific settings that are not generalizable to more diverse tasks.
GraphFormers (Yang et al., 2021) jointly train a GNN with a PLM so as to learn text-informed node representations. However, they require a complex progressive learning strategy that iteratively utilizes manipulated and raw data. GIANT (Chien et al., 2022) predicts graph structure using PLMs to provide better initial embeddings for GNNs. However, the language model embeddings cannot be jointly optimized during the GNN training phase. GLEM (Zhao et al., 2023) uses a variational expectationmaximization (EM) framework that alternately updates a PLM and GNN separately using pseudolabels predicted by the other module. While it enables improved scalability, the training procedure is complex and relies on the availability of taskspecific target labels. In contrast, ConGraT is a general representation learning method for both graph nodes and texts, applicable to any inductive or transductive setting without such assumptions or complex training paradigms.

heading: 3 Methodology
text: We consider a directed or undirected TAG, each node of which is associated with a set of one or more texts. The goal is to learn a shared latent space that allows us to place the embeddings of nodes and texts in semantically meaningful locations within that space. Formally, let 𝐺 = (𝑉,𝐸) be a graph, with 𝑉 the set of nodes and 𝐸 ⊆ 𝑉 × 𝑉 the set of edges. Also, let 𝑇 (𝑣) = {𝑡(𝑣)𝑖 }𝑁𝑣𝑖=1, for 𝑣 ∈ 𝑉 , be the set of node 𝑣’s texts, with 𝑁𝑣 the number of texts corresponding to node 𝑣. We model 𝑡(𝑣)𝑖 , the 𝑖-th text of node 𝑣, as a finite sequence of tokens over a vocabulary 𝑊 , where 𝐿(𝑣)𝑖 is the length of 𝑡 (𝑣) 𝑖 : 𝑡 (𝑣) 𝑖 = (𝑆0, 𝑆1, 𝑆2, ..., 𝑆𝐿(𝑖)𝑣
). The first and last tokens are always special start and end tokens.
Our training framework involves a text encoder, a function 𝐹𝑇 : ∪∞𝑖=1 ⊗𝑖 𝑊 → R𝑑 from the set of all token sequences to a 𝑑-dimensional Euclidean embedding space. Similarly, we have a node encoder, a function 𝐹𝐺 : 𝑉 → R𝑑 from nodes to an embedding space of the same dimension. (Note that while its domain is nodes, not edges, 𝐹𝐺 also depends on the edge set 𝐸.) We aim to train the two encoders such that they learn a joint latent space between the text and graph node embeddings. This will allow us to use geometric properties of a common space to relate nodes and texts to each other for downstream inferential purposes.

heading: 3.1 Approach
text: The text and node encoders in ConGraT (a PLM and GNN, respectively) are connected at the output layers by a batch-wise contrastive training objective inspired by CLIP (Radford et al., 2021). The encoders are trained to align their representations in a joint latent embedding space. As in CLIP, each encoder is set behind an adapter module which generates embeddings of the same dimension. Each adapter consists of two fully connected layers with a GeLU activation (Hendrycks and Gimpel, 2020) in between, followed by layer normalization (Ba et al., 2016), and dropout (Srivastava et al., 2014). This approach is flexible and allows use of many different kinds of both text and node encoders. On the text side, we illustrate this flexibility with experiments employing both causal and masked PLMs.
Training objective. We augment the standard InfoNCE loss (Oord et al., 2019) in CLIP with additional graph-specific elements. Unlike the visionlanguage case, in graphs, there are easily com-
putable measures of how similar pairs of nodes are, such as their SimRank (Jeh and Widom, 2002) or their number of mutual in- and out-edges. We use these measures to incorporate information about the most likely second, third, and further choices for the nodes a text may originate from as well as the texts that may be associated with a node. The method is visualized in Figure 2.
More formally, let 𝑋 = ∪𝑣∈𝑉 {(𝑣, 𝑡(𝑣)𝑖 )} |𝑇 (𝑣)| 𝑖=1 be a dataset of (node, text) pairs, and let 𝐵 = {(𝑣𝑖, 𝑡(𝑣𝑖)𝑖 )}𝑁𝐵𝑖=1 ⊆ 𝑋 be a minibatch of size 𝑁𝐵 sampled from 𝑋 . Now, fix an ordering of nodes, with 𝑣𝑗 the 𝑗-th node. Then, in terms of the text and node encoders 𝐹𝑇 and 𝐹𝐺, the matrix 𝐶 given by
𝐶𝑖𝑗 ≜ 𝑒𝜏 𝐹𝑇 (𝑡
(𝑣𝑖) 𝑖 ) · 𝐹𝐺(𝑣𝑗)
‖𝐹𝑇 (𝑡(𝑣𝑖)𝑖 )‖ · ‖𝐹𝐺(𝑣𝑗)‖
is the 𝑁𝐵 × 𝑁𝐵 matrix of cosine similarities between texts and nodes in the batch. (Note that 𝐶 is square but not symmetric: rows are texts and columns are nodes.) The matrix is multiplied by a scalar factor 𝑒𝜏 , where 𝜏 is a log-temperature parameter that allowing some learnable control over the learning rate, reducing sensitivity to the choice of learning rate. We empirically initialize 𝜏 = 3.5 based on our experiments (see Appendix F).
Further, let 𝑆𝑇 (·, ·) and 𝑆𝐺(·, ·) be graph-based similarity functions for texts and nodes, respectively, assigning non-negative continuous similarity scores. Then, we define graph-based similarity distributions for texts and nodes. Where 𝐾𝑇 (𝑖) = ∑︀𝑁𝐵 𝑘=1 𝑆𝑇 (𝑡 (𝑣𝑖) 𝑖 , 𝑡
(𝑣𝑘) 𝑘 ) and 𝐾𝐺(𝑖) =∑︀𝑁𝐵
𝑘=1 𝑆𝐺(𝑣𝑖, 𝑣𝑘), then ∀𝑖, 𝑗, we have
𝑠 (𝑖) 𝑇 (𝑗) =
𝑆𝑇 (𝑡 (𝑣𝑖) 𝑖 , 𝑡 (𝑣𝑗) 𝑗 )
𝐾𝑇 (𝑖) , 𝑠
(𝑖) 𝐺 (𝑗) =
𝑆𝐺(𝑣𝑖, 𝑣𝑗)
𝐾𝐺(𝑖) .
The target distributions are mixtures of these distributions and indicator variables for the true source node of a text and matching text of a node. For each example 𝑋𝑗 = (𝑣𝑗 , 𝑡 (𝑣𝑗) 𝑗 ) in the minibatch, fixing some hyperparameter 𝛼 ∈ [0, 1], we define the target distribution in the text case by D(𝑗)𝑇 (𝛼) = (1 − 𝛼)1𝑖{𝑣𝑖 = 𝑣𝑗} + 𝛼𝑠𝑇 (𝑗), where 𝑠𝑇 (𝑗) is the vector of 𝑠 (𝑖) 𝑇 (𝑗) values for all 𝑖. In the graph case, where 𝑠𝐺(𝑗) is defined analogously, we have D(𝑗)𝐺 (𝛼) = (1− 𝛼)1𝑖{𝑡𝑖 = 𝑡𝑗}+ 𝛼𝑠𝐺(𝑗). Then where 𝐻 is the cross-entropy and 𝐶𝑖,:, 𝐶:,𝑖 are the 𝑖-th row and 𝑖-th column of 𝐶, our loss is
ℒ(𝐵;𝛼) = 1 2𝑁𝐵
𝑁𝐵∑︁
𝑖=1
𝐻(𝐶𝑖,:,D (𝑖) 𝑇 (𝛼))
+𝐻(𝐶:,𝑖,D (𝑖) 𝐺 (𝛼)).
With 𝛼 = 0, this loss is equivalent to the average of cross-entropies for predictions of which node in the minibatch goes with which text and which text goes with which node. With higher values of 𝛼, the target distributions are mixtures of indicators for the true source node and text and the distribution of other nodes and texts by graph similarity. If similar nodes produce similar texts, as suggested by the homophily principle (De Choudhury et al., 2010), positive 𝛼 values should allow the model to learn more efficiently. Even if not all graph nodes are closely related to their texts, this objective should be able to learn from those that are.
Similarity function. For undirected graphs, we base our similarity function on a node pair’s number of mutual neighbors. If 𝐴 is the graph adjacency matrix, we compute 𝐴𝐴𝑇 to find the number of mutual in- or out-neighbors of each node pair, and find the cosine similarity of each row 𝑖 and column 𝑗 of 𝐴𝐴𝑇 to measure the similarity of nodes 𝑖 and 𝑗. A benefit of this function over alternatives like SimRank (Jeh and Widom, 2002) is its lower computational cost and faster runtime for large graphs. On the text side, because we are interested in leveraging graph information, we approximate the similarity of a pair of texts as that of the associated nodes. The digraph case is more complicated, as it requires a directed similarity function that can distinguish between edges (𝑖, 𝑗) and (𝑗, 𝑖). We defer choosing and validating such a function to future work; thus, all experiments with 𝛼 > 0 in Section 4 discard edge directions.

heading: 3.2 Theoretical View
text: From a theoretical perspective, the above similarity function allows our training objective to be viewed as a continuous relaxation of the contrastive CLIP objective across a node’s two-hop neighborhood. (Only nodes with shared neighbors, which are in each other’s two-hop neighborhoods, will have positive similarity.) Different choices of similarity correspond to different choices of how to relax the contrastive objective across the graph; in particular, restricting to the one-hop neighborhood amounts to using a coarse, binary indicator of similarity, with 𝑆𝐺(𝑢, 𝑣) the indicator function for edge (𝑢, 𝑣).
This view indicates a connection to label smoothing (Szegedy et al., 2015), but with the smoothing distribution based on graph similarity rather than being uniformly random. It also casts our model as an extension to a graph setting of prior work on
similarity-based smoothing in non-graph contexts (Liu and JaJa, 2021). Note that this is a different sense of “smoothing” than the aggregation over neighbor representations that the term refers to in the context of message-passing GNNs.

heading: 4 Experimental Setup
text: We evaluate our approach on three tasks: node category classification, link prediction, and language modeling. Link prediction and language modeling are fundamental modality-specific metrics that measure how well our node and text encoders retain the ability to model their individual modalities. We perform node classification using each encoder’s embeddings in order to measure how effective the learned representations are for downstream tasks. Further details are provided in Appendix F.

heading: 4.1 Datasets
text: We evaluate on three datasets, comprising one each of citation, link, and social graphs: (1) the Pubmed dataset (Sen et al., 2008), (2) a dataset of Wikipedia articles represented by their introductory paragraphs and the hyperlinks between the articles in those paragraphs, selected from the broader T-REx Corpus (Elsahar et al., 2018), and (3) a novel Twitter dataset comprising high-profile public figures, which includes the tweets, the follow graph over the associated users, and some demographic information about them (age, gender, United States political party, etc.). We include the last Twitter dataset to demonstrate the performance of our method on social network TAGs, which is a setting that has been less explored in prior work on joint graph and language learning. Table 1 shows descriptive statistics of the datasets. Because we use entirely separate train, validation, and test splits, without shared graph structure, our results below are in an inductive (rather than transductive) setting. More information about the datasets and our collection procedures for Twitter data and the other datasets’ raw text are provided in Appendix E.

heading: 4.2 Models
text: For each dataset, we train two ConGraT variants with masked and causal PLMs, initializing with weights from MPNet (Song et al., 2020) and DistilGPT-2 (Sanh et al., 2019), respectively. Specifically, we use the pretrained all-mpnet-base-v2 and distilgpt2 models from the sentence-transformers toolkit (Reimers and Gurevych, 2019). For the graph node
encoder, we use a graph attention network (GAT) (Veličković et al., 2018) with 3 layers of 2 attention heads each, randomly initialized and trained from scratch. All text and graph embeddings have dimension 768, and we obtain text-level representations from the PLM text encoder by mean pooling.
We examine models with (𝛼 = 0.1) and without (𝛼 = 0.0) graph similarity information included in the loss. We also examine models which consider edge directions (and thus have 𝛼 = 0.0).1 In all, between these three factors (masked or causal PLM, 𝛼 = 0.0 or 𝛼 = 0.1, directed or undirected edges), and without experiments with 𝛼 = 0.1 for directed edges, there are 6 possible model combinations on each dataset, for a total of 18 combinations.

heading: 4.3 Baselines
text: For node representations, we compare against embeddings from a GNN-only baseline: a unimodal GAT autoencoder with the same architecture as the ConGraT node encoder, trained as usual to reconstruct the adjacency matrix without added similarity information. For text representations, in addition to unimodal masked and causal PLM baselines finetuned on each dataset, we also compare against two models leveraging both modalities: LinkBERT (Yasunaga et al., 2022) and Social-LM, a modified implementation of SocialBERT (Karpov and Kartashev, 2022) and LMSOC (Kulkarni et al., 2021). Because LinkBERT uses a masked language modeling objective, it is used as a baseline only for the masked versions of ConGraT. Initial node representations for all GNN models are sentence embeddings of text associated with each node: for Pubmed, the concatenated text of the title and abstract sections; for Twitter, user bios; for T-REx, the Wikipedia article text. Further implementation details are given in Appendix F.
1Recall that because we defined a similarity function with 𝛼 > 0 only for undirected graphs, there are no experiments with directed edges where 𝛼 = 0.1.

heading: 5 Results

heading: 5.1 Node Classification
text: We train logistic regression models to perform node classification on the Pubmed, Twitter, and T-REx datasets using the embeddings generated from each ConGraT model or baseline. Overall, ConGraT models achieve high performance on this set of tasks relative to baselines on all three datasets.
Table 2 shows AUCs for article label classification on Pubmed and T-REx, and Table 3 shows AUCs for demographic classification tasks on age, gender, occupation, party, and region on users in our Twitter dataset. (At the text level, the dependent variable is that of the corresponding node.) The best ConGraT model achieves the highest node classification performance on all datasets, and the differences from the nearest baseline are statistically significant by a bootstrap test (𝑝 < 10−4) in all cases. Even the less performant ConGraT model outperforms all baselines in 26 of 28 experiments.
Notably, we see some of ConGraT’s largest improvements when one modality has less signal than the other. For example, tweet text is less useful than graph data in predicting users’ geographic region. Many Twitter edges are geographically nearby (Takhteyev et al., 2012), and our method is more effective than baselines at infusing this information into an encoder which at inference time sees only text.
The more discriminative nature of representations learned by ConGraT can also be seen visually; Figure 3 shows a 2D UMAP plot comparing ConGraT and GAT embeddings on the Twitter data
subset with U.S. political party labels, which validates that ConGraT embeddings have a much more clearly separated class boundary.

heading: 5.2 Link Prediction
text: We evaluate link prediction performance using inner product decoding (Kipf and Welling, 2016) to derive edge existence probabilities from embeddings. As a baseline, we use the same GAT architecture as in our jointly trained ConGraT models and train it directly on link prediction using the same inner product decoding.
Results are shown in Table 4. All ConGraT models outperform the baselines, despite those baselines being specifically trained for link prediction. In some cases, these improvements are quite large, with the best-performing model on the Twitter dataset recording an AUC of 0.806 vs. 0.723 for the best-performing baseline, a relative increase of 11.5%. Training with graph-based similarity information (𝛼 = 0.1) often also leads to further improvements. Performance is similar for directed and undirected models, demonstrating our
approach’s adaptability to different types of graphs. Notably, our model’s high performance is zero-shot, with no additional training on link prediction.

heading: 5.3 Language Modeling
text: Previous works that jointly trained LMs with GNNs on TAGs (Yang et al., 2021; Chien et al., 2022; Zhao et al., 2023) evaluated on node classification tasks using the representations learned by each module, but did not study in depth how joint training affected the LM component’s capabilities. We perform this analysis, evaluating joint pretraining’s impact on downstream language-modeling performance. To do this, we attach a randomly initialized LM head to the ConGraT text encoder and further train both the encoder and head on causal language modeling. We evaluate with perplexity, and thus limit evaluation to causal-LM variants of our model (those initialized from DistilGPT-2). As a baseline, we finetune the baseline DistilGPT-2 LM on each dataset’s texts.
Table 5 shows the mean perplexity of causal LMs trained using ConGraT with 𝛼 = 0 and 𝛼 = 0.1 compared against the causal LM baseline. For all datasets, LMs trained using ConGraT achieve consistently lower average perplexity, and these differences are statistically significant by a paired 𝑡-test at the 5𝜎 level (𝑝 < 5.7× 10−7).

heading: 5.4 Application: Community Detection
text: To illustrate ConGraT’s broad usability in applications, we compare it to other methods of detecting communities in the Twitter data. As baselines, we use Louvain community detection (Blondel et al., 2008) on the follow graph, and a clustering-based approach on the GAT baseline’s embeddings using UMAP (McInnes et al., 2020) and HDBSCAN (McInnes et al., 2017). For ConGraT, we use the same clustering approach with embeddings from the 𝛼 = 0.0 variant. We expect these methods will find different kinds of communities: while Louvain communities are entirely determined by graph structure, and the GAT baseline can take some advantage of text via use of sentence embeddings as initial node vectors, we expect ConGraT to be most able to infuse textual information into network communities.
Because we want to determine how informed each set of communities is by the text associated with the graph, we evaluate by predicting community labels from text embeddings. For each of the above community detection methods, we first compute the centroid of each node’s text embeddings and label it with the user’s community. Then, we fit a logistic regression model on the training split and predict the test set community label from these centroid text features.
The results, in Figure 4, demonstrate exactly the expected pattern: graph-based Louvain communities are poorly predictable from text, while communities clustered from baseline GAT embeddings are more predictable. The closest relationship to
textual content occurs for communities detected with ConGraT embeddings. This pattern highlights a potential application of our method: detecting more discursively or textually grounded communities in social graphs, rather than ones based only on graph information (e.g., communities informed by discussion among political figures on Twitter).

heading: 6 Conclusion
text: We propose ConGraT (Contrastive Graph-Text pretraining), a self-supervised pretraining framework for jointly learning text and graph node representations using pretrained language models (PLMs) and graph neural networks (GNNs) on a text-attributed graph (TAG). ConGraT uses a batchwise contrastive learning objective to train text and graph encoders to align their representations within a common latent space. The framework is inductive, generalizable to any text or graph encoder architecture, and does not depend on the structure of the TAG or a particular downstream task. In experiments on citation, link, and social graphs, our method outperforms baselines on various downstream tasks, including node classification, link prediction, and language modeling. Our results also highlight the value of incorporating graph structure into our contrastive learning objective, with nonzero values of the 𝛼 parameter often improving performance. Finally, an application to community detection, in which our method finds more textually grounded communities than alternative methods, highlights the broad applicability of this form of representation learning to many domains.

heading: A Sensitivity Analysis on 𝛼
text: To examine the hyperparameter 𝛼’s impact on downstream performance, we conduct a sensitivity analysis on all three evaluation tasks, using the Pubmed dataset, with 𝛼 = 0, 0.1, 0.5, and 1.0. We use only the causal model variant for the LM task.
We find that 𝛼’s impact varies by task. For link prediction and node classification, we see an intuitive pattern: Performance is best for 𝛼 between 0 and 1, especially compared to 𝛼 = 1. That is, both components of our objective—matching nodes to texts, and matching nodes and texts to similar nodes and texts—add value. This pattern is not universal, however; while 𝛼 > 0, particularly 𝛼 = 0.1, consistently outperforms 𝛼 = 0, LM performance is best with 𝛼 = 1.0. We conjecture this pattern may be due to inter-document similarity in language use, which 𝛼 = 1.0 more effectively trains into the PLM. Overall, our results suggest both that 𝛼 = 0.1 is a reasonable default and that it may be worth tuning this parameter in practice.

heading: B Embedding Space Geometry Analysis
text: To complement evaluation on downstream applications like node classification, link prediction and language modeling, this section pursues certain analyses of the geometry of the joint embedding space. We compare these jointly learned embedding spaces to the null model of separate spaces, as learned by the unimodal LM and GAT baselines.
We expect in particular that ConGraT’s joint pretraining should align the two embedding spaces with each other, as well as with non-embedded distance metrics based entirely on the graph. We focus on examining how these distances (in the embedding spaces and the graph) relate to each other, because the geometric properties of a metric space
are chiefly determined by the underlying metric. A finding of significantly increased alignment between distance metrics would indicate the models are effectively integrating information across language and graph modalities. As discussed below and shown in Table 7, this is in fact exactly what we do see.
Inter-Embedding Distance Correlation. First, we examine the correlation of the distance between pairs of texts with the distance between the corresponding pairs of nodes. That is, we sample text pairs (𝑡(𝑢)1 , 𝑡 (𝑣) 2 ) from nodes 𝑢 and 𝑣, and examine over many such samples the correlation between the text-embedding distance 𝑑𝑇 (𝑡 (𝑢) 1 , 𝑡 (𝑣) 2 ) and the graph-embedding distance 𝑑𝐺(𝑢, 𝑣). We operationalize both in practice as the cosine distance.
With ConGraT pretraining, the cosine distance between text embeddings is substantially more correlated than in the separately trained case with the distance between the parent nodes. We see this effect for all model variants against the separatespaces baseline on all datasets, and the increases are significant by a bootstrap test (𝑝 < 10−6). Supporting our hypothesis, the two spaces have become systematically more aligned geometrically.
Embedding-Graph Distance Correlation. Next, we relate the cosine distance in the text embedding space to a purely graph-based distance — SimRank (Jeh and Widom, 2002) in our case. This extends the previous analysis by grounding the text-embedding distance more directly in the graph. In 34 out of 36 cases, we observe a significant increase over the separate-spaces baseline in the correlation between the text embedding distance and graph SimRank (at the 𝑝 = 10−6 level, using a similar bootstrap test).
B.1 Retrieval
Finally, as an additional test of geometric alignment and cross-modal data integration, we consider a simple retrieval task: identifying the node associated with a given text. For each text, we select the node whose embedding has the highest cosine similarity to the text’s embedding, and report the top-𝑘 accuracy for 𝑘 from 1 to 10. This task might itself be an important downstream measure in a retrieval setting, but for purposes of geometric analysis we consider only the comparison to separate embedding spaces here. Note that as with CLIP, this use
of our representations can be thought of as zero shot transfer for text or node classification (where objects in the other modality are the classes).
Results are shown in Figure 5. Top-𝑘 accuracy is substantially higher than the separately-trained baseline for all models at all values of 𝑘. All differences are significant at the 𝑝 = 10−6 level according to a bootstrap test. Moreover, the top-k accuracies achieved are often high relative to the size of the datasets. With 1,996 articles (i.e., nodes) in the Pubmed test set, the best-performing model includes the correct article for a text snippet in its top 10 most similar articles (0.5% of the test set) 94.3% of the time.

heading: C Robustness Check: SVD-Based Initial Vectors
text: In this section, we replicate the analysis described in Section 4, with a twist: instead of the sentence embeddings used there as initial node representa-
tions for those models which rely on them, we use vectors from the truncated SVD of the graph adjacency matrix. We train and evaluate entirely new models, in which all other properties of training, inference and datasets besides the choice of initial node vectors are the same as in the main text. Doing so provides an additional check on the soundness of our approach and gives evidence of its adaptability to various kinds of attributed graphs.
We truncate the SVD embeddings produced by scikit-learn’s implementation (Pedregosa et al., 2011) to 768 dimensions, so that the embedding dimensionality is the same as for the sentence embeddings. To avoid leakage between train and test, we use the 𝑉 * matrix from the train set to generate test-set embeddings (thus relying only on test-set nodes’ connections to nodes in the training set).
C.1 Node Classification
Node classification performance is similar to that reported in the main text, with the best-performing
ConGraT model outperforming all of our baseline models in all tested cases.
Twitter. Table 8 shows AUCs for the demographic classification tasks. Results are similar to what we observe with sentence embeddings as initial node representations: the best-performing ConGraT model beats all baselines in evaluation with both graph and text embeddings.
Unlike in the maint text, however, we do not present results for the political-party outcome variable. This is because the SVD-based embeddings are too predictive: all graph models (and thus also all joint models) are able to perfectly separate the two classes. This phenomenon is a good example of the Twitter follow graph’s powerful organizing principle of homophily: Users tend to be connected to other users who are similar to them (Barberá, 2015), in this case politically.
Pubmed and T-REx. AUCs for article category classification are shown in Table 9. As with the models using sentence embeddings, the best ConGraT model outperforms our baselines in all experiments, using both graph and text embeddings.
C.2 Link Prediction Link prediction results, given in Table 10, are broadly similar to those in the main text and its Table 4. On all three datasets, we see ConGraT node encoders deliver much better than chance performance as zero-shot link predictors.
Performance is often very similar to the levels achieved with sentence embeddings, as on the
Pubmed and undirected T-REx datasets. The most notable difference in relative performance vs baseline is on the Twitter dataset, where in fact ConGraT performance is quite close to that in Table 4 — the GAT baseline, however, performs much better with SVD-based embeddings.
C.3 Language Modeling
Results here are similar to those presented in the main text, though ConGraT model performance
declines slightly on the Pubmed dataset.

heading: D Additional Related Work
text: Many traditional NLP tasks focus on learning graph structures that exist within a text, such as dependency parsing (Kübler et al., 2009) or grammar induction (Klein and Manning, 2001; Kim et al., 2019). More recent lines of work have extended this focus on graph structures to knowledge graphs, where the graph structure is over knowledge-base entities which may appear in the text, and citation or social networks, where the relevant graph is the one between the entities which write or contain the texts. Social science applications have also frequently motivated approaches to learning from joint graph/text data.
Knowledge graphs. Work on knowledge graphs (KGs) uses graph representations to encode facts about the world (Hogan et al., 2022), with real-
world entities as nodes and edges used to encode inter-entity relationships. Previous works have presented models for jointly representing texts and KG entities, or entire graphs associated with each text (Toutanova et al., 2015), often focusing on question answering or reasoning (Zhang et al., 2021; Sun et al., 2020; Yasunaga et al., 2021) or text generation (Ke et al., 2021). Our work differs in the architecture employed (a contrastive text/graph matching objective) and in the setting it is specialized for: text-attributed graphs, where the graph is over entities which produce (e.g., Twitter users) or contain (e.g., academic articles) their associated texts.
Applications. Many studies in the joint graph/text domain have focused on social-science questions in addition to machine-learning ones. Learning the community structure of social networks is a common application (Martin, 2017; Waller and Anderson, 2021), as is detecting hate speech, misinformation or fake news (Vijayaraghavan et al., 2019; Chandra et al., 2020). Some work has also focused on understanding political polarization or ideological differences between groups (Milbauer et al., 2021; Lyu and Luo, 2022).

heading: E Dataset Details
text: Twitter. We created a Twitter dataset of 167,558 tweets posted by a set of 8,721 influential users in media, politics, and entertainment, and the follow graph among these users, which consists of 2,373,956 edges. We included up to 20 tweets per user in the dataset, sampled from each user’s most recent 3,200 tweets as of May 9, 2021. We also collected certain demographic data about these users (region of residence, age, gender, politician vs. entertainer occupation, and political party) by matching them to Wikipedia and Ballotpedia2.
To obtain the age and gender of Twitter users, we connected the accounts to their corresponding Wikipedia pages and used Wikidata to infer those two features. Users also self-report locations in their Twitter bios; from these locations, we created four regional categories. Finally, we used data from Ballotpedia to label whether a user is a politician or not and to identify their political party. Note that politician status and party are derived in different ways, from different data fields, with politician status being defined more strictly. These variables,
2https://ballotpedia.org/
used as targets in node classification tasks, are broken down in Table 12.
Pubmed. We built from scratch a version of the popular Pubmed graph learning benchmark (Sen et al., 2008) that includes the titles and abstracts of each article; widely available versions of the dataset do not include any text. We began with the standard list of PMIDs for the articles in the dataset and fetched the title, abstract, and list of references using the Pubmed API. We kept directed citation edges only to other articles in the dataset. One PMID was not found in the Pubmed database and was thus left out. The final dataset includes 19,716 nodes, 61,110 edges, and 59,381 texts, including both titles and abstracts. The included articles are about diabetes and the standard node categories are from the Pubmed database: type-1 diabetes, type-2 diabetes, or experimental evidence.
T-REx. We used the articles in the T-REx corpus (Elsahar et al., 2018) of Wikipedia articles that were labeled with the “Robots” category or any of its descendant categories. From these categories, we constructed several binary target label sets for the T-REx prediction task. However, since the most commonly occurring category was only associated with 526 (roughly 5.7%) of the articles, we expanded each article’s labels to include both first and second level ancestors in the category hierarchy to obtain better class label balance. From the initial set of 1,433 unique categories, this expansion yielded a total of 6,643 unique categories, with the most frequent (“Spacecraft”) occurring on 1,342 articles. We then selected five categories to use as labels for separate binary prediction tasks, choosing frequent categories that generally had small overlap with each other (i.e. were associated with mostly disjoint document sets.) Note that not every data point in the dataset, then, received a label. The resultant categories we selected are listed in Table 12.
Splitting. We divide the datasets into a 70% train set, 10% validation set, and 20% test set, splitting at the node level so that every text associated with a given node is in the same split. Because evaluation is inductive, any graph edges which cross split boundaries are dropped.

heading: F Model Architectures and Training Details
text: We estimate that training all of our joint and baseline models together used 263 hours of GPU time. Because the assumptions made for this value are conservative, the actual value is likely slightly less.
F.1 ConGraT Models
We trained all ConGraT models on either a single NVIDIA RTX A6000 GPU or a single NVIDIA A100 GPU. For masked LM experiments, we used the pretrained all-mpnet-base-v2 model (Song et al., 2020) from the sentence-transformers toolkit (Reimers and Gurevych, 2019), which has 12 layers of 12 heads each, producing 768-dimensional embeddings. It was pretrained constrastively on several corpora from similar domains to those we consider here,3 making it a good match for our work. Our causal LM experiments used the pretrained distilgpt2 model (Sanh et al., 2019), distilled from the larger GPT-2 model (Radford et al., 2019), with 6 layers of 12 heads each, producing 768-dimensional embeddings.4 For the graph node encoder, all models used a graph attention network
3See the model card for details: https://huggingface. co/sentence-transformers/all-mpnet-base-v2.
4Again see the model card for more details: https:// huggingface.co/distilgpt2.
(GAT) (Veličković et al., 2018) with 3 layers and 2 attention heads in each layer. As in a standard transformer, each graph convolutional layer is separated from the next by a linear layer, with layer normalization (Ba et al., 2016) applied afterwards. Hidden representations are 64-dimensional, and the final output vectors are 768-dimensional so that baseline model outputs have the same shape as language model outputs.
Parameter counts are as follows: distilgpt2, 81.9 million; all-mpnet-base-v2, 109.4 million; our GAT encoder, 199.7 thousand. The jointly trained models, including the adapter layers after the text and graph encoders, have 83.9 million parameters (causal / distilgpt2) and 110.9 million parameters (masked / all-mpnet-base-v2).
Training is sensitive to the learning rate; we found that a good compromise between speed of training and instability was a value of 1e-4. At a variety of learning rates, there were also intermittent large spikes in the norm of the gradient, which derailed training unless the gradients were clipped. We clipped the gradient at each step to a norm of 1. In order to reduce memory consumption and fit larger batches onto a single GPU, we used 16-bit mixed precision training (Micikevicius et al., 2018). We encountered numerical overflow problems with FP16, however, related to large gradient values at certain layers, and found it necessary to reduce the init-scale parameter of the gradient scaler from its default value of 216 to 256 in order to avoid overflow. We initialized the log-temperature parameter 𝜏 to 3.5 and constrained it to be within (− log 100,+ log 100) in order to avoid training instability. We trained all models with PyTorch (Paszke et al., 2019) and pytorch-lightning (Falcon et al., 2020), also using pytorch-geometric (Fey and Lenssen, 2019) for graph encoders and GAT baselines, and Huggingface Transformers (Raffel et al., 2020) for textual baselines and text encoders.
We also found that performance suffers if each batch is not unique on nodes (i.e., if each node has multiple texts, only one text per node can be in any given batch). We experimented with simply dropping duplicates from uniformly sampled batches, but this discarded too much data. Instead, we randomly sorted the texts on each epoch so as to minimize the number of within-batch duplicates (assuming minibatches are taken consecutively from the sorted dataset), and dropped any remaining duplicates.
Finally, because the objective is batch-wise con-
trastive, the problem becomes quadratically more difficult as the batch size increases. We used the largest batch size we could consistently fit into available hardware, but future work should explore the question of returns to scale.
All models used the AdamW optimizer (Loshchilov and Hutter, 2019) with 𝛽 values of (0.9, 0.999) and without weight decay. All joint models used a probability of 0.3 for dropout applied to text and node embeddings. Learning rates and batch sizes for our various models are shown in Table 13.
F.2 Unimodal Baselines
To better understand the effects of multi-modal pretraining, we also trained unimodal models, either language models or graph attention transformers, and evaluated these unimodal models on the downstream tasks. For textual models, we fine-tuned pretrained all-mpnet-base-v2 and distilgpt2 on the training splits of the evaluation datasets. Language models were fine-tuned for 3 epochs. For graph models, we trained graph attention network (GAT) models to do non-variational graph autoencoding (Kipf and Welling, 2016), also known as link prediction, on the network structure of the evaluation datasets. GAT models were trained from between 30 to 100 epochs with early stopping based on validation AUC, with patience of 3 epochs and minimum delta of 0.01. We compare these unimodal baselines against ConGraT. Parameter counts for the text and graph baselines are the same as reported for the appropriate modality’s joint encoder in Subsection F.1. Batch sizes and learning rates, as for joint models, are reported in Table 13. Our unimodal baselines were trained on NVIDIA RTX A6000 GPUs, or on up to four NVIDIA GTX 1080 Ti GPUs.
F.3 Social-LM
We implemented a baseline Social-LM, as a modified version of SocialBERT5 (Karpov and Kartashev, 2022) (also very closely related to LMSOC (Kulkarni et al., 2021)), which uses pretrained, frozen node embeddings to prime language model pretraining. Specifically, we added a special node token [G] at the beginning of texts and used the pretrained GAT model to obtain the corresponding node embedding paired with each tweet or article, which was used to replace the token embedding
5The SocialBERT authors did not publish their code.
for [G]. During the language model pretraining, we froze the node embeddings and only fine-tuned the language model to generate texts conditioned on the node embeddings. Our Social-LM implementation has some key differences from SocialBERT and LMSOC: (1) for masked LM experiments, we used all-mpnet-base-v2 to replace BERT, to be consistent with other experiments for a fair comparison; (2) we also experimented with a causal language model distilgpt2 under the Social-LM baseline, whereas LMSOC and SocialBERT only used the masked language model BERT; (3) we injected the node embedding as the zero token embedding of texts as SocialBERT suggests, whereas LMSOC appends the node embedding at the end. We adopted the zero token injection approach because the same strategy is adaptable for both causal and masked language modeling, while last token injection does not work for causal LMs like distilgpt2; (4) we used our unimodal GAT model trained on the graph autoencoding task to generate node embeddings for each tweet or article, whereas LMSOC uses node2vec and SocialBERT uses vectors from SVD and Deep Walk. We used the GAT in order to be consistent with ConGraT and the unimodal baseline, to ensure that the com-
parisons were fair, and because it was likely to be a stronger baseline than using SVD. Social-LM models were fine-tuned for 3 epochs with the same hyperparameters used for the language modeling baseline, and have the same number of parameters as all-mpnet-base-v2, our masked LM baselines and the joint masked text encoders.
F.4 LinkBERT
We implemented and trained LinkBERT (Yasunaga et al., 2022) as described in the original paper, with the only difference being that we used the same all-mpnet-base-v2 architecture as the other baseline models (instead of BERTBase) in order to maintain consistency across experiments. We initialized weights from the pretrained all-mpnet-base-v2 model from sentencetransformers, and fine-tuned it on the masked language modeling (MLM) and document relation prediction (DRP) tasks for 3 epochs. Hyperparameters used for training are listed in Table 13. Note that because of its MLM training objective, we used LinkBERT as a baseline for masked language model variants of ConGraT only. All LinkBERT models have the same number of parameters as all-mpnet-base-v2, as the DRP head is dropped
at inference time. We created training instances for LinkBERT by sampling contiguous, linked, or random text segment pairs for the DRP training objective from each dataset, with the three options appearing uniformly (1/3, 1/3, 1/3). For the Pubmed and Twitter datasets, we sampled 100,000 text pairs for each category, for a total of 300,000 pairs. For T-REx, which is a substantially smaller dataset, we sampled 10,000 text pairs for each category, for a total of 30,000 pairs. Text pairs consisted of anchor text segment 𝑋𝐴 and paired text segment 𝑋𝐵: (𝑋𝐴, 𝑋𝐵). The specific methods we used to sample pairs for each dataset were as follows:
Pubmed. Text segments in each pair consisted of individual sentences from the abstracts of each article in the dataset. Anchor segments 𝑋𝐴 were taken by sampling a random abstract, then sampling a random sentence from that abstract. For contiguous pairs, 𝑋𝐵 was chosen as the sentence immediately following 𝑋𝐴 in the abstract (𝑋𝐴 could not be the last sentence of the abstract). For linked pairs, 𝑋𝐵 was chosen as a random sentence from the abstract of one of the articles that was connected to 𝑋𝐴’s corresponding article in the citation graph. For random pairs, 𝑋𝐵 was chosen as a random sentence from an abstract whose article was not connected to 𝑋𝐴’s corresponding article in the citation graph.
T-REx. Text segments in each pair consisted of individual sentences from the introductory paragraphs of each article in the dataset. Anchor segments 𝑋𝐴 were taken by sampling a random article, then sampling a random sentence from that article’s introductory paragraphs. For continuous pairs, 𝑋𝐵 was chosen as the sentence immediately following 𝑋𝐴, with the same restriction as in Pubmed that 𝑋𝐴 could not be the last sentence. For linked pairs, 𝑋𝐵 was chosen as a random sentence from the introductory paragraphs of one of the articles connected to 𝑋𝐴’s corresponding article in the link graph. For random pairs, 𝑋𝐵 was chosen as a random sentence from an article not connected to 𝑋𝐴’s corresponding article in the link graph.
Twitter. Twitter has a different graph-text structure than Pubmed and T-REx; rather than the nodes consisting of texts themselves, the nodes are users who can each produce multiple tweets. Therefore, the notion of what constitutes continuous or linked text segments (tweets) is less clearly defined. We defined these relationships as follows. For contigu-
ous pairs, we sampled a random tweet as 𝑋𝐴, and chose 𝑋𝐵 as a different tweet from the same user as 𝑋𝐴. For linked pairs, we sampled 𝑋𝐴 from the set of tweets that mentioned other users that were present in our dataset. Then, 𝑋𝐵 was chosen as a random tweet from the mentioned user. Random pairs were taken by randomly sampling two tweets from different users to use as 𝑋𝐴 and 𝑋𝐵 .
F.5 Node Classification Methodology We use the standard scikit-learn (Pedregosa et al., 2011) implementation of logistic/softmax regression with the default L2 regularization, balancing our sometimes very imbalanced classification problems by downsampling before fitting. For performance reasons we use the liblinear solver for problems with no more than 5000 training data points and the saga solver otherwise. To ensure convergence, we increase the maximum iterations for the solvers from the default of 100 to 10000.

heading: G Licenses and Terms of Use
text: All software and pretrained models we used were available under open-source licenses which permit our use for research purposes. Our non-Twitter datasets were available under Creative Commons or other licenses allowing research use. We have access to Twitter data pursuant to an agreement with Twitter permitting use of data for research and publication. The agreement permits releasing the tweet IDs, which can be used to get the corresponding tweets from the public Twitter API. Along with the tweet IDs, we plan to release the demographic data collected from Wikipedia and Ballotpedia. Our code and datasets, when released upon publication, will be subject to an open-source license allowing use for research purposes.

heading: Future Work
text: We used the largest batch size we could consistently fit into available hardware, but future work should explore the question of returns to scale. All models used the AdamW optimizer (Loshchilov and Hutter, 2019) with 𝛽 values of (0. 9, 0. 999) and without weight decay. All joint models used a probability of 0. 3 for dropout applied to text and node embeddings. Learning rates and batch sizes for our various models are shown in Table 13. F. 2 Unimodal Baselines
To better understand the effects of multi-modal pretraining, we also trained unimodal models, either language models or graph attention transformers, and evaluated these unimodal models on the downstream tasks. For textual models, we fine-tuned pretrained all-mpnet-base-v2 and distilgpt2 on the training splits of the evaluation datasets. Language models were fine-tuned for 3 epochs. For graph models, we trained graph attention network (GAT) models to do non-variational graph autoencoding (Kipf and Welling, 2016), also known as link prediction, on the network structure of the evaluation datasets. GAT models were trained from between 30 to 100 epochs with early stopping based on validation AUC, with patience of 3 epochs and minimum delta of 0. 01. We compare these unimodal baselines against ConGraT. Parameter counts for the text and graph baselines are the same as reported for the appropriate modality’s joint encoder in Subsection F. 1. Batch sizes and learning rates, as for joint models, are reported in Table 13. Our unimodal baselines were trained on NVIDIA RTX A6000 GPUs, or on up to four NVIDIA GTX 1080 Ti GPUs. F. 3 Social-LM
We implemented a baseline Social-LM, as a modified version of SocialBERT5 (Karpov and Kartashev, 2022) (also very closely related to LMSOC (Kulkarni et al. , 2021)), which uses pretrained, frozen node embeddings to prime language model pretraining. Specifically, we added a special node token [G] at the beginning of texts and used the pretrained GAT model to obtain the corresponding node embedding paired with each tweet or article, which was used to replace the token embedding
5The SocialBERT authors did not publish their code. for [G]. During the language model pretraining, we froze the node embeddings and only fine-tuned the language model to generate texts conditioned on the node embeddings. Our Social-LM implementation has some key differences from SocialBERT and LMSOC: (1) for masked LM experiments, we used all-mpnet-base-v2 to replace BERT, to be consistent with other experiments for a fair comparison; (2) we also experimented with a causal language model distilgpt2 under the Social-LM baseline, whereas LMSOC and SocialBERT only used the masked language model BERT; (3) we injected the node embedding as the zero token embedding of texts as SocialBERT suggests, whereas LMSOC appends the node embedding at the end. We adopted the zero token injection approach because the same strategy is adaptable for both causal and masked language modeling, while last token injection does not work for causal LMs like distilgpt2; (4) we used our unimodal GAT model trained on the graph autoencoding task to generate node embeddings for each tweet or article, whereas LMSOC uses node2vec and SocialBERT uses vectors from SVD and Deep Walk. We used the GAT in order to be consistent with ConGraT and the unimodal baseline, to ensure that the com-
parisons were fair, and because it was likely to be a stronger baseline than using SVD. Social-LM models were fine-tuned for 3 epochs with the same hyperparameters used for the language modeling baseline, and have the same number of parameters as all-mpnet-base-v2, our masked LM baselines and the joint masked text encoders. F. 4 LinkBERT
We implemented and trained LinkBERT (Yasunaga et al. , 2022) as described in the original paper, with the only difference being that we used the same all-mpnet-base-v2 architecture as the other baseline models (instead of BERTBase) in order to maintain consistency across experiments. We initialized weights from the pretrained all-mpnet-base-v2 model from sentencetransformers, and fine-tuned it on the masked language modeling (MLM) and document relation prediction (DRP) tasks for 3 epochs. Hyperparameters used for training are listed in Table 13. Note that because of its MLM training objective, we used LinkBERT as a baseline for masked language model variants of ConGraT only. All LinkBERT models have the same number of parameters as all-mpnet-base-v2, as the DRP head is dropped
at inference time. We created training instances for LinkBERT by sampling contiguous, linked, or random text segment pairs for the DRP training objective from each dataset, with the three options appearing uniformly (1/3, 1/3, 1/3). For the Pubmed and Twitter datasets, we sampled 100,000 text pairs for each category, for a total of 300,000 pairs. For T-REx, which is a substantially smaller dataset, we sampled 10,000 text pairs for each category, for a total of 30,000 pairs. Text pairs consisted of anchor text segment 𝑋𝐴 and paired text segment 𝑋𝐵: (𝑋𝐴, 𝑋𝐵). The specific methods we used to sample pairs for each dataset were as follows:
Pubmed. Text segments in each pair consisted of individual sentences from the abstracts of each article in the dataset. Anchor segments 𝑋𝐴 were taken by sampling a random abstract, then sampling a random sentence from that abstract. For contiguous pairs, 𝑋𝐵 was chosen as the sentence immediately following 𝑋𝐴 in the abstract (𝑋𝐴 could not be the last sentence of the abstract). For linked pairs, 𝑋𝐵 was chosen as a random sentence from the abstract of one of the articles that was connected to 𝑋𝐴’s corresponding article in the citation graph. For random pairs, 𝑋𝐵 was chosen as a random sentence from an abstract whose article was not connected to 𝑋𝐴’s corresponding article in the citation graph. T-REx. Text segments in each pair consisted of individual sentences from the introductory paragraphs of each article in the dataset. Anchor segments 𝑋𝐴 were taken by sampling a random article, then sampling a random sentence from that article’s introductory paragraphs. For continuous pairs, 𝑋𝐵 was chosen as the sentence immediately following 𝑋𝐴, with the same restriction as in Pubmed that 𝑋𝐴 could not be the last sentence. For linked pairs, 𝑋𝐵 was chosen as a random sentence from the introductory paragraphs of one of the articles connected to 𝑋𝐴’s corresponding article in the link graph. For random pairs, 𝑋𝐵 was chosen as a random sentence from an article not connected to 𝑋𝐴’s corresponding article in the link graph. Twitter. Twitter has a different graph-text structure than Pubmed and T-REx; rather than the nodes consisting of texts themselves, the nodes are users who can each produce multiple tweets. Therefore, the notion of what constitutes continuous or linked text segments (tweets) is less clearly defined. We defined these relationships as follows. For contigu-
ous pairs, we sampled a random tweet as 𝑋𝐴, and chose 𝑋𝐵 as a different tweet from the same user as 𝑋𝐴. For linked pairs, we sampled 𝑋𝐴 from the set of tweets that mentioned other users that were present in our dataset. Then, 𝑋𝐵 was chosen as a random tweet from the mentioned user. Random pairs were taken by randomly sampling two tweets from different users to use as 𝑋𝐴 and 𝑋𝐵. F. 5 Node Classification Methodology We use the standard scikit-learn (Pedregosa et al. , 2011) implementation of logistic/softmax regression with the default L2 regularization, balancing our sometimes very imbalanced classification problems by downsampling before fitting. For performance reasons we use the liblinear solver for problems with no more than 5000 training data points and the saga solver otherwise. To ensure convergence, we increase the maximum iterations for the solvers from the default of 100 to 10000